How do you model a rolling shutter in the frequency domain

Say you have an image in time domain: $$a(x, y, t)$$

The Fourier transform is then: $$A(k_x, k_y, f)$$

So far so good. What if we now introduce a rolling shutter. How would one model that. I'd expect the rolling shutter to have an effect similar to a rect window, albeit a 3 dimensional one. Any references on this? It would be nice if I can avoid deriving it all.

A rolling shutter introduces a displacement of the lines of an image that is proportional to the relative velocity of the camera (or target) and the velocity of the shutter. In terms of "simulating" this through a video, it translates to "borrowing" a few lines of pixels from each frame to compose one final frame. Notice here how the shearing produced by the rolling shutter, which could be seen as a displacement of $x$ in 2 directions just got translated to a simple "vertical" motion in 3 dimensions.

So, all that we have to do is look at what exactly this operation does in the time domain and see how (if possible) we can replicate it in the frequency domain.

From your question, I suppose that you want to create one image (let's call it $I$) from the "integration" of the frames in $a$. Before we construct $I$, we have to make some assumptions about how $a$ was acquired. So, let's assume that $a$ was acquired with a "perfect" camera. One that does not suffer from rolling shutter itself and one that shoots at a very high frame rate. So the period of sampling $T_s = t_k - t_{k-1}$ of $a$ is $T_s \ll T_e$, where $T_e$ is the frame exposure time (the time it takes for the shutter to roll and create one image). Furthermore, let's assume that the "width" of the shutter, in terms of image lines, is $W_s$.

Then, just considering the shutter, this is indeed a "white" rectangle that moves parallel to axis $y$ across all time instances of $a$ and it looks like:

$$S(x,y,t) = rect\left(\frac{x-\frac{N_x}{2}}{\frac{N_x}{2}},\frac{y-t \times T_e}{\frac{W_s}{2}}\right)$$

This is a two dimensional rectangle. It is centred in the middle of the $X$ dimension of the frame and is as wide as the frame and in the $Y$ direction it is $W_s$ lines tall, moving downwards.

The time instances that make up $I$ are:

$$I(x,y,t) = a(x,y,t) \times S(x,y,t)$$

But if you "replay" this, all that you see is a band of pixels from each frame moving downwards.

To get rid of $t$ in $I$, you need to integrate this signal with respect to $t$. In other words, you have to sum all pixel time courses to generate just one image that is independent of $t$.

Now, this is a linear time invariant system of operations in three dimensions. To find its coefficients, you need to first transform $S_{FT} = \mathscr{F}(S(x,y,t))$.

To produce $I$ we multiplied $a$ with $S$. Multiplication in the time domain is convolution in the frequency domain. Therefore to express $I(x,y,t)$ (notice the $t$ still there):

$$I_{FT}(k,m,f) = \mathscr{F}(a(x,y,t)) \ast \mathscr{F}(S(x,y,t))$$

Here of course, notice the change of variables, $m,n,f$ represent their frequency domain $x,y,t$ counterparts.

Right, so now, we want to integrate the three dimensional complex $I_{FT}(k,m,f)$ with respect to time. We look up the properties of the Fourier Transform and realise that integration in time is equivalent to multiplication in frequency by a $\frac{1}{j \omega}$ factor plus a static DC component scaled by $\pi \delta(\omega)$. So, all we have to do now is multiply our $I_{FT}$ coefficients and offset them by these factors, in the direction of the time vector. Or in other words, in the dimension that corresponds to frames.

$$I_{FT}(k,m,f) = I_{FT}(k,m,f) \times \frac{1}{e^{2 \pi i f/N_f}} + \pi I_{FT}(0,0,0)$$

(And of course then, above, obtain the inverse to get back into "time" domain)

Of course, you can express the two as one operation and do the transformation in one step.

Hope this helps.

• Very helpful, thanks. By the way, I thought of a as continuous with infinitе domain. Something like the projection of the scene on an infinite plane i.e. perfect camera. – user110971 Sep 24 '16 at 10:58
• Indeed. I was thinking common cameras shooting at something like 30fps which would result in quite a lot of rolling shutter distortion for anything less than minutes of a recording. Of course you may have a very slowly varying to immovable scene and a "moving" apparatus. So better work with times and let everything sort itself out. – A_A Sep 25 '16 at 12:27

assuming that the shutter rolls from top to bottom, you might multiply $a(x,y,t)$ by

$$s(x,y,t) = \sum\limits_{n=-\infty}^{+\infty} \operatorname{rect}\left( \frac{y - r \, t - n \, H}{H/2} \right)$$

the height of the whole frame is $H$. the height of the shutter window is $H/2$. the width of the shutter window is across the whole frame. note that there is no dependency on $x$.

the rate of movement of the shutter window is $r$. i think the shutter window is moving down if "down" is in the $+y$ direction.

so to find out what this does in the frequency domain, you might have to Fourier Transform this mess. then you gotta do some convolution in the frequency domain.