2
$\begingroup$

I want to test some motion estimation algorithms which compute motion with sub-pixel accuracy. I can create images shifted by the order of pixel (e.g. [2,2]) using MATLAB or OpenCV but I am not clear on how can I shift images with sub-pixel (e.g. [3.57,4.83]) accuracy. Any help regarding this will be much appreciated.

$\endgroup$
2
$\begingroup$

Let me provide you an ideal sub-pixel shifter in 1D, which can guide you for further study in 2D and using different methods as well:

Now it's simple to show an integer delay operator in discrete time as: $$ y[n] = T\{x[n]\} = x[n-d]$$ The system is LTI and the impulse response turns out to be $$h_d[n] = \delta[n-d]$$

In order to shift an input sequence $x[n]$ by the amound of $d$, therefore, all you have to do is convolve it with the sihft-kernel $h_d[n]$ to produce the result: $$y[n] = x[n] \star h_d[n] = x[n] \star \delta[n-d] = x[n-d]$$

Coming to the fractional delay operation, first it's very intuitive to express the integer delay as $$h_d[n] = \frac{ \sin(\pi(n-d))} {\pi (n-d)} $$

It can easily be seen that, when $d$ is an integer, the numerator is always zero, and when $n=d$ the $0/0$ is resolved as $h_d[d] = 1$. Therefore we conclude that $\delta[n-d] = \frac{ \sin(\pi(n-d))} {\pi (n-d)} $ ;

Now without a proof can we boldly assume that replacing integer $d$ with a noninteger $\tau$, we may achieve the effect of a non-integer delay? i.e., does $$h_{\tau}[n] = \frac{ \sin(\pi(n-\tau))} {\pi (n-\tau)} $$ provide the samples of a fractionally shifted (subpixel shifted image) signal $x(t-\tau)$?

The answer is yes. But keep in mind that in discrete-time systems and signals any non-integer argument makes no sense. Therefore it's not right to consider the non-tinteger delay operator as $h_\tau[n] = \delta[n-\tau]$ as was the case of integer delay, but we still have the impulse response of the fractional delay system in discrete-time as: $$h_{\tau}[n] = \frac{ \sin(\pi(n-\tau))} {\pi (n-\tau)}$$

Note that this is a non-causal, infinite length (therefore ideal) impulse response. You can get good approximations to it however...

Hence the sub-pixel shifted elements will be provided to you with the operation as: $$y[n] = x[n] \star h_\tau[n]$$

Note that this operation can also be considered as an interpolation, or resampling as Marcus have pointed out.

$\endgroup$
2
$\begingroup$

A simple solution would be to start out with much higher resolution images. Shift by integer pixels, then resize to the target resolution. Otherwise you might train your estimator against interpolation artifacts.

$\endgroup$
1
$\begingroup$

The science of finding values in between those points where you actually have samples is called Interpolation.

It's a rich field, and every signal processing toolkit (be it Matlab or OpenCV, or …) has functions for it.

Picking the right interpolator has a lot to do with the nature of your input, and the purpose of you doing the interpolation – so, no general advice can be given on what specific interpolator to use.

$\endgroup$
1
$\begingroup$

It seems your question is not only about how to create images with sub-pixel shift but more about motion estimation for which you need those images to compute a likelihood.

If this is the case, you may be interested in the Condensation algorithm by Isard and Blake. Motion is represented by floats and therefore allows for sub-pixel estimation.

Such a method was used for motion estimation in this paper (the python code is also available; disclaimer: I am the author of this paper). If you wish I may describe the method in more details.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.