I need to do a FFT on an image for noise reduction, but the problem is that I do not need the complete image, but only a circle in the middle. The borders are a fixed rig, thus I am not interested in what it displays, but it has an impact on the result of the FFT.
Is there any way to just cut out a circular part and use that for the FFT? Note that if I use black background, the edge between background, and image data will have quite an impact.
Instead of having a hard edge between the image data of interest and the background, you could use a two-dimensional tapered window function, as is often done in spectral analysis. You might start by trying a Gaussian window, which for a two-dimensional case would look something like:
$$ w[x,y] = e^{-\frac{\left(x-\frac{N_x-1}{2}\right)^2}{2\left(\sigma_x \frac{N_x-1}{2}\right)^2}} e^{-\frac{\left(y-\frac{N_y-1}{2}\right)^2}{2\left(\sigma_y \frac{N_y-1}{2}\right)^2}} $$
$N_x$ and $N_y$ are the dimensions of the desired transform in the $x$ and $y$ directions, respectively, and $\sigma_x$ and $\sigma_y$ are parameters that allow you to control the shape of the window; for small $\sigma$ values, most of the energy in the window function will be concentrated toward the center, with that effect decreasing as you increase $\sigma$.
Use a tapered flat-top window function with a flat top and edge taper, such as a tapered cosine window or Tukey window (spin it around a polar axis to make it a circular 2D template), and zero-pad as needed for any fast-convolution filtering. The window going to zero at the edges will reduce the effect of the edge discontinuities from the circle on the FFT. After your filtering or other processing, blend the result back in using the subtractive inverse of your window function on the existing image data.
A simple solution could be to spread the boundary conditions of your non-rectangular region of interest into the rectangle around it.
You can perform this with a nearest neighbor algorithm.
