You might try converting your image to greyscale first with
greycells = rgb2gray(rgbcells);
where rgbcells contains the color image of your cells (see here).
Then use Otsu's threshold on the greyscale image like so
threshcells = graythresh(greycells);
like so (see here).
After OP explained what image should be the resulting image.
If you remap a local patch around a feature point to log–polar coordinates (with the origin in the point of interest), scale changes correspond to a translation along the log–radial axis, while rotations correspond to translations (with wrap-around) along the angular axis. If you then calculate the two-dimensional Fourier transform, translations in the ...