Moiré patterns in sampled imaging are usually effects of aliasing.
As usual, with aliasing, it's most visible in high-frequency bins, as that's where low-frequency energy gets aliased to. It's also the case that high-frequency energy gets aliased to low frequencies.
So, the first approach would be to compare the spectra (absolute of the Fourier transform) of images with and without Moiré patterns. You'll probably see more energy at higher frequencies. Note the rough ratio of how much of the overall Nyquist bandwidth is affected. (For example, if the top 10% of frequencies contain more energy in pictures with Moiré patterns, note 0.1)
Then design a real 1D bandpass FIR filter to pass through everything in frequency that is higher than that ratio, and lower than (1-ratio). Make a circularly symmettric 2D kernel from that, and convolve your image with it. (details on how to deal with edge effects will depend on what you're doing with the results later on, so you'll have to figure out for yourself what is appropriate – padding, cutting or circular repetition.)
With the Moire patterns out of the way, try your Mellin transform.