# Which image filter can be applied to remove gridded pattern from corrupt jpegs?

I have around 1,400 jpegs that have been corrupted somehow and have lost the backup images. They all seem to have the same gridded pattern of lines over each (i.e. the gridding does not shift from image to image.

Here's what one of these images looks like:

Are there any image filtering techniques in Matlab particulary or otherwise that will remove or smooth this gridding pattern?

• can you give us more information on how these images came to be corrupted? This is a strange pattern, since it's very pixel-local, which requires high resolution in frequency domain, which would imply a horribly broken JPEG encoder, maybe? Jan 3 '17 at 23:03
• I'm not entirely sure, sorry. The images are part of a database that has changed hands a few times and people haven't been as careful as they should have (...undergrads). As far as the pixel-local issue, I agree. In some photos that include darker water surfaces, the gridding pattern is very light. Jan 3 '17 at 23:46
• @MarcusMüller: A horribly broken JPEG decoder seems more likely to me, although I suppose either way is possible. Anyway, based on the uneven and non-power-of-2 spacing of the lines, it seems to me that the images have probably been scaled up and re-encoded after the corruption occurred, so, alas, trying to fix them in the DCT domain is probably futile. The inpainting solution by Maximilian Matthé below is probably the OP's best bet. Jan 4 '17 at 2:42
• Oh, and the OP should definitely save a backup of the images before attempting to fix them in any way, just in case someone ever wants to reanalyze them. Inpainting, however well done, is always a lossy operation, and has the potential to introduce bias (since it basically amounts to making up fake data to replace the corrupted pixels). And the same goes for median filtering or frequency removal too, or anything else likely to hide this kind of damage. Jan 4 '17 at 2:47
• @IlmariKaronen Thanks for the tip. Will definitely try to be more careful with these images. Jan 4 '17 at 4:14

You can use a standard inpainting algorithm. These algorithms replace marked pixels in an image with the pixel values that surround these marked pixels. The challenge here is to detect the grid (my tests seem to show that it is not a completely regular grid). So, I came up with this solution:

from PIL import Image
import requests
from io import BytesIO
import cv2

url = "http://i.stack.imgur.com/Ahrnl.jpg"
response = requests.get(url)
img = Image.open(BytesIO(response.content))

plt.imshow(img)
A = np.array(img)
A2 = A.copy()
A_gray = cv2.cvtColor(A, cv2.COLOR_RGB2GRAY)

# Do some rough edge detection to find the grid
sX = cv2.Sobel(A_gray, cv2.CV_64F, 1, 0, ksize=3)
sY = cv2.Sobel(A_gray, cv2.CV_64F, 0, 1, ksize=3)
sX[sX<0] = 0
sY[sY<0] = 0

plt.subplot(221)
plt.imshow(sX)

plt.subplot(222)
plt.imshow(sY)

plt.subplot(223)
# the sum operation projects the edges to the X or Y-axis.
# The 0.2 damps the high peaks a little
eX = (sX**.2).sum(axis=0)
eX = np.roll(eX, -1) # correct for the 1-pixel offset due to Sobel filtering
plt.plot(eX)

plt.subplot(224)
eY = (sY**.2).sum(axis=1)
eY = np.roll(eY, -1)
plt.plot(eY)

plt.figure()
plt.subplot(221)
plt.imshow(A)

plt.subplot(222)
plt.imshow((A2))

restored = cv2.inpaint(A, mask, 1, cv2.INPAINT_NS)
plt.subplot(223)
plt.imshow(restored)


The program output is as follows:

To detect the grid I did a quick-and-dirty solution. It can be improved a lot, but it shows the initial idea. The general flow is:

1. detect the grid
2. create a mask that describes which pixels are corrupted by the grid
3. inpaint the corrupted pixels.

For inpainting I used OpenCV inpaint operation. For detecting the grid, I performed edge detection in X and Y direction using a Sobel filter. Then I add all edge values in the X-direction and Y-direction to find peaks, where the grid lines are. Then, I choose the highest peaks as the coordinates where the grid lines are estimated. It's not working perfect (e.g. strong edges in the image are falsely detected as grid lines), but it shows the idea. It can be improved by e.g. Hough transformation to find lines, kicking out very strong edges etc.

Alternatively, if the grid is really the same for all images, then you can perform the grid detection jointly for all images, which would yield a much better accuracy (just do the technique above, but before choosing the peaks, sum up the results from all pictures). In more detail, you would calculate eX for all images and add all these eX together into a single vector. This vector will have a much clearer peak structure and the thresholding can be done easier.

• Thank you for your input! Your result here is really nice! I'll give this a try. Disclaimer: I'm very much a novice in image processing so this will take me sometime to work through on my own, but will mark it solved since it work so well on your end. In what environment are you running this algorithm? I think undesirable edge detection on instrumentation and infrastructure in the image wont be much of a problem as the majority of the images don't include any of that and are just Tundra and/or water. How you go about adding up the peaks from all the pictures? Jan 4 '17 at 0:11
• Thanks! The environment is Python and I just added a note on what I mean with summing up the peaks for all images. Jan 4 '17 at 6:18

I tried a really simple algorithm of running a 3x3 median filter on the R and G channels of that image and it works quite well. The python code is really simple:

import scipy.signal as sp
from scipy import ndimage

image_filtered = np.array(image)
for i in range(2) :
image_filtered[:,:,i] = sp.medfilt2d(image[:,:,i])


Alternatively you can use frequency domain filtering as discussed in this question: https://stackoverflow.com/questions/34027840/removing-periodic-noise-from-an-image-using-the-fourier-transform

The Fourier transform of your image clearly shows some repeated "dots" in the spectrum corresponding to this periodic noise.

As Maximilian pointed out, this latter method works well only if the noise is perfectly periodic, which does not seem to be the case here.

I tried running a really stupid filter that zeros out 5x5 squares of frequency bins centered around multiples of 9 in both x and y directions and it (sort of) suppresses the noise but introduces artifacts in locations that don't contain noise (eg. the sky).

One can perhaps do better with careful notch-filter design instead of directly zeroing FFT bins (never do that in practice!) and only applying the filter in regions of the image where the noise is present (i.e. don't filter the sky).

• in your last line, I believe you just filter the red channel twice (last index should be i, not 0) Jan 3 '17 at 22:51
• @MaximilianMatthé good catch! (Luckily my actual code that I ran was ok :P) Jan 3 '17 at 22:54
• Regarding the Fourier-Transform method: This only reliably works, if the grid is really regular (i.e. same distance between all lines). I could not (at least not quickly) find parameters such that I could draw a regular grid on top of the corrupted lines. Then, the Fourier-Method would also not be able to remove this not-exactly-periodic noise. Jan 3 '17 at 23:00
• @MaximilianMatthé you are right - FFT method is tricky as the noise isn't perfectly periodic pattern. But it might work with careful notch filter design. Maybe. Jan 3 '17 at 23:27
• Thank you for your input! I did try a 3x3 median filter in Matlab and while it did remove the gridding (mostly) I didn't like the decrease in detail of the image (they need to appear in a pop-up on a web-mapping application. Jan 3 '17 at 23:54