2
$\begingroup$

I have an IP webcam that generates very low light images of a plain background. The camera spits out JPEG only files. The following is one such frame... raw frame and this is with graphic equalisation applied... equalised frame You can see that there is a lot of noise, but also a discernable image of the stepped lens mount. These JPEG files are approximately 20KB in size.

I want to determine what proportion of the 20KB is due to the fixed image (of the lens mount), and what proportion is entirely due to noise. It's important to reiterate that I'm only concerned with the proportions of the JPEG file, and not the expanded raster images.

So noise.jpg(part) + mount.jpg(part) = 20KB.

My instinct suggested repetitively sampling the static image and frame stacking to isolate the lens mount. That fundamentally doesn't work.

How can I determine these proportions?

Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ One approach could be to draw (in inkscape or something) a clean image representing what you think the sensor is seeing. Compress it using JPEG at the same quality level as the camera. The difference should be noise. Another approach could be to put the camera in the freezer; this reduces sensor noise, so you might be able to extrapolate the result to zero noise. $\endgroup$
    – MBaz
    Dec 9, 2016 at 18:41
  • $\begingroup$ @MBaz I can reliably get the camera to +30 deg., +4 deg. and probably -20 deg. Wouldn't I have to extrapolate to -270 degrees though to get the y-axis (file size) intercept, which is such an extent that it would probably be totally inaccurate? $\endgroup$
    – Paul Uszak
    Dec 10, 2016 at 2:16
  • 1
    $\begingroup$ I'm not sure how easy/accurate the extrapolation would be, but in the abscence of other alternatives, it may be worth a shot. There may also be technical information about the noise/temperature relationship in CMOS devices. $\endgroup$
    – MBaz
    Dec 10, 2016 at 16:48
  • 1
    $\begingroup$ @MBaz could the answer be noise=20, mount=0? If I draw my mount, the image has low frequency and information content. It is then overlaid with high frequency high entropy noise. Does the low frequency information get totally swamped by the higher frequency information when it is then compressed? JPEG is like a lossy zip. $\endgroup$
    – Paul Uszak
    Dec 10, 2016 at 22:25
  • $\begingroup$ You are trying to measure noise using noise as a reference. You could instead get rid of square sampling and random noise by taking pictures of a white circle on the dark repeatedly shifting the circle with known random lateral shifts. Then you perform stacking of the resulting JPEG shifting them back to centre of the camera view to obtain the averaged circle image. The random shifts will destroy the square patterns while the statistics within the circle like standard deviation will provide you with the estimate of noise. $\endgroup$
    – mbaitoff
    Dec 11, 2016 at 7:34

1 Answer 1

Reset to default
1
$\begingroup$

Your problem here is twofold:

  1. Image Noise / low SNR due to darkness
  2. problems to average things out due to JPEG compression

The first point is something we cannot really change per se; your averaging approach surely is a good one.

The second problem now is due to how the JPEG compression works in your camera:

  1. Gamma correction is applied to the RGB image first. That is a nonlinear operation, "pushing together" high values and stretching the lower values. That way, you don't get "pitch black" everywhere for dark images. Sadly, this also makes our quantization noise problem harder.
  2. The resulting R'G'B' pixel data, is converted to the Y'CbCr colorspace, meaning you convert the threedimensional red,green,blue signal to the threedimensional brightness,Blue-difference,Red-difference signal.
  3. The Cb/r channels are low-pass filtered and downsampled
  4. The original-resolution Y', and the reduced-resolution Cb and Cr images are split into 8x8 pixel blocks separately
  5. These blocks are separately DCT'ed
  6. A quantization matrix is chosen according to pick a quality/compression trade-off. This results in different DCT coefficients being omitted, saved with fewer or more bits.

As shown in the other answer, averaging on the reconstructed RGB image is a bad idea, since the random noise must lead to you basically seeing an effect of the IDCT of the quantization matrix in each block – meh.

So, first of all, notice that it's pretty certain that the for the DCT coefficient that defines the constant intensity offset ("DC component", an EE would say), the highest quantization level was chosen.

Conversely, at least for most higher-frequency components in the 8x8 blocks, quantization error energy is higher.

Thus, inherently, the DCT's constant bin is the most reliable – also, it happens to be the average of the 64 pixels in the block, and thus reduce noise already (if noise's frequency distribution was white, that would be the case for all components, but neither is it reliably white in cameras nor is the gamma correction linear...).

So, what you probably want to do as a first step is

  • work on the Y'CbCr, not the RGB reconstruction
  • only look at the Y component first
  • ignore everything that is not the DC component in each 8x8 block

You'd get an image of $\frac18$ the width and height of your original photo. It's probably still going to be noisy – with the shot noise that is probably relevant in a low-light setting contributing to the average in each frame. However, averaging out that over time will reduce that effect.

You can probably get an noise reduction, too, by summing up the non-zero-frequency coefficient, weigh them with a factor < 1, and subtracting them from the DC component. The idea behind that is that when you set a single pixel in a 8x8 frame, you get energy in all coefficients (the cosine transform of a dirac has energy at all frequencies, if you think about it). Since you do not get all coefficients, most of them being thrown away or quantized heavily, you'll never be able to fully remove the effect of shot noise from the average coefficient – but you can, at least on average, reduce the effect. Choice of that factor would depend on camera sensor, gamma factor, quantization matrix, the way the DCT is implemented (truncating fixed point?) and thus, too many unknowns, but you could probably just empirically try a couple of values and minimize the variance of the result over multiple frames.

Improve this answer
$\endgroup$
2
  • $\begingroup$ energy in all coefficients I'm starting to think that it is impossible to separate the two components of the JPEG file. I'm not trying to reconstruct the background image. I want to know how many of the 20KB represent the background. I'm starting to think all of them. If the JPEG wasn't effectively losslessly zipped then it might be otherwise? I'm going to try some simulations in reverse by programmatically adding noise to a fixed image and then creating JPEG s of it... $\endgroup$
    – Paul Uszak
    Dec 10, 2016 at 22:46
  • 1
    $\begingroup$ I tried to address exactly that fact (JPEG being part of your problem) in my answer. And of course, you can never separate noise from background based on a single observation unless you have a signal model that allows you to do that. And to answer your question of "how much of my data is noise?" yep, that can basically be answered to "almost all". Noise has much information content. You can quantize information in bits. That leads to information-theoretical entropy, and noise has very much entropy. Read the Wikipedia article on Information Theory if you're interested in that. $\endgroup$
    – Marcus Müller
    Dec 11, 2016 at 3:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.