Why are some JPEG files of the same scene much larger than others?

Question

I'm using a Foscam FI8910W ip camera to view a static scene under constant lighting conditions. When I pull back a frame grab, it's about 35 KB in size. I can do this over and over and it's always around 35 KB, but fluctuates somewhat due to various noises inherent to electronic image capture. This random fluctuation is only of the order of 1 KB at most.

About every 2500 frames, the image size of a frame is suddenly in the order of 70 KB. There's no gradual creep upwards if you're thinking thermal noise as the camera warms up. 1 frame will be 70 KB(ish) and then they return to the 35 KB sized frames.

This has happened before with another run looking at a different scene. The common file size was 39 KB then, and out of 10,000 frames, 4 were of the order of 77 KB. The image size histogram looked like this:-

Before you ask, I've managed to save one of these frames and it looks exactly like all the others bar the expected noise fluctuation. They have roughly the same number of unique colours at around 23,000. So it's not a moth randomly landing on the lens for exactly 1 frame and then flying off. For completeness, I've done another run of images and this is an example typical image (the reflection is the IR illuminator):-

This is the anomaly image:-

You can see there is no difference. Excuse the hippo. I'm fairly familiar with the JPEG algorithm, and I can't see how this might happen other than a coding error with Foscam's implementation. But, could there be something inherently chaotic within some of the JPEG transformation functions (like the discrete cosine transform or quantization)? Statistically, one would expect a normal distribution of file size and this is what I see around 39 KB. Then there are several outliers at 77 KB. So it doesn't appear stochastic.

The reason this is in CS and not in hardware is I'm asking could this be a programming code phenomenon relating to the JPEG encoding algorithm? Seems unlikely, but the anomalies are random and infrequent and there is no human interaction with the device. Is JPEG encoding stable?

The reason you might be unfamiliar with this phenomenon is that as the images look the same, no one really looks at the file sizes. The file size is of critical importance to me so I noticed. How can this happen roughly every 2500 frames?

Supplemental:-

Posting these images is not going to work guys, as the imgur software re-samples the uploaded files. So whilst I posted 37K and 73K files, imgur has re-sampled both to 35K. This seems to be a Stack Exchange problem which seems ironic for a site dealing with image processing, data compression and analysis.

This is my processing of the images. It's the normalised difference between a normal image and the anomaly. The image is as you would expect, with JPEG noise in the high frequency regions. This is an RGB image even though it looks monochrome. There are 8000 unique colours in the colour cube (representing the noise).

Supplemental 2:-

As requested, 4 normal frames and 2 abnormal frames can the downloaded from sample frames. It is a different scene, but the anomalous behaviour still occurred, so this proves that it's consistent.

Answer 1

My guess is on autofocus or aperture briefly changing in a manner where the resulting image contains more high-frequency elements.

For example, if the focus moves from a smooth object to a textured object (like the smooth hippo to the cloth drape) in a manner where the latter's detail texture surfaces, JPEG will tend to take quite a larger size.

As someone else already stated: it seems like a good idea to check the image EXIF data for changes in core parameters like aperture and focus distance. For such a striking difference in image size, it is quite likely that some fundamental parameter is different in the camera's opinion.

Answer 2

It's common with 'CMOS' sensors to encounter a phenomenon known as "PURPLE FRINGING" or possible "Sensor Bloom" issues, Which I will explain in more detail in a sec.

I must preface this, however by telling you that there is actually some debate about wether P.F. is the cause of Sensor Bloom or visa versa but the bottom line: both can be the result of anomalies in the lens or camera sensor itself - or both. These effects can be the result of a culmative effect that causes an overload if you will, in the sensor, in turn causing a spike in the light captured. MAKING THE FILE LARGER.

I belive the overload occurs in the magenta (or purple) range, and this phenomenon is pretty rare.

Think of the sensor like a huge ice cube tray ..... laid out in a grid like fashion. If one compartment OVERFILLS with water (light) because of the anomaly.... it can spill over into the adjacent compartments and so on, causing a batch of slightly larger volume ice cubes for that batch. (Possibly explaining larger file size...AND color data)

Now this is a best guess and I found a link you may find helpful with more info on the above - AS WELL AS some aditional technical information which may help isolate the problem for you if my assesment is incorrect.

Check out this link http://toothwalker.org/optics/chromatic.html

Remember RGB is a subtractive color space. Color via light (vs pigments) are manipulated by removing light in certain wavelengths. Some color wavelengths being longer than others.

There is a great optics lesson on the page that can also help explain the anomaly as well.