1
$\begingroup$

Security camera and recorder vendors tout HVEC/H265 codec support in their systems. As far as I can tell, this is the industry top-of-the-line. I’ve been using such a system for a few years now and I am continually frustrated by the (I believe) characteristics of the HVEC for this application of forensic video limited by the available settings in the (also terrible) management software. It seems to me that camera sensors are capable of VERY crisp image capture, but that the lossy encoding makes most video recordings useless for moving faces and plates at reasonable distances. In security camera footage, almost all content is redundant and uninteresting. When the scene is still, it seems I could zoom in and count the blades of grass across the street. I can clearly read distant street signs or the license plates of parked cars. But when a vandal of foot smashes a car window, or a drunk driver sideswipes a parked car, all I can recover is a useless lossy blur. It’s very frustrating to have clear boring footage and blurry action footage.

The camera sensors MIGHT be to blame if the issue is motion blur because of shutter speed, but it doesn’t seem to me to be the main problem. I’ve tried a variety of CBR/VBR, resolutions, framerates to try to get better results. The software enforces bandwidth limits per camera feed and among all feeds, but I wonder if they are too conservative or if the real limit is the realtime processing power needed to encode.

HVEC was developed for entertainment applications mostly, wasn’t it? Where subjects of interest are much closer to the camera, we get familiar with the characters, and lossy-related motion blur is more acceptable.

Is there is a better codec choice out there for forensic video? Is there a codec in which still regions are de-prioritized and motion or even prerecognized areas of interest, such as faces, figures, letters or areas of just high contrast edges, are prioritized in the encoding process? Is this all possible in HVEC but just apparently not implemented in the commercial-professional recommended systems that I’ve seen?

$\endgroup$
1
  • $\begingroup$ Still parts of the image should allready be «deprioritized». Depending on rate control policy and VBR/CBR, regions with movement will have significant motion vectors and significant residue after motion compensation. If these regions are still too low quality it might be (sensor integration time) motion blur, too low bitrates or some pre encoder noise reduction. A state of the art video codec using inter-coding probably is your best bet. Intra-coding or even lossless coding would provide be great if bitrate was not an issue, but usually it is. $\endgroup$
    – Knut Inge
    Jun 6, 2020 at 19:44

1 Answer 1

2
$\begingroup$

Don't discount the contribution of physics to the blur in your pictures: Especially in low light, you need to have long exposure times, to get a picture that isn't completely dominated by grain. Conversely, things become less sharp if they're moving.

In fact, security cameras tend to have low frame rates (that might actually be for historical reasons – because nobody actually cared about the lawn growing in 25 fps, or about getting 100 grainy shots of someone smashing a window in 4 seconds, instead of 10 shots that have better exposure). So, that might actually be the limiting factor here.

HVEC was developed for entertainment applications mostly, wasn’t it?

Certainly, the highest money flow would be from the movies industry, but they have the same problem, namely that action scenes are harder to compress than still live shots of landscape. So, HEVC tries very hard to be better at accomodating different types of video content, so, I'd assume it's the best standard solution.

All in all, HEVC does a lot of things to make action scenes work best – motion prediction on a sub-pixel level, high dynamic range, you name it.

Especially, it compresses still-standing parts very efficiently, leaving more headroom on storage space when there's movement. Of course, the encoder has to be configured to actually spend the bitrate exactly when necessary. Maybe your vendors misconfigure it. Impossible to tell. Maybe look at the instantaneous bitrate over time; if it doesn't change much when there's something appearing in the picture, the encoder was configured to aim for a constant bitrate rather than a constant quality.

Of course, there are numerous approaches to making surveillance video capture more efficient. (I wouldn't be surprised if a lot of that wasn't available for civilian purposes, though. Automated video surveillance is very much something that states do.) I know that several "smart" home cameras draw rectangles around areas they consider noteworthy based on some more or less artificially intelligent algorithm.


Source: Pyimagesearch/Adrian Rosebrock

Of course, if you attach enough neural network processing power to your camera, and train accordingly, you can make sure your system reliably doesn't even store background, but recognizes e.g. cats and people in the picture.

Tuco smart camera screenshot
Source: Micah Scott (Scanlime): Tucoflyer Smart DNN-based cat tracking indoor camera gimbal system

If you know what you don't care about and that you care about the rest, and only record the latter, you can encode more efficiently by not even trying to record your grass growing. But: this takes a solid amount of processing power and hence electricity.

Taking that thought further:

Video compression is literally nothing but taking a high-dimensional input and mapping it to a lower-dimensional description and quantizing that cleverly.

That is also a pretty good description of what a video object classifier neural network does. Thus, there's plenty of approaches to encode video through neural networks.

You might then store video regions of interest not even as "collection of picture information", but simply as coefficients necessary to recreate e.g. the same face from a neural network optimized at representing faces by feature vectors with little error. This works pretty well. In fact, so well, that by synthesizing such feature vectors, we can create convincingly real-looking faces. For example This Person Does Not Exist:

This Person Does Not Exist

It seems at first that the quality as evidence of a face picture recreated from a camera neural network, which you cannot prove to be right, is very limited. Or at least, should be, in a court of law.

My algorithm says this is the visual representation of the lowest-error feature vector of a face that would match what the camera saw at 3 a.m. when someone broke into my car. I cannot formally prove an upper bound for the error, but it looks just like the defendant. So, it's the word of my trained-on-thousands-of-pictures neural network against the word of the defendant! If that doesn't prove him guilty, what will?

shouldn't be an all too successful argument when in dubito pro res is a principle in criminal justice.

Then again, all video compression claims the same; got a JPEG picture of the thief breaking into your store done with your iphone? These JPEG coefficients just are the lucky coincidence of image content, shot noise and the JPEG encoder.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.