3
$\begingroup$

TWO PART QUESTION - We've all seen the movies/TV shows where the police/feds/spies use computer software to take a grainy photo and do a "clean-up" to see a better picture and more details. I assume the concept is executed by some sort of uber-power pixel-smoothing or anti-aliasing type of algorithms to fill in the blanks based on deductive processing.

PART 1: How real is this technology in the public/commercial software world? I'm not asking about any speculation on alleged secret gov software or such, I just want to know where we actually are with this concept today? How much is fully automated vs human-assisted.

PART 2: Assuming there actually is some reality with this technology for photographs the second part of this question is how (if at all) has this been applied to videos? Again the issue of fully automated vs human assisted is of interest here.

At the heart of this post is the ultimate question of how viable is today's software for being able to take an old VHS or DVD recording and process the frames to create a new HD-resolution remaster. Considering that doing this would mean cleaning up tens-of-thousands of frames for even a simple wedding video I am not expecting this technology to be fast of course.

NOTE: Per in-topic and meta discussions I went ahead and cross-posted this in two other suggested SE forums to acquire their viewpoints and expertise on this matter. So far (it is still a little "early in the day" so to speak) I have received some pretty interesting information in 2 of the 3.

When this is all done (when good answers have been selected) I would appreciate a way to merge these for the benefit of all three SE communities:

Improve this question
$\endgroup$
2
  • $\begingroup$ NOTE: This question was originally asked in the Computer Graphics SE but after discussion there and in the meta forum for the Video SE it has been cross-posted here. $\endgroup$
    – O.M.Y.
    Jan 9, 2016 at 7:27
  • $\begingroup$ I came across VideoCleaner and it comes pretty close to what you want. It isn't magic, but it is the software used by law enforcement. Best of all, it is ad-free and open source. $\endgroup$
    – user24269
    Oct 18, 2016 at 4:56

2 Answers 2

Reset to default
3
$\begingroup$

Entertainment industry greatly exaggerates what video enhancement can do.

2016 answer:

Spatial and temporal noise reduction

Noise reduction by Neat Video and MSU Denoiser Filter, which are somewhere at (according to a comparison from 2007) or near state-of-the-art for hard noise, takes advantage of that adjacent frames usually are of the same scene but with independent noise. So noise can be filtered both spatially and temporally. But that kind of processing is in my experience just cosmetic. It does not really reveal things that a human would not already see through the noise, it just makes the video look nicer. Both filters have an option to adjust settings automatically, but because noise reduction seems always a compromise between noise reduction and introduction of artifacts, the final "strength" of noise reduction should be adjusted to taste manually.

Motion estimation and superresolution

MSU Super-Resolution Filter reconstructs a new video at 2x horizontal and vertical resolution. It follows a similar paradigm as the above two filters, but also estimates motion between frames. The original pixelization (and other artifacts that do not follow the motion such as blocking and sensor Moiré) can be reduced by compensating for motion and then filtering temporally. This improves resolution and makes detailed things like pixel-deprived text and faces more clear. Resolution can be increased only if the pixelization is not properly anti-aliased. Artificial pixelization of faces, if not done carefully, could be reduced by the same or similar processing. This is a kind of enhancement that the human visual system is not accustomed to do natively, and can actually reveal hidden useful information.

2022 update:

Deep learning-based methods

As is happening in many fields, these days the highest-quality results are obtained using deep learning-based methods. See for example Liang, Jingyun and Cao, Jiezhang and Fan, Yuchen and Zhang, Kai and Ranjan, Rakesh and Li, Yawei and Timofte, Radu and Van Gool, Luc. 2022. VRT: A Video Restoration Transformer. arXiv preprint arXiv:2108.10257, Github.

Improve this answer
$\endgroup$
2
$\begingroup$

I have recently discussed image restoration with a team of one of the French forensics police departments. They have issues related to dirty/spider web nested lens, wrong illumation, occlusions related to poor camera location, very low frame rates in MPEG compression, shaking with low-res smartphone lens, proprietary video file formats. Problems are different from some idealized models used on academic papers (known psf, contraints), generally more complex, so movies/TV shows performance are overrated, as said in the following cartoon:

enter image description here

This team was using Amped Five solutions, but I understood that they would have been interesed in having muuch better.

Lecture notes in passing: Limits on Super-Resolution and How to Break Them, Bakker and Kanade, 2002.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.