I would like to make a large set of videos more visually pleasing by adjusting colors, saturation, contrast, etc. However, the techniques that I can find online roughly fall into these two categories:

  1. Clever mathematics heuristics to automatically adjust sharpness, color balance, contrast, etc.
  2. Deep learning models operating on pixels directly, producing pixels as output.

But I want to go beyond heuristics and apply "intelligence" to the color adjustments, eg. so that skin color does not become weirdly red. However, applying neural networks to produce each frame is undesirable because it is often very compute intensive, since I would then need to process all frames (30 frames per second) instead of just applying the neural network to eg. 1 frame per second, smoothing the output and then apply a cheap transformation to all frames.

So I was hoping to find some research, benchmark datasets, huggingface models, etc. on models that could suggest a filter to apply, either from a set of pre-defined filters, or by outputting a complete histogram transformation to apply to the image.

However, I haven't been able to find anything.

Is that because this is not a good idea for some reason?? Or what is this kind of research called, eg. what should I search for? And do there exist any benchmark datasets for this kind of stuff?

  • 2
    $\begingroup$ you will need to define what "enhance" means to you; I think that's really why your research turned up few results: you didn't look for the right things. $\endgroup$ Feb 20 at 19:22
  • 1
    $\begingroup$ "undesirable because compute intensive": 90% of these arguments on this site boil down to "can you actually put a number on what you consider to be compute intense", and then either it turns out that things aren't as complicated as thought, or even the simplest possible operation is more intense than what the asker demands, and the requirement is impossible to fulfill. So, make a clear statement on what you find "too intense"; for example, calculating the necessary color adjustments doesn't have to happen every frame. So, does it really matter how complex it is? $\endgroup$ Feb 20 at 19:24
  • 1
    $\begingroup$ "frames can't be modified individually due to flickering" so, what? Give your adjustment's coefficients (no matter how these look like) some memory. After all, you're building a control system here, and any control system has a trajectory that you can influence by choosing integral / differential aspects of your control. $\endgroup$ Feb 20 at 19:25
  • $\begingroup$ @MarcusMüller Thanks for taking the time to point out the many unclear things about my question. Very useful feedback. I have tried improving the question. Defining enhancement: I think you are right. I need to figure out what professionals call this. But to me, it is just making the video visually pleasing to most people - very vague, I know. Compute intensive: I just don't want to be forced to run on every frame, that's it. As long as I can skip some frames I can tweak it and find the sweet spot. Never mind the flickering I mention. Poorly explained and not important. $\endgroup$
    – elgehelge
    Feb 20 at 21:00
  • $\begingroup$ you will have to modify every frame, so it's not logical at all to say you don't want to "run" it. Anyways, nobody forces you to. It's just that if you don't analyze every frame, then some frames will simply not affect the result. Whether or that is acceptable: impossible for us to know, and essentially again inconsequential, since your control system would sort-of "interpolate" between the information you get. $\endgroup$ Feb 20 at 21:02


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.