1
$\begingroup$

I assume a standard/ old school super resolution techniques use Polynomial interpolation and Upsampling approaches. Why would deep neural networks achieve better results?

$\endgroup$
  • 2
    $\begingroup$ This is assuming they do... $\endgroup$ – Marcus Müller Apr 23 '18 at 8:29
  • 2
    $\begingroup$ There is a generic discussion on an extended topic on Is deep learning killing image processing/computer vision?. $\endgroup$ – Laurent Duval Apr 23 '18 at 10:40
  • 1
    $\begingroup$ @MarcusMüller what do you mean? $\endgroup$ – 0x90 Apr 23 '18 at 13:57
  • 2
    $\begingroup$ @0x90 you're simply assuming that you use a deep neural network that delivers better results. That's a big claim – these things are trained to something specific, and you don't even say what "better results" are, so without a metric, it's hard to make any qualified statement. So, if you ask me, I'd argue that they won't in general, but would possibly do for specific classes of images $\endgroup$ – Marcus Müller Apr 24 '18 at 7:45
6
$\begingroup$

The objective is to derive a higher resolution image from a low resolution image. When performing this upsampling, inevitably, you have to fill in information that you do not have.

The differences between the two approaches are in how you fill that information in.

In the standard techniques an underlying model is assumed. So, for example, in nearest neighbour interpolation we decide to fill in the unknown pixels with the value of the nearest pixel. An improvement over that is linear interpolation where we derive a weighted sum of the pixel values around the unknown pixel to set its value. An improvement over that are spline and cubic interpolation which are trying to derive the value of the unknown pixel from a bigger patch of known pixels around it.

In all of these cases, we are assuming a model for filling in the pixel values which is based on physics (e.g. due to the Point Spread Function) or assumptions.

But, how about deriving the "model" that constraints the values of the pixels from the images themselves and then use that model and the limited known data to guess the unknown pixels? Instead of deciding on an interpolant arbitrarily, we derive it.

This is an optimisation problem where we are trying to discover the function that connects the low and high resolution image versions. To do this, you obviously need both the high and low resolution images. But what you can do, is take (known) high resolution images, degrade them (reduce their resolution) and workout the relationship between the high resolution and low resolution.

The surprising result here is that images depicting different content end up having very similar interpolants. Of course, content does play a role in the sense that if you derive models from a set of images with a specific bandwidth, you will discover models that support that specific bandwidth. You cannot train the model on smoothly varying images with lots of low frequency content and expect it to reconstruct high contrast high details parts of unknown images.

Therefore, the technique is more suitable to classes of images and so far it has been applied to medical imaging (and of course Magnetic Resonance Imaging where there is a long history of reconstruction from limited data) and radio astronomy

For more information you might want to see this and this.

Hope this helps.

$\endgroup$
1
$\begingroup$

In super resolution problem, the goal is finding sub pixel values based on multiple observations or a prediction.If we know values of black, red, blue and yellow pixels, there is absolutely no way to know even guess the middle pixel, if we do not have any priory knowledge. They are just four random values on a grid that may or may not have any relation to each other.

enter image description here

To extend of my knowledge, prediction of the value works based on some assumption or a priory knowledge. For example one of the common assumptions is that natural images are mostly low pass and there is no sudden sharp transition in images. Pixel interpolation works based on this assumption.

If we do not have a model, then we have to train one. That is where machine learning based super resolution enters. But, train it on the basis of which features? If we look more carefully we notice typical images have typical feature. Image of an ambush always has a bunch of trees, sky (partly) and so on. The problem is that these features are so diverse and even hard to find, that it is hopeless to try to gather all and train our model based on those for all types of images in the world. A Deep Neural Network automatically finds features associated with each theme and class of images and other than that when it is presented with an image it can associated the image content with appropriate feature. Like, DNN might say aha! This low quality photo seems to be a jungle so these green things maybe are trees, I know how this type of trees (or leaf) look like, so it modifies that part of the picture with its own image of the tree or leaf.

I would like to give an analogy . I think this is somehow similar to what the brain of a painter, let say a nature painter, does based on his memories of a scene . Since he have seen and explorer lots of sample scenes closely and in high resolution he has developed a very detailed image of a flower, a pile of snow or anything else in his mind. So, when he is asked to draw, he sketches a rough image then fills the image elements with details he already seen.

$\endgroup$
  • $\begingroup$ Is it connected to phase contrast imaging? $\endgroup$ – 0x90 Apr 24 '18 at 0:33
  • $\begingroup$ @0x90 unfortunately I am not familiar with that topic.But skimming through the link you've mentioned, I guess no . $\endgroup$ – MimSaad Apr 24 '18 at 7:59
  • $\begingroup$ I respectfully disagree with the background on this. Human's are slow and have limited experience. I think it would be natural that an entity/AI could examine a million cats it could achieve a higher score than me. What I don't accept is the AI communities tendency to abrogate responsibility and not try to see what a superior algorithm is doing. Extending our intuition improves us; letting somebody (robot) do the work doesn't. I have always thought (and used) modeling to improve my understanding; not point to a modeling result and say "hay there it is"; take it or leave it. $\endgroup$ – rrogers Apr 24 '18 at 20:35
  • $\begingroup$ Thank you for sharing your thoughts. I would agree in limited experience of humans and our superior ability of generalization, maybe someday more accurate DNN models show up that can even mimic that ability, I wish I had but I do not have much neuroscience background :). The analogy given here meant for intuitive understanding of what a DNN does. You say it is a wrong analogy? well, maybe , who am I to oppose ;) $\endgroup$ – MimSaad Apr 24 '18 at 21:18
  • $\begingroup$ No, I was not disagreeing with you just "venting" a little. If you like I will delete the comment as unhelpful. In fact, I came back here to do that. $\endgroup$ – rrogers Apr 24 '18 at 21:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.