0
$\begingroup$

In the last few years, there have been many breakthroughs in the image processing world regarding repairing "damaged" images, images with corrupt pixels, or even reversing artefacts from formats such as JPEG, which have received media attention. I am trying to determine if these methodologies can similarly be used to predict future samples of audio, rather than pixels of an image.

Some examples:

much of which derives from an NVIDIA thesis published in EECV 2018: Image Inpainting for Irregular Holes Using Partial Convolutions, EECV 2018.

In theory, this basic method should have a direct analog to audio: where, rather than predicting missing pixels, we're trying to predict the next ms or so of future audio from the last second. So, it can really be thought of as a type of nonlinear prediction, which clearly performs very well.

QUESTION: What is the state of the art in using this method for audio processing rather than image processing? Is there any published literature on this? Does anyone have any references?

Below are some examples of the technique that I am talking about in the image processing realm. In all these situations, the missing pixel data is reconstructed from the remaining pixels, which is basically 2D sample prediction; I am hoping to get some insight into the 1D analog of this.

Repairing damaged images Reversing artefacts Comparison between ground truth and prediction enter image description here

$\endgroup$
13
  • 3
    $\begingroup$ uff, I basically always write the same to questions that ask "what is the current state of the art": You'll have to read all the papers in the field, and then have to ask all researchers, including these that don't regularly publish. That doesn't sound like a feasible thing to ask for – I think you didn't even mean to ask that. You meant to ask for a solution for a class of problems you'd like to solve! Maybe instead of adding a lot of images that are unrelated to the problem you're trying to solve, maybe actually describe that class of problems in more detail and ask how to solve them :) $\endgroup$ – Marcus Müller Oct 24 '20 at 21:48
  • 2
    $\begingroup$ @MikeBattaglia This paper will give you a hook into some of the machine learning literature in that area: arxiv.org/abs/2003.07704 $\endgroup$ – datageist Oct 25 '20 at 3:14
  • 3
    $\begingroup$ Do you wish to minimize the total of the error between each original and the corresponding predicted sample value, or to generate audio that sounds convincing or in the least not disturbing? If future original samples are not input to the generator, then this is audio outpainting or extrapolation. In packet audio with enough buffering at the audio sink, packet loss concealment might be best done by inpainting. Outpainting is simpler to study, as the pattern of missing samples is always the same ...000000000111111111... $\endgroup$ – Olli Niemitalo Oct 25 '20 at 7:27
  • 3
    $\begingroup$ Consider that the original audio samples are random and from identical and independent zero-mean distributions, so white noise. In the first approach, predicting that there is only silence minimizes the square of the prediction error at each sample. As an example of the second, you would generate noise that sounds the same, so that the listener could not tell that the audio was generated rather than the original audio continuing. $\endgroup$ – Olli Niemitalo Oct 25 '20 at 8:03
  • 1
    $\begingroup$ Definitely they are of the second type, as they do not impose that the distributions of the samples are independent. Unless you are interested in lossless compression, I think the second type is what you should ask about. Some inpainting methods of the second type sample from an approximate joint distribution of the samples in the training data, conditioned on the surrounding image data. But you probably shouldn't limit your question to just those methods by specifying that, unless it is vital to, say, avoid exacerbating racial biases in the generated data. $\endgroup$ – Olli Niemitalo Oct 26 '20 at 6:18
2
$\begingroup$

i am also quite skeptical of the presented image restoration.

in audio, there are papers like this and this. i am sure there are more. i just stumbled upon them by googling "AES interpolating gaps in audio data".

$\endgroup$
2
$\begingroup$

These Images are FAKE !!!

There is absolutely no way, no algorithm, no nothing that exist that can tell you whether the eyes of a girl are blue or black by just looking at her lips !!!

You can guess, something, good or bad, based on your prior training and the probabilities based on that, but that's just a guess.

The fact that the synthesized images look so natural is the key to deceive people into thinking that the algorithm correctly predicts the impossible to fix missing information.

No algorithms can tell whether she had a tattoo in her screwed part of the image !!!

What the algorithm is doing is that you feed it with correct information in many many other training images, from different angles, different illuminations etc. Then the algorithm will try at best (and may succeed just as painting artist would do) to replace the missing pixels from those of the available set... And if she had a tattoo in the mean time; forget about it !

Same is for audio...

$\endgroup$
7
  • 1
    $\begingroup$ "These Images are FAKE !!!" - alrighty then. $\endgroup$ – Mike Battaglia Oct 26 '20 at 1:07
  • $\begingroup$ Anyway, these images are examples from published literature, which is explicitly linked to, of a deep learning-based image inpainting algorithm, which I am looking for audio analogues of. $\endgroup$ – Mike Battaglia Oct 26 '20 at 1:10
  • 1
    $\begingroup$ @MikeBattaglia If it's possible to predict missing information (hence future), then DO PREDICT my next comment using you AI deep learning resources ;-))) $\endgroup$ – Fat32 Oct 26 '20 at 1:30
  • 2
    $\begingroup$ i am also quite skeptical. but if you have some cited references for these reconstructed photos, i would be interested in seeing such. $\endgroup$ – robert bristow-johnson Oct 26 '20 at 2:36
  • 2
    $\begingroup$ The models don't recover ground truth, they just fill in the gaps with something plausible (cf. youtube.com/watch?v=gg0F5JjKmhA) $\endgroup$ – datageist Oct 26 '20 at 6:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.