8
$\begingroup$

I have a 22 kHz mono audio recording, which is mainly speech, a reading. I would like to upsample somehow to 44 kHz, to improve the audible quality. I have read about there are AI methods for upsampling pictures, even videos to a higher resolution. Maybe there is some similar methods for audio too.

The recording comes from radio broadcast, so the recording was excellent studio quality, streamed in 128 kbps, 44 kHz, stereo, but somebody, who did the recording, downsampled it from the original 44 kHz broadcast to 22 kHz, mono. Hence the sound quality is far from optimal, sounds like an old telephone, I think, due to the missing high frequencies.

The sound sample https://drive.google.com/file/d/1AqdQomHjpKqYM4Wq8SdHGOPDR8Irzjj3/view?usp=sharing

Improve this question
$\endgroup$
6
  • 5
    $\begingroup$ Normal speech seldom has frequencies above 11 kHz, so upsampling is unlikely to improve quality. Do you know what is the cause of the low quality? Maybe there are methods to improve that specific scenario. $\endgroup$
    – MBaz
    Dec 7, 2020 at 17:33
  • 4
    $\begingroup$ Notice that reproducing high-rate audio from low-rate data is exactly the problem a voice codec tries to solve; modern codecs did both: all the voice modelling that says "if you're voicing this and that thing, these are the frequency / harmonics involved" as well as the psychoacoustic modelling that says "if you hear this mixture of tones, these are the ones that don't need to be reproduced to give the impression of high quality". These things have been refined for 40 years – so maybe look into a high-quality vocoder to encode your 22 kHz audio, and play it back at 44 kHz. $\endgroup$
    – Marcus Müller
    Dec 7, 2020 at 17:36
  • 4
    $\begingroup$ As MBaz said, speech quality will (almost) not change even if you upsample 22 kHz to 44.1 kHz. But you should instead look for speech enhancement techniques, they will tend to make speech more intelligible, rather than hi-fidelity, according to various criteria that marcus have commented on. $\endgroup$
    – Fat32
    Dec 7, 2020 at 17:43
  • 3
    $\begingroup$ Perhaps this is not a traditional signal processing issue, but simply an issue to run it through an audio editor program like Audacity? It can be used to upsample, remove noise and perhaps apply compression to make it more intelligible. $\endgroup$
    – Justme
    Dec 7, 2020 at 18:14
  • 2
    $\begingroup$ Cheapest solution, DIY or via mechanical turk: transcribe the recording, record anew read by a voice actor. Unless there are other, unstated requirements (e.g. unique voice, unknown language, music, etc.), what computer and AI can do, is, ultimately, what humans can do, just faster. If the data set is small, humans are cheaper; the startup cost only makes sense when you have thousands of hours of recordings. For the kicks though? Go for it! 🖤 $\endgroup$
    – Dima Tisnek
    Dec 8, 2020 at 3:23

4 Answers 4

Reset to default
12
$\begingroup$

What you want to do is called "Super Resolution" in the field of imaging.
This is an ill posed problem.
Hence in order to solve it you need some prior / model about your audio data.

For example you may look at the model in Speech Super Resolution Using Parallel WaveNet.

Improve this answer
$\endgroup$
2
  • 3
    $\begingroup$ You probably meant "ill-posed" not "ill-poised" ... but only you can fix the typo. $\endgroup$
    – CrimsonDark
    Dec 9, 2020 at 11:21
  • $\begingroup$ @user02814, Indeed. Feel free to edit next time. Thanks... $\endgroup$
    – Mark
    Dec 12, 2020 at 20:37
9
$\begingroup$

While I don't know about tools for upsampling using AI, I am challenging the assumption that the main problem with your sample is lost high frequncies due to resampling from 44kHz to 22kHz, so "just" upsampling is unlikely to solve your root problem.

The possibility of AI upsampling itself is sensible, though: While standard (mathematical) upsampling yields the answer to the question "which 44kHz signal sounds most like this given 22kHz signal?", AI-based upsampling is intended to answer the completely different question "which 44kHz real-world signal is most likely to sound like this when you (re)sample it at 22kHz". If the down-sampling is in fact the root of your problem, AI upsampling might very well fix it, as long as the AI is trained with signals of the right kind. E.g. if you have an AI trained for upsampling music, the AI might "assume" the speaker is actually singing all the time, and add some tonality that doesn't belong into your signal - so be careful on what AI methods use use.

On the other hand, I see two more likely causes than being sampled to 22kHz that might cause the "old telephone line" effect. Old telephone lines are way worse than 22kHz/16 bit sampling. 16kHz sample rate is already considered to be "HD telephony" nowadays. If the sound of the voice itself resembles old telephone lines, it has been treated considerably more badly than being filtered to a bandwidth of 10kHz (what is needed for resampling to 22kHz). An actual telephone line limits the frequencies to 0.3 to 3.4 kHz, and coarsely matches 8kHz sample rate.

I suppose the root problem is one of these:

  • The 22kHz file has been encoded in a lossy way (like MP3 at 64kbit/s or even lower). Especially for old MP3 encoders, it's very common to put a low-pass filter in fron of enoding to limit the amount of frequencies that might need encoding and thus reduce the amount of data. If lossy encoding is the root of the problem, you still might have success using AI to reconstruct a better sounding signal, but you need an AI that is trained to losses caused by low-bitrate MP3 encoding, not an AI that is trained to losses caused by resampling to 22kHz.

  • The file you have is monaureal. It does not convey any information about the room the speaker(s) is/are in. It sounds like the/all speaker(s) are directly in front of you. If there is not much "room sound" in it, you can "fix" the problem by using a stereo reverb process that puts the sample into an artificial room and provides different signals to the left and right ear just due to the fact that the artificial reflections from the left or right wall sound different. Furthermore, if there are multiple speakers, you might need to seperate them (put them at different locations in the virtual room). The easiest way would be to pan the sample to different stereo positions for different speakers. If multiple speakers are talking at the same time, separating them is a hard task. Even after panning the signal, adding some stereo reverb might help the sound.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Yes, higher frequencies are missing, due to downsampling, when encoding, so speech lose the clarity. It sounds like it is coming from behind a mask... Also the encoder was lossy mp3 - of course. $\endgroup$
    – Konstantin
    Dec 9, 2020 at 0:18
  • 3
    $\begingroup$ In that case (yor sample is in fact MP3 encoded), I maintain the stance that the most significant damage to the sound quality is not from down-sampling it to 22kHz, but from encoding it to MP3. If I connect "monaureal", "22kHz" and "MP3", my guess is at a bitrate between 32kbit/s and 64kbit/s. Especially 32kbit/s is detrimental to quality. The positive side is: There is some research into "better sounding MP3 reproduction", e.g. the SBR (spectral band replication) approach used in MP3Pro. $\endgroup$
    – Michael Karcher
    Dec 9, 2020 at 0:31
4
$\begingroup$

As pointed in some of the comments, there are several issues involved in your problem that you should consider to go for an optimal solution:

So, apart from the Speech Super Resolution approach proposed in the previous post, you could try some other approaches such as:

Cheers.

Improve this answer
$\endgroup$
2
$\begingroup$

Add high frequency noise.

While the other answers provide extremely valid points, adding noise is a very simple "solution" (deliberately placed in quotes) to make the sound appear more "brilliant".

Obviously, it makes more sense to add high frequency noise only in proportion to already higher frequency signal already present in the signal.

Now, noise does not provide any more data; and it will not make the signal more intelligible. It will just add to the listener's impression that there is high frequency content.

It appears that the original apt-X codec (aptX since 2010) has this as a side effect and if therefore often considered to "reproduce high frequency signals better than SBC".

Note that I am aware that this answer has to do more with psychoacoustics than with signal processing. I just want to provide a different angle to the issue. I am very much well aware that "adding noise" is usually not a solution but rather a problem.

This has nothing to do with CNG (comfort noise generation), which is meant to "remove silence".

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.