0
$\begingroup$

I was trying to analyse the lossy compression of MP3 audio files through their Spectrograms and the results were something like this (where the upper is 80Kbps and lower is 128Kbps).

enter image description here

It is obvious that the compression works by deleting higher frequencies from the audio. This is visible from the Spectrogram itself so I wanted to know

  1. How exactly does the compression take place behind the simple ffmpeg commands?
  2. Are spectrograms the best way to analyse the change in bitrates of an audio file?
  3. The STFT used, gives us the phase and the amplitude from which we take the magnitude and plot the spectrogram. So how different can I expect the phases of the higher and lower bitrates to be?
  4. Is it possible to somehow recover the lost frequencies because ffmpeg based bitrate upscaling doesn't produce accurate results on the Spectrogram

I am from a CSE background so I will be grateful if someone can answer these questions or atleast refer me to some article or paper regarding the same.

$\endgroup$
7
  • 1
    $\begingroup$ 1. is "the simple ffmpeg command applies an encoder", and MP3 is really explained in a thousand places on the internet (also, describing an MPEG audio codec "exactly" in an answer is a bit too much for an answer here) $\endgroup$ – Marcus Müller Dec 3 '20 at 9:32
  • $\begingroup$ 2. no; 4. don't know what you mean with "accurate results" $\endgroup$ – Marcus Müller Dec 3 '20 at 9:33
  • $\begingroup$ 1. I am aware about the "applies an encoder" part but I was looking for a mathematical approach to it. Thats why I asked to be linked to a suitable article so that I can read it up myself than someone having to explain it here 2. So what is the best way? 4. "Accurate results" meant proper reconstruction of the frequency spectrum. I know this is a region under active research but what I meant to ask was "What is the current best way to recover the spectrum from the years of research and what are the problems faced here" $\endgroup$ – Darshan Deshpande Dec 3 '20 at 10:11
  • $\begingroup$ Re1: If in doubt, start at wikipedia: their MP3 article links to articles and the ultimate reference: the mpeg standard. 2. looking at the bitrate? I'm not sure what else there is to do when looking for a bitrate - but maybe I'm misunderstanding your estimation problem. 4. the spectrogram is exactly accurate when you want that spectrogram – you'll find it mathematically impossible to get arbitrary temporal and frequency resolution, so it's not quite clear what is more or less accurate. Maybe explain what kind of information you want to extract from the spectrum? $\endgroup$ – Marcus Müller Dec 3 '20 at 10:30
  • $\begingroup$ (Phrasing it "What's the current state of the art in spectral estimation" basically makes an answer to your question require a whole library to be shipped to you, so that's really too broad. Really, maybe you could describe what you're doing, and most importantly, why, so that we have a good chance at specifically writing an answer.) $\endgroup$ – Marcus Müller Dec 3 '20 at 10:31
0
$\begingroup$

How exactly does the compression take place behind the simple ffmpeg commands?

MP3 is a perceptual coder. It creates a time/frequency representation and quantizes in the frequency domain primarily by calculating the spectral and temporal masking thresholds of human auditory perception. It maximizes the masking of the quantization noise to minimize the audibility of the quantization noise.

Are spectrograms the best way to analyse the change in bitrates of an audio file?

No. Depends a bit on what you want to analyze, but a spectrogram doesn't correlate well with the audibility of compression artifacts. Most of problematic artifacts are temporal anyway, the loss of a few high frequencies is a comparatively smaller perceptual problem.

The STFT used, gives us the phase and the amplitude from which we take the magnitude and plot the spectrogram. So how different can I expect the phases of the higher and lower bitrates to be?

Phase will be fairly close for all spectral components that are above the masking threshold. Not so much for low energy components.

Is it possible to somehow recover the lost frequencies because ffmpeg based bitrate upscaling doesn't produce accurate results on the Spectrogram

No. If that would be useful in any way, the original inventors of MP3 (or AAC, DOlby, DTS, WMA, etc.) , would have included it in the codec. Many people have tried but it turns out the harm outweighs the benefit.

$\endgroup$
7
  • $\begingroup$ Thanks a lot for the answer! You said that "most artifacts are temporal anyway". Could you explain what this means exactly? How exactly does the encoder introduce these temporal artifacts? As I said I am an ML Engineer and I have gotten decent results with the spectrogram reconstruction ( sample here ) but the audio still doesn't sound the same and the answer probably lies with what you said about the temporal artifacts. It would be great if you could help me understand this! $\endgroup$ – Darshan Deshpande Dec 3 '20 at 13:08
  • $\begingroup$ @hilmar spectral band replication is a standard tool in some of the AAC flavours. Depending on viewpoint, you could say that it is about «recovering lost frequencies», or embedding a parametric model into the codec? $\endgroup$ – Knut Inge Dec 3 '20 at 13:54
  • $\begingroup$ If the OP is a machine learner, it might be possible to train ML to recover the most likely source signal from a compressed stream, given a suitable training set and assuming the non-ideal compressed streams out there. If that technique turns out to be efficient, one would expect it to be embraced by codecs. The advantage of the latter is that it would be a closed loop rather than guesswork. $\endgroup$ – Knut Inge Dec 3 '20 at 13:59
  • $\begingroup$ @KnutInge Yes I have managed to capture the temporal features through the spectrogram and I am able to achieve a decent result (if not the ideal. I am actively working on improving it) of recovering the lost frequencies but unfortunately my problem is that the re-created audio doesn't "sound" a whole lot different from the input data which got me to this conclusion that deletion of frequencies cannot be the only factor for the "irrecoverable" nature. I was just asking what could be the other factors $\endgroup$ – Darshan Deshpande Dec 3 '20 at 14:18
  • 1
    $\begingroup$ Lossy music codecs use some time/frequency analysis filterbank where different time/frequency values are quantized differently. It is not only about «deleting frequencies», but also representing energy in an approximate manner. If your ML could truely «understand» the score of the music played and the instrumentation used, then it could possibly replace distorted musical sounds with cleaner sounds $\endgroup$ – Knut Inge Dec 3 '20 at 15:01
0
$\begingroup$
  1. Encoding isn’t specified in MP3, only decoding, so there isn’t really a answer. That being said, the compression used is Huffman encoding. Huffman works best when data is repeated, that’s why you see so many empty frequencies, because compressing a bunch of zeroes is very efficient.

  2. The frame/metadata would be more accurate.

  3. Anywhere from 0 to 2pi radians I suppose. Even if you found some correlation, it would only apply to one or several encoders.

  4. No, that data is quantized to zero, it’s gone forever. Anything you added would only be to suit your personal tastes, and would not conform to the MP3 spec.

$\endgroup$
5
  • 1
    $\begingroup$ Thanks for the answer but Huffman encoded strings are possible to decode. Am I missing something here? $\endgroup$ – Darshan Deshpande Dec 3 '20 at 14:26
  • $\begingroup$ Huffman encoding certainly can be decoded. MP3 does not specify how the input data is analyzed/quantized before Huffman encoding. For that reason, the maths behind encoding can be proprietary. However, the objective is to set data to zero to improve the efficiency of the compression. $\endgroup$ – Dan Szabo Dec 3 '20 at 14:42
  • $\begingroup$ Got it. I don't see why someone would keep the math proprietary and still release the idea into the market for everyone to use, especially considering the amount of encoders present in the market. Anyway, thanks for the explanation! $\endgroup$ – Darshan Deshpande Dec 3 '20 at 15:46
  • $\begingroup$ As I understand it, the decoders are spec’d to ensure consistency between products, but the encoders aren’t to foster innovation. That being said, there are open source implementations you could use for reference. LAME comes to mind. However, my experience is that encoders are hard to understand, and efforts are better spent looking at how decoders work. All decoders should give the same output for the same input. Cheers. $\endgroup$ – Dan Szabo Dec 3 '20 at 16:30
  • 1
    $\begingroup$ Basically, the decoder will specify a framework describing how a given bitstream will be interpreted in a «human-relevant» time-frequency representation before being decoded to a waveform. This architecture strongly hints at a possible encoder implementation, but there is still room for proprietary stuff in how the encoder prioritize bits within that framework, depending on its model of human hearing and testing of those decelopers. $\endgroup$ – Knut Inge Dec 3 '20 at 17:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.