Audio compression with STFT

Let STFT[frame, freqbin] be the short-time Fourier transform of some audio data. (here frame is the index in time, and freqbin is the index in frequency).

Very often, STFT[k, f0] will be close to STFT[k+1, f0]...

... then it could be interesting to store, for each frequency bin f0, only the differences :

STFT[0, f0], STFT[1, f0] - STFT[0, f0], STFT[2, f0] - STFT[1, f0], STFT[3, f0] - STFT[2, f0], etc.


(such processing is known as delta encoding) ; these differences will be small on average, and this could lead to a reduction of the average bit usage for storage of the audio data.

Can this idea be turned into an efficient audio data compression ?

(I don't speak here about psychoacoustic effects that could lead to use different levels of quantizations for each frequency / or frequency masking ; I know that such ideas are used in MP3 compression).

Edit: as mentionned by @pichenettes, the graph above showing continuity between adjacent frames is the "magnitude plot". The "phase / angle plot" is a bit less continuous :

I see two big drawbacks to your method:

• The image you posted is not the STFT, but its magnitude. The difference between the magnitude of adjacent STFT frames is small; so for your scheme to work you would have to store the magnitude and phase separately. But then, there would be no compression at all on the phase data that would still represent 50% of the data and would be required for reconstructing the signal. Try plotting the phase (angle) of the STFT to see how ugly it is.
• Assuming the deltas are encoded using variable-length integers or Golomb codes (I assume that's the idea behind it...), your method would compress large and temporally stable values in the spectrum (the red "lines" on the spectrogram image). It's obvious that these represent only a small fraction of the values. There would be no gain for the background "noise" which amounts for a very large portion of the spectrogram magnitude data.

A much more efficient way of getting rid of these stable/predictable peaks in the spectrogram is to whiten the signal and code the residual+the filter coefficients. This is (among other things) the principle of the FLAC lossless codec.

• Great answer. I'll have to check out the FLAC CODEC. – user2718 Mar 28 '14 at 14:53
• Thanks @pichenettes for your answer. I assumed that phase/angle would have some sort of continuity as well, but I was wrong : I edited the question and added a plot of the phase : it is terribly uncontinuous! But : is there absolutely no way to show some regularity in the phase of adjacent frames ? Is it similar to a random gaussian vector ? /// About your second point, yes I planned to use Golomb codes. For "red lines", the differences will be small, but for the background noise, it is already small, so I expected the difference to be not so big as well. What do you think ? – Basj Mar 28 '14 at 14:54
• @pichenettes Could you explain your last paragraph : what is the benefit of whitening the signal? It will show less regularity if it is whitened, or am I wrong ? Then it will be harder to compress ? – Basj Mar 28 '14 at 15:11
• You could measure the difference in angle between the phase at time t and t-1 ; and add this to the phase at time t to get an estimate of the phase at t+1. Then the difference modulo 2 pi between this estimate and the actual phase at t+1 would be small in the parts of the spectrum corresponding to stable sounds. In other words, the second order derivative of the angle plot would be continuous... modulo 2pi... in some areas. – pichenettes Mar 28 '14 at 16:15
• @Basj : The phase may look random. But the first derivative of the unwrapped phase of fftshifted data may be fairly stable for stationary spectral peaks. – hotpaw2 Mar 28 '14 at 17:47

Audio compression has been done with wavelet transforms which have some relation to the STFT. The wavelet transform, like the STFT is a time/frequency transform but has forms that are specifically designed to be invertible whereas practical inversion of the STFT is problematic. The STFT resembles continuous wavelet transforms which are better suited for analysis than synthesis (reconstruction). The discrete wavelet transform was developed in a way that is more suitable for synthesis and thus applications such as compression. The practical ability to invert discrete wavelet transforms is achieved by defining the discrete wavelet transform such that redundant information is eliminated between the coefficients computed using the transform.

Unlike the STFT which employs the complex exponential as its kernel, many different functions have been developed for use as wavelet transform kernels, some of which are real functions. Wavelet kernels can be developed for specific tasks. The kernel, which can be loosely referred to as a wavelet, need only satisfy a set of admissibility conditions.

The coefficients that are computed using a discrete wavelet transform follow an order associated with time duration. The lowest order coefficients represent long time duration (often associated with low frequencies) while higher order coefficients represent short rime duration (representing detail and often associated with high frequencies). With band limited signals having a lowpass characteristic (such as voice), the higher order coefficients tend to vanish. One technique that is used for signal compression is to eliminate all coefficients above a certain order (think higher frequencies). There is an obvious trade off between the highest order of coefficients retained for signal reconstruction and fidelity. This method is distinctly different than the one you proposed which uses the difference between adjacent frequency bins.

A good non technical introduction to wavelet analysis is:

"The World According to Wavelets" by Barbera Burke Hubbard. There are lots of sites on the web dedicated to the theory and application of wavelets. Google wavelet audio compression and you will find many links.