Librosa stft + istft - Understanding my output (which always seems too perfect) at varying window lengths

Question

I've just started to use Python with Librosa for a DSP project I'll be working on. First thing I've been trying to do is determine my preferred parameters for the FFT window size, and hop-distance.

The domain is music, and my plan is to try a variety of values for the window size and hop distance, and for each of them, do a forward STFT and then an inverse STFT and write the result back out to wav file. I'll then listen to results and choose based on which values I think capture the information in the input the best.

My simple code is as follows:

import librosa.core as lc
import numpy as np
import scipy

_n_fft=80
print(str(_n_fft))
_hop_length=_n_fft/4

data, sampleRate = lc.load("13_Hate_To_See_Your_Heart_Break.wav", sr=44100, duration=20, mono=True)

stftMat = lc.stft(data, n_fft=_n_fft, hop_length=_hop_length, center=True)
iStftMat = lc.istft(stftMat, hop_length=_hop_length)

scipy.io.wavfile.write("testOut.wav", 44100, iStftMat)

powerMat = np.abs(stftMat)
print("powerMat shape = " + str(powerMat.shape))

The behavior I'm experiencing, however, is not what I would have expected.

When I use an incredibly short window length (as in the code above) - I get the correct number of window frames for my FFT length and hop-distance:

powerMat shape = (41, 44101)

44101 window makes sense, and as you can see the frequency resolution is low, with only 41 frequency bands. I would expect the resulting testOut.wav to sound pretty terrible, as the frequency resolution is so low. I can visibly see the effects on a rendered spectrogram as the subtleties in frequency changes are smeared together. Listening back, however, the resulting track sounds great - pretty much like the original input.

Compare this with a much wider window size of 44100 (1 window = 1 second of audio, hop-distance of 1/4*Window size):

powerMat shape = (22051, 81)

Again this output makes sense - in the 20 seconds of audio, with a window length of a second and a hop distance of a quarter second, there would be about 80 window frames. This is pretty poor time resolution, but fairly high frequency resolution with 22051 frequency bins. Again I would expect the resulting testOut.wav to sound poor in the time domain.

Once again the resulting track sounds great - pretty much like the original input. These extreme values, and everything in between, pretty much yield the same output testOut.wav, even though on the real power spectrum I can visibly see the differences when changing the parameters.

Is there a fundamental misunderstanding I'm having with the STFT and it's inverse? Am I simply not understanding the library?

Answer 1

The information you think is missing, due to either poor visible frequency resolution or poor visible time resolution, is actually still contained in the sequence of FFTs, but not in a form that can be easily visualized, especially by just displaying FFT magnitudes or spectrograms. That's because the information is "hidden" in the FFT phase results.

In the sequence of short FFT windows case, some of the frequency resolution information is represented in the difference in phase between adjacent or overlapped windows. In the long FFT window case, much of the time locality information is represented in both the absolute phases and in the difference in phases within groups of nearby FFT result bins. None of that is visible in FFT magnitude plots or spectrograms; and FFT phase and phase delta plots are hard to read.

But complete complex IFFTs do use the phase information contained in the real and imaginary components of the complex FFT results, and thus can reproduce both the time and the frequency information accurately and completely.

This phase information can be used in certain forms of FFT analysis to gain back some of the "missing" resolution. For instance, the phase vocoder algorithm deduces or uses greater frequency resolution by looking at the phase deltas between adjacent or overlapped FFT windows. And the time shift property of the FT shows how the phases between DFT result bins rotate relative to each other as an event (an AM or "beat" modulation, etc.) is rotated to different positions within a DFT window.

Answer 2

The behavior that you describe is perfectly normal. Actually, if you compute the short-term discrete Fourier (STDFT) transform of a time-domain signal first and then compute the inverse transform the output signal should be identical to the input signal, not just "pretty much" the same. Independent of the block length, the STDFT of a time-domain signal is a complete, invertible representation, containing all information that is required to go back to the time-domain and obtain the original signal.

So - to answer your two questions - it seems that there is a fundamental misunderstanding about how the short-term discrete Fourier transform and its inverse work. The output you obtain from the librosa library is perfectly fine.

I suggest you gather some information about short-term discrete Fourier transforms. Maybe it is a good idea to take step-by-step approach:

1. Read about the discrete Fourier transformation, which is the basic building block for the STDFT
2. Read about the (weighted) overlapp-add (WOLA) method
3. Combine both to obtain the STDFT

Your observation that the properties of the representation in the STDFT-domain vary depending on the block lengths and hop sizes, however, is correct. Usually, these parameters are chosen to obtain a satisfying compromise between time and frequency resolution when the goal is to do analysis or processing in the transform domain. If the values in the STDFT-domain are modified by some kind of processing algorithm, the behaviour usually changes in dependence on the specific choice of STDFT parameters. Then, the time-domain signal that is obtained by inverse transform also, indirectly, depends on these parameter values.

Answer 3

The STFT is an instance of a large family linear, redundant, invertible transformations belonging to time-frequency transforms. They can be expressed (forward and inverse transformation) in terms of redundant overlapped filter banks.

With classical orthogonal transforms, one usually have one sole inverse. With invertible redundant transformations, one turns the original $N$ samples of the signal to $M>N$ (complex) coefficients, in such a way that several inverses are possible.

In the most classical version, for specific window types, perfect inversion is secured to several hop distances.