3
$\begingroup$

Spectrograms convert an amplitude-time signal into a frequency-time signal using Short Time Fourier Transform (STFT). The window size L used in STFT is a design choice. As L increases, you get higher frequency resolution and lower time resolution and vice versa.

If you performed STFT with a large L to get spectrogram A with high frequency resolution then performed STFT with small L to get spectrogram B with high temporal resolution, can you combine spectrogram A and B somehow to get high temporal and frequency resolution?

My gut feeling is no, as changing L is simply "changing how you interpret the amplitude-time data."

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ You could even take it to the next level and analyze the data for chirps so that you have high resolution on rising and falling frequencies. Those are smudged by STFT of any window size. $\endgroup$
    – Olli Niemitalo
    Commented Oct 31, 2017 at 9:05
  • $\begingroup$ One reasonable alternative is take the STFT's of length L every L/2 samples. That is to say, make the windows overlap. The window function applied will tend to 0 near the edges of the windows, so using this hop size of L/2 helps to ensure that each sample will be close to the middle of at least one STFT. This effectively doubles the temporal resolution without a loss of frequency resolution. $\endgroup$
    – MSalters
    Commented Aug 3, 2020 at 11:07

2 Answers 2

Reset to default
2
$\begingroup$

Improving the time/frequency resolution balance remains a standard research issue, trying to adapt the STFT, inventing novel/better time-frequency distributions (Wigner, Cohen), using together or combining the above. For instance in music or speech source separation, a couple of STFTs is often used with both short and long windows, to be able to separate transient from larger signals.

The ways you can combine such SFTF depend a lot on the exact processing or features you want to extract, as detailed by @hotpaw2. Here are a few pointers to references suggesting methods. The first one suggests a point-wise combination of two spectrograms as $$S_c(f,t) = (|S_1(f,t)||S_2(f,t)|)^{1/2},$$ and the second generalizes it to the geometric mean, the minimax or the reciprocal average. The third encompasses it in a more generic merging approach.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Is it possible for these combined multi-resolution spectrograms to be invertible? $\endgroup$
    – Sevag
    Commented Jan 24, 2021 at 18:03
  • $\begingroup$ Yes, if each full "spectrogram" is invertible, their union is invertible as well. If you only have magnitudes, due to the redundancy, there are options still for recovery (quite technical) $\endgroup$
    – Laurent Duval
    Commented Jan 24, 2021 at 18:27
1
$\begingroup$

Yes. Of course you can combine them. The question is how to choose which way to combine them, and whether the combined result provides any useful information gain (highlighting certain types of spectra), or value (e.g. looks "nicer", more colorful, etc.)

One method might be to upsample or resample each STFT set, one in the time dimension and one in the frequency dimension, until you get a matching set of 2D matrix sizes and grid points. Then combine time-frequency grid points using some algorithm chosen depending on your purpose. For instance you can give each matrix a different color look-up table, and then mix the two colors (with a 2D LUT, or do 3 STFTs and mix RGB). Or pick the max or min or average or use some non-linear 2D filter kernel, such as a threshold gate, or median filter, or highest contrast decision mux, or best looking color scheme chooser, and etc.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.