What is a more accurate alternative to a spectrogram that can be represented in an image? Also, what is a python script/exe that can be used to convert audio into the image format then back from image to audio?
closed as unclear what you're asking by Stanley Pawlukiewicz, MBaz, hotpaw2, lennon310, Peter K.♦ Jun 20 at 15:59
Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.
The ltfatpy 1.0.16 package
is a partial Python port of the Large Time/Frequency Analysis Toolbox (LTFAT), a MATLAB®/Octave toolbox for working with time-frequency analysis and synthesis.
Among linear an quadratic time-frequency, there is a large number of options for sharper analysis tools converting a 1D signal into 2D data. You can even get more information by decomposing onto 3D or 4D spaces (coming back on that later). Quadratic (or higher-order) representations are usually more difficult to invert than linear ones.
For real valued signals (i.e. contains only positive frequencies) the Wigner-Ville (WV) distribution is a popular method. Check out this page for more information.
The WV method provides some better localization than your typical spectrogram is capable of. That webpage I linked has some great examples, namely the linear-frequency modulation "chirp" signal example. While the WV method has some "spreading" at the start/end of the signal, it's time/frequency resolution is greatly superior to the spectrogram method in the example shown. WV does have some issues with highly transient signals, so you specific application may influence the results.
Depending on your signal and what you're trying to do, other decomposition methods of signals into an "image" exist as well, notably cyclo-stationary methods like the spectral correlation function, as well as ambiguity functions. These methods aren't universal, but they can be really useful for things like automatic modulation recognition.
Usually when talking about accuracy the classical Fourier Uncertainty principle
enters the conversation (I would skip the quantum mechanics stuff). Essentially you can have good resolution in frequency or good resolution in time but there is a tradeoff between the two.
In order to go back to a time series from an image, you at a minimum need to retain phase information which is not included in the image. There is also the issue of how you map a magnitude level of a STFT bin to a pixel value, which usually involve some nonlinear transform (including clipping). The eye and ear are very different in how they perceive a level. It is much simpler to just keep a copy of the time series and use the image to index back into the section of signal you might be interested in.
The utility of STFT based images is that given the roughly 100ms delay that a human brain needs to perceive something, for audio signals, you can craft a real time implementation where you can simultaneously listen and look at the signal. One can use tricks like averaging, maxing , along with decimation to get a useful "real time" waterfall time/frequency spectral analyzer. These sorts of processing are specific to particular application. You would use different techniques for something like looking for low SNR tones in SONAR than those for clean music. This makes going from image information back to time series more difficult ( i really want to say impossible but one needs to be careful with that word).
The STFT is linear. The STFT of 2 signals is the sum of the STFTs of each.
are also linear and can produce images and relatable to the STFT
For high SNR signals, people have also created time/frequency images using AR models by stitching together segment line by line.
These typically have good frequency accuracy of peaks. but not so good at troughs.
Before I go further, there is an example that might be useful about redundancy in time/frequency images. Led Zepplin's, Stairway to Heaven is 482 seconds long. At a nominal 44100 samples/second, the time series is 21,168,000 samples. If I choose a 1024 point FFT at 50% overlap, I will have a 41,344 by 512 array, which I can reasonably create a scrollable image.
If you want higher "time resolution" you increase the overlap, so 75% overlap produces 82,687 by 512 array which can be scrolled but it listening and seeing simultaneously becomes more problematic.
One can use some tricks to reduce the scan rate of wavelet grams but it is harder if you want to listen and see simultaneously. The same is true of AR grams.
As mentioned in the other answer by @mathewpollard there is Wigner Ville (WV) which is nonlinear. WV is a special case of the "Cohen Class" which also includes Choi Williams.
I had the privilege of attending a joint lecture by Leon Cohen and Al Nuttal a long time ago. I remember a quote from Nuttal, "Wigner Ville is like small child, when good it is very good, when bad it is very bad".
A problem with WV is that it is nonlinear, there are cross terms. The image of 2 simultaneous signals is not the sum of the individual images (neglecting pixel level mapping issues) . There are artifacts in the image that don't belong there. There are modifications to WV that mitigate the cross term problem. People do use WV. You need to understand it.
The other issue (which may not be your issue) is the redundancy of WV. In its simplest form, you update it every new sample. The pixels will fly by your display faster than you can perceive them in human real time.
The strength of STFT based grams for audio signals is the ability to listen and see simultaneously. It is much easier for an analyst to relate the spectral feature to the sound it makes and identifying the cause . Understanding the data is more important in many cases than accuracy.
A long enough waveform plot at a high enough resolution allows reproducing sound from an image. Old movie film projectors reproduced sound this way (converting optical density images into amplitudes).