0
$\begingroup$

Spectrogram renders of audio are limited in their representation of audio. For example, one could not use a Spectrogram to re-render the original audio.

What are some alternatives to Spectrogram charts for rendering representations of audio?

Preferably this would be methods which are more accurate (in every possible way/in every possible audio dimension) for representing audio in graphical format (an image)?

A good measure of accuracy would be for example;

  1. Could the image be used to (at least to some extent) re-create the audio file?
  2. Could the image be used to match the similarity between two songs in some or all dimensions?

And, are there any scripts/tools/software's that can be used for converting audio into an image format?

$\endgroup$

closed as unclear what you're asking by Stanley Pawlukiewicz, MBaz, hotpaw2, lennon310, Peter K. Jun 20 at 15:59

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 3
    $\begingroup$ What do you mean by "more accurate"? How do you define accuracy? $\endgroup$ – Florian Jun 19 at 10:01
  • 2
    $\begingroup$ I've been praying for years for a tractable definition of accuracy ;) $\endgroup$ – Laurent Duval Jun 19 at 15:23
  • $\begingroup$ I've largely abandoned this project. $\endgroup$ – itisyeetimetoday Aug 9 at 23:42
  • 1
    $\begingroup$ I have updated the question to make it much clearer. Can we please re-open it now? There are some relevant answers already. Feel free to delete this comment. $\endgroup$ – Roel Van de Paar Sep 6 at 11:56
  • 1
    $\begingroup$ If your edits clarify the question, they change it a lot. Are you related to the original author? $\endgroup$ – Laurent Duval Sep 6 at 12:07
2
$\begingroup$

The ltfatpy 1.0.16 package

is a partial Python port of the Large Time/Frequency Analysis Toolbox (LTFAT), a MATLAB®/Octave toolbox for working with time-frequency analysis and synthesis.

Among linear and quadratic time-frequency methods, there is a large number of options for sharper analysis tools converting a 1D signal into 2D data. You can even get more information by decomposing onto 3D or 4D spaces (coming back on that later). Quadratic (or higher-order) representations are usually more difficult to invert than linear ones.

And here is a documentation for ltfatpy.

$\endgroup$
  • $\begingroup$ thank you! How can one take a WAVE (.wav) file and decompose it into a 3D or 4D space? Could you please give an example? I am trying to judge the human quality of WAVE files in some automated way. $\endgroup$ – Roel Van de Paar Sep 6 at 11:35
1
$\begingroup$

For real valued signals (i.e. contains only positive frequencies) the Wigner-Ville (WV) distribution is a popular method. Check out this page for more information.

The WV method provides some better localization than your typical spectrogram is capable of. That webpage I linked has some great examples, namely the linear-frequency modulation "chirp" signal example. While the WV method has some "spreading" at the start/end of the signal, it's time/frequency resolution is greatly superior to the spectrogram method in the example shown. WV does have some issues with highly transient signals, so you specific application may influence the results.

Depending on your signal and what you're trying to do, other decomposition methods of signals into an "image" exist as well, notably cyclo-stationary methods like the spectral correlation function, as well as ambiguity functions. These methods aren't universal, but they can be really useful for things like automatic modulation recognition.

$\endgroup$
  • $\begingroup$ real signals contain negative frequencies. you need a Hilbert transform to have only positive frequencies $\endgroup$ – Stanley Pawlukiewicz Jun 19 at 12:38
  • $\begingroup$ I mean to say the signal can’t be centered on zero and have full bandwidth across the entire frequency swath, I.e at baseband. I am aware they can contain negative frequencies. Trying to keep the concepts simple for the original poster. $\endgroup$ – matthewjpollard Jun 19 at 18:33
1
$\begingroup$

Usually when talking about accuracy the classical Fourier Uncertainty principle

https://en.wikipedia.org/wiki/Fourier_transform#Uncertainty_principle

enters the conversation (I would skip the quantum mechanics stuff). Essentially you can have good resolution in frequency or good resolution in time but there is a tradeoff between the two.

In order to go back to a time series from an image, you at a minimum need to retain phase information which is not included in the image. There is also the issue of how you map a magnitude level of a STFT bin to a pixel value, which usually involve some nonlinear transform (including clipping). The eye and ear are very different in how they perceive a level. It is much simpler to just keep a copy of the time series and use the image to index back into the section of signal you might be interested in.

The utility of STFT based images is that given the roughly 100ms delay that a human brain needs to perceive something, for audio signals, you can craft a real time implementation where you can simultaneously listen and look at the signal. One can use tricks like averaging, maxing , along with decimation to get a useful "real time" waterfall time/frequency spectral analyzer. These sorts of processing are specific to particular application. You would use different techniques for something like looking for low SNR tones in SONAR than those for clean music. This makes going from image information back to time series more difficult ( i really want to say impossible but one needs to be careful with that word).

The STFT is linear. The STFT of 2 signals is the sum of the STFTs of each.

Continuous Wavelets

https://en.wikipedia.org/wiki/Wavelet#Continuous_wavelet_transforms_(continuous_shift_and_scale_parameters)

are also linear and can produce images and relatable to the STFT

https://en.wikipedia.org/wiki/Wavelet#Comparisons_with_Fourier_transform_(continuous-time)

.

For high SNR signals, people have also created time/frequency images using AR models by stitching together segment line by line.

https://en.wikipedia.org/wiki/Autoregressive_model

These typically have good frequency accuracy of peaks. but not so good at troughs.

Before I go further, there is an example that might be useful about redundancy in time/frequency images. Led Zepplin's, Stairway to Heaven is 482 seconds long. At a nominal 44100 samples/second, the time series is 21,168,000 samples. If I choose a 1024 point FFT at 50% overlap, I will have a 41,344 by 512 array, which I can reasonably create a scrollable image.

If you want higher "time resolution" you increase the overlap, so 75% overlap produces 82,687 by 512 array which can be scrolled but it listening and seeing simultaneously becomes more problematic.

One can use some tricks to reduce the scan rate of wavelet grams but it is harder if you want to listen and see simultaneously. The same is true of AR grams.

As mentioned in the other answer by @mathewpollard there is Wigner Ville (WV) which is nonlinear. WV is a special case of the "Cohen Class" which also includes Choi Williams.

I had the privilege of attending a joint lecture by Leon Cohen and Al Nuttal a long time ago. I remember a quote from Nuttal, "Wigner Ville is like small child, when good it is very good, when bad it is very bad".

A problem with WV is that it is nonlinear, there are cross terms. The image of 2 simultaneous signals is not the sum of the individual images (neglecting pixel level mapping issues) . There are artifacts in the image that don't belong there. There are modifications to WV that mitigate the cross term problem. People do use WV. You need to understand it.

The other issue (which may not be your issue) is the redundancy of WV. In its simplest form, you update it every new sample. The pixels will fly by your display faster than you can perceive them in human real time.

The strength of STFT based grams for audio signals is the ability to listen and see simultaneously. It is much easier for an analyst to relate the spectral feature to the sound it makes and identifying the cause . Understanding the data is more important in many cases than accuracy.

$\endgroup$
0
$\begingroup$

A long enough waveform plot at a high enough resolution allows reproducing sound from an image. Old movie film projectors reproduced sound this way (converting optical density images into amplitudes).

$\endgroup$
  • $\begingroup$ isn't that modulation? the movies didn't encode spectra of the audio. $\endgroup$ – Stanley Pawlukiewicz Jun 19 at 16:03
  • 1
    $\begingroup$ Historical note: on the beginning of the www-internet, mp3 sharing was banned in some places because it swallowed too much bandwidth. But JPEG was still allowed. There were a program to convert mp3 into a (crappy) jpeg, easily decodable. $\endgroup$ – Laurent Duval Jul 2 at 19:32

Not the answer you're looking for? Browse other questions tagged or ask your own question.