Original Question: https://ux.stackexchange.com/q/23040/16006

I've only taken some basic signal analysis courses, so I might be missing some things.

Purely theoretical question:

What methods exist for representing audio?

What methods could be made for representing audio, more specifically musical audio?

So far, I'm aware of:

  • Viewing the waveform (Soundcloud does this), mostly useless except for seeing "loudness"

  • Spectral analysis (Example), good for seeing frequency and "loudness"

Essentially I'm wondering if there is a way one could "see" the notes, beats, and so on of a song, visually.

Right off the top of my head I can think of displaying 3 differently colored waves over time representing treble, mid, bass in a soundcloud-like container with the section playing (or moused-over) being magnified, with the surrounding waveforms being compressed into the corners (like a wide-angle lens effect).

EDIT: I don't know where this could be used, this was just born out of my frustration with current audio visualizing technology.

I imagine having a 3d graph of a spectral analysis over time (Ninja Edit: apparently known as spectrogram) would be the "best" solution since you see everything but it might not be the most elegant and it might not be portable to places like soundcloud.

Even current spectrum analysis is hard to decipher (Too low level for images):

FL Studio wave editor

I'm essentially wondering what might work for casual users, and for people wondering ahead of time how the song will play out.

  • $\begingroup$ "Best" is too subjective unless you've got some specific feature in mind you want to illustrate. I've fooled around with trying to cram as much data as possible into a waveform display (intending it to be used in waveform editing software like Audacity), showing the waveshape (not just the peaks) through density (dsp.stackexchange.com/q/184/29), the spectral content by mapping audio spectrum to visual spectrum, etc. Here's an example with density and spectral centroid as color: flic.kr/p/7S8oHA $\endgroup$ – endolith Jun 29 '12 at 21:12
  • 2
    $\begingroup$ As for "seeing" the notes and beats of a song, spectrogram is definitely better than amplitude waveform, and is already built into things like Adobe Audition, Sonic Visualiser, freesound.org, etc. $\endgroup$ – endolith Jun 29 '12 at 21:24
  • 1
    $\begingroup$ Perhaps rephrase your question (since it is unanswered and it is too general) and ask how can you determine what is in the audio track. (Later on, you can consider (or ask) how to visually show this to the user.) See my answer on UX for more details. $\endgroup$ – Danny Varod Jun 29 '12 at 21:55
  • 1
    $\begingroup$ @dudeoea: Probably a spectrogram (though maybe using continuous complex Morlet wavelet transform, which is called a scalogram though I think spectrogram is still appropriate). Our ears basically work like spectrum analyzers. Though our perception of phase differences between the ears would not be shown. Not sure how important that is. $\endgroup$ – endolith Jun 29 '12 at 23:14
  • 1
    $\begingroup$ @dudeoea: Yes, this makes me think it's more like a wavelet transform than a spectrogram, because the FFT bins are equal widths for all frequencies, while our ears have narrower filters at low frequencies: en.wikipedia.org/wiki/Critical_band#Auditory_filters $\endgroup$ – endolith Jul 6 '12 at 22:39

What a human (or their ear-brain) perceives in sound is a psychoacoustic phenomena, and may or may not be exactly related to the actual audio as recorded. e.g. the exact notes, beats and instruments that a human "hears" may be influenced by visual cues, memory of other similar music, and the musical context around the actual sound of the note in question.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.