4
$\begingroup$

Original Question: https://ux.stackexchange.com/q/23040/16006

I've only taken some basic signal analysis courses, so I might be missing some things.

Purely theoretical question:

What methods exist for representing audio?

What methods could be made for representing audio, more specifically musical audio?

So far, I'm aware of:

  • Viewing the waveform (Soundcloud does this), mostly useless except for seeing "loudness"

  • Spectral analysis (Example), good for seeing frequency and "loudness"

Essentially I'm wondering if there is a way one could "see" the notes, beats, and so on of a song, visually.

Right off the top of my head I can think of displaying 3 differently colored waves over time representing treble, mid, bass in a soundcloud-like container with the section playing (or moused-over) being magnified, with the surrounding waveforms being compressed into the corners (like a wide-angle lens effect).

EDIT: I don't know where this could be used, this was just born out of my frustration with current audio visualizing technology.

I imagine having a 3d graph of a spectral analysis over time (Ninja Edit: apparently known as spectrogram) would be the "best" solution since you see everything but it might not be the most elegant and it might not be portable to places like soundcloud.

Even current spectrum analysis is hard to decipher (Too low level for images):

FL Studio wave editor

I'm essentially wondering what might work for casual users, and for people wondering ahead of time how the song will play out.

Improve this question
$\endgroup$
7
  • $\begingroup$ "Best" is too subjective unless you've got some specific feature in mind you want to illustrate. I've fooled around with trying to cram as much data as possible into a waveform display (intending it to be used in waveform editing software like Audacity), showing the waveshape (not just the peaks) through density (dsp.stackexchange.com/q/184/29), the spectral content by mapping audio spectrum to visual spectrum, etc. Here's an example with density and spectral centroid as color: flic.kr/p/7S8oHA $\endgroup$
    – endolith
    Jun 29, 2012 at 21:12
  • 2
    $\begingroup$ As for "seeing" the notes and beats of a song, spectrogram is definitely better than amplitude waveform, and is already built into things like Adobe Audition, Sonic Visualiser, freesound.org, etc. $\endgroup$
    – endolith
    Jun 29, 2012 at 21:24
  • 1
    $\begingroup$ Perhaps rephrase your question (since it is unanswered and it is too general) and ask how can you determine what is in the audio track. (Later on, you can consider (or ask) how to visually show this to the user.) See my answer on UX for more details. $\endgroup$
    – Danny Varod
    Jun 29, 2012 at 21:55
  • 1
    $\begingroup$ @dudeoea: Probably a spectrogram (though maybe using continuous complex Morlet wavelet transform, which is called a scalogram though I think spectrogram is still appropriate). Our ears basically work like spectrum analyzers. Though our perception of phase differences between the ears would not be shown. Not sure how important that is. $\endgroup$
    – endolith
    Jun 29, 2012 at 23:14
  • 1
    $\begingroup$ @dudeoea: Yes, this makes me think it's more like a wavelet transform than a spectrogram, because the FFT bins are equal widths for all frequencies, while our ears have narrower filters at low frequencies: en.wikipedia.org/wiki/Critical_band#Auditory_filters $\endgroup$
    – endolith
    Jul 6, 2012 at 22:39

2 Answers 2

Reset to default
1
$\begingroup$

What a human (or their ear-brain) perceives in sound is a psychoacoustic phenomena, and may or may not be exactly related to the actual audio as recorded. e.g. the exact notes, beats and instruments that a human "hears" may be influenced by visual cues, memory of other similar music, and the musical context around the actual sound of the note in question.

Improve this answer
$\endgroup$
1
$\begingroup$

I'm just getting into signal processing, mostly through playing with 3D spectrograms.This is a .3 second excerpt of a middle C on piano.

This is a .3 second excerpt of a middle C on piano. The plane facing the screen shows amplitude and frequency (spectrogram), whereas the plane facing left shows the waveform (amplitude and time). I like this method of visualizing sound because it has the best of both worlds. You can see the moment-to-moment amplitude modulations of a waveform and the frequency-specific info from a spectrum. There are a few programs I've found that generate this for you in real time. This in-browser tool gives you a top-down view of the 3D spectrogram using your computer mic. It also has a few sample sounds to visualize.

Hope this helps!

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.