Update 8

Unhappy of having to upload tracks to the service and looking at the new release of RekordBox 3 I've decided to take another look for an offline approach and a finer resolution :D

Sounds promising though still in a very alpha state:

Johnick - Good Time

enter image description here

enter image description here

Note that there's no logarithmic scale nor palette tuning, just a raw mapping from frequency to HSL.

The idea : now a waveform renderer has a color provider which is queried for a color for a specific position. The one you are seeing above gets the zero crossing rate for the 1024 samples next to that position.

Obviously there's still a lot to do before getting something robust but it seems like a good path...

From RekordBox 3:

enter image description here

enter image description here

Update 7

The final form that I'll adopt, much like in Update 3

enter image description here

(it's been a little Photoshopped to get smooth transitions between colors)

Conclusion is I was close months ago but did not consider that result thinking it was bad X)

Update 6

I've unearthed the project recently, so I thought about updating here :D

Song : Chic - Good Times 2001 (Stonebridge Club mix)

enter image description here

enter image description here

enter image description here

enter image description here

It's way better IMO, beats have a constant color etc ... it's not optimized though.

How ?

Still with http://developer.echonest.com/docs/v4/_static/AnalyzeDocumentation.pdf (page 6)

For each segment :

public static int GetSegmentColorFromTimbre(Segment[] segments, Segment segment)
    var timbres = segment.Timbre;

    var avgLoudness = timbres[0];
    var avgLoudnesses = segments.Select(s => s.Timbre[0]).ToArray();
    double avgLoudnessNormalized = Normalize(avgLoudness, avgLoudnesses);

    var brightness = timbres[1];
    var brightnesses = segments.Select(s => s.Timbre[1]).ToArray();
    double brightnessNormalized = Normalize(brightness, brightnesses);

    ColorHSL hsl = new ColorHSL(brightnessNormalized, 1.0d, avgLoudnessNormalized);
    var i = hsl.ToInt32();
    return i;

public static double Normalize(double value, double[] values)
    var min = values.Min();
    var max = values.Max();
    return (value - min) / (max - min);

Obviously there's a lot more code needed before you get here (upload to service, parse JSON etc) but that's not the point of this site so I'm just posting the relevant stuff for getting the above result.

So I'm using the first 2 functions of the analysis result, there is certainly more to do with but I still need to test. If I ever find something cooler than above I'll come back and update here.

As always, any hints on the topic are more than welcome !

Update 5

Some gradient using harmonic series

enter image description here

The color smoothing is ratio-sensitive otherwise it looks bad, will need some adjustment.

Update 4

Rewrote the coloring to happen at source and smoothed colors using an Alpha beta filter with 0.08 and 0.02 values.

enter image description here

A little better when zoomed out

enter image description here

Next step is to get a great color palette !

Update 3

enter image description here

Yellows represents mediums enter image description here

Not so great when unzoomed, yet. enter image description here

(the palette needs some serious work)

Update 2

Preliminary test using second 'timbre' coefficient hint from pichenettes enter image description here

Update 1

A preliminary test using an analysis result from EchoNest service, note that it's not aligned very well (my fault) but it's much more coherent than the above approach.

enter image description here

For people interested in using this great API, start here : http://developer.echonest.com/docs/v4/track.html#profile

Also, don't get confused by these waveforms as they do represent 3 different songs.

Initial question

So far this is the result I get using a 256-samples FFT and calculating each chunk's spectral centroid.

Raw result of the calculations enter image description here

Some smoothing applied (form looks much better with it) enter image description here

Waveform produced enter image description here

Ideally, this is how it should look like (taken from Serato DJ software) enter image description here

Do you know what technique/algorithm I could use to be able to split the audio when average frequency changes over time ? (like the above image)

  • 1
    $\begingroup$ I think you are on the right track by using the spectral centroid; maybe you just need a non-linear mapping of the result to get a good distribution of colors. It looks like only the blue/cyan segment of your palette is currently used. $\endgroup$ May 5 '13 at 14:02
  • 1
    $\begingroup$ Is your question about the pseudo-color algorithm or about splitting the audio into chunks in an automated way? $\endgroup$ May 5 '13 at 14:59
  • $\begingroup$ @pichenettes : right, I'm going to address this issue using a logarithmic scale. $\endgroup$
    – aybe
    May 5 '13 at 15:31
  • 1
    $\begingroup$ This is exactly what freesound.org uses to color their waveforms. See their spectral_centroid code. What I'd really like is to map the audio spectrum onto the color spectrum, so a low frequency is red, high frequency is blue, combination of both is magenta, middle frequency is green, log sweep is rainbow, white noise is white, pink noise is pink, red noise is red, ... I can't figure out how to do it though, since one spectrum is linear and the other is log. :) flic.kr/p/7S8oHA $\endgroup$
    – endolith
    May 6 '13 at 13:50
  • 1
    $\begingroup$ @Aybe: Yes, I know how to make a log sweep look like a rainbow, but when measured with log frequency spacing (proportional width bands), white noise has a slanted spectrum, not a flat spectrum, so it would not produce white light. Our names for colors of noise are based on constant width bands (linear spacing), but our perception of tones is on a log spacing (proportional width bands). May I ask what you plan to do with this when you get it working? $\endgroup$
    – endolith
    May 7 '13 at 0:02

You can try first the following (no segmentation):

  • Process the signal in small chunks (say 10ms to 50ms in duration) - if necessary with a 50% overlap between them.
  • Compute the spectral centroid on each chunk.
  • Apply a non-linear function to the spectral centroid value to get a uniform distribution of the palette color used. Logarithm is a good start. Another option is to compute first the distribution of the centroid value other the entire file, and then color according to percentiles of this distribution (CDF). This adaptive approach guarantees that each color in the palette will be used in equal proportion. The drawback is that what is plotted in blue in one file won't be similar sounding to what is plotted in blue in another file if this adaptive approach is used!

It is not clear from the picture if Serato does this, or indeed goes one step further and attempts to segment the signal - it might not be surprising that it does since knowing the instants at which notes are present in a music signal can help synchronizing them! The steps would be:

  • Compute a Short-Term Fourier Transform (spectrogram) of the signal - using the FFT on short overlapping segments (start with a FFT size of 1024 or 2048 with 44.1kHz audio).
  • Compute an onset-detection function. I recommend you to look at this paper - the proposed approach is very effective and even a C++ implementation takes less than 10 lines. You can find an implementation in Yaafe - as ComplexDomainOnsetDetection.
  • Detect peaks in the onset-detection function to get the position of note onsets.
  • Compute the spectral centroid over all the time segments delimited by the detected onsets (don't forget to window and/or zero-pad!)
  • And don't forget the non-linear map! Note that the gradient-effect that appear between each note on the Serato waveform might be artificially generated.

Once you get this working, a "low hanging fruit" would be to compute a few other features on each segment (moments, a handful of MFCCs...), run k-means on these feature vectors, and decide on the color of the segment using the cluster index. See section II of Ravelli's paper.

  • $\begingroup$ I've started working on your solution number 2, fortunately the service (EchoNest) I am using provides onset-detection in their analysis so that's great news and a real-time saver ! I've updated my answer with a small test I did and it looks promising (note that it's not aligned properly since it's really quick and dirty). I'll post a better result when I've made signifcant progress. Thank you ! $\endgroup$
    – aybe
    May 5 '13 at 19:29
  • 1
    $\begingroup$ Then you don't have any work to do. The second "timbre" coefficient in the EchoNest analysis is very related to the centroid, so you don't have any computation to do (unless they got rid of the DC component in their basis, in which case it is the first coefficient) - just use the beat or segment-level value of the "timbre" vector. $\endgroup$ May 5 '13 at 19:41
  • 1
    $\begingroup$ The "Brightness" as measured by the second "timbre" coefficient in the EN analysis is very related to spectral centroid (the difference being that the EchoNest analyzer uses some perceptual pre-processing - middle-ear filtering, time/frequency masking - rather than taking a raw FFT). Color coding using the maximum bin in the 'pitch' vector would be useful for very pure monophonic, melodic material, but is rather meaningless for a drum loop. $\endgroup$ May 5 '13 at 21:53
  • 1
    $\begingroup$ Ubounded? It's still bounded by the signal energy in the analyzed frame... I think that for this feature it's worth keeping the scale consistent between tracks. What I would do: collect a dozen very diverse songs to get a feel for the distribution of the value. If necessary apply a simple non-linear map to make it look more gaussian. Then remove mean and divide by standard deviation. Then apply atan to make it bounded - bringing back outliers closer to the 1st or 9th deciles without causing clipping. $\endgroup$ May 7 '13 at 14:43
  • 1
    $\begingroup$ When the waveform is zoomed out, segment your signal according to larger elements like bars or sections - rather than attempting to coloring individual notes/beats. $\endgroup$ May 8 '13 at 17:33

Not the answer you're looking for? Browse other questions tagged or ask your own question.