# How to build a program that gives you on the fly the chords of the song while listening to it?

I am trying to think how can I program a SW that can recognize Chords on the fly. since I know nothing about music, Can I assume that the chords are orthogonal and unique? I guess I can since it is linear combination of sinusoidal signals in different frequencies.

a. What is the rate I have to sample the song?

b. Are Chords Unique?

c. Can I assume I can store and have all the chords? is it finite set?


I found this question but it doesn't answer my questions

a. What is the rate I have to sample the song?

Chords are usually played by instruments with f0s in the 100 - 1kHz range, so if your algorithm can work with only the fundamental of each note, a sampling rate of 2kHz is enough. You can't go lower. There is absolutely no point sampling above 16kHz for such a recognition task; and pretty much all the literature on chord and key detection uses this sampling rate or lower (Mauch, Papadopoulos : 11kHz. Ellis, Harte : 16kHz). Keep in mind that a large fraction of the adult population has lost their "top octave" and cannot hear much above 10kHz; this doesn't prevent aging musicians from distinguishing chords easily!

b. Are Chords Unique? c. Can I assume I can store and have all the chords? is it finite set?

Not sure what you mean here ; but the same chord (say C) can be played :

• At various octaves,
• At various inversions (C, E, G ; or E, G, C ; or G, C, E...),
• With various combination of timbres of instruments,
• With or without background drums, with or without a singing voice,
• With all kinds of audio transformations applied during recording, mixing, mastering, and mp3 compression.

This means that there is an extremely wide array of observable signals of a "C" chord.

If you want to look into automatic chord detection, please read Matthias Mauch's publications, with his Thesis giving the most complete body of work on the topic. To give you an idea how complex the task is, his "baseline" algorithm consists in:

• Extracting a STFT of the signal with rather large windows (in the hundreds of ms range).
• Mapping this to a constant-Q representation by upsampling and projection.
• Applying several contrast enhancing and whitening operations to the constant-Q spectrogram to compensate for the "horizontal" decay (when a note is played its amplitude decays with time) and "vertical" decay (the spectrum of a musical sound has decreasing energy as frequency increases)
• Using non-negative least-square to describe each slice of the constant-Q spectrogram as a sum of a small number of positively weighted harmonic combs.
• Converting the f0 of the extracted combs to a 12-tones scale (so called "chroma vectors").
• Matching the chroma vectors to a dictionary of hundreds of manually defined templates corresponding to chords.
• Using a HMM model to smooth the sequence of detected chords.

The latest bit is important: his approach (reputedly state of the art) is not causal, so it cannot work "on the fly". Furthermore the processing time needed by the NNLS decomposition and the very large FFT windows make it run 3x slower than real time. 75% accuracy on the Beatles works. Just to make you realize that getting something reliable and realtime will be a very hard task! If you are looking for something simpler, you might start by reading part II of this paper by Laurent Oudre.