How to build a program that gives you on the fly the chords of the song while listening to it?

I am trying to think how can I program a SW that can recognize Chords on the fly. since I know nothing about music, Can I assume that the chords are orthogonal and unique? I guess I can since it is linear combination of sinusoidal signals in different frequencies.

a. What is the rate I have to sample the song?

b. Are the chords unique?

c. Can I assume I can store and have all the chords? Is it a finite set?


I found this question but it doesn't answer my questions

a. What is the rate I have to sample the song?

Chords are usually played by instruments with f0s in the 100 - 1kHz range, so if your algorithm can work with only the fundamental of each note, a sampling rate of 2kHz is enough. You can't go lower. There is absolutely no point sampling above 16kHz for such a recognition task; and pretty much all the literature on chord and key detection uses this sampling rate or lower (Mauch, Papadopoulos : 11kHz. Ellis, Harte : 16kHz). Keep in mind that a large fraction of the adult population has lost their "top octave" and cannot hear much above 10kHz; this doesn't prevent aging musicians from distinguishing chords easily!

b. Are Chords Unique? c. Can I assume I can store and have all the chords? is it finite set?

Not sure what you mean here ; but the same chord (say C) can be played :

• At various octaves,
• At various inversions (C, E, G ; or E, G, C ; or G, C, E...),
• With various combination of timbres of instruments,
• With or without background drums, with or without a singing voice,
• With all kinds of audio transformations applied during recording, mixing, mastering, and mp3 compression.

This means that there is an extremely wide array of observable signals of a "C" chord.

If you want to look into automatic chord detection, please read Matthias Mauch's publications, with his Thesis giving the most complete body of work on the topic. To give you an idea how complex the task is, his "baseline" algorithm consists in:

• Extracting a STFT of the signal with rather large windows (in the hundreds of ms range).
• Mapping this to a constant-Q representation by upsampling and projection.
• Applying several contrast enhancing and whitening operations to the constant-Q spectrogram to compensate for the "horizontal" decay (when a note is played its amplitude decays with time) and "vertical" decay (the spectrum of a musical sound has decreasing energy as frequency increases)
• Using non-negative least-square to describe each slice of the constant-Q spectrogram as a sum of a small number of positively weighted harmonic combs.
• Converting the f0 of the extracted combs to a 12-tones scale (so called "chroma vectors").
• Matching the chroma vectors to a dictionary of hundreds of manually defined templates corresponding to chords.
• Using a HMM model to smooth the sequence of detected chords.

The latest bit is important: his approach (reputedly state of the art) is not causal, so it cannot work "on the fly". Furthermore the processing time needed by the NNLS decomposition and the very large FFT windows make it run 3x slower than real time. 75% accuracy on the Beatles works. Just to make you realize that getting something reliable and realtime will be a very hard task! If you are looking for something simpler, you might start by reading part II of this paper by Laurent Oudre.

• a. At least 32 kHz
• b. no
• c. finite but large. 1000s. If you factor in inversions and "over" notations, tens of 1000s.

This is actually a really hard problem. Chords are made of individual notes. However these notes have fundamentals and harmonics. They are not pure sinusoid. Let's say you play C7/#9 chord (C seven sharp nine) on a guitar which is typically played as C E Bb Eb using four strings. Each of the four strings produces its fundamental and many harmonics. Since you typically get 6-10 significant harmonics per note, the spectrum is quite complicated.

Even if you can reliably extract the notes, the chord detection is also tricky. Chords are certainly not unique. For example C6 (C six) and am7 (A minor 7) have basically the same notes. So have C_dim, Eb_dim, F#_dim, A_dim. Furthermore the stack up of the notes matters (so called inversions), i.e. which note is the highest and which is the lowest. In many cases it also depends on the musical context. A certain set of notes will form a different chord depending on what happened before and what happens after.

Chord notation is not super scientific. In each key there are dozens of different chord flavors and people transcribe them in different ways. There is also a whole second set of chord flavors that use the "over" notation. For example B7+/C means "play a B flat major seven chord (Bb D F A) over a C root". If you are interested to hear how that sounds like: this is actually the chord used in the first 15 seconds of this song http://www.youtube.com/watch?v=V7dg8vRDM68.

The first issue with your problem description is that each note within each chord within a "song" is not just a sinusoid or sinewave at the pitch frequency of the note, but a far more complicated, likely time varying, possibly only pseudo-periodic waveform, potentially composed of dozens of overtones. Then you have a unknown number of these harmonically rich waveforms mixed together to form each chord. Then any sustain pedal or effect will potentially overlap and mix sequential chord sounds together. These combine to make the search space many orders of magnitude larger than just the table of possible chords, (unless you stick with an extremely restricted electronic synthesizer as the only instrument).

I realize that trying to account for the changing timbral envelope, as you go up and down the instrument's pitch range would make things far too complex to begin with. I am talking about playing a note which would the root of a major chord. You multiply the frequency of the root by two constants to get the third and fifth. You would use the simplest DSP chip you could find since it is simply a concatenate(root,third,fifth) command to create a major chord. There would be clip-on mic to hear the note, and a speaker to play the resulting major chord. Also there would be a turn pot (nominally 10k$$\Omega$$ input impedance) with the center tap tied to DSP's input line so you could control the volume and not drown out the rest of the band.

The only question for me is whether to use the equal temperament scale of today or the scales of the past to make the math even simpler in determining the frequencies of the third and fifth. Traditionally, the notes a major chord form a 4:5:6 ratio. Frequency of the third = 5/4$$\cdot$$frequency of the root. Frequency of the fifth = 3/2$$\cdot$$frequency of the root.

In the equal temperament scale, the smallest unit of pitch, the semitone, is a multiplying factor,$$\sqrt[12]{2}=2^{\frac{1}{12}}$$, the twelfth root of 2, to get from one note to the next higher pitch. $$(\sqrt[12]{2})^{12}=2$$, meaning after 12 multiplications of the twelfth root of 2, you get 2. The doubling of frequency means you have gone up an octave. The third is $$2^{\frac{4}{12}}=2^{\frac{1}{3}}=\sqrt[3]{2}$$ or the cube root of 2$$\cdot$$frequency of the root. The fifth is $$2^{\frac{3}{12}}=2^{\frac{1}{4}}=\sqrt[4]{2}$$ or the fourth root of 2$$\cdot$$frequency of the third.

I would opt for the first choice, because that's what human ears are attuned to, as well as the fact that's what your instrument will actually play in terms of harmonics produced. This approach would make the most sense for the higher pitched one note a time instruments. Full disclosure, I have not yet built such a device, nor have I written any code whatsoever for a DSP chip in my life. It's a thought experiment worth trying since we are talking \$5-10 dollars for parts, at most. The idea is that this could help school bands, or any other new band sound better right away and keep them interested in playing music. I would give them away for free. As a musician got better, he or she could dial down the gain knob attached the turn pot. I am going to make a few myself, as soon as I can get my hands on the necessary parts, which is easier said than done. If I am wrong, it would be an experiment well worth trying, and I would welcome your unmerciful flogging as a result. I will use MathJax next time. (It's 2AM local time as I am finishing up here, and I would like to get a few hours sleep before daybreak ~7:15AM.)

• This sounds similar to the «auto-chord» function found on electronic keyboards and organs? Play the fundamental and get a major triad automatically? That sounds doable, for a clean monophonic input. I have some doubts about the musical value, but as a project it could still be interesting. Oct 25, 2021 at 11:19
• I would like to address the OP's question directly, which I did not do last night. There is no earthly reason to build such a program, since many of the most popular songs consist of only three chords (I root, IV perfect fourth, V perfect fifth). You should be able to pick these three chords out by ear. I would guess most of popular music has between 3 to 7 different chords. They change in predictable, repeatable ways every 8,12,16,or 24 bars in a sequence known as a chord progression or the "changes". You need to pick these chords out by ear, if you going to be any kind of musician. Oct 25, 2021 at 14:18
• To address Knut Inge's point: I strongly agree that if your instrument can play multiple notes at the same time (keyboard or guitar, among others), you can't use auto chord and call yourself a musician. I play alto sax mainly, but can play tenor or bari sax as well. None of these can play more than one note at a time. This would give me the ability play lead while allowing the sax to create its own accompaniment. This could be expanded to all useful 3 and 4 note chords. The major chord only version would take a few hours to create, and new musicians would enjoy the sound they make much more. Oct 25, 2021 at 14:47
• Eventide used to market «Harmonizers» and half the bass players I know used to have an «octave pedal» in their arsenal. I guess today you can get boxes for harmonizing vocal (being your own backup singers). In the long run, having only major triads is going to be limiting. Oct 25, 2021 at 17:28
• Mr Steele that was a bizarre comment. I wish you good luck. Oct 27, 2021 at 14:07

The answer to the question is that it's a complete waste of time writing a program of such complexity, as evidenced by the other answers, when all you have to do is listen to the song. Most popular songs have a small number of chords, usually at least 3, usually fewer than 10, that you could pick out by ear once you studied the sheet music. You would learn far more about the structure of a good song, and good music in general, by taking this approach, rather than have some program do it for you.

Warning! Blatant music math geekiness ahead. The TL;DR is there for a reason. The frequency of a given pitch in Hertz $$(Hz)$$ can be found by the following formula: $$f(n)=440\cdot2^{(n/12)}$$ Where $$n$$ is the number of semitones that a pitch is away from the standard reference pitch of A4 (440Hz). If $$n>0$$, the pitch is higher than A4. If $$n<0$$, the pitch is lower than A4. If $$n=0$$, the pitch is A4 itself. $$f(0)=440\cdot2^{(0/12)}=440\cdot2^0=440\cdot1=440Hz$$ The three main points of the formula are:

1. A4 is the reference pitch by which all other pitches are determined.
2. When you double the frequency, you go up an octave. When you halve the frequency, you go down an octave.
3. All twelve pitches of the chromatic scale are evenly spaced from each other, unlike in the past.

A complicating factor arises if you play a transposing instrument, like I do. I play the saxophone (alto sax mostly these days). The four most common saxophones in use today are the soprano, alto, tenor, and baritone (bari for short). The soprano and tenor are in the key of B$$\flat$$, meaning if you play a C written in sheet music, you get B$$\flat$$ as the tone you produce. A similar situation exists for the alto and bari saxes, which are in the key of E$$\flat$$. The transpositions are not only of pitch, but of octaves as well. All music for the saxophone family is written with a treble clef, from the smallest to the hugest. It's just something that the composer or arranger of the song has to deal with. The following table will show you what I mean.

B$$\flat$$ Soprano Sax -2 semitones

E$$\flat$$ Alto Sax -9 semitones

B$$\flat$$ Tenor Sax -14 semitones

E$$\flat$$ Bari Sax -21 semitones

Here are a couple of examples to illustrate the point. The highest key on the soprano sax I have is the high F key (not F$$\sharp$$). The high F is 20 semitones above A4. Subtracting 2 semitones from the table above you have 18 semitones higher. Plugging +18 into the formula, you get: $$f(18)=440\cdot2^{(18/12)}=440\cdot2^{(3/2)}=440\cdot2^{1.5}\approx1,244.5Hz$$ The bari sax has a low A key. No other saxophone has one, even the much larger and much lower saxophones. The low A (2 ledger lines below the treble clef) is written an octave (12 semitones) below A4, making it A3. Adding -12 and -21, you get -33. Using the formula again: $$f(-33)=440\cdot2^{(-33/12)}=440\cdot{(-11/4)}=440\cdot2^{-2.75}\approx65.4Hz$$

Before I go, I want to tell you about the tubax (mash up of tuba and sax). It is a wondrous creation of musical technology. For it allows a very low pitched sax to realized in the relatively compact form of a tuba, rather than a saxophone that is as taller or taller than the person playing it. There are two kind of tubaxes: a contrabass tubax in E$$\flat$$ and a subcontrabass tubax in B$$\flat$$. Why don't we calculate the lowest frequency of each? Should be fun.

The E$$\flat$$ contrabass tubax is pitched an octave below the bari sax. We can build off what we have done before. $${f(-33)}\cdot{1/2}=440\cdot2^{(-33/12)}\cdot2^{-1}=440\cdot2^{-11/4}\cdot2^{-1}=440\cdot2^{(-11/4)-(4/4)}=440\cdot2^{-15/4}=440\cdot2^{-3.75}\approx32.7Hz$$

The B$$\flat$$ subcontrabass tubax is pitched an octave lower than the B$$\flat$$ bass saxophone, which itself is pitched an octave lower than the B$$\flat$$ tenor sax. So the subcontrabass tubax is pitched two octaves lower than the tenor sax. Start by calculating the frequency of the lowest pitch of the tenor sax. The low B$$\flat$$ is written 11 semitones below A4. Adding -11 and -14 you get -25. Once again with the formula. $$f(-25)=440\cdot2^{(-25/12)}\approx103.8Hz$$ Then we have $$f(-25)\cdot{1/4}=440\cdot2^{(-25/12)}\cdot2^{-2}=440\cdot2^{(-25/12)-(24/12)}=440\cdot2^{-49/12}\approx26.0Hz$$

End of TL;DR

That's all, folks! No more braggadocious boasting and blustering, until I can create and physically instantiate an idea in both embedded hardware and firmware. (DSP hardware and firmware in this case).