# How to get the melody from a signal

I'm trying to retrieve a melody from a signal (simple sine with 3 notes).

I've windowed the signal and performed autocorrelation to get the fundamental frequency of each window and then converted frequency to note.

So for a signal with notes "G A E" I am getting something like "G G G A A B E E E E F". It makes sense that notes are repeated and there are some mistakes... But how should I deal with this? If I use a bigger window then I might miss some notes with shorter duration. I can think of a really naive algorithm to get the melody for this example but what are some real techniques that are usually applied?

Some common approaches:

Simple post-processing

Apply a mode filter on your resulting sequence of pitches. This will increase temporal inaccuracies though!

Pre-segmentation

Instead of analyzing your signal according to a grid of fixed-size windows, try to detect first the boundaries between notes using an onset detection algorithm. Once you have performed the onset detection, your signal is divided into segments in which you can make the assumption that the pitch is constant. You then apply your pitch detection algorithm to each segment. This works best for instruments with a sharp attack - like guitar or piano (playing monophonic melodies) - but not very well in which the pitch of a held note is modulated without sharp attacks between notes (flute, whistling, singing voice).

Dynamic programming

• You define a function $C_i(f)$ which indicates how unlikely it is that frequency $f$ is the fundamental frequency of the $i$-th signal window. You can use, for example, the reciprocal of the autocorrelation function; or sort your pitch hypotheses. For example, if the 4 highest peaks in the autocorrelation function of the 6th signal window are at 220 Hz, 440 Hz, 500 Hz and 250 Hz, $C_6(220) = 0$, $C_6(440) = 1$, $C_6(500) = 2$ and so on. The higher the function, the most unlikely the argument is to be true fundamental frequency.
• You define a transition cost $T(f_1, f_2)$ which indicates how unlikely it is that a window with fundamental frequency $f_1$ will be followed by a window with fundamental frequency $f_2$. This transition cost can embed basic musicological knowledge (which notes are more likely to follow other notes) but will mostly serve as temporal smoothing.
• You use dynamic programming to find the sequence $f^*$ that minimizes:

$$\sum_i C_i(f^*_i) + T(f^*_{i - 1}, f^*_i)$$

Intuitively, this searches for a sequence of pitches that is both compatible with your observations, but also doesn't change too much. Adjusting the relative weight between $C$ and $T$ will allow you to control the amount of smoothing.

HMM

Manually annotate a large collection of files. Train a HMM using autocorrelation vectors as your feature vectors, and musical notes as your states.

To transcribe a melody, find the most likely sequence of states that explain your autocorrelation vectors. This is done by the Viterbi algorithm, which will take exactly the same form as the dynamic programming method given above. The only difference is that there is this time a rigorous statistical framework for defining and learning the scoring and transition functions.