# How to determine if certain notes exist within polyphonic music?

Given a segment of audio, if you were to calculate the histogram of frequency amplitudes for all standard musical note frequencies present in the audio, how would you check to see if 2-3 specific musical notes exist in the audio?

This is a type of polyphonic detection, similar to this question. Except I'm not trying to comprehensively find all notes present in the audio. I already know what notes I'm looking for and am just trying to check to see if they're present.

My current (naive) approach is to:

1. Calculate the average amplitude across all frequencies to use as a threshold for noise filtering. Any frequencies with an amplitude below this I ignore as background noise.
2. For each note I'm searching for, I calculate the frequencies for the first 3 harmonics, lookup the amplitude for each of those frequencies, and if they're all above the average, then I assume that note is present.

I find this sort-of works, but isn't 100% reliable. The main problem I'm running into is that, given the type of musical instrument, the amplitudes of all the note's harmonics can be very inconsistent across the instrument's range, and this makes setting the noise threshold very error prone.

For example, on an acoustic guitar, playing the high E4 note, the fundamental frequency is very strong and larger than all the other harmonics. However, for the lower E2 note, the fundamental is so small, it's often excluded as background noise. And there's not always a consistent pattern within the bass strings. Some of the low bass string notes have a very strong fundamental as well.

How do I solve this? I know open-ended real-time polyphonic detection is a very difficult and unsolved problem, but are there any solutions for a constrained version where you're only checking for the existence of a few notes and their specific harmonics?

– Bob
May 31 '21 at 22:57
• @Bob That's interesting, although that question and answer are about applying a single classification label to an audio sample. I'm essentially trying to apply two or more classification labels to an audio sample. And even then, the answer is just a vague "train a model". Yes, but how would you train a model for this problem? That Teachable Machine webapp looks interesting. I was looking for a solution that didn't require collecting audio samples for specific instruments, but I'm not above doing that. I'm more concerned it looks like it can only apply a single classification label. Jun 1 '21 at 0:14
• The problem is- in white noise all notes exist! This goes to say that you need to specify a level, and perhaps an SNR if that is your real interest at which point you can make a quantifiable estimate. Jun 1 '21 at 0:16
• @DanBoschen In the case of white noise, that's fine. For my application, where a musician is recording their instrument, it can be safely assuming there's not a ton of blaring white noise. The main problem is a slight din of background noise obscuring fundamentals and any noise from sympathetic string vibrations. Jun 1 '21 at 0:27
• I think this could be a good application of the probability of false alarm vs probability of detection (based on your threshold setting) combined with a simple correlation for the notes of interest (the FT is a correlation to all notes, while in this case you just want two so a direct correlation would be simpler). You would also need to set a time duration. Jun 1 '21 at 0:33

Since you don't want to collect audio samples the best I could suggest is to have some filtering of the compressed spectrum.

for instance

$$\underbrace{\int_{-\infty}^{\infty} \left( \sum_{k=1}^{\infty} u_k \cdot c(h(\omega/\omega_1) X\left( \frac{k\cdot \omega}{\omega_1}\right) \right) d\omega}_{\textrm{Harmonics}} + \underbrace{\int_{-\infty}^{\infty} \left( \sum_{k=2}^{\infty} v_k \cdot c(h(\omega/\omega_1) X\left( \frac{\omega}{k\cdot \omega_1}\right) \right) d\omega}_{\textrm{Subharmonics}}$$

Where

• $$h(w)$$ is a function that will take the spectral leakage in to account when scoring each harmonic/subharmonic.
• $$u_{k}$$ is a weight to the harmonic $$k$$
• $$v_{k}$$ is a weight to the hamonic $$v$$
• $$c(I)$$ is a function that translate amplitude to importance, maybe you want to use a logarithm function since human perception of sound intensity is rouhgly logarithmic. Maybe simply a square root to avoid detecting a harmonic of the fundamental, for instance suppose that some timber has a very strong third and fourth harmonics and you play a C4 and a G3, if you just search for use intensity directly you could detect a G5. With this compression function you will give more importance to the detection of multiple harmonics instead of looking to the intensity of a particular harmonic.

The sums and the integral are declared to be infinite, but since $$u_k$$ and $$v_k$$ tend to zero as $$k$$ increases, you can set a limit for the summations. And since the signal has a limited spectrum you can set a bounds to the integration.

Then you guess all of these parameters without collecting samples, and if you are lucky you will have a good result. If you are doing this based in your prior experience you are implicitly using collected data as well.