1
$\begingroup$

My specific question is: is there a standard way to do the kind of analysis I describe below? If not, is there a useful method that I have missed?

I'm exploring ways to produce a spectral histogram of audio signal magnitude based on musical notes. I wish to calculate a spectrum spanning 10 octaves, or 120 notes. This covers a frequency range from ~20Hz to ~21,000Hz. The sample rate will be at least 44,100Hz.

The final desired result is a numerical value per note, that I can then plot on an axis of note bins. This is essentially a logarithmic x axis, with non-linear note bins. I wish to do this in realtime, on a desktop computer, and display visually at ~10Hz update rate or better.

I initially looked at the DFT (via FFT), however this has some drawbacks - the size of the DFT needs to be large enough to provide sufficient resolution at the lowest frequencies. E.g. to get ~1Hz resolution at 44100 Hz I will need to use an FFT of size 65536, which is longer than 1/10th of a second of signal (for 10Hz update). Secondly, the frequency bins for the DFT are linearly spaced in frequency, so there needs to be a warped mapping from these linear frequency bins to the non-linear note bins. In some cases many frequency bins will map to a single note bin. I understand that the shape of the frequency bins is related to the window function, so I imagine I would use something like sinc interpolation to collect frequency bins into corresponding note bins.

I've also come across some other options:

Someone claimed to have found a way to tweak the DFT to use note frequencies as the basis functions, however I wasn't able to understand how they did this and there was no math provided to back it up. Is this possible? Can the set of basis functions for the DFT be changed to match note frequencies, and if so what are the implications for scaling, phase (not so important in my application). I'm a layman but my feeling is that if the basis functions are not orthogonal then there's going to be overlap which could result in some terms counting energy more than once.

Another suggestion involves applying the Goertzel algorithm 120 times. However based on the rule-of-thumb on Wikipedia, this would be significantly more expensive than using the FFT above.

Another idea involved applying a much smaller FFT to a single octave, with filtering and downsampling to shift each octave down. This would result in 12 smaller FFTs and 11 frequency-shift/filter operations. This may be more efficient than the large FFT above.

One final point - the system can be fine-tuned in realtime, so "A440" may not actually be exactly 440Hz and may shift slightly by up to half a semitone in either direction. Therefore I need a method that can be "tuned" accordingly, or is precise enough to correctly capture energy within +/- half-semitone of a note.

$\endgroup$
  • $\begingroup$ For ~10Hz visual update, at 44100 Hz sample rate, this implies a maximum FFT size of 4096. This results in a linear frequency resolution of ~10Hz. Unfortunately, at the lowest octave, this covers seven notes. $\endgroup$ – davidA Jan 9 '17 at 22:29
  • 1
    $\begingroup$ you can run 120 simultaneous comb filters. each comb filter tuned to each of the 120 notes you want to detect. when a low note lights up notes that are an octave or two octaves or 19 or 31 semitones above and does that at the same attack time you will have to pick the correct note. and when one of those higher notes lights up an octave below, you will also need to sense that and pick the correct note. $\endgroup$ – robert bristow-johnson Jan 10 '17 at 0:01
  • $\begingroup$ rb-j, you might want to add a link to what a comb filter is; the idea of frequency-domain periodic filters is pretty but not inherently intuitive, I guess $\endgroup$ – Marcus Müller Jan 10 '17 at 1:01
  • $\begingroup$ An FFT spectral histogram will not reliably show musical note pitches. That's because many pitched musical sounds have more energy in their overtones series, and some of the higher harmonics don't end up near one of the 12 ET frequencies per octave. $\endgroup$ – hotpaw2 Jan 10 '17 at 16:12
  • $\begingroup$ @meowsqueak : A 10 Hz update rate does not limit the size of the FFTs to 4k. Much longer FFT windows can be used by overlapping them. $\endgroup$ – hotpaw2 Jan 10 '17 at 16:16
1
$\begingroup$

Have you looked into a Chroma/pitch-class analysis? That sounds more like what you want overall (expect perhaps for the octave-folding/collapsing).

Check out librosa's CQT-based chromogram. Code to generate plot below on github: enter image description here

You can use the tuning parameter to tune the musical scale, but study the full paramter set.

tuning : float
    Deviation (in cents) from A440 tuning

Finally, checkout these examples showing "enhanced chroma", to say isolate harmonic content:

enter image description here

Regarding performance of the CQT computation, the FFT + a kernel can be used to make the computation much faster. See Judith Brown & Miller "Super" Puckette's An efficient algorithm for the calculation of a constant Q transform.

$\endgroup$
  • $\begingroup$ Thanks, I did start looking at the Constant-Q Transform independently, which seems related to this and could indeed be promising. LibROSA seems like a nice library for exploring this kind of thing too. I do need to calculate/measure the latency that it introduces like the STFT though. $\endgroup$ – davidA Jan 10 '17 at 2:15
  • 1
    $\begingroup$ Modified my answer to address the performance/latency question of the CQT. $\endgroup$ – ruoho ruotsi Jan 11 '17 at 0:07
  • $\begingroup$ That is an interesting paper - thank you. I also found this one: Constant-Q Transform Toolbox for Music Processing and there's an open-source implementation too. $\endgroup$ – davidA Jan 11 '17 at 2:49
  • $\begingroup$ That's a solid paper, Dr. Anssi Klapuri is a giant in the field of MIR. I'm gathering your preference for reference implementations are in C++ versus python? $\endgroup$ – ruoho ruotsi Jan 11 '17 at 4:36
  • $\begingroup$ My target is C++, however I don't mind either way - I can translate from one to the other as necessary if the source is available under a suitable license. $\endgroup$ – davidA Jan 11 '17 at 20:13
0
$\begingroup$

A bank of 120 filters (Goertzel or other IIR or FIR) or a bank of near-Constant-Q FFTs would work. (I've also used windowed 1-bin DFTs for each filter.) You can use overlaps as needed to meet your display update rate. Either of these will likely not overload any modern (laptop, smartphone, R.Pi) processor, even for real-time display.

The lower frequencies would require more filter or FFT window latency for semitone resolution (depending on some assumptions about the sound). You didn't say what your purpose was for visualization; but if to roughly match human perception of frequency, some psychoacoustic experiments seem to indicate that human estimation of the note of a mid to low pitch also requires more time, very roughly proportional to some number of periods of the pitch.

$\endgroup$
  • $\begingroup$ Thanks. The purpose of the visualisation is to complement a parametric equalizer for before/after (visual) comparisons. It's not for accurate measurement however it does need to be consistent across the spectrum. $\endgroup$ – davidA Jan 11 '17 at 20:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.