1
$\begingroup$

What exactly is one looking for in audio data to find different instruments or energy sources?

So the FFT can extract frequencies, but how can e.g. percussion be detected alongside e.g. a bass line?

Would the frequency bins be checked into a frequency mapping of instruments? This would lead to collisions in possible instruments.

Are the frequencies of a FFT output directly mappable to pitch frequency charts or should there be a conversion first?

$\endgroup$
3
  • 1
    $\begingroup$ I can answer, but first, what do you want to do: recognition ("this audio clip contains a double bass and a piano"), transcription ("there is a double bass playing a C1 and a piano playing a C3"), or audio separation? Because in the first case one only needs to look at superficial features. $\endgroup$ Aug 8, 2013 at 11:08
  • $\begingroup$ I would call this "decomposition task" $\endgroup$
    – Val
    Aug 8, 2013 at 13:32
  • $\begingroup$ "Look" may be the wrong word. Humans can be fooled by being shown one instrument, when another is actually playing in the audio mix. These visual cues, and other assumptions and expectations, may help humans "find" instruments and sound sources, whether they are actually in the sound mix or not, or the signal is ambiguous. $\endgroup$
    – hotpaw2
    Aug 8, 2013 at 16:16

2 Answers 2

2
$\begingroup$

There seem to be four questions here. I can't give complete answers for each of them but let me address them in turn.

What exactly is one looking for in audio data to find different instruments or energy sources?

There's not really a single answer to this. There's lots of research into many closely related but ultimately different problems. In general instruments vary in the way their spectral properties vary over time. Tracking the changes of all the harmonic and inharmonic components of a note amongst other sounds is no small task.

So the FFT can extract frequencies, but how can e.g. percussion be detected alongside e.g. a bass line?

Generally you'd use a Short-time Fourier transform so that there's time data as well as the spectral information. From here you've got a few problems to think about:

Source separation: look up techniques such as ICA, NMF and CASA.This will divide up the energy between the detected sources allowing you to detect the separate instruments.

Fundamental frequency detection: I recommend summary auto-correlation of frequency bands rather than purely time or frequency domain methods.

also Source identification (which I don't know much about).

Would the frequency bins be checked into a frequency mapping of instruments? This would lead to collisions in possible instruments.

You need to apportion the frequency information not just assign it all to one instrument.

Are the frequencies of a FFT output directly mappable to pitch frequency charts or should there be a conversion first?

The fundamental frequency, which is given on a pitch-frequency chart, may not actually present in a note ('the missing fundamental problem'). So a fundamental frequency detection algorithm such as the one I suggest above will be needed before you can use the chart.

$\endgroup$
1
  • 1
    $\begingroup$ Note that the ICA model assumes instantaneous mixtures, whereas actual sound on multiple mics is modeled better by convolutive mixtures, which pure ICA cannot solve. (+1) nonetheless. $\endgroup$ Aug 8, 2013 at 14:05
1
$\begingroup$

Some of the frequencies (often the vast majority), in an FFT of audio data, represent some overtone of a musical pitch. Perhaps even a combination of different overtone numbers of different pitch sources. One needs to figure out which overtone of which pitch source(s) a frequency represents before any note pitch mapping can take place. Since the overtones of many instruments can overlap, you may need to figure out how to unmix them as part of the process as well.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.