1
$\begingroup$

What exactly is one looking for in audio data to find different instruments or energy sources?

So the FFT can extract frequencies, but how can e.g. percussion be detected alongside e.g. a bass line?

Would the frequency bins be checked into a frequency mapping of instruments? This would lead to collisions in possible instruments.

Are the frequencies of a FFT output directly mappable to pitch frequency charts or should there be a conversion first?

$\endgroup$
  • 1
    $\begingroup$ I can answer, but first, what do you want to do: recognition ("this audio clip contains a double bass and a piano"), transcription ("there is a double bass playing a C1 and a piano playing a C3"), or audio separation? Because in the first case one only needs to look at superficial features. $\endgroup$ – pichenettes Aug 8 '13 at 11:08
  • $\begingroup$ I would call this "decomposition task" $\endgroup$ – Val Aug 8 '13 at 13:32
  • $\begingroup$ "Look" may be the wrong word. Humans can be fooled by being shown one instrument, when another is actually playing in the audio mix. These visual cues, and other assumptions and expectations, may help humans "find" instruments and sound sources, whether they are actually in the sound mix or not, or the signal is ambiguous. $\endgroup$ – hotpaw2 Aug 8 '13 at 16:16
2
$\begingroup$

There seem to be four questions here. I can't give complete answers for each of them but let me address them in turn.

What exactly is one looking for in audio data to find different instruments or energy sources?

There's not really a single answer to this. There's lots of research into many closely related but ultimately different problems. In general instruments vary in the way their spectral properties vary over time. Tracking the changes of all the harmonic and inharmonic components of a note amongst other sounds is no small task.

So the FFT can extract frequencies, but how can e.g. percussion be detected alongside e.g. a bass line?

Generally you'd use a Short-time Fourier transform so that there's time data as well as the spectral information. From here you've got a few problems to think about:

Source separation: look up techniques such as ICA, NMF and CASA.This will divide up the energy between the detected sources allowing you to detect the separate instruments.

Fundamental frequency detection: I recommend summary auto-correlation of frequency bands rather than purely time or frequency domain methods.

also Source identification (which I don't know much about).

Would the frequency bins be checked into a frequency mapping of instruments? This would lead to collisions in possible instruments.

You need to apportion the frequency information not just assign it all to one instrument.

Are the frequencies of a FFT output directly mappable to pitch frequency charts or should there be a conversion first?

The fundamental frequency, which is given on a pitch-frequency chart, may not actually present in a note ('the missing fundamental problem'). So a fundamental frequency detection algorithm such as the one I suggest above will be needed before you can use the chart.

$\endgroup$
  • 1
    $\begingroup$ Note that the ICA model assumes instantaneous mixtures, whereas actual sound on multiple mics is modeled better by convolutive mixtures, which pure ICA cannot solve. (+1) nonetheless. $\endgroup$ – Tarin Ziyaee Aug 8 '13 at 14:05
0
$\begingroup$

Some of the frequencies (often the vast majority), in an FFT of audio data, represent some overtone of a musical pitch. Perhaps even a combination of different overtone numbers of different pitch sources. One needs to figure out which overtone of which pitch source(s) a frequency represents before any note pitch mapping can take place. Since the overtones of many instruments can overlap, you may need to figure out how to unmix them as part of the process as well.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.