I'm building a neural network for instrument recognition based on the IRMAS data set. However, I'm a bit at a loss for what features to use. MFCC seems to be quite popular but extracting this per sample didn't give better results than random guessing, and I'm quite lacking in audio background.

Any suggestions on good features to select?



Well the state of the art performance on such tasks is achieved by deep neural networks, and especially, the convolutional ones (CNNs) set you free of extracting hand crafted features. The network learns those features as well as the weights. This way you avoid the hustle or the rots o'ruck. For sequential data input, such as audio signals, recurrent neural nets stand out to be promising.

There are many toolboxes such as TensorFlow, Torch, or CNTK and they would allow you to quickly prototype your ideas.

I have used deep learning successfully in many scenarios, but unfortunately, instrument recognition is not one of them. However, looking into the academia, I already see that there exists some works on instrument recognition and a huge amount of works on understanding of music.

  1. First, a comprehensive list of use in music: https://github.com/ybayle/awesome-deep-learning-music

  2. An approach targeting the similar problem published in IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP):

Deep convolutional neural networks for predominant instrument recognition in polyphonic music, Yoonchang Han, Jaehun Kim, and Kyogu Lee, 2017


  1. A youtube demo that uses CNNs: https://www.youtube.com/watch?v=ZtxNEOIpVPs
  • $\begingroup$ Have you ever used any of these deep neural network tools that actually worked well for instrument separation? $\endgroup$
    – dsp_user
    Dec 30 '17 at 10:25
  • $\begingroup$ Updated my answer to include that. $\endgroup$ Dec 30 '17 at 10:41
  • $\begingroup$ +1 for providing that additional information. I'm always suspicious of systems/algorithms that claim being able to do instrument separation. $\endgroup$
    – dsp_user
    Dec 30 '17 at 11:48
  • $\begingroup$ Well indeed deep learning is something to be suspicious of, but it has been proven to work well in many domains. $\endgroup$ Dec 30 '17 at 19:03

well, i'm quite lacking in NN background. there are so many things our brain uses to recognize an instrument.

  1. the range of pitch. (pitch is a perceptual parameter most often associated with the physical parameter we call the "fundamental frequency" which is the reciprocal of the period of a quasi-periodic function of time.)
  2. the nature of the note attack or the note onset. some stringed instruments are plucked, others are bowed. some horns have a recognizable note onset. other horns sound like they gradually fade in. even a pipe organ has some kinda noisy attack. in MIDI we associate "key velocity" with this note attack.
  3. the timbre profile. this is a multi-element vector with the relative amplitudes of each of the harmonics. and this timbre profile can change with different loudness or key velocities. some timbre profiles are more difficult because, say with tubular bells, the "overtones" or "partials" are not harmonic (which means they are not at frequencies that are an integer multiple of the fundamental frequency that we associate with pitch).
  4. tremelo (in amplitude) and vibrato (in pitch). instruments like an acoustic piano don't have this. but electric pianos have a tremelo knob. violins and other stringed instruments have vibrato. a vibraphone actually has tremelo, but not vibrato of pitch. a marimba has neither.

extracting these parameters from recordings of multinote music or chords will be (how shall we say?) a copulating female canine. rots o' ruck.


I've previously performed instrument recognition on the IRMAS data set using simple classifiers (shallow neural network, SVM, decision tree, knn) and a range of audio features (averages and standard deviations of time series audio features to be precise). Most of the audio features that were used are described in this source, including MFCCs. MATLAB code snippets for audio feature extraction can be found on Alexander Lerch's Audio Content Analysis website (that accompanies his book that he has written on the topic). Note however that the field of audio content analysis is on the decline with the advent of deep learning techniques in music information retrieval. The paper linked by Tolga Brital proposes a method using a CNN that outperformes previous attempts of (predominant) instrument recognition (the authors also used the IRMAS data set). Depending on your purposes, it might be beneficial to skip classical audio feature extraction and concentrate on deep learning approaches instead.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.