# Melody/Vocal extraction

I'm new to the whole DSP area, but I've built a program which plays music and does FFT to produce some cool visuals for the music in realtime.

The FFT algorithm I chose was from a C# library called NAudio. I applied a hamming window before shipping it off to the FFT, from which I get an array of complex. Nothing special. Works fine.

But now I want to display this data to the user, so I converted the bins to magintudes following this formula: magnitudes[i] = sqr(real[i]² + imaginary[i]²)

This with some amplification yields a pretty good result to display for the user, but I feel that most of the levels being displayed are just random noise, I want to show the user what he/she "hears", where the melody or vocal are being amplified OVER the other background music.

The thing I'm unsure of is how to approach this problem. What can I do to filter out everything else and only have the melody/vocals left. I can understand there's not 100% clean way you can do this, but what method could give me a decent result?

Thank you

• I'm afraid there is no simple method to achieve what you need. – Jazzmaniac Apr 8 '15 at 17:38
• What people hear is a psycho-acoustic issue, not something an FFT can separate. For typical music recordings, there is no way to filter out everything else but voice or melody, as the spectrums are mixed and overlapping. The practical solution is to use a multi-track recording where voice and music are already on completely separate tracks. – hotpaw2 Apr 8 '15 at 20:29

I remember reading a similar question (either here or on stack overflow) abouylt melody extraction. In any event someone did create such a project as a PhD thesis (luckily I bookmarked the site). He even offers his work (for research/personal/non-commercial use) for windows mac and linux http://www.justinsalamon.com/melody-extraction.html#demo if you could somehow integrate this to your application you can use the extracted melody to display the visuals, but still just play the original song from speakers/headphone

• Very cool project! Exactly what I've been looking for. I'll check it out more in-depth later today, and hopefully implement this. Thank you! – Tokfrans Apr 10 '15 at 4:10

As you are going at it, you won't be able to see much detail. There won't be (much) visible difference between loud sounds and quieter sounds, not even if you multiply the magnitudes to make them larger.
I don't think you really need to separate the sounds into "melody+voice" and "background." That would be a (seriously) non-trivial problem. I think you just need to make the difference more obvious.

To make the difference more obvious, change the magnitudes to decibels. The Decibel is a logarithmic representation.

What you need to do is this:

magnitudes[i] = 20* log((sqr(real[i]² + imaginary[i]²))

This will give you a logarithmic view of the music.

The formula won't give you dBs that relate to any real volume level - the levels are relative to one another, so that while you can say that a particular frequency is 20dB louder than another one, you can't say what the absolute loudness is for any of them. This is fine, though, since you are just looking at it and not using it for measurements.

• This surely gave me better results! Just added an arbitrary number to the output to make it fit in my scale. Good tip! – Tokfrans Apr 10 '15 at 4:43