# How can I modify a sound signal based on it's MFCC

I'm working on an NN algorithm with which I want to filter noise from sound signals.

I have some clean speech sound samples and noise samples, by combining them I get the signal that I want to filter.

I get the 24 MFCC values which I feed in a NN and the output is another 24 MFCC values.

• How do I modify the original sound sample using the MFCC values to actually see the difference in it?

I've read about this topic here and he does something similar, but doesn't quite explains how does he modify the sound signal using those MFCC values. By the way, im using keras and tensorflow in jupyter notebook.

• uh, are you adding the sample-wise same noise to all your speech samples? That'd be a ML sin! Dec 6 '18 at 8:09

The authors in the page you are linking are using the analog of an equaliser but the more accurate representation of what they do is a vocoder.

The typical application of a vocoder in music is to modulate one signal with the spectrum envelope of another. This sounds like this or this or this and so many other uses today, I hope these three examples are descriptive enough.

In this particular application, the Neural Network (NN) is "learning" what is signal and what is noise, something that looks very much like adaptive filtering.

Therefore, the signal path looks like this:

Input -+-----------------> Vocoder --> Output
|                      ^
|                      |
+-- MFCC --> NN -------+


Hope this helps.

EDIT:

In terms of actually doing this you would first have to set up the basic pipeline that splits the Input into frames of N samples, calculates the Discrete Fourier Transform of each frame, derives the spectrum, modulates the spectrum with the coefficients of the NN and then does the inverse DFT to go back to samples in the time domain.

This basic pipeline is usually handled with one of overlap-add or overlap-save methods and Python's Scikit module includes everything you need to implement that.A good start would be the fftpack' module. In addition to this, you can search for Python implementations of these two methods and see if there is something there already that you could draw inspiration from. (e.g. this one).

A detail here would be that if you want to use MFCC coefficients, you would have to split your DFT coefficients in the same bands (as the MFCC coefficients are derived from).

This would cover the Vocoder` part which accepts the spectrum of the signal at its input and the MFCC "weights" and applies the weights to the spectrum.

The training / operation of the Neural Network can be handled with a number of other modules like those that you mention (or maybe even Shogun which is an older but very useful toolkit even for large problems).

EDIT 2:

The act of filtering each of the signal frames that result from an overlap-add or overlap-save method is a typical Digital Signal Processing operation.

To do that, all you have to do is multiply the DFT spectrum of your signal with the frequency response of the filter. The operation is concluded when you apply the inverse transform and you go from the frequency domain to the time domain to playback your signal.

What you get from the NN is basically the filter. So, the short answer is that you need to multiply your NN coefficients with the spectrum of a given sound frame.

The difference here is that instead of using all the possible coefficients of the DFT spectrum, you divide the spectrum in 22 bands. In addition to that, these bands are not at equal spacing to each other.

For example, in a typical filtering application, you could decompose your spectrum in 20 DFT coefficients and those would look like equally spaced band pass filters. But the Mel Scale or the Bark Scale are not dividing the frequency range at equal spaces.

So, prior to applying the "weights" that you get from the NN to the signal, you need to construct and apply those bandpass filters that divide the spectrum according to the Mel or Bark scales.

• This looks like Exactly what i was looking for! And the examples are pretty useful :)) I just have to find a way to implement it in python though. Thanks a lot! Dec 6 '18 at 17:43
• So yeah, if you know any way to implement that in python, as I can't really find any straightforward method to do that, please let me know. Thanks! Dec 6 '18 at 19:08
• @DariusChira Glad to hear you found this useful, please see edits for notes on implementation.
– A_A
Dec 6 '18 at 21:22
• well that's an incredible amount of help, thanks a lot! Dec 8 '18 at 11:37
• I've basically done everything, the only part that was missing and that I didn't exactly know how to do was how to modulate the signal with the NN MFCC output Dec 8 '18 at 11:50