I'm trying to implement source-filter separation of some pre-recorded speech in Max/MSP. I'm unsure of whether one uses an FFT to enter the cepstral domain, or an IFFT. Some of the textbooks and dissertations I've found that cover this topic say that I should perform an FFT to get into the cepstral domain and then an IFFT to exit back into the frequency domain; others say that I should perform an IFFT to enter the cepstral domain and then an FFT to get back into the frequency domain! The first way makes more sense to me, but something about my patch isn't quite working quite right, and I'm not sure what I'm doing wrong.
Here is how I understand the procedure (Please tell me if you notice anything incorrect):
1. Perform an FFT on the time-domain signal to enter the frequency domain.
2. Do Mel scaling (optional) and then logify.
3. Do an FFT (or an IFFT, depending on who you ask) to enter the cepstral domain.
4. Separate the low bins (the filter) from the high bins (the source / excitation signal).
5. Perform an IFFT (or an FFT, depending on who you ask) on both source and filter to exit back into the frequency domain.
6. Exponentiate both source and filter.
7. Modify amplitude of filter bins.
8. Multiply the modified filter with the source signal.
9. Un-Mel-scale (optional).
10.Perform an IFFT to exit back into the time domain.
In Step 3, should I use an FFT or an IFFT? (And then Step 5 will be the opposite of whatever Step 3 is.)
P.S.: I am not trying to modify phase information, only change some amplitude information of the filter portion of the signal (the lower FFT bins in the cepstral domain) before I add/multiply them back together.