Convolving two signals

I saw a video where this guy used a program to do a frequency analysis on a voice signal and a sawtooth wave (I'm assuming this was FFT). Then he saved the plots as images and combined them pixel by pixel through multiplication using photoshop. He then put this picture back into the program and it did the inverse transform, turning it back into a sound. He said that this was an implementation of a vocoder but the low quality made it hard for me to tell if he was correct.

If so, then couldn't one simply implement a vocoder as a convolution operation? You would just have your two signals then select a window from each of these of the same width at the same position, then perform the convolution operation on these two windows (and probably use a window function as well, such as Hanning). You would, of course, have to do this for every sample, so you would be doing this as many times as you have samples in your tracks (and the windows would sometimes lie partially outside of a track, so they would have to be zero padded).

This seems like it might work because the convolution theorem says convolution in the time domain is itemwise multiplication in the frequency domain, so if it doesn't implement a vocoder, it at least implements exactly what the man in the video was doing (at higher quality). And, I'm not just asking this question blindly, I actually tried it. I get a very cool voice effect but I'm not sure it's the same as a vocoder. In fact, it sounds nothing like a vocoder. If some kind individual could tell me exactly what is happening here, I would be very grateful.

From your description, here is what is happening in the video:

• The short-term Fourier transform (aka spectrogram) of the signals is computed. The output of this operation is a matrix of complex values, which cannot be represented as images. Thus, the magnitude or the square of the magnitude is extracted to yield a single positive real value converted into a pixel intensity.
• The magnitudes (pixel intensities) are multiplied.
• The inverse short-term Fourier transform of the product is synthesized, presumably with made-up phases (or with the original phase of the carrier).

There are so many reasons why this has nothing to do with a convolution. I'm sure other members will point a few more:

• What is multiplied are magnitudes or their squares; but not the actual complex values produced by the Fourier analysis.
• Even if the quantities that were multiplied were the actual complex values, keep in mind that when dealing with discrete signals of finite length, the inverse discrete Fourier transform of the product of the discrete Fourier transforms is the Circular Convolution, not the convolution.
• And still... If you split two signals $x$ and $y$ into blocks of length N, and compute the pair-wise circular-convolution of length N of these blocks, you'll get something very different from $x * y$.

Convolving two audio-signals is a rather meaningless operations. One usually convolve an audio signal with the impulse response of a system; and when convolution is performed in the frequency-domain, the details are trickier than just pair-wise multiplications of FFTs (there's some blocking issues; + necessary overlap add or save).

What you saw would be more accurately described as spectral cross-synthesis. And there is one major reason why this is a very different beast from a vocoder. The goal of a vocoder is to apply the spectral envelope of the modulator onto the carrier. I emphasize on spectral envelope, because when you use a vocoder to make a stack of sawtooth waves say "hello", you apply the formants and overall loudness envelope of the original speech signal to the saw waves, but the last thing you want is the pitch information of the speech signal to get involved. The analysis filter bank of the vocoder should be designed to make abstraction of the individual spectral peaks in the modular signal - what matters is the rough spectral envelope - the location of the bumps (formants). This is why a vocoder doesn't need more than 12-30 channels - too few channels and it doesn't capture the formants, too many channels and it starts capturing pitch-dependent fine spectral peaks. Just like the features used in speech recognition...

Let me give you an example. Let us say you have a modulator speech signal with a 200 Hz f0 ; and formants at 1kHz and 1.5kHz - its spectrum is a sequence of narrow peaks at 200 Hz, 400 Hz, 600Hz, 800 Hz, 1000 Hz, 1200 Hz, 1400 Hz, 1600 Hz, 1800 Hz ; with the peaks at 1000 Hz and 1400 / 1600 Hz emphasized (formants). Let's say your carrier signal is a 140 Hz sawtooth - the spectrum is made of narrow peaks at 140 Hz, 280 Hz, 420 Hz... with a decreasing $1/n$ amplitude. If you go with what you described in your question (product of STFTs), you wouldn't have much left because the two spectra have very little overlap - the first common frequency in the sequence would be at 1400 Hz! What you want to do is somehow capture that the modulator spectrum has a bump at 1kHz and 1.5kHz, and use this information to boost the carrier spectrum in this frequency area. That's how vocoders work.

To do so, and if you really want to go the STFT route, an option would be to smooth the modulator spectrum with a rather wide kernel before doing the multiplication - so that what you are really doing is applying the spectral envelope of one signal onto the other one. This would be akin to applying a motion-blur on the Y axis in Photoshop using this silly image manipulation presentation...

• Yes, it seemed odd to me that the operation with the images was so symmetrical and I suspected that there was something missing. And I was wondering where phases and complex values came into this, so you've answered that. I've been trying over and over (and over) to program a good vocoder. So could I run a STFT over my modulator and my carrier wave then run a boxcar filter or maybe a Hanning window over the absolute values of the frequency domain for the modulator, then multiply the result of that with the frequency domain of my carrier wave (complex), and finally pass that through the ISTFT? Commented Aug 4, 2012 at 22:35
• Two more things: First, you make the STFT sound like a sub-optimal solution, what would you recommend instead? I've tried bandpass filters (twice) but I could never get it working well. Unless you can give me specific advice on that I don't think I'll ever get it to work. Second, and not to cling to a bad idea, but could I somehow "blur" the frequency domain on my modulator then perform the convolution I was talking about for a vocoder effect? Commented Aug 4, 2012 at 22:43
• Not sure what you mean exactly by "run a boxcar filter or Hanning window", but for extracting a spectral envelope from a STFT frame, you could filter the spectrum by a gaussian response. It wouldn't be a very good solution because it would do the smoothing on a linear frequency scale. A better solution would be to multiply the STFT frame by a matrix with triangular responses (just like when computing MFCCs) to get a small number of coefficients corresponding to the energy in a half-octave filter bank, then multiply by the transposed of this matrix to get a smoothed spectral envelope. Commented Aug 4, 2012 at 23:20
• The STFT is overkill for a vocoder. It's not because you didn't get the filter-bank approach to work that it's not the way to go... You could get away with a STFT, but you'll have to be smart with the blurring. Also, to the risk of repeating myself, there's nothing in the whole process that will deserve to be called "convolution". Commented Aug 4, 2012 at 23:23
• The crux of the problem is that to be musically pleasing and relevant, a vocoder requires a coarse representation of the input signal in half or third-octave band. The STFT doesn't yield such a representation. Commented Aug 4, 2012 at 23:25