Convolving two signals

Question

I saw a video where this guy used a program to do a frequency analysis on a voice signal and a sawtooth wave (I'm assuming this was FFT). Then he saved the plots as images and combined them pixel by pixel through multiplication using photoshop. He then put this picture back into the program and it did the inverse transform, turning it back into a sound. He said that this was an implementation of a vocoder but the low quality made it hard for me to tell if he was correct.

If so, then couldn't one simply implement a vocoder as a convolution operation? You would just have your two signals then select a window from each of these of the same width at the same position, then perform the convolution operation on these two windows (and probably use a window function as well, such as Hanning). You would, of course, have to do this for every sample, so you would be doing this as many times as you have samples in your tracks (and the windows would sometimes lie partially outside of a track, so they would have to be zero padded).

This seems like it might work because the convolution theorem says convolution in the time domain is itemwise multiplication in the frequency domain, so if it doesn't implement a vocoder, it at least implements exactly what the man in the video was doing (at higher quality). And, I'm not just asking this question blindly, I actually tried it. I get a very cool voice effect but I'm not sure it's the same as a vocoder. In fact, it sounds nothing like a vocoder. If some kind individual could tell me exactly what is happening here, I would be very grateful.

Answer 1

From your description, here is what is happening in the video:

• The short-term Fourier transform (aka spectrogram) of the signals is computed. The output of this operation is a matrix of complex values, which cannot be represented as images. Thus, the magnitude or the square of the magnitude is extracted to yield a single positive real value converted into a pixel intensity.
• The magnitudes (pixel intensities) are multiplied.
• The inverse short-term Fourier transform of the product is synthesized, presumably with made-up phases (or with the original phase of the carrier).

There are so many reasons why this has nothing to do with a convolution. I'm sure other members will point a few more:

• What is multiplied are magnitudes or their squares; but not the actual complex values produced by the Fourier analysis.
• Even if the quantities that were multiplied were the actual complex values, keep in mind that when dealing with discrete signals of finite length, the inverse discrete Fourier transform of the product of the discrete Fourier transforms is the Circular Convolution, not the convolution.
• And still... If you split two signals $x$ and $y$ into blocks of length N, and compute the pair-wise circular-convolution of length N of these blocks, you'll get something very different from $x * y$.

Convolving two audio-signals is a rather meaningless operations. One usually convolve an audio signal with the impulse response of a system; and when convolution is performed in the frequency-domain, the details are trickier than just pair-wise multiplications of FFTs (there's some blocking issues; + necessary overlap add or save).

What you saw would be more accurately described as spectral cross-synthesis. And there is one major reason why this is a very different beast from a vocoder. The goal of a vocoder is to apply the spectral envelope of the modulator onto the carrier. I emphasize on spectral envelope, because when you use a vocoder to make a stack of sawtooth waves say "hello", you apply the formants and overall loudness envelope of the original speech signal to the saw waves, but the last thing you want is the pitch information of the speech signal to get involved. The analysis filter bank of the vocoder should be designed to make abstraction of the individual spectral peaks in the modular signal - what matters is the rough spectral envelope - the location of the bumps (formants). This is why a vocoder doesn't need more than 12-30 channels - too few channels and it doesn't capture the formants, too many channels and it starts capturing pitch-dependent fine spectral peaks. Just like the features used in speech recognition...

Let me give you an example. Let us say you have a modulator speech signal with a 200 Hz f0 ; and formants at 1kHz and 1.5kHz - its spectrum is a sequence of narrow peaks at 200 Hz, 400 Hz, 600Hz, 800 Hz, 1000 Hz, 1200 Hz, 1400 Hz, 1600 Hz, 1800 Hz ; with the peaks at 1000 Hz and 1400 / 1600 Hz emphasized (formants). Let's say your carrier signal is a 140 Hz sawtooth - the spectrum is made of narrow peaks at 140 Hz, 280 Hz, 420 Hz... with a decreasing $1/n$ amplitude. If you go with what you described in your question (product of STFTs), you wouldn't have much left because the two spectra have very little overlap - the first common frequency in the sequence would be at 1400 Hz! What you want to do is somehow capture that the modulator spectrum has a bump at 1kHz and 1.5kHz, and use this information to boost the carrier spectrum in this frequency area. That's how vocoders work.

To do so, and if you really want to go the STFT route, an option would be to smooth the modulator spectrum with a rather wide kernel before doing the multiplication - so that what you are really doing is applying the spectral envelope of one signal onto the other one. This would be akin to applying a motion-blur on the Y axis in Photoshop using this silly image manipulation presentation...