How does Adobe After Effects generate its "audio spectrum" effect?

Question

I'm trying to replicate the "audio spectrum" effect from Adobe After Effects. An example can be seen in this video:

Obviously, it has to be some variant of a fourier transform, but I've tried taking the fft of an audio sample in realtime and my implementation looks more like a big block of bars:

I imagine some kind of intermediate function is being applied to the fft, but I can't really figure out what it might be.

Answer 1

I've had a try at reproducing the effect, and I think these are some of the key elements:

• You need a high-resolution FFT (large size; windows overlapping in time so that this doesn't come at the expense of frame rate) and to discard all but the lowest-frequency bins. You can tell this from the video because there are only rarely any harmonics visible and then only a second harmonic. The music you're hearing contains many more harmonics, so the display must be showing only the lower frequencies, not all the way up to $f_s/2$.

• You need to dynamically adjust the vertical offset so that only the significant peaks are shown, hiding the broad-spectrum sounds whether they are continuous 'noise' or sharp drumbeats. What I did was take the median of the current frame's data and subtract it (as well as a manually set additional term), but there may be better strategies.

• If still needed once you've discarded the high-frequency components, apply a frequency-varying offset so that the levels are even across the spectrum (rather than decreasing with frequency as your picture does). Whether this is actually needed depends on the frequency content of your input signal — thus the frequency response of the microphone it was recorded with, and so on.

• The nice rounded rather than spiky peaks should be apparent once you've zoomed in on the low frequencies, but they are also dependent on your choice of window function. Use a window function with a large main lobe.

Pseudocode:

fft_length = 32768
display_length = 1024
freq_varying_offset = fft_length * 0.000005
for frame from 1 to ...:
    let start = frame / video_frame_rate * audio_sample_rate
    let samples = audio_data.subarray(start, start + fft_length)

    let buf = log_fft(window(samples), fft_length).subarray(0, display_length)
    let offset = fixed_offset - median(buf)
    for i from 0 to buf.length:
        buf[i] = buf[i] - offset + i * freq_varying_offset

    // now draw the bars on the video frame using the contents of buf

Answer 2

You need to apply windowing to your audio data before passing it to the FFT. FFT assumes the audio is infinitely continuous so if the chunk of audio you have were to play back to back in a loop, any discontinuities in the audio end up containing a lot of higher frequency data than is present in the actual signal. Windowing is a scaling up of the audio up from zero at the start of the audio and then back down to zero at the end of the chunk. This insures no discontinuity in the audio exists and all of the extra high frequency 'noise' in your FFT will be gone. There are various windowing techniques that are in use. Any one will work but they all have different properties so you may want to try a few to see what gives you the most pleasing result for your application. This article is a nice introduction to FFT and Windowing. https://download.ni.com/evaluation/pxi/Understanding%20FFTs%20and%20Windowing.pdf