First of all - I have only a limited of Fourier transforms and the mathematics involved, so please pardon any potentially bad terminology or huge gaps in conceptual knowledge.
I'm experimenting with creating an additive audio synth and want it to be able to achieve the following:
- capture a FFT window from some audio source
- feed the resulting spectrum from its bins into the synth
- have the synth cleanly play back that spectrum. By that I mean that the sound should be "smooth", eliminating artefacts from the phase/frequency relations of the individual oscillators. I kind of expect to get a "smudgy" sound if the captured FFT frame contains transient material, and a "smooth" sound if the input is a simpler harmonic sound.
my synth implementation has free-running sine oscillators where I can set the frequency / amplitude / initial phase.
first question: is it sufficient to set the oscillator freq/amp/phase to certain initial values from a FFT frame and just play them in free-running mode to achieve smooth playback?
From some limited experimentation I've noticed the following: If I take a 1024-sample windowed FFT, I get 512 frequency bins. So I use 512 oscillators, and set their frequencies linearly from 0-22050Hz (samplerate/2), so evenly 21.533203Hz apart. I then set the oscillators' amplitude and phase to the values I get from the FFT frame.
Now when playing back this configuration, one thing that is immediately (audibly) noticeable is that the ~21.53Hz frequency defined by the frequency bins is the dominant component in the output. The oscillators are "beating" against each other because (from my intuitive understanding) there are strong phase correlations between the bins. These are not part of the sampled input signal, and should be eliminated from the desired output signal.
Now the question is, what do I have to do to get rid of that? Is it actually possible to just use the raw frequency bin magnitudes and phases to get a continuous signal?
Or, do I need to calculate some other frequencies for the oscillators? I'm aware that the FFT frequency bins do not represent actual frequency peaks in the signal accurately. Is it necessary to calculate/guess actual peaks?
Or, do I need to modulate the oscillator phases over time in a way such that the beating is eliminated..? (the synth can do modulation, but just blindly guessing here)