First of all - I have only a limited of Fourier transforms and the mathematics involved, so please pardon any potentially bad terminology or huge gaps in conceptual knowledge.

I'm experimenting with creating an additive audio synth and want it to be able to achieve the following:

  • capture a FFT window from some audio source
  • feed the resulting spectrum from its bins into the synth
  • have the synth cleanly play back that spectrum. By that I mean that the sound should be "smooth", eliminating artefacts from the phase/frequency relations of the individual oscillators. I kind of expect to get a "smudgy" sound if the captured FFT frame contains transient material, and a "smooth" sound if the input is a simpler harmonic sound.

my synth implementation has free-running sine oscillators where I can set the frequency / amplitude / initial phase.

first question: is it sufficient to set the oscillator freq/amp/phase to certain initial values from a FFT frame and just play them in free-running mode to achieve smooth playback?

From some limited experimentation I've noticed the following: If I take a 1024-sample windowed FFT, I get 512 frequency bins. So I use 512 oscillators, and set their frequencies linearly from 0-22050Hz (samplerate/2), so evenly 21.533203Hz apart. I then set the oscillators' amplitude and phase to the values I get from the FFT frame.

Now when playing back this configuration, one thing that is immediately (audibly) noticeable is that the ~21.53Hz frequency defined by the frequency bins is the dominant component in the output. The oscillators are "beating" against each other because (from my intuitive understanding) there are strong phase correlations between the bins. These are not part of the sampled input signal, and should be eliminated from the desired output signal.

Now the question is, what do I have to do to get rid of that? Is it actually possible to just use the raw frequency bin magnitudes and phases to get a continuous signal?

Or, do I need to calculate some other frequencies for the oscillators? I'm aware that the FFT frequency bins do not represent actual frequency peaks in the signal accurately. Is it necessary to calculate/guess actual peaks?

Or, do I need to modulate the oscillator phases over time in a way such that the beating is eliminated..? (the synth can do modulation, but just blindly guessing here)

audio example

440Hz saw wave original

440Hz saw wave taken as FFT frame and played by additive synth

  • $\begingroup$ This is in the larger topic areas of the phase-vocoder but, more likely, sinusoidal modeling. You do not want an oscillator for each of the bins, but you want to identify, from the Fourier Transform output, the discrete frequencies that you want to pass to discrete oscillators for resynthesis. You want to make sure that the phase of each oscillator stays continuous as you transition from one frame to the next. This means adjusting the phases of each oscillator you have in frame $n+1$ to the phase of the corresponding oscillator in frame $n$. Frequency-domain audio can be hard to do right. $\endgroup$ – robert bristow-johnson May 29 '19 at 20:35
  • $\begingroup$ cool thx! got it going, looking promising. using parabola fitting to detect the top n peak frequencies and setting the oscillators to these freqs. some minor things unclear and code is a bit messy, will update the post / answer once cleaned up. audio: dropbox.com/s/pkkrbwyvqdgp219/resynth_rtm.mp3?dl=0 - this is recorded online and running on GPU $\endgroup$ – Jakob May 30 '19 at 11:08
  • $\begingroup$ What are you doing this with? C/C++? or MATLAB? or something else (like Python)? $\endgroup$ – robert bristow-johnson May 30 '19 at 20:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.