1
$\begingroup$

I'm currently working on a phase vocoder implementation based on the Short-Time Fourier Transform; it's heavily based on the models described here and here. I have successfully completed the analysis phase, wherein

  • $m$ FFT frames of size $N$ are derived from a time-domain signal $x[n]$
  • These frames are sampled from equidistant, overlapping locations by a number of samples defined as $hop_a$
  • Assuming that $hop_a$ is known and $M$ is the total number of samples being analyzed via the STFT, I believe $m$ can be found via the equation $m = \dfrac {M - N} {hop_a} + 1 $

After analysis, the frames are overlap-added back together. When time-stretching/pitch-shifting, the overlap is changed such that the resynthesis hop size is defined as $hop_s = S * hop_a$, where $S$ is the stretching ratio. Theoretically, will result in a buffer of samples that is of length $S * M$, which can then be resampled back into a buffer of size $M$, which will then have a pitch equal to $S$ times its original pitch. For instance, to pitch a 1024-sample block up an octave, it would be put through the STFT, analyzed, and then its frames re-layered into a block of 2048 samples which would finally be resampled back down to 1024 samples, resulting in an octave pitch shift. In the formulas, this is apparently achieved by simply having the synthesis hop size $hop_s$ be equal to $S * hop_a$

Here is my confusion: As far as I am able to discern, the math suggests that simply setting $hop_s$ to $S * hop_a$ will not result in an output block of size $S*M$. Taking the above example, assuming $M=1024$ and $N=256$ with $hop_a = 64, S=2.0$, $$m = \dfrac {1024 - 256} {64} + 1 = 13,$$ $$ m = \dfrac {M_s - N} {hop_s} + 1,$$ $$\therefore M_s = (m - 1)*hop_s + N = (13 - 1) * 128 + 256 = 1792,$$ $$ S * M = 2048 \neq 1792$$ By this logic, using $S=2.0$ would only result in a time-stretch and resulting pitch shift with a ratio of 1.75, or about 9.7 semitones; not the intended 12. I have tried this math out with many input values. I'm fairly certain the issue lies with this pitch-stretching step, as my implementation worked exactly as expected until this point.

What am I missing here? It's been many years since I've been in a math class, so all of this has been self-taught and it wouldn't surprise me if I've missed a very fundamental part of the calculations. As such, my notation might also be confusing; let me know if I can clarify! Thank you!

EDIT: I believe my misunderstanding came from trying to think about continuous-time algorithms as if they were to be applied to a finite chunk of samples. For a real-time algorithm, you are simply resampling $hop_s$ samples back into a buffer of size $hop_a$, which will always confer the stretching ratio $S$ you are looking for. Mathematically, this can be thought of as $m$ approaching infinity, thereby negating the issue of $N$ not being divisible by $hop_s$.

Improve this question
$\endgroup$
8
  • $\begingroup$ Are you pitch shifting or are you time scaling? Is this something you are trying to implement for real-time operation? Or is it something that you want to apply to sound files? $\endgroup$
    – robert bristow-johnson
    Jan 22, 2022 at 9:26
  • $\begingroup$ @robertbristow-johnson I'm intending to use this for pitch-shifting, and the end goal is for it to be able to operate in real-time. I'm doing this all with the intention of making a plugin that allows more fine user-level control over the pitch-shifting algorithm for more creative possibilities. $\endgroup$
    – Jackson Kaplan
    Jan 22, 2022 at 22:57
  • $\begingroup$ Well, then there are two approaches, if you're doing the STFT and phase vocoder. One approach is to do time-scaling first (so that, at first, the pitch is unchanged) that time-stretches or time-compresses the audio, then do resampling (with interpolation) of that stretched or compressed audio back to the original number of samples. The other approach is to move the spectral peaks in the STFT from one location to another in the frequency domain. But then some kind of interpolation in the frequency domain is needed. It's tough, choose your poison. $\endgroup$
    – robert bristow-johnson
    Jan 22, 2022 at 23:12
  • $\begingroup$ Now, If you want this to run real-time, what do you want the throughput delay to be? Can you handle a throughput delay of 100 or 200 ms? Because that is the ballpark of the minimum delay if you do this in the frequency domain. This is because you will have to input at least 2048 or 4096 samples before you can even begin to pass them to the FFT. Even with an infinitely fast computer, you still have to wait until you get enough samples before any decent Fourier Transform. There are time-domain techniques that require a decent pitch detector that are popular for live, real-time operation. $\endgroup$
    – robert bristow-johnson
    Jan 22, 2022 at 23:17
  • $\begingroup$ At current, I'm planning on following the time-stretch and resample method. Though it will be a real-time algorithm, I plan on allowing for a large latency to account for user parameter automation. I've got all that under control though—my question is a specific conceptual one: why does the calculated size of the time-stretched buffer not exactly equal the original pitch ratio times the original buffer size? $\endgroup$
    – Jackson Kaplan
    Jan 23, 2022 at 20:12

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.