Pitch shifting with bins creating large feedback

Question

I am attempting to create a pitch correction algorithm. I started by performing a test. The test goes as such:

1. Get WAV file
2. Split it into bins of size n (512 in my case)
3. Shift each bin by 2 semitones (using a high level pitch shifting algorithm)
4. Aggregate all shifted bins together to recreate audio file

However, when I do this, a large amount of noise relative to the size of "n" is generated (spectral leakage maybe?) The smaller the size n, the larger the amount of noise

How do I implement the pitch shifting on a bin by bin basis while minimizing the noise, and how should I adjust for the phase shifting that the individual pitch shifts create?

Audio clips and code here (last clip in notebook is the one with all the added noise): https://colab.research.google.com/drive/1cpRhPpvXY_9XZidjOLKk_wW15EnkqLEX?usp=sharing

My code that attempted to fix the problem, but failed:

def win_taper(N, a):
    R = int(N * a / 2)
    r = np.arange(0, R) / float(R)
    win = np.r_[r, np.ones(N - 2*R), r[::-1]]
    stride = N - R - 1
    return win, stride

def pshift(key, x, f, G, overlap=0):

    notes = frequencies(key)

    N = len(x)
    y = np.zeros(N)

    win, stride = win_taper(G, overlap)

    for n in range(0, len(x) - G, stride):
      w = manipulate(x[n:n+G] * win, sr, f)
      y[n:n+G] += w
    return y

Tapered function taken from here: https://lcav.gitbook.io/dsp-labs/granular-synthesis/effect_description

Answer 1

...when I do this, a large amount of noise relative to the size of "n" is generated (spectral leakage maybe?) The smaller the size n, the larger the amount of noise

No. This is not "spectral leakage". The flutter that you hear comes from the fact that you are pitch shifting each signal frame without taking into account the phase shifting caused by the pitch shifting of the earlier blocks.

To put it in a different way: When you work with an audio recording, the "zero" for time is the first sample and that is the starting point when evaluating phase. By pitch shifting block $n$, you alter its phase (relative to the first captured sample) by some $d\theta_n$ but when you pitch shift block $n+1$, you are not taking into account the $d\theta_n$ you just applied. Therefore, subsequent blocks are out of phase.

What you hear at play-back are those phase discontinuities.

How do I implement the pitch shifting on a bin by bin basis while minimizing the noise/leakage?

Exactly how you are going to take this into account depends on the algorithm used. If the algorithm is a linear phase algorithm, then you might get away easily by correcting the phase for a constant term. Another thing you can do is to process the audio in overlapping blocks (for example, one block from 0 to 99 and the next from 60 to 159, rather than 0..99 followed by 100..199), still as if they were independent of each other. The final result is produced by averaging over the overlapping region. This does not correct the "problem" but it softens the phase discontinuities. By tuning the overlap (and possibly using a low pass filter too), you can supress those "jumps" to the point that they are not too noticeable. In a real-time setting, the overlap can be implemented as preserving a proportion of the currently playing output buffer which you average with the right segment of the next processed frame as the final step.

But also, saying this, you might be able to find an "online" version of a phase-vocoder with all these considerations implemented.

Hope this helps.