Software implementation of pitch correction

Question

I have a question about the physics of pitch correction of sound.

My background is physics, I'm mostly active on physics.stackexchange. (Check out my profile.)

The background of my question:

Before computerized recording there was only analog recording of sound. Let's say you are a singer and you have a recording of a piano accompinament, but you want to rehearse it at a lower pitch. Some audio tape players have adjustable tape speed. Adjusting the tape speed means that the pitch of the music and the tempo of the music change in lock-step.

It is my understanding: with the advent of computerized recording: software was developed with the capability of changing pitch and tempo independently. So then it was possible to change pitch whitout change of tempo, or change of tempo without change of pitch, or a desired combination of different shifts of pitch and tempo.

Given my physics background (knowing how sound is a superposition of many frequencies) I am stumped by the existence of software that can change pitch and tempo independently. I know it exists, I have used it, but I can't even begin to guess how it is implemented.

About viewing waveform on a computerscreen:
Years ago I copied music from a bunch of vinyl albums to CD. Some of the records had a bad scratch, causing 10 or 20 loud pops every turn at the location of that damage. Today there is software that can filter out pops like that automatically, but the software I used back then didn't have that. So what I did was: I would zoom in all the way to that spike, and sometimes I could copy an adjacent section, just a couple of miliseconds, and use that to replace a damaged section of that duration. I was never able to make it seamless, the act of pasting would always leave some artifact. That artifact was hardly audible, so it was OK, but I could never make it seamless.

I'm giving this information so that you know at what level you can answer this question.

How is pitch correction implemented? It must be a process that is fundamentally different from how pitch is changed when manipulating an analog recording.

I assume it cannot be a process of cutting up the music an putting it together again; my assumption is that that would create a ton of artifacts.

---

Response to the answers and comments.
In some ways my expectation has been confirmed in the answers. The original version has to be deconstructed and resynthesized at a very abstract level. Among the requirements for the level of reconstruction/resynthesization is high fidelity pitch detection, but in the real world it can be ambiguous in which octave the pitch is.

My impression is: when it comes to pitch correction there is no entry level understanding that gets you 90% of the way in, while only being 10 % wrong.

My impression is: when it comes to understanding pitch correction algorithms it's a deep dive from the get go.

Answer 1

@hotpaw2 give you nice tricks about how to do it in time domain and how try avoid glitches...

The basic trick step is find where cut to crossfade and splice the two parts again, you can search in waveform where grains have best match, we usually apply some kind of autocorrelation algorithm to find periodicity in waveform and cut ... now you can crossfade to try kill all glitches!

we can change the pitch or time individually using frequency or time domain aproach, when you talk:

It must be a process that is fundamentally different from how pitch is changed when manipulating an analog recording

Its remember me about manipulation of an analog recording is when you put your fingertip on a vinyl and spin it at a different speed, its will give to you tempo and pitch change at same time, it is easly reproduced in digital audios just playng it in a different samplerate, one pitch correction not works in this way, to keep the tempo intact one way is change the audio tempo (stretch or shrink) its will not afect the pitch, to do it copy grains to strech or cut grains to shrink your audio (choose where slice your audio using autocorrelation and applying crossfade to do it glitch free), now resample/interpolate with an inverse factor used to stretch or shirnk, this process give to you an pitch shift audio without tempo change ...

To know what factor to use to strech/shrink I usually create an table frequency from all notes, an pitch track run parallel to get the current pitch and compare with table note, so now you know the pitch correction factor to use in each grain!

The big problem of this method is that you will lose all formants :-(

To do pitch correction, after 32 years, I still love the keith Lent method, why ??? this algorithm keep the formants intact :-), one problem here is that you need know the pitch and not just the best autocorrelation match:

I try and build an automatic Pitch Correction after read Lent and RBJ paper, my simple matlab tests here:

https://www.youtube.com/watch?v=6ns5K1FHtd4

https://www.youtube.com/watch?v=MYxQVTwHK_o

Answer 2

Okay, so I'll start with the simplest time-domain pitch shifting (no FFT) that is often used in live guitar effects.

What is going on are two parallel processes. One is a pitch detector in which audio goes in and out comes a computed period length, and from that the fundamental frequency and pitch (log frequency) can be computed.

The other process is a moving-tap delay line having variable fractional-sample delay. Ideally, this would be some windowed-sinc interpolation, but, if the signal is already over-sampled (that is the bandwidth is much lower than Nyquist), perhaps linear interpolation is good enough.

input_index++;
input_index &= index_mask;       // modulo-2 arithmetic
delay_buffer[input_index] = input_sample;
...
delay += delay_increment;
integer_delay = (int)delay;
fractional_delay = delay - (float)integer_delay;
...
output_index = input_index - integer_delay;
output_index &= index_mask;
output_sample = delay_buffer[output_index--];
output_index &= index_mask;
output_sample += fractional_delay*(delay_buffer[output_index]-output_sample);   // this is linear interpolation

Now the input pointer is always an integer, but the output pointer has an integer and fractional value.

Now, when upshifting, you're moving the output pointer faster than the input pointer, so the delay must be decreasing. When downshifting, the delay is increasing. This delay_increment is

delay_increment $= 1 - 2^{cents/1200} $

Where $cents$ is the pitch shifting amount in units of 1/1200 of an octave.

Now here is where it gets tricky. If the output pointer is moving at a different rate than the input pointer, eventually you will wrap around in your buffer and you get a nasty click. So you have to start fading out the audio at the current output pointer and start fading in the audio at a new output pointer that is farther back from the edge.

For a glitch-free splice, the difference between the current output delay (being faded out) and the new output delay (being faded in) should be one period of the audio and that information must come from the pitch detector. There are some necessary nasty details in lining up the pitch detector with the mean delay of the output pointers.

The code to do the cross-fading (or splicing) is a little more involved, but I might write you some right here if you give me some time.

Answer 3

It can be (but not necessarily is) just a process of cut and splice.

There are (at least) two main classes of time-pitch modification, time domain, and frequency domain analysis/resynthesis. The one you mention is a time domain approach, where entire periods of pitched audio are (almost) repeated or removed as needed to expand or contract the time. Then the waveform can be fractionally resampled to adjust the frequency. This works better for monophonic sound or tracks.

Beyond just slicing, some artifacts can be removed (or not introduced) by several means. The timing at the slice points can be sub-sample interpolated to minimize phase glitches. The amplitude around the splice points can be adjusted by ramping or cross-fading to minimize envelope changes or glitches. Filtering can be applied to intermediate results to remove any new frequency artifacts, or to shape the averaged spectral envelope closer to that of the original. Convolution can be applied in an attempt to reshape the room (reverb) back to the original room size. And etc.