What mathematical model of audio can make possible pitch shifting of individual notes in (timbrally complex) polyphony?

My question is: What mathematical model of polyphonic sound can make possible the changing (i.e. pitch shifting) of individual musical notes in a multi-voice-in-single-channel audio recording of a polyphonic acoustic musical instrument? By 'changing notes in polyphonic audio', I mean doing something like editing sound with the so-called 'Direct Note Access' feature in celeony's Melodyne software.

According to wikipedia, what Melodyne uses to model audio signal of a single line melody played on an acoustic (and thus timbrally complex) musical instrument is something like what Henning Thielemann describes in his paper entitled 'Untangling phase and time in monophonic sounds' However, I cannot find any reference to models of audio signals of polyphonic musical instruments; according to an interview on Youtube of Peter Neubacker (transcribed below), Melodyne's feature for dealing with editing polyphonic audio requires an approach unlike the one described by Thielemann.

One clue from another youtube clip is that Neubacker's model works better with audio record of one KIND of instrument only (i.e. only piano, only guitar, only strings, only winds, etc). Another clues is yet another clip showing the ability to not only shift pitch of a note but also the (starting and ending) timing of it.

Below is transcript of youtube video which mentioned that 'polyphonic materials calls for a different approach' (in case you have not time to watch it from from 22:00).

• The question, from which Melodyne arose: how can I obtain a sound from a 3 dimensional form like this [gesturing with stone in hand]? By which means, the sound can then be freed from its dependence upon continuous time? This sculpture is actually what emerged from this... It's a piece of plastic.... This was derived directly from musical data. This object is [plucking a note on lute] this note. It is best visualized as this, from left to right. Time runs in this direction [gesturing left to right]. And that is the amplitude [gesturing big and small with finger opposing thumb]. If I turn it, it ... represents the timbre of this sound at any given instance. You can see very clearly here a structure [pointing to cross section at bottom of sculpture] that is somewhat triangular; that is because in this sound, the third overtune (the fifth) is particularly loud, and that introduces an element of threeness.

Since Melodyne didn't exist yet and I was simply experimenting with the translation of the sound into this shape, I worked for almost a year with this one sound. ... I know this sound inside and out and by heart. This also provides a good illustration of local sound. I can, not only play back the sound [clicking mouse], but I can also enter the sound of any point, and moved through it as slowly or quickly as I like. I can even linger in the sound, or move forwards and backwards, so if I examine one place here ... go around it. ... Ten years ago it was new.

Recently dna (direct note access) was added. With it, I can also edit polyphonic music. In other words, I can edit individually notes that sounds simultaneously, such as for example a guitar recording. If I now play a small chord [selecting Poly -> Separate Notes on screen], we see here the 3 notes I've just played as separate entities. Let's just listen again [computer plays minor chord]. And now, as if by moving my finger to a higher fret, I can raise this one note [dragging a note on screen up; computer plays major chord]. For the divided-up audio, I can isolated this one note, and can move it up or down at will now, to any pitch I please.

Why it was no one able previously to isolate individual tones within complex material in this way? I honestly don't know. In science, the natural tendency is to begin with something simple, a sine wave for example, or individual notes, and analyze that first, only to discover when the material becomes more complex, or has to be treated in its entirety, that the system doesn't work. My approach is different. I actually begin with complex signals, and it's only when I want to examine something in detail that I go back to simpler ones, but first, i have to have an overall impression of what is actually happening in reality.

Does the secret perhaps lies in this roll? Heheh, this is actually a loo roll. The question originally raised by the stone was how can I translate a given sound into a three-dimensional form. Here, I have arranged the individuals sampling values of the sound, indicated here by one two three and so, in a spiral. And it turns out, that if you interpolate between the points [gesturing across the spiral], a landscape emerges that also represents the individual cross-sections in the sound [gesturing cross sections of sculpture].

How old is the roll? 12 years. So that idea is the well-spring of Melodyne, of all that we've seen today ... ? Yes, but this manner of coiling up the the sound would no longer be of use for polyphonic materials, which calls for a different approach.

• No time now, but you might want to read some of Bill Sethares' work on Consonance. I'll try to digest your post and answer more fully over the next few days.
– Peter K.
Oct 5, 2011 at 12:41
• I'm not sure what the question is. Isolating individual notes and "coiling up the sound" makes me think of wrapping a spectrum in a spiral so that the harmonics of a note line up with each other: nastechservices.com/Spectrograms.html nastechservices.com/Spectratune.html Oct 5, 2011 at 15:39

TL;DR? Google Scholar for harmonic partial separation.

A good starting point would be sinusoidal modeling techniques that separate the signal into sines+noise (deterministic and stochastic) components. The deterministic component, made up of sines, can be resynthesized convincingly:

http://mtg.upf.edu/files/projectsweb/sms-piano-original.wav

http://mtg.upf.edu/files/projectsweb/sms-piano-deterministic.wav

The sines are subtracted from the signal and the noisy/stochastic portion remains.

http://mtg.upf.edu/files/projectsweb/sms-piano-stochastic.wav

The stochastic portion is synthesized by putting noise through some noise-shaping filter. Some others have extended this to a sines+noise+transients model which helps preserve transient stochastic features in time stretching.

https://ccrma.stanford.edu/~jos/sasp/Sines_Noise_Modeling.html

http://mtg.upf.edu/technologies/sms

Once you have the sinusoidal parameters of a signal, it is possible to separate the sines of overlapping notes by looking for harmonic ratios and grouping by onset, etc. Partial tracking turns up a lot of results on Google Scholar.

http://recherche.ircam.fr/equipes/analyse-synthese/lagrange/research/papers/lagrangeIcassp05.pdf

http://dream.cs.bath.ac.uk/software/sndan/mqan.html

Hidden Markov Models, polynomials, and Macaulay-Quatieri are some of the methods. I am stumped about separating the stochastic leftovers into two notes. I don't know how Melodyne addresses this.

The approach used in melodyne requires 2 separate frequency domain operations. Firstly, polyphonic transcription techniques are used to group frequency components (from a standard frequency transform) of polyphonic audio into note activations. In other words, group harmonic subsets according to most likely note activations. See my response to "Inverse polyphonic chord recognition" post on this forum for references and mathematical models.

The second operation is that of frequency domain pitch shifting of the harmonic subsets extracted above. I'm not certain but I would almost guarantee that Melodyne uses a phase vocoder approach to achieve this. You can also carry out time stretching using this technique. We use techniques similar to these in Riffstation and they work quite well.

One possibility might be analysis/re-synthesis using a statistical pattern matching approach. If you know or can reasonably guess the mix of instruments involved and have templates (including initial transients, spectrum plus spectral evolution, etc.) for the instrument sounds for all expected notes, you could try a statistical matching of a large number of sane chord combinations using the template sound patterns to estimate the most likely polyphonic combination(s). This would very likely be a very computationally intensive search for global minima's, where various "AI"-like search techniques might be useful. You could then take the various individual chord probabilities and then use decision theories to pick the most likely polyphonic sequences in time.

Then take the estimated notes and re-synthesize them at your chosen key pitch and duration.