My question is: What mathematical model of polyphonic sound can make possible the changing (i.e. pitch shifting) of individual musical notes in a multi-voice-in-single-channel audio recording of a polyphonic acoustic musical instrument? By 'changing notes in polyphonic audio', I mean doing something like editing sound with the so-called 'Direct Note Access' feature in celeony's Melodyne software.
According to wikipedia, what Melodyne uses to model audio signal of a single line melody played on an acoustic (and thus timbrally complex) musical instrument is something like what Henning Thielemann describes in his paper entitled 'Untangling phase and time in monophonic sounds' However, I cannot find any reference to models of audio signals of polyphonic musical instruments; according to an interview on Youtube of Peter Neubacker (transcribed below), Melodyne's feature for dealing with editing polyphonic audio requires an approach unlike the one described by Thielemann.
One clue from another youtube clip is that Neubacker's model works better with audio record of one KIND of instrument only (i.e. only piano, only guitar, only strings, only winds, etc). Another clues is yet another clip showing the ability to not only shift pitch of a note but also the (starting and ending) timing of it.
Below is transcript of youtube video which mentioned that 'polyphonic materials calls for a different approach' (in case you have not time to watch it from from 22:00).
The question, from which Melodyne arose: how can I obtain a sound from a 3 dimensional form like this [gesturing with stone in hand]? By which means, the sound can then be freed from its dependence upon continuous time? This sculpture is actually what emerged from this... It's a piece of plastic.... This was derived directly from musical data. This object is [plucking a note on lute] this note. It is best visualized as this, from left to right. Time runs in this direction [gesturing left to right]. And that is the amplitude [gesturing big and small with finger opposing thumb]. If I turn it, it ... represents the timbre of this sound at any given instance. You can see very clearly here a structure [pointing to cross section at bottom of sculpture] that is somewhat triangular; that is because in this sound, the third overtune (the fifth) is particularly loud, and that introduces an element of threeness.
Since Melodyne didn't exist yet and I was simply experimenting with the translation of the sound into this shape, I worked for almost a year with this one sound. ... I know this sound inside and out and by heart. This also provides a good illustration of local sound. I can, not only play back the sound [clicking mouse], but I can also enter the sound of any point, and moved through it as slowly or quickly as I like. I can even linger in the sound, or move forwards and backwards, so if I examine one place here ... go around it. ... Ten years ago it was new.
Recently dna (direct note access) was added. With it, I can also edit polyphonic music. In other words, I can edit individually notes that sounds simultaneously, such as for example a guitar recording. If I now play a small chord [selecting Poly -> Separate Notes on screen], we see here the 3 notes I've just played as separate entities. Let's just listen again [computer plays minor chord]. And now, as if by moving my finger to a higher fret, I can raise this one note [dragging a note on screen up; computer plays major chord]. For the divided-up audio, I can isolated this one note, and can move it up or down at will now, to any pitch I please.
Why it was no one able previously to isolate individual tones within complex material in this way? I honestly don't know. In science, the natural tendency is to begin with something simple, a sine wave for example, or individual notes, and analyze that first, only to discover when the material becomes more complex, or has to be treated in its entirety, that the system doesn't work. My approach is different. I actually begin with complex signals, and it's only when I want to examine something in detail that I go back to simpler ones, but first, i have to have an overall impression of what is actually happening in reality.
Does the secret perhaps lies in this roll? Heheh, this is actually a loo roll. The question originally raised by the stone was how can I translate a given sound into a three-dimensional form. Here, I have arranged the individuals sampling values of the sound, indicated here by one two three and so, in a spiral. And it turns out, that if you interpolate between the points [gesturing across the spiral], a landscape emerges that also represents the individual cross-sections in the sound [gesturing cross sections of sculpture].
How old is the roll? 12 years. So that idea is the well-spring of Melodyne, of all that we've seen today ... ? Yes, but this manner of coiling up the the sound would no longer be of use for polyphonic materials, which calls for a different approach.
