I'm looking for candidate algorithms that might work for a use-case comparable to Rocksmith's analysis system. To compare a clean electric guitar signal with a tablature reference.
A higher latency is acceptable, as well as compromises in accuracy, but the goal is to have near real-time information on whether what you actually played, is within a threshold of what you were expected to play.
The output would be a probability value per expected strum (or pluck or tap). Which is then matched against a configurable threshold.
My current line of thinking is to take multiple algorithms and predetermine which algorithm should be used per segment of the song.
- One for chords (compare lowest note & 12-bin PCPs?)
- One for sustained / single notes (FFT cross-correlation?)
- One for fast or polyphonic riffs (Supervised training? Classification models? Don't know.)
- One for bends... etc.
My two question are:
- Is a (context-sensitive) combination of sub-algorithms required for such a use-case?
- Which algorithms would make good candidates for the individual tasks, given this use-case?
do any "master" algorithms already exist to do such switching / tweaking of sub-algorithms?
What is considered acceptable processing time depends on the reference. For example if you're instructed to play a sustained single note for 3 bars, it's perfectly acceptable to do a slow analysis of the plucking envelope.
However if you're instructed to play a solo, alternating picking and hammering, 1/16th notes and bends. You're working with shorter samples, more human error and residual noises while playing. But we still need to attribute a binary decision if it was "close enough" for each note within, say ~200ms after each note was supposed to be released.
The guitar signal comes directly from a dedicated USB guitar audio interface, making it single instrument and a good quality signal.
The tablature is highly detailed. Assume a modern guitar pro format. Making relatively accurate training samples possible, but they will not contain the characteristics of the exact guitar, pickups, noise, etc. that the guitar signal would have.