# Matching guitar signal with tablature, near real-time algorithms?

I'm looking for candidate algorithms that might work for a use-case comparable to Rocksmith's analysis system. To compare a clean electric guitar signal with a tablature reference.

A higher latency is acceptable, as well as compromises in accuracy, but the goal is to have near real-time information on whether what you actually played, is within a threshold of what you were expected to play.

The output would be a probability value per expected strum (or pluck or tap). Which is then matched against a configurable threshold.

My current line of thinking is to take multiple algorithms and predetermine which algorithm should be used per segment of the song.

• One for chords (compare lowest note & 12-bin PCPs?)
• One for sustained / single notes (FFT cross-correlation?)
• One for fast or polyphonic riffs (Supervised training? Classification models? Don't know.)
• One for bends... etc.

My two question are:

1. Is a (context-sensitive) combination of sub-algorithms required for such a use-case?
2. Which algorithms would make good candidates for the individual tasks, given this use-case?

Bonus question: do any "master" algorithms already exist to do such switching / tweaking of sub-algorithms?

Extra details,

What is considered acceptable processing time depends on the reference. For example if you're instructed to play a sustained single note for 3 bars, it's perfectly acceptable to do a slow analysis of the plucking envelope.

However if you're instructed to play a solo, alternating picking and hammering, 1/16th notes and bends. You're working with shorter samples, more human error and residual noises while playing. But we still need to attribute a binary decision if it was "close enough" for each note within, say ~200ms after each note was supposed to be released.

The guitar signal comes directly from a dedicated USB guitar audio interface, making it single instrument and a good quality signal.

The tablature is highly detailed. Assume a modern guitar pro format. Making relatively accurate training samples possible, but they will not contain the characteristics of the exact guitar, pickups, noise, etc. that the guitar signal would have.

• so you want to do pitch tracking and match to a template, perhaps a MIDI file? the polyphonic stuff is too hard for me. ask Peter Neubacker. fitting a pitch contour of a monotonic note to a MIDI file is sorta like fitting a line to data. there's a least-squares way to define the degree of matching to the template. – robert bristow-johnson Dec 23 '16 at 22:59
• How would you distinguish if the user used picking or legato techniques to play a given line? – Matt L. Dec 25 '16 at 10:55
• @MattL. While I don't have an answer, I think Rocksmith has opted not to implement a distinction there. And I would expect that's the best route for my case too, as a "compromise in accuracy". Especially considering that we don't have any characteristics of the specific instrument or player to match against, only the tablature file. Perhaps taking a look at envelope, but not sure how that would work in polyphonic signals. – Beanow Dec 25 '16 at 11:36
• I haven't tried Rocksmith, I just know that a student of mine happily used it for a while. It would be interesting to see how accurately it works, even though I don't believe that it can really judge playing quality. I once saw a similar thing for checking if a singer sings in tune, and a really bad singer who had worked with the system for some time scored much better than a professional singer who didn't know the system. No doubt who was really the better singer ... – Matt L. Dec 25 '16 at 12:54
• Yes the evaluation is limited, so for teaching I would say it can be part of the formula but it's not enough on it's own once beyond beginner level. Also, some of the modding community created exercises in the song format. But all this is quite off-topic for the question. I would point you to customsforge.com as a starting point for this. – Beanow Dec 25 '16 at 13:46

• the "FIR" method of making a comb filter is specifically "using delays and adding the signal on top of itself". that's how you do an FIR comb. so the difference between a comb filter and a set of notch filters is that the comb filter has the notches equally spaced in frequency. in a couple of answers to this question i tried to start to lay out a general "theory" of comb filters (first you need a precision delay line, then you can replace the $z^{-1}$ of digital filters with $e^{-sT}$ where $T$ is the delay. – robert bristow-johnson Dec 25 '16 at 3:16