0
$\begingroup$

I'm looking for candidate algorithms that might work for a use-case comparable to Rocksmith's analysis system. To compare a clean electric guitar signal with a tablature reference.

A higher latency is acceptable, as well as compromises in accuracy, but the goal is to have near real-time information on whether what you actually played, is within a threshold of what you were expected to play.

The output would be a probability value per expected strum (or pluck or tap). Which is then matched against a configurable threshold.

My current line of thinking is to take multiple algorithms and predetermine which algorithm should be used per segment of the song.

  • One for chords (compare lowest note & 12-bin PCPs?)
  • One for sustained / single notes (FFT cross-correlation?)
  • One for fast or polyphonic riffs (Supervised training? Classification models? Don't know.)
  • One for bends... etc.

My two question are:

  1. Is a (context-sensitive) combination of sub-algorithms required for such a use-case?
  2. Which algorithms would make good candidates for the individual tasks, given this use-case?

Bonus question: do any "master" algorithms already exist to do such switching / tweaking of sub-algorithms?

Extra details,

What is considered acceptable processing time depends on the reference. For example if you're instructed to play a sustained single note for 3 bars, it's perfectly acceptable to do a slow analysis of the plucking envelope.

However if you're instructed to play a solo, alternating picking and hammering, 1/16th notes and bends. You're working with shorter samples, more human error and residual noises while playing. But we still need to attribute a binary decision if it was "close enough" for each note within, say ~200ms after each note was supposed to be released.

The guitar signal comes directly from a dedicated USB guitar audio interface, making it single instrument and a good quality signal.

The tablature is highly detailed. Assume a modern guitar pro format. Making relatively accurate training samples possible, but they will not contain the characteristics of the exact guitar, pickups, noise, etc. that the guitar signal would have.

Improve this question
$\endgroup$
5
  • $\begingroup$ so you want to do pitch tracking and match to a template, perhaps a MIDI file? the polyphonic stuff is too hard for me. ask Peter Neubacker. fitting a pitch contour of a monotonic note to a MIDI file is sorta like fitting a line to data. there's a least-squares way to define the degree of matching to the template. $\endgroup$
    – robert bristow-johnson
    Commented Dec 23, 2016 at 22:59
  • $\begingroup$ How would you distinguish if the user used picking or legato techniques to play a given line? $\endgroup$
    – Matt L.
    Commented Dec 25, 2016 at 10:55
  • $\begingroup$ @MattL. While I don't have an answer, I think Rocksmith has opted not to implement a distinction there. And I would expect that's the best route for my case too, as a "compromise in accuracy". Especially considering that we don't have any characteristics of the specific instrument or player to match against, only the tablature file. Perhaps taking a look at envelope, but not sure how that would work in polyphonic signals. $\endgroup$
    – Beanow
    Commented Dec 25, 2016 at 11:36
  • $\begingroup$ I haven't tried Rocksmith, I just know that a student of mine happily used it for a while. It would be interesting to see how accurately it works, even though I don't believe that it can really judge playing quality. I once saw a similar thing for checking if a singer sings in tune, and a really bad singer who had worked with the system for some time scored much better than a professional singer who didn't know the system. No doubt who was really the better singer ... $\endgroup$
    – Matt L.
    Commented Dec 25, 2016 at 12:54
  • $\begingroup$ Yes the evaluation is limited, so for teaching I would say it can be part of the formula but it's not enough on it's own once beyond beginner level. Also, some of the modding community created exercises in the song format. But all this is quite off-topic for the question. I would point you to customsforge.com as a starting point for this. $\endgroup$
    – Beanow
    Commented Dec 25, 2016 at 13:46

1 Answer 1

Reset to default
1
$\begingroup$

Assuming that the tablature is well defined; let's say it's a MIDI file complete with pitch bend commands. you can make a synthesizer where each note is actually a tracking comb filter tuned to the midi note (and offset by the pitch bend command) with notches at each integer harmonic (including the fundamental). if those tracking comb filters processes a sound file playing that MIDI file and the two are properly aligned in time, the tracking comb filters should null out most of the input audio, even polyphonic.

Then magnitude-square-LPF the result and a minimum value means a pretty good match.

Just an idea. No equations, but I have posted about pitch detection before. And I thought I did about tuned comb filters, too (you need a precision delay line to tune the comb filter to whatever fine-tuned pitch that is specified).

Improve this answer
$\endgroup$
7
  • $\begingroup$ So if I understand correctly, I would take each note, plus a number of expected overtones and work out the delay in samples for those frequencies. Then apply them sequentially with a FIR style comb filter? $\endgroup$
    – Beanow
    Commented Dec 24, 2016 at 10:04
  • $\begingroup$ no, if can be an IIR combfilter too. but either way your "synthesizer" controlled my the MIDI files is a comb fliter with notches at the note frequencies and their harmonics. so when the note passes through, it should be nulled out. $\endgroup$
    – robert bristow-johnson
    Commented Dec 24, 2016 at 18:09
  • $\begingroup$ Oh right, so not actually using delays and adding the signal on top of itself, but using a set of notch filters. I'll have a go at that. How different would this be from using an FFT and knocking out the relevant frequency bins there? $\endgroup$
    – Beanow
    Commented Dec 24, 2016 at 19:25
  • $\begingroup$ the "FIR" method of making a comb filter is specifically "using delays and adding the signal on top of itself". that's how you do an FIR comb. so the difference between a comb filter and a set of notch filters is that the comb filter has the notches equally spaced in frequency. in a couple of answers to this question i tried to start to lay out a general "theory" of comb filters (first you need a precision delay line, then you can replace the $z^{-1}$ of digital filters with $e^{-sT}$ where $T$ is the delay. $\endgroup$
    – robert bristow-johnson
    Commented Dec 25, 2016 at 3:16
  • $\begingroup$ about using the FFT for filtering (sometimes called "fast convolution", you need to learn about the overlap-add and overlap-save techniques. you are using the FFT to implement an FIR. so unless just "knocking out" a few bins corresponds to a given FIR, you can't just do that with the FFT and not suffer consequences with discontinuities between adjacent frames. $\endgroup$
    – robert bristow-johnson
    Commented Dec 25, 2016 at 3:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.