pyo: Retrieve amplitudes of audio signal frequenciy ranges

Question

I'm playing around with audio signal processing for the first time and just to start somewhere, I decided to use the Python library pyo. My goal is to analyze an audio input signal from the microphone to detect different types of strikes on a drum and to emit an event (or call a callback, whatever) in that case. My first step is to define some frequency ranges, I want to observe and then to have a function called periodically and pass in the peak amplitudes within those defined frequency ranges.

I found that there is something like FFT/DFT and my naive idea is, that I can setup an FFT object with my input signal and create an instance of the Pattern class, that calls a function periodically. In that function I can then call the FFT object in a way, so that it will return me a list with 1024 items. The "whole" frequency spectrum is sliced into 1024 slices and each item in the list contains an amplitude value for that specific frequency range slice at the given time.

I tried the following code and now I don't know how to interpretate the values provided by the FFT.get() method. My question is: Are my expectations wrong or how do I use the FFT class to get my list?

from pyo import *

server = Server(nchnls=1, buffersize=128, duplex=1, winhost="asio").boot().start()


class FrequencyRangeAmplitudeProvider:

    __frequency_ranges = ()
    __listener = lambda x: x

    __fft = None
    __pattern = None

    def __init__(self, input, frequency_ranges, listener):
        self.__frequency_ranges = frequency_ranges
        self.__listener = listener

        self.__fft = FFT(input, size=1024, overlaps=4, wintype=0)
        self.__pattern = Pattern(function=self.__extract_and_emit, time=0.06).play()

    def __extract_and_emit(self):

        #print(self.__fft.get('real', True))
        #print(self.__fft.get('imag', True))
        #print(self.__fft.get('bin', True))

        my_list = []  # <-- How to get the data, I'd need?

        peak_amplitudes = self.__extract_peak_amplitude_in_frequency_ranges(my_list, self.__frequency_ranges)
        self.__listener(peak_amplitudes)

    @staticmethod
    def __extract_peak_amplitude_in_frequency_ranges(amplitudes, frequency_ranges):
        return (('A', -2.34), ('B', -.65))


frequency_ranges_to_observe = (('A', 430, 450), ('B', 870, 890))


def do_something_with_peak_amlitudes_in_frequency_ranges(amplitudes):
    print(amplitudes)


provider = FrequencyRangeAmplitudeProvider(
    Input(),
    frequency_ranges_to_observe,
    do_something_with_peak_amlitudes_in_frequency_ranges)

server.gui(locals())
```

Answer 1

pyo is not really the issue here at all. The question still stands if you were to substitute it with a million other packages of similar functionality.

What is at the centre of this question is "How to build a classifier to distinguish between different types of hits on a drum". Once you have that sorted, you can then implement it with whatever package you like, including pyo.

One of the simplest classifiers you can use for this task is the Nearest Neighbour Classifier.

The end effect of this classifier is not too different than your current thinking: I will split the frequencies of the sound and then try to work with if-then-else constructs, the final outcome of which would be the classification. Every if-then-else, puts a binary decision boundary on the outcome. For example, if you monitor the audio volume then you can create a rule that tells you whether there is something playing or not (if the volume is above a threshold, then something is playing). The nearest neighbour classifier over spectral features (of the FFT), does exactly the same thing but puts $N_{DFT}$ if-then-else on a given frame (it is not exactly if-then-else, but the end-effect is the same).

To get it to work, you would need a few representative recordings of each drum shot. For example, 20 different rimshots, 20 different straight hits. If you pass each one through the Discrete Fourier Transform (DFT) (of which the FFT is a quick implementation of) this will give you $c_{k,i,N_{DFT}}$ coefficients. That would be one set of $N_{DFT}$ coefficients for the $i^{th}$ recording of the $k^{th}$ type of drum hit.

Find the centroids (means of each DFT coefficient) that correspond to each drum hit and this will give you $k$ points in feature space that represent what each drum hit tends to sound like (obviously, here $k=2$).

To figure out what sort of drum hit has just been recorded (a drum hit you have not encountered during the "training" phase), take the waveform of it, pass it through DFT, take its DFT coefficients and work out which centroid they are closer to.

Doing this in real time might be a bit more challenging as you would have to establish an audio "loop" that continuously records frames from the audio interface, transforms them and passes them over to the classifier to make a decision.

Eventually, you will probably also need a way to describe a "third class" for sounds that are not drum hits and in which case you would like your classifier to remain "silent".

Overall, the system will also require some tuning to bring it as close as possible to a real time identification of drum hits, presumably, from a clean audio source, but it would be a start.

Hope this helps.