Audio fingerprint questions

Question

I am developing an application that requires audio fingerprints. I have been reading a lot of articles and PDFs, now i think i have gotten myself confused. Based on my present understanding, I have some questions

1. After decoding the audio to its raw format, resampling and extracting the mono channel. Do i convert the bytes to integers or floats?

2. What is convolution and why is it necessary to convolve the audio samples

3. What is a window and its length

4. FFT transforms a signal from time domain to frequency domain. Am i correct? If yes, does the frequency components of an audio determine the contents/sound/noise/volume?

5. What operation can be carried out on two audios of same content but different bitrate to normalize the bitrate and get the data that they share

6. What are low pass and high pass filters and how are the derived. What are their uses in relation to audio fingerprint operations

7. In the text Computer vision for music identification. After convolving the signals with a low pass filter and extracting every 8th sample. Then, a short term fourier transform with a window size of 2048 samples with successive windows offset by 64 samples. Also divide the power between 300Hz and 2000Hz in 33 logarithmically spaced bands.

   • Will these operations be applied to each of the samples retrieved after the convolution process? Can i get a simpler explanation?

8. 32 learned filters and thresholds are applied to get a 32-bit descriptor for every time step (11.6 ms) of signal. This series of signals are known as the signature.

   • what are these learned filters and thresholds?

Wow :), that's a lot of questions. I am sure a lot of beginners will find the answers helpful.

Thanks

Answer 1

1. There are rare situations when processing is done with integers (embedded systems with no floating point units), but usually, audio signal processing is done with floats.

2. (Handwaving) Convolution is the operation through which a filter is applied to a signal. If the authors of the paper you're looking at found that applying a specific filter to the data (for example to remove noise, any information not processed by the human auditory system, etc.) yielded good results, then you have to trust them on this!

3. Since audio signals are not stationary, they are processed in short blocks, referred to as "windows" or "frames". The analysis/processing is thus done on short overlapping segments. For example you will process segment 0.00s:0.02s ; then segment 0.01s:0.03s ; then 0.02s:0.04s etc. In this example, the window size is 20ms and the overlap 50%. There is no "universal" window size for audio applications, look at what is used in the paper describing the technique you're implementing.

4. Not really, but going from the time domain to frequency domain is "one step closer" to what is ultimately happening in the human auditory system.

5. No operation could do that. If there was one, we would send all our files with the lowest quality and apply this magical operation to recover the highest quality. What this means is that the features used for fingerprinting must be robust enough to capture only the overall "shape" of the sound rather than focus on details - which would be lost by an increased amount of MP3 encoding.

6. Look at a signal processing textbook. These are very basic signal processing operations and can be used for many things inside or outside the context of a fingerprinting system - reducing sample rate, smoothing temporal sequences, emphasizing details, removing noise...

7. Let's say you have 800000 samples. You apply the convolution, you get 800000 samples out of it. Now you extract every 8th sample, you get 100000 samples. Then you apply your FFT on samples 0..2047 to get your 33 coefficients ; then on samples 64..2111 etc... so in the end you have a list of roughly 100000 / 64 = 1562 vectors of 33 coefficients.

8. (Handwaving). I am assuming you are referring to the Ke et al system (later refined in the works of Baluja et al), which is itself inspired by the Viola-Jones face detector. Once you have converted the signal into a sequence of vectors of 33 coefficients, you compute weighted sums of the coefficients in adjacent bins or along the time axis (a filter), and you compare the result to a threshold. This gives you a binary "feature". There are 32 such binary features, each of them computed with a different filter and a different threshold. They are "learned", because they were obtained the following ways: the authors built a database with millions of pairs of songs, some of them identical (but maybe at difference bitrate, volume, noise level etc.), some of them different. Using machine learning techniques (boosting on decision stumps), they evaluated which filters were the best at binning together identical content and separating unrelated content, and which threshold were being used for that. You can find the list of those filters/thresholds in the source code of an open-source implementation of lastfm's Ke-style audio fingerprinter.