# Audio Signal Separation - Identifying Interest points

I asked this question here: Audio Separation of .wav signal but it wasn't clear, so, here is my second attempt:

First off, assume that I have a .wav file containing a sentence as follows:

"My name is Michael" I would like to extract, from this, the following:

"My" -> Phoneme (1)

"Name" -> Phoneme (2)

"Is" -> Phoneme (3)

"Michael" -> Phoneme (4)

This means that I have taken my 1D signal, and split it into a 2D signal (vector) that contains these particular words/phonemes which I can then analyse and identify. I would therefore, like to compute this in the time domain and not the frequency domain. Just to clarify again:

I take in a 1D signal containing a sentence, split this sentence up into different parts which contain this data: vect[0], vect[1], .... vect[4] Let's say in matlab I did the following command wavwrite(vect[0], ....) then it would output the word "My" and putting all the blocks together would give me the full sentence back.

Here is my "real-world problem" instead of Phonemes, I have bat calls, the length of each bat call is unknown at this stage. But here is a typical sample of a bat call: Here and for each of these bat calls, these need to be separated from the inputted signal and stored inside a vector (Just like the example above), this, then allows me to identify each of the bats and perform analysis on them.

This, like, the sample would give me a 2D vector containing each of the bat calls: "Bat1", "Bat2" ..... "Bat[n]" it is unknown the amount of time the bats have been recorded for, or, what the length of each of the bat call therefore is.

What I have done so far:

I have obtained the bat signal, processed it and I am given the following (Which is plotted):

I have also Emphasised the signal using the following formula:

rawSignal[i] = rawSignal[i] - (0.95 * rawSignal[i-1]);

And then I have Compressed the signal using the following:

    float param = 1.0;
for(unsigned i=0; (i < rawSignal.size()); i++)
{
int sign = getSignOf(rawSignal[i]);
float norm = normalise_abs_value(rawSignal[i]);
norm = 1.0 - pow(1.0 - norm, param);
rawSignal[i] = denormalize_value(norm, sign);
}


Which then gives me an output of the following:

I'm unclear to where I should go from here in identifying single elements ("calls") from this signal. Since, if I use zero-crossing and/or calculating the total energy of the signal and thus using a threshold then it will just remove the noise and I'm left with a compressed version of the signal.

Speaking to someone, they suggested that I should try and use the Cochleagram domain however, I'm not familiar with this and there is very little research on this available.

If anyone has any suggestions, or the algorithms that I could use then please suggest them.

• So therefore, would Blind Source Separation (BSS) work here? – Phorce Sep 17 '13 at 19:08
• Blind source separation is unrelated. That's for taking recordings of multiple simultaneous sources and unmixing them, like people talking over each other at a party. – endolith Sep 18 '13 at 18:59
• @endolith Yeah, I did some research on this and found that to be the best method of BSS. I'm implementing a un-voiced/voiced analysis on the signal to determine and separate the signal.. Using energy and thresholding.. Think this is the best method? – Phorce Sep 18 '13 at 19:44
• I'm confused. I thought you were trying to isolate individual bat calls in a recording with multiple bat calls at different times. So I said that BSS does not work here, because BSS is for separating things that happen at the same time and get mixed together, usually involving multiple microphones to be able to pull them apart again. Sounds like cochleagram is a time-frequency represention. Did you try STFT? – endolith Sep 18 '13 at 20:37
• I would have thought that bat sounds will be very similar to the BSS cocktail-party problem: the bats are not necessarily polite enough to NOT talk over each other... so there are multiple bat chirps at the same time and, potentially, at the same frequency. BSS may very well be your best bet. – Peter K. Sep 18 '13 at 21:40

## 2 Answers

(a follow-up to my suggestion on the previous question), you can use the spectrogram and ICA to help:

A similar shorter sound file:

import wave, struct, numpy as np, matplotlib.mlab as mlab, pylab as pl
def wavToArr(wavefile):
w = wave.open(wavefile,"rb")
p = w.getparams()
s = w.readframes(p[3])
w.close()
sd = np.fromstring(s, np.int16)
return sd,p

def wavToSpec(wavefile,log=False,norm=False):
wavArr,wavParams = wavToArr(wavefile)
print wavParams
return  mlab.specgram(wavArr, NFFT=256,Fs=wavParams[2],detrend=detrend_mean,window=window_hanning,noverlap=128,sides='onesided',scale_by_freq=True)

wavArr,wavParams = wavToArr("bat_speech.wav")
hf = pl.figure(); ax=hf.add_subplot(1,1,1)
ax.plot(wavArr)


Now take a look at the spectrogram:

Pxx, freqs, bins = wavToSpec("bat_speech.wav")
Pxx += 0.0001
freqs += (len(wavArr) / wavParams[2]) / 2.
hf=pl.figure(figsize=(12,12));
ax = hf.add_subplot(2,1,1);
#plot spectrogram as decibals
hm = ax.imshow(10*np.log10(Pxx),interpolation='nearest',origin='lower',aspect='auto')
hf.colorbar(hm)
ylcnt = len(ax.get_yticklabels())
ycnt = len(freqs)
ylstep = int(ycnt / ylcnt)
ax.set_yticklabels([ int(freqs[f]) for f in xrange(0,ycnt,ylstep) ])


We can clip this at 8000Hz or so it looks like, or don't bother cleaning it up.

Now you have frequencies which can proxy for multiple sources in BSS, so you can play with PCA, ICA, normalization, etc. For example, see if you have some components you can isolate:

from sklearn.decomposition import PCA, FastICA
ncomps = 7
# reduce dimensionality with PCA
pca = PCA(n_components=ncomps)
y = Pxx.copy().T
pc = pca.fit(y).transform(y)
# run ICA
ica = FastICA(n_components=ncomps,random_state=42)
z = ica.fit(pc).transform(pc).T
hf = pl.figure()
for p in xrange(ncomps):
ax = hf.add_subplot(ncomps,1,p+1)
ax.plot(z[p])


or see if the spectrogram is enough to let you do your segmentation:

hf = pl.figure()
ax = hf.add_subplot(1,1,1)
ax.plot(np.sum(Pxx,axis=0))


EDIT: Just realized you said you didn't want to use the frequency domain, but it may help you isolate your phonemes which you can extract. Anyway, running on your SeroWeb.wav I get this:

• Thank you for your answer. Let's assume I have carried out the ICA of the signal. I've come up to the point in your last figure; can I therefore use a PeakFinding algorithm to determine the number of phonemes within the spectrum? – Phorce Jan 4 '14 at 16:14

This seems like an excellent opportunity to apply ideas from feature learning. You have a signal that's assumed to be a sum of a small set of distinct calls, each with a different temporal offset, and you'd like to learn from the signal alone what the calls look like ?

In math, if your signal is $x$, then the assumption is that $x$ can be expressed as $$x(t) = \sum_{\tau, \alpha, i\, \in\, Z} \alpha\phi_i(t - \tau)$$ where $\Phi = \{\phi_1, \dots, \phi_k\}$ is a "codebook" or "dictionary" of "basis signals" and $Z$ is an encoding that specifies which basis signal $\phi_i$ is applied at time offset $\tau$ with magnitude $\alpha$.

If this describes your situation, then you might find an excellent paper by Smith & Lewicki [1] to be of interest. Basically, the approach they take is to apply the Matching Pursuit [MP; 2] sparse coding algorithm to an effectively extremely long audio recording of "natural sounds" (as opposed to sinusoids or triangle waves or the like) using convolution to do the encoding. In general, MP can be used to compute $Z$ given $x$ and $\Phi$, and Smith & Lewicki extend this coding algorithm with a learning approach that uses gradient descent to compute $\Phi$ as well. They show that the learned basis signals $\Phi$ closely match the auditory receptive fields from cat cochlear neurons, much like the celebrated sparse coding result from Olshausen & Field [3] in vision.

If you already know $\Phi$ (e.g., you already have recordings of the bat calls you know are in $x$) then you could just use the MP variant that Smith & Lewicki use to derive $Z$.

[1] Smith, E & Lewicki, M. "Efficient auditory coding." Nature, 2006.

[2] Mallat, S & Zhang, Z. "Matching pursuits with time-frequency dictionaries." IEEE Transactions on Signal Processing, 1993.

[3] Olshausen, B & Field, D. "Emergence of simple-cell receptive field properties by learning a sparse code for natural images." Nature, 1994.

• @imjohns3 Thank you for your reply. This is really appreciated. I'll take a look at the reading materials that you suggested. At the moment, I'm just interested in identifying the bat calls from the signal.. My main concern is "splitting" the signal into segments for processing, since, with speech I used 256x100 and I don't believe that the same can be applied for natural sounds. Any ideas? :) – Phorce Sep 19 '13 at 0:58
• Given that the calls in your signal are emitted at different times, it's unlikely you'll be able to split your signal into chunks of equal size and retain the right information. If you just need to identify the start of each call you could even just do some sort of threshold on the power in the signal. Or, if you need to be more specific, you could identify the formant frequencies of the calls you're interested in, and then threshold your signal using the power in those formant bands. Hope that makes sense. – lmjohns3 Sep 19 '13 at 3:49