# Smoothing signal / detecting bumps in a data stream

(EDIT: This question follows from Extracting Binary Magnetic-Strip Card Data from raw WAV)

Here is my signal (top line) and a basic IIR filter applied (bottom line)

(EDIT: my task is to break the signal into binary 0 (frequency F) and binary 1 (frequency 2F) -- that's why it is called F2F. So I need to process it in such a way that guarantees no false peaks. While the screenshot makes it look trivial, there is a potential problem of getting a double peak, and also of getting false positives in the trough between real peaks.)

My question is, what methods are available for smoothing this signal? Is IIR my best bet?

I can see at least three possibilities:

• IIR y[n] = 0.9*y[n-1] + 0.1*x[n] where y[x] =0 when x < 0

• Moving / windowed average -- place a Bell curve with area 1.0 over the surrounding say w=10 samples each side and integrate bellSmooth(x) = integral[x-w,x+w] { bell(k).samp(k) }dk

• Determine the expected frequency and FFT / remove higher order bins / reverse FFT

I may have answered my own question, but probably this is incomplete and I'm sure I am using the wrong terminology. Also I can't really predict the pros and cons. The last method is less attractive as it requires knowledge of the basic signal frequency. But then so does the second method; I need to choose an appropriate window length.

Are there any other methods?

• What characteristics of the original signal are you trying to preserve/measure? For example, timing between peak points, height of peak, amount of time above a threshold, something else? – Martin Thompson Dec 13 '11 at 11:50
• Timing between peaks... and even this doesn't need to be too accurate -- it is an F2F signal (I will amend the question to reference the signal source and give a context) – P i Dec 14 '11 at 3:05
• Is this for online or offline processing? – user42 Dec 14 '11 at 15:41
• My thesis topic is noise smoothing in the data stream. Do you like this article or topic? – user16985 Aug 13 '15 at 6:41

Effects of Averaging

Using a moving average filter will smooth out the irregularities in the signal. The noise becomes E/N where N is the length of the moving average filter. The side effect of using a MA is that the signal peaks become wider and shallower.

In addition, the frequency content of the signal will change. A moving average filter in the time domain is the same thing as convolving the frequency domain signal by a sinc function everything gets smudged out.

Peak Detection Algorithm Peak detection is a common problem in 9/10 engineering problems. (not really, but a TON depend on them)

Typically this is what is done:

Median Thresholding

1) Look for all peaks in your signal. (i.e., a point that is larger than the two
2) take this list of points and for each one of them compute:
med_threshold = median(Peak,Width) + constantThresholmedian   where median is the
median value of the data centered at "Peak" with Width being the number of
points to look at.
a) The Width(usually written as Lambda in literature) and constantThreshold
(usually written as C) are determined by trial and error and using the ROC
curve (Acronym below)
3) if the peak's magnitude is above this threshold accept it as a true peak.
Else: Discard it, its a false peak
4) Generate a Receiver Operating Characteristic Curve(ROC) to how well the algorithm
is performing.


Here is an example:

suppose we have the signal X = [ 0 0 0 0 1 3 **9** 2 1 1 **2** 1 1 ]
1) 9 and 2 are both potential peaks
2) Lets use a window of 5 and  a threshold =2
so at 9 we have [1 3 9 1 2] -> [1 1 2 3 9]  so Median(9,5) = 2
9 > 2 +2, therefor its a peak
Lets take a look at 2: [ 1 1 2 1 1] -> [1 1 1 1 2 ] Median(2,5) = 1
2 < 1+2, therefor it is NOT a peak.


Determining Frequency

Now that You've effectively found the time localization of the peak try to find their frequency:

1) Use the locations of the peaks to generate a pulse train
a) this means create sum(Dirac_delta[t-L(n)]) where L(n) is the nth time that
you've localized through median thresholding
2) Apply FFT Algorithm
3) Look for largest peak.


Alternate Frequency Estimation

1) Think of this like a beat in a piece of music (I learned about thresholding by
researching Onset Detection.
2) Compute the average time distance between detected peaks.
3) now call your results BPM or PPM (pulses per minute)


While you may be satisfied by the peaked signal as is, there are algorithms that are applied to a whole different beast of problems called Onset Detection.

Onset Detection is a big area in Music Information Retrieval Research. Its used to determine when a note is being played.

If you think of your tape head signal as a highly sampled signal, you can apply many of the algorithms that you would find in this paper:

http://www.elec.qmul.ac.uk/people/juan/Documents/Bello-TSAP-2005.pdf

• "[1 3 9 1 2]" How are you getting the number 2 if your window is width = 5? – Spacey Apr 11 '12 at 21:19
• Notice how I lined up the numbers. the median is the middle number in an ordered set. [ 1 1 2 3 9] <- ordered, middle number is 2. – CyberMen Apr 12 '12 at 13:07
• Look at your numbers, they are [1 3 9 1 1]. Where did you get the 2 from? – Spacey Apr 12 '12 at 15:44
• @Mohammad went through a few edits while working on it, got deleted. Fixed. – CyberMen Apr 12 '12 at 18:04
• Cool thanks! Now, when you say 'note' in the musical context, does that mean single frequency, or many frequencies? Or does it not matter? I am asking to see if this is a way to also be used with some other applications that are narrow-band, (single tone). – Spacey Apr 12 '12 at 18:21