0
$\begingroup$

Hey Stack Overflow community,

I've been given a series of datasets that resembles a square pulse, and ideally looks like the following:

enter image description here

This particular plot is composed of many sets of data.

I'm wondering if there is a way to reduce the noise found in the data, and retrieve just the area under the square pulse with the assumption that "spikes" may appear before or on the actual pulse itself.

The method I've tried is to cut everything before and after the pulse, and look for spikes that may occur on the actual pulse by looking at how much each value deviates from the mode peak found. However, this seems less than adequate as my implementation requires going through each point multiple times generating quite a large time complexity, and the file sizes are pretty huge.

Thanks for helping out!

$\endgroup$
1
$\begingroup$

A good strategy in the presence of white noise would be to do a moving average over the number of samples that equals the length of the pulse. The peak of this moving average will be your area with the noise optimally minimized.

So if you use MATLAB or Octave it would be as simple as this, assuming the pulse is 200 samples wide:

out = filter(ones(200,1),1,in);
area = max(out);

Or otherwise convolve your data with a sequence of 200 ones.

Using the Python scipy.signal package this is equivalently:

out = sig.lfilter(np.ones(200,1),1,in) 
area = np.max(out)
$\endgroup$
  • $\begingroup$ Hi! Thanks for the input, really helping me out here, and I appreciate that. I have a question, is the maximum of the filtered signal the area under the pulse, or should I apply a moving average over the filtered signal? $\endgroup$ – under267 May 3 '20 at 4:09
  • $\begingroup$ @under267 The average over the number of samples N is the area of the curve under N if you multiplied the average by N— so the “moving average” filter isn’t scaled by N and is simply the sum of all N samples in the filter. As the entire signal moves through the filter, the output at any given time represents the average under the portion of the signal that is in the filter at that time. Therefore when it is at maximum, given your signal pictured, will occur when the entire pulse is in the filter. $\endgroup$ – Dan Boschen May 3 '20 at 10:30
  • $\begingroup$ This serves to minimize noise (optimally so if the noise is white and identically distributed in each sample) since the standard deviation of the noise increases as $\sqrt{N}$ while the signal of interest increases at N in the addition process. $\endgroup$ – Dan Boschen May 3 '20 at 10:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.