2
$\begingroup$

I have signal that looks like this: enter image description here

As you can see there are some almost periodic pulses at the background with amplitude about 1000. I am saying almost, because, if you zoom in, you may see that they slightly differ from each other, plus there may be intervals without them, something like between 35000 and 40000.

How can I get rid of these pulses?

csv with data

Improve this question
$\endgroup$
3
  • $\begingroup$ It would be useful to see the waveform or spectrum of the signal without noise, if you can provide it (even an estimate). In many cases, how much you can reduce the noise depends on how much the signal and noise spectra overlap. $\endgroup$ – MBaz Dec 9 '15 at 0:51
  • 1
    $\begingroup$ Pulses related to electric heater operation have typical waveform - sharp rise, decay, plateau, sharp fall. This could be modeled as a mother wavelet in wavelet transform. Thus other typical waveforms with different period would act as child waveforms. A wavelet "spectrum" in timestamp - scale domain would present blobs nearly periodic in timestamp and slowly varying in scale. Muting out this blob and transforming back into timestamp - signal domain would yield the signal with electric heater patterns removed. $\endgroup$ – mbaitoff Jan 12 '16 at 19:15
  • $\begingroup$ You can also compute deltas (differences between current and next reading). Arrangement of their maxima would be at major on/offs. Elementary Fourier analysis on these deltas gives fundamental frequency and harmonics of the oscillations (for your data they were peaks at 43, 85, 176 etc). Calculating the corresponding period gives you the running average base size to attenuate those oscillations. $\endgroup$ – mbaitoff Jan 13 '16 at 14:57
1
$\begingroup$

I think you have a difficult problem here. Smoothing you signal (for example, with a moving average) probably won't help you. I guess you could try passing your signal through a narrowband notch filter whose notch center frequency is the frequency of your unwanted pulses. MBaz is correct. To think about an appropriate filter means you'd have to know what your signal looks like if it were not contaminated by those unwanted pulses. For example, does your uncontaminated signal look like the following?

enter image description here

I realize I'm not answering your question in any meaningful way, but I was thinking: I assume your noisy signal is the output of an A/D converter sampling an analog voltage signal that's badly contaminated by some sort of electrical noise, some electrical interference. Instead of eliminating the noise in the digital samples I think you should do everything you possibly can to eliminate the electrical noise in your original analog signal.

Dec. 12, 2015 Update based on Vlad's Comment: @Vladimir: Ah, interesting! What you wrote makes much sense. Those pulses are not noise contamination, they are part of your signal. To attenuate those pulses, three things off the top of my head:

[1] I'd like to see the result of applying your signal to a moving average process.

[2] Although I have no meaningful experience with such a filter, I wonder what would be the result of applying your signal to a median filter.

[3] There's a scheme in the "DSP Tricks" chapter of my DSP book describing how to attenuate very narrow impulsive noise in a signal. I don't know if that scheme would be useful with regard to your pulses but who knows? If you'd like me to send you information on this scheme, send me a private e-mail at: R_dot_Lyons_@_ieee_dot_org.

Improve this answer
$\endgroup$
1
  • $\begingroup$ This signal is a measure of the energy consumption as a function of time at some house. Periodic pulses may be related to an electric heating that turns on and of, trying to maintain desired temperature. I am guessing that it looks periodic only because temperature in the house within this time interval is not changing too fast. So it does not have any pure periodic source that is responsible for this noise. And yes, uncontaminated signal that you show is good enough for my task. $\endgroup$ – Vladimir Dec 9 '15 at 17:20
0
$\begingroup$

One thing (specifically for a problem like this) you can do is:

  1. Smooth the signal
  2. Get the peaks of the signal
  3. Connect the peaks to get rid of the pulses.

Let us say x contains the noisy signal

%Define a smoothing filter
h = ones(1, 10)./10;

smoothedSignal = filtfilt(h,1,x); 
[~,locs] = findpeaks(smoothedSignal);

% Connect the peaks together
plot(locs,smoothedSignal(locs));

Example Plot

This is just an example and you can of course play with different smoothing methods and select the one which works the best.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.