# Filtering Surge Signals

I am working on a matlab program which needs me to detect the pattern of surges in a signal and then remove them. I tried out various filters like median, hampel, butterworth etc. but couldn't get an effective output. Here's a snap of my signal :

Now, I need to filter out the surges. Could anyone please suggest me a method on how to approach this problem? Sampling frequency is 4khz and No. of data points is 40k.

Sounds like a job for dedicatedly identifying these surges and then actually subtracting them.

I'd start with a low-pass filter to find the slightly time-varying mean of the signal.

Use that to define lower and upper thresholds above or below you count something as surge.

Identify the samples lying outside the thresholds. Find a signal model for surges, e.g. a (truncated) sin x/x function or a quadratic function. Fit that model to the surge samples and then subtract from the signal.

Very similar to what Marcus Müller suggested, this is what I did:

The raw noisy waveform is compared (via comparators) with upper and lower thresholds. The upper threshold is simply a constant (0.8 here) plus the long term average (LTA). Likewise, the lower threshold is the LTA minus 0.8. The value 0.8 was chosen by inspection of the raw waveform. If a positive spike exceeds the upper threshold, the upper comparator output goes high; otherwise, it is low. If a negative spike goes below the lower threshold, the lower comparator output goes high; otherwise, it is low. The two comparator outputs are inputs to an OR gate. If a spike is not detected, the OR gate stays low and causes the SPST switch to simply pass the raw waveform: no filtering takes place. But, if a spike is detected, the OR gate goes high and causes the SPST switch to replace the spike value with the LTA. The results are shown below, for sampling at 4 kHz and a 10 second record:

Of course, a zero value could be inserted, instead of the LTA, and then post processing could replace those zeroes with whatever makes sense. This is easy because the comparator outputs can be recorded, thereby identifying exactly where replacements were made. This makes it simple to see how many spikes were replaced, their polarities, how long they lasted, and so on. For post-processing purposes, additional operations could be performed, e.g., adding in a little noise or a random sample from somewhere on the raw waveform that did not have a spike. There are lots of post-processing possibilities. I hope this helps!