2
$\begingroup$

If needed, you can find my first post for this problem here. I am trying to clean the following signal:

Raw signals

As proposed in the comment, I tried to use this post proposing 2 methods: median filtering and total variation denoising. I'm a python user, and I didn't find a good implementation of the second, thus I only kept the median filtering, implemented with scipy.

from scipy.ndimage import median_filter
window_size = 10
data_filtered = median_filter(data, size=window_size, mode='nearest')

The result is, at first look, not too bad:

filtered signals

However, the ramp is not completely preserved. The first one is slightly shifted to the right; while the second remains at the correct position. If I zoom on the low amplitude signal, I get:

Zoom on ramps of low amplitude signal

My goal is to measure the width of the square. This is directly impacting my measure. What is causing this phenomenon, why is it not symmetric and can it be fixed?

N.B: This is impacting my measure because I would like to measure the width at a us scale. Obviously, as those first measurements were done with a sampling frequency fs = 1e6, this resolution is not possible. Would this problem be solved if I increase the sampling frequency? (Max is 1e9). Which sampling frequency would you recommend?

Thank you for the guidance, Mathieu

Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ You got this obliquely in a comment to your earlier question, but let me reiterate it here, more strongly. This is an XY problem. You want to measure width. You think that you first need to get a good-looking filtered version, and then you'll do something else. That's not necessarily the case. What you should do is ask "How do I measure the width of a square pulse in noise?" There is filtering involved, but it's not necessarily what you think. $\endgroup$ – TimWescott Aug 19 '20 at 20:27
  • 2
    $\begingroup$ You actually have a pretty good looking pulse there -- why can't you just look for the thing crossing a threshold, and reject events like the one around time $t = 110$ as being too narrow? $\endgroup$ – TimWescott Aug 19 '20 at 20:30
  • $\begingroup$ @TimWescott So, my way of thinking was not fixed on step 1 filter, step 2 measure. Mainly, I want to compare both the width measured on the raw data, the width measured on filtered data. I want to get confident that the width I measure is actually the correct one. The measure part is not finalize, but I am basically correlating a ramping down and a ramping up with the beginning and the end of the square. But even without the measure part, the width is clearly altered here by the filter. I do understand why the median shifts the signal to the right, but why does it not occur for the ramp up? $\endgroup$ – Mathieu Aug 19 '20 at 21:45
  • 1
    $\begingroup$ So, the way I'd solve your actual problem is to filter with a moving average that's $N$ points long. Call its latest output $y_k$. I'd look at the latest output, and the output $N$ points ago -- $y_{k-N}$. When $y_k - y_{k-N}$ are above some threshold and reach a peak, then there was an edge $N$ points ago. To really measure the location of the peak accurately, fit the rising and falling edges of $w_k = y_k - y_{k-N}$ to straight lines, and find their intersection -- that's the time of the "actual" falling edge. I dunno if that's optimal, but it should be pretty close. $\endgroup$ – TimWescott Aug 20 '20 at 1:00
  • $\begingroup$ @TimWescott I'll keep that in mind. I have a very good method fitting ramp up and ramp down on the signal to determine the change position. I'm not that concerned about it. I would like to apply median filtering to get a better estimate of the charge (width times amplitude). The median filtering improve the amplitude reading, but deteriorate the width reading because of the shift observed in my post. Could we recenter on the question, I do not understand why the shift is not symmetrical (beginning and end of the square)? How can I apply this filter forward and backward as with filtfilt ? $\endgroup$ – Mathieu Aug 20 '20 at 11:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.