Denoising a signal

I'm starting hydraulic experiments, where I'd have to measure velocity in an unsteady flow with a device called Acoustic Doppler Velocimeter. In DSP terms, I'd have a nonstationary signal in a shape of waves (In the figure below, the instantaneous velocity (cm/s) as function of time (s) in one point, the period is about 70 sec in my case). This signal contains the mean component (mean velocity) and noise (turbulence). My goal is to extract the mean velocity.

I have looked up DSP and found many interesting models (Huang-Hilbert Transform, Wavelet Transform, Short Fourier Transform) to denoise. The only problem is that, in steady case, they need about 3 minutes measuring in one point so that they can average (arithmetic averaging) and filter out this noise. Since I'm in unsteady, I'd probably need more. Besides, my signal lasts about 1.5 minute. So I'm a little bit lost: can I still apply the denoising models (They're applied in the literature) ?

Thank you!

• You've named some pretty complex methods (and you noted that they are); have you looked at simpler methods, for example a low-pass filter of some type? In this case it looks like your signal of interest is that "wave", which is relatively low frequency. What is the sampling rate of your device/system? You've noted record times, but it may be helpful to include the sample rate as well – matthewjpollard Jul 19 '18 at 13:46
• I've chosen those methods because I want my final curves to be smooth. Besides, it's the methods I've come across in my research (I needed some dsp key words while researching). I admit I didn't pay attention to low-pass filters. The sampling rate is 100 Hz. – Yassine Jul 19 '18 at 15:05
• I add that I've selected those methods (and I'd likely work with HHT) because my signal is nonlinear and nonstationary. – Yassine Jul 19 '18 at 15:15
• I believe a lowpass filter or a Kalman filter will get what you want. – ZHUANG Jul 20 '18 at 2:26
• OK, what about the fact that we need more samples to do the arithmetic average, can those filters be trusted as we have less data than required for the arithmetic average? – Yassine Jul 20 '18 at 8:03

As @matthewjpollard has already indicated, you are acting too complex for a possibly simpler problem. You shall always begin with the simplest solution.

I hope the following OCTAVE code can convince you on the justification of this principle. Note that I've used the simplest (yet more complex than simple polynomials) model of a nonlinear wave fitting into your description. A more realistic model might be required for a through investigation.

clc; clear all; close all;

N = 256;       % cosine period (and signal length)
M = 32;        % tail length of signal
n=[0:N-1];     % time-index

x1 = 1-cos(2*pi*n/N);      % wanted signal
x1 = [x1, zeros(1,M)];     % append same tail (for practical pupose here)

x2 = 0.15*randn(1,N+M);    % enough wideband gaussian noise, to be added to our signal

x = x1 + x2;               % total signal as noise + wanted

b = fir1(64,0.01);         % simple FIR LP filter of order 64 and wc=0.01*pi
y = filter(b,1,x);         % Filter the input signal and obtain the result.

figure,plot(x1);title('what you want')
figure,plot(x);title('what you have')
figure,plot(y);title('what you get')


The result is: (note the delay of the filter...) Depending on your accuracy requirement, you may investigate other filter types too.

• Hello, thank you I'll check this out it sounds interesting. However the question remains: what about the fact that we need more samples to do the arithmetic average, can those filters be trusted as we have less data than required for the arithmetic average? – Yassine Jul 20 '18 at 8:04
• (by the way, here is an example of my signal (first figure with 2 waves) i.imgur.com/mXW9hJm.png – Yassine Jul 20 '18 at 8:14
• Yes the filter will work, you may have to adjust its length and cutoff frequency though. – Fat32 Jul 20 '18 at 10:46
• I'm still curious though if you don't mind, I don't understand the term "a priori" when they compare HHT to other methods : "the frequency band associated with each IMF is not known a priori, since the process carried out by the HHT is an adaptive filtering.." ? Thanks – Yassine Jul 23 '18 at 21:55
• Yes you're right the cutoff is rather wc. Thank you! – Yassine Jul 23 '18 at 22:43

I think that you want to estimate the velocity of the flow with respect to time. It seems that you want to obtain the velocity which has a wave type of profile in time. Therefore, you want to preserve the low-frequency content of the given signal. Hence, I would recommend you to use any low pass filter to denoise your time history as the noise in the given time history consists of the high-frequency component. So, you can use a low pass IIR filter like a 6th order Butterworth filter.

• Yes I want, in the end, to have a mean time-varying velocity, so as you said, I want to preserve the low frequency content (so I need to figure out how to compute the cutoff frequency). I've come across the Butterworth filter but realised that it's used for linear processes. Mine is nonlinear, that's why I'm spending time on the HHT method since it seems to be more suited for nonlinear and nonstationary processes. I'll look up the low pass filter. Thank you – Yassine Jul 19 '18 at 15:24
• I think if you plot the FFT plot you would easily find out the cut-off frequency. And can you please send me the link where it states that Butterworth filter can be only used for linear processes? What I know, for nonstationary processes instead of doing the FFT directly to the whole signal, Short Time Fourier Transform or Wavelet Transform can be used to visualize the frequency component in small time steps. – Debasish Jana Jul 19 '18 at 15:34
• electronics-tutorials.ws/filter/filter_8.html, where they said "Of these five “classic” linear analogue filter approximation functions only the Butterworth Filter...". Yes I can use STFT or wavelet too. I'll look up how to determine the cut off frequency. The main problem actually, besides which method to use, is that we need more data (about 10 000 samples) to average and find the mean velocity and I'm wondering if we can we avoid this by applying one of these filtering methods. – Yassine Jul 19 '18 at 15:38
• I think it says the filter is linear. It does not state that it can't be applied to the nonlinear process. While filtering a signal, we are convoluting the signal with a filter. If the signal is nonlinear and the filter is linear, the end convoluted result will be nonlinear. – Debasish Jana Jul 19 '18 at 15:43
• OK thank you! Here is an example of my signal imgur.com/a/Qd72HjO with 2 waves, for streamwise component (first curve) – Yassine Jul 19 '18 at 15:54

It seems like you're understanding of the underlying signal processing math isn't quite that solid (which isn't a bad thing!). Just because your signal is non-linear does not mean that you are unable to use a filter which may be linear. As a similar example, consider audio signals, those are also "non-stationary" and "non-linear" in the way you are using those words, though simple digital and analog filters are used VERY often to reduce noise on signals like these.

It looks like this is just a simple de-noising case where you have some low frequency components you wish to keep, and the high frequency components are almost entirely due to noise. In this, a really simple low-pass filter will likely do the trick. You could use an FIR filter or an IIR filter, that's up to you. Personally, I would use an FIR filter here since the group delay is well behaved. When you filter your signal, make sure you correctly compensate for the group delay (an easy way of doing this would be to use MATLAB's built-in "filtfilt" command).

You could certainly use other methods (ones you've mentioned, as well as a whole mess of others), but I feel they might just make this problem a bit more difficult than it needs to be. Often times, a lot of common problems like this can be solved with just a relatively simple filter. If the low-pass filter doesn't work out for you, you might want to look into Singular Spectrum Analysis (SSA), and use that to de-noise the data a bit (though I think you'll find its performance will be pretty similar to the low-pass filter). If you're interested in that method, check out this MATLAB tutorial on the subject. I've used SSA a bunch for work/in research, and found it can be very useful in problems like these where you know the signal you care about is band-limited and has a decent SNR value, and you can isolate it from other frequencies, though you may not know exactly where the signal is with respect to frequency, making a priori filter design difficult. Might be worth a look if you're wave data can shift around a lot.

• Hello, thank you. I don't have a solid background in signal processing actually. It's just that I've come across those methods in the literature. I think I'd end up using a low pass filter as you said. However the question remains: what about the fact that we need more samples to do the arithmetic average, can those filters be trusted as we have less data than required for the arithmetic average? – Yassine Jul 20 '18 at 8:11
• (by the way, here is an example of my signal (first figure with 2 waves) i.imgur.com/mXW9hJm.png – Yassine Jul 20 '18 at 8:14
• Simple moving average filters are certainly one way you could smooth, but there are loads of other low-pass filters you can use too! Simple moving average filters are just that: simple; you’re right that if you use a large window size for the filter, it might smear your signal too much to be useful. So with that I’d say check out other low-pass filters (least squares, parks-McClellan, etc) and see if they can accomplish what you need – matthewjpollard Jul 20 '18 at 18:32
• Thanks. I'm still curious though, I don't understand the term "a priori" when they compare HHT to other methods : "the frequency band associated with each IMF is not known a priori, since the process carried out by the HHT is an adaptive filtering.." ? Thanks – Yassine Jul 23 '18 at 21:55
• A priori is a Latin phrase which means “from the earlier”. If you know something a priori, you have knowledge of it going into the filtering. For example one might know the center frequency of a signal a priori. If you do not have a priori knowledge about a signal, you (mostly) do not know anything about the signal before you start. I’d advise you consult google/Wikipedia if you need a better explanation for that one since that’s a very common mathematics term. – matthewjpollard Jul 23 '18 at 23:46