37
What you've implemented is a single-pole lowpass filter, sometimes called a leaky integrator. Your signal has the difference equation:
$$
y[n] = 0.8 y[n-1] + 0.2 x[n]
$$
where $x[n]$ is the input (the unsmoothed bin value) and $y[n]$ is the smoothed bin value. This is a common way of implementing a simple, low-complexity lowpass filter. I've written about ...
17
[EDIT: added some history, old papers (I love them) and punch cards]
You used, with $a=0.2$ the form:
$$y(n) = y(n–1) + a[x(n) – y(n–1)]\,,$$
sometimes written as:
$$y(n) = ax(n) + (1 – a)y(n–1)\,.$$
The first above version avoids one multiply.
It yields a linear, causal, infinite impulse response filter. Story goes back to and through Poisson, Kolmogorov-...
15
Because of the way the Savitzky-Golay filter is derived (i.e as local least-squares polynomial fits), there's a natural generalization to nonuniform sampling--it's just much more computationally expensive.
Savitzky-Golay Filters in General
For the standard filter, the idea is to fit a polynomial to a local set of samples [using least squares], then replace ...
14
L1 norm minimization (compressed sensing) can do a relative better job than conventional Fourier denoising in terms of preserving edges.
The procedure is to minimize an objective function
$$
|x-y|^2 + b|f(y)|
$$
where $x$ is the noisy signal, $y$ is the denoised signal, $b$ is the regularziation parameter, and $|f(y)|$ is some L1 norm penalty. ...
11
The first equation you give is the difference equation for a lowpass FIR filter, or a linear filter with an impulse response that is finite in duration. I'll write it a bit differently (so that it is expressly discrete in time and causal):
$$
f_s[n] = 0.1 f[n-2] + 0.8 f[n-1] + 0.1 f[n]
$$
$f_s[n]$ is the smoothed version of the discrete-time input sequence ...
10
It should not, the need really depends on your application. However, this is a safe bet for most needs, and almost mandatory when you want to control the information lost by the downsampling.
Blurring is often another word for low-pass filtering. When an image contains high-frequency content (fast variations), downsampling can produce visually weird or ...
9
Are there better approaches or further study on solutions to this which I should look at?
The normal approach for audio meters is a "lossy peak detector".
if new_value > current_value
current_value = new_value;
else
current_value = current_value * decay;
This reacts immediately to any new or peak or transient in the signal but it lingers on for a ...
8
Typically "smoothing" means "replace the current value with average over the neighboring ones". Most common is energy smoothing, where the smoothing results in the energy average over the smoothing interval and the phase information is lost. Complex smoothing can be done as well but it's tricky business because of phase wrapping.
Energy smoothing can be ...
8
In more standard DSP terms, you have the following filter:
$$
y[n] = (1-a) x[n] + a y[n-1]
$$
where $x[n]$ and $y[n]$ are the input and output signals at time $n$ respectively.
The transfer function (which you didn't ask for) is:
$$
H(z) = \frac{1-a}{1 - az^{-1}}
$$
so here is your single pole, at $z=a$ in the complex plane. This filter is also known as ...
7
Getting a sub-sample resolution
A very cheap (in terms of code size) solution is just to upsample your signal. In matlab, this can be done with interp(y ,ratio). A slightly more complicated solution consists in naively detecting peaks ; and for each peak, fitting a parabola through y[peak - 1], y[peak], y[peak + 1] ; then using the point at which this ...
7
If you apply two filters in a series cascade, then the behavior of the cascade can be expressed in two different ways. In the time domain, the overall system's impulse response can be calculated by convolving the impulse responses of $y[n]$ and $y_2[n]$ together. For IIR filters, this can be somewhat cumbersome.
In the frequency domain, the overall system's ...
6
You can consider anisotropic diffusion. There are many methods based on this technique. Generally spoken, it is for images. It is an adaptive denoising method which aims to smooth non-edge parts of an image, and preserve edges.
Also, for Total variation minimization, you can use this tutorial. Authors provide MATLAB code also. They recognize the problem as ...
6
Chaohuang has a good answer, but I will also add that one other method that you can use would be via the Haar Wavelet Transform, followed by wavelet co-efficient shrinkage, and an Inverse Haar Transform back to the time-domain.
The Haar wavelet transform decomposes your signal into co-efficients of square and difference functions, albeit at different ...
6
Boyd has A Matlab Solver for Large-Scale ℓ1-Regularized Least Squares Problems. The problem formulation in there is slightly different, but the method can be applied for the problem.
Classical majorization-minimization approach also works well. This corresponds to iteratively perform soft-thresholding (for TV, clipping).
The solutions can be seen from the ...
6
Yes, in general, your #2 is correct. That being said, both of the filters stink (with your triangle filter being a little better).
No, f3 does not have the same frequency response as f4. To get an idea of why that is so, you generally have to zero-pad the impulse response before DFT'ing it to get a reasonable idea of what its frequency response looks like....
6
It probably depends more on your data. Just know, since differentiation is a linear operation, if you choose any linear filter to smooth f' and f'', it is equivalent to smoothing f using that same filter, then taking its derivatives.
Can you post some pictures or more information about the signal you want to differentiate? Probably what you're looking for ...
6
The sum of a gaussian kernel cannot be zero, because all the elements are going to be positive. The first kernel you have shown, is most likely an edge detection kernel, (which is a type of high pass filter), so the elements add up to zero because you want to completely null out any DC/constant component.
The second kernel you have shown however, is a low ...
6
The usual approach to change detection is the CUSUM algorithm.
I've done an implementation that just addresses the level (mean) change issue. It's included (in R) below.
The black line is the noise-free data, the red line is the noisy data and the blue bars are the detected breaks (for this realization).
This just addresses the level change; to address ...
6
Around US DoD contractor circles, this particular filter is frequently called an "alpha filter", because it can be characterized with one parameter that is traditionally named "alpha".
It is directly analogous to a simpe analog RC low-pass filter.
They are extremely simple, have serious limitations, but they have the undeniable advantage over more complex (...
6
Hmmmmmmmmm, interesting question.
Since you want to use the second derivative as your criteria, it would seem that you would want to have the maximum second derivative absolutie value for as short of a duration as possible. This would suggest piecing together parabolas, matching the first derivatives at the joints.
How to do this algorithmically will take ...
6
Not sure if this has a name, but it is a nonlinear low pass filter that uses different smoothing constants depending on the input signal deviation from the filtered output. Small deviations are typically assumed to indicate consistency with the smooth estimation and result in little adaption to the input, while large deviations indicate a relevant state ...
5
One method would be to resample your data so that it is equally spaced, then you can do whatever processing you like. Bandlimited resampling using linear filtering isn't going to be a good option since the data isn't uniformly spaced, so you could use some sort of local polynomial interpolation (e.g. cubic splines) to estimate what the underlying signal's ...
5
I think using cross-correlation and interpolating the peak would work fine. As described in Is up-sampling prior to cross-correlation useless?, interpolating or upsampling before the cross-correlation doesn't actually get you any more information. The information about the sub-sample peak is contained in the samples around it. You just need to extract it ...
5
While I'm not familiar with this specific type of filter, based on the plot you've shown, I would guess that the maxima that aren't found by your process are just butting up against the time resolution inherent in the process. Any kind of "smoothing" implies that there is some time-local smearing of the signal of interest, such that if there are two nearby ...
5
According to (digital) sampling theory, signals should be properly bandlimited, before they are (down) sampled.
A digital filter limits the bandwidth of the signal and makes it suitable for downsampling without aliasing.
A Gausssian filter is very suitable as a filter, as it has a number of nice features. The Gaussian function is mathematically tractable. ...
4
As to your point 1, F1 appears to smooth more because it is wider, in terms of its 2nd moment width, an thus has a slightly lower and sharper transition. But a rectangular filter will have terrible stop-band ripple in exchange.
Low stop band ripple does require a filter not to have any sharp transitions, at the ends especially as the 2nd derivative gets ...
4
This is a hard question to handle generally. Smoothing with a rectangular window is used all the time (often called a "moving average"), so that's not necessarily a problem. I'm not sure what ringing you're referring to, perhaps the sidelobes of the rectangular window's frequency response.
Differentiation is inherently a highpass operation; the ideal ...
4
The Savitzky-Golay filter provides smooth estimates of the signal and first few derivatives.
A MATLAB implementation can be found here.
4
"As a cheap alternative, one can simply pretend that the data points are
equally spaced ...
if the change in $f$ across the full width of the $N$ point window
is less than $\sqrt{N/2}$ times the measurement noise on a single point,
then the cheap method can be used."
$\qquad -$ Numerical Recipes pp. 771-772
(derivation anyone ?)
("Pretend equally spaced" ...
4
When you talk about bezier curves, it sounds like you think about them from an "illustrator" point of view, which is not totally right when it comes to spline interpolation (most probably what you are looking for).
Splines are piecewise curves that pass thrhough points. Bezier curves are third degrees splines between two points, and a series of them can ...
Only top voted, non community-wiki answers of a minimum length are eligible
