37
What you've implemented is a single-pole lowpass filter, sometimes called a leaky integrator. Your signal has the difference equation:
$$
y[n] = 0.8 y[n-1] + 0.2 x[n]
$$
where $x[n]$ is the input (the unsmoothed bin value) and $y[n]$ is the smoothed bin value. This is a common way of implementing a simple, low-complexity lowpass filter. I've written about ...
18
Warning: include some history, old papers (I love them) and punch cards!
You used, with $a=0.2$ the form:
$$y(n) = y(n–1) + a[x(n) – y(n–1)]\,,$$
sometimes written as:
$$y(n) = ax(n) + (1 – a)y(n–1)\,.$$
The first above version is less natural, but it avoids one multiply, and is somehow more efficient.
Both formulae yield a linear, causal and infinite ...
16
Because of the way the Savitzky-Golay filter is derived (i.e as local least-squares polynomial fits), there's a natural generalization to nonuniform sampling--it's just much more computationally expensive.
Savitzky-Golay Filters in General
For the standard filter, the idea is to fit a polynomial to a local set of samples [using least squares], then replace ...
11
The first equation you give is the difference equation for a lowpass FIR filter, or a linear filter with an impulse response that is finite in duration. I'll write it a bit differently (so that it is expressly discrete in time and causal):
$$
f_s[n] = 0.1 f[n-2] + 0.8 f[n-1] + 0.1 f[n]
$$
$f_s[n]$ is the smoothed version of the discrete-time input sequence ...
10
It should not, the need really depends on your application. However, this is a safe bet for most needs, and almost mandatory when you want to control the information lost by the downsampling.
Blurring is often another word for low-pass filtering. When an image contains high-frequency content (fast variations), downsampling can produce visually weird or ...
9
Typically "smoothing" means "replace the current value with average over the neighboring ones". Most common is energy smoothing, where the smoothing results in the energy average over the smoothing interval and the phase information is lost. Complex smoothing can be done as well but it's tricky business because of phase wrapping.
Energy smoothing can be ...
9
Are there better approaches or further study on solutions to this which I should look at?
The normal approach for audio meters is a "lossy peak detector".
if new_value > current_value
current_value = new_value;
else
current_value = current_value * decay;
This reacts immediately to any new or peak or transient in the signal but it lingers on for a ...
8
In more standard DSP terms, you have the following filter:
$$
y[n] = (1-a) x[n] + a y[n-1]
$$
where $x[n]$ and $y[n]$ are the input and output signals at time $n$ respectively.
The transfer function (which you didn't ask for) is:
$$
H(z) = \frac{1-a}{1 - az^{-1}}
$$
so here is your single pole, at $z=a$ in the complex plane. This filter is also known as ...
7
If you apply two filters in a series cascade, then the behavior of the cascade can be expressed in two different ways. In the time domain, the overall system's impulse response can be calculated by convolving the impulse responses of $y[n]$ and $y_2[n]$ together. For IIR filters, this can be somewhat cumbersome.
In the frequency domain, the overall system's ...
6
The sum of a gaussian kernel cannot be zero, because all the elements are going to be positive. The first kernel you have shown, is most likely an edge detection kernel, (which is a type of high pass filter), so the elements add up to zero because you want to completely null out any DC/constant component.
The second kernel you have shown however, is a low ...
6
Boyd has A Matlab Solver for Large-Scale ℓ1-Regularized Least Squares Problems. The problem formulation in there is slightly different, but the method can be applied for the problem.
Classical majorization-minimization approach also works well. This corresponds to iteratively perform soft-thresholding (for TV, clipping).
The solutions can be seen from the ...
6
The usual approach to change detection is the CUSUM algorithm.
I've done an implementation that just addresses the level (mean) change issue. It's included (in R) below.
The black line is the noise-free data, the red line is the noisy data and the blue bars are the detected breaks (for this realization).
This just addresses the level change; to address ...
6
Around US DoD contractor circles, this particular filter is frequently called an "alpha filter", because it can be characterized with one parameter that is traditionally named "alpha".
It is directly analogous to a simpe analog RC low-pass filter.
They are extremely simple, have serious limitations, but they have the undeniable advantage over more complex (...
6
Hmmmmmmmmm, interesting question.
Since you want to use the second derivative as your criteria, it would seem that you would want to have the maximum second derivative absolutie value for as short of a duration as possible. This would suggest piecing together parabolas, matching the first derivatives at the joints.
How to do this algorithmically will take ...
6
Not sure if this has a name, but it is a nonlinear low pass filter that uses different smoothing constants depending on the input signal deviation from the filtered output. Small deviations are typically assumed to indicate consistency with the smooth estimation and result in little adaption to the input, while large deviations indicate a relevant state ...
6
According to (digital) sampling theory, signals should be properly bandlimited, before they are (down) sampled.
A digital filter limits the bandwidth of the signal and makes it suitable for downsampling without aliasing.
A Gausssian filter is very suitable as a filter, as it has a number of nice features. The Gaussian function is mathematically tractable. ...
4
A simple method that often works is to apply a median filter.
4
"As a cheap alternative, one can simply pretend that the data points are
equally spaced ...
if the change in $f$ across the full width of the $N$ point window
is less than $\sqrt{N/2}$ times the measurement noise on a single point,
then the cheap method can be used."
$\qquad -$ Numerical Recipes pp. 771-772
(derivation anyone ?)
("Pretend equally spaced" ...
4
To solve optimization problems with TV penalty, we use a recently proposed algorithm called Fast Gradient Based Algorithms for Constrained Total Variation Image Denoising and Deblurring Problems (FISTA), which has better convergence rate than conventional iterative methods, such as ASD-POCS.
4
The window length should be equal to your transform length, not necessarily the length of your entire data set. The two are the same, of course, if you are going to transform the entire data set at once, but if you are planning to do shorter transforms then you should make the window length equal to the length of those transforms.
4
Let's call the window length $M$, and $N$ is the order of the polynomial. One important thing to realize is that you use a polynomial of order $N$ to approximate $M$ data points. This means that you use $M$ points to compute $N+1$ polynomial coefficients (an $N^{th}$ order polynomial has $N+1$ coefficients). So if $N+1=M$ then you do not smooth at all but ...
4
2 point discrete differentiation is bound to produce highly noisy results.
try the 5-points stencil. you can also generate coefficients (i.e. more points) yourself using derivation of Lagrange polynomials.
4
Since the discussion in the existing answers and comments has mainly focused on what Savitzky-Golay filters actually are (which was very useful), I will try to add to the existing answers by providing some information on how to actually choose a smoothing filter, which is, to my understanding, what the question is actually about.
First of all, I'd like to ...
4
@Greyfrog. Here are the descriptions of four different kinds of averaging operations:
4
Assuming you meant to produce something similar to the green line:
What about
$$\text{output}[n] = \max\{\text{input}[n-k], \text{input}[n-k+1], \ldots ,\text{input}[n]\}$$
i.e. you just find the maximum along a sliding window over the last $k$ input values?
3
That's a tricky problem. First you will need to define how to design "integrity". What do you value more: time domain shape, spectrum, phase, envelope, short term Fourier spectrum envelope, etc. You will probably have to trade between those, so prioritizing would be helpful.
In general this is done bu doing "cross fades", i.e. you overlap the two signals ...
3
Yes, the combination of the two first order IIR filters would be called a 2nd order IIR filter. The process of combining two first order filters to form a second order filter is called cascading.
3
Just like what Jim says, unless you are FFT-ing the entire data set at once, without splitting the data into shorter frames, then you will most likely use the length of data set. A quick example is shown below.
Ts = 50e-6; % Sampling Time(s)
Fs = 1/Ts; % Sampling rate, Sampling Freq (Hz)
f0 = 50; % ...
3
In the particular case where $f(y)=\|y\|_1$, the objective function can be written as
$$
\|x-y\|^2 + b\|y\|_1 = \sum_i(x_i - y_i)^2 + b\sum_i |y_i|,
$$
minimizing it requires to minimize each entry of the sum:
$$
\hat{y_i} = argmin \{(x_i-y_i)^2 + b|y_i|\}
$$
Using subdifferentials it is possible to show that the minimizer is the soft-thresholding ...
3
I know this is old, but I thought I would add something here. You can smoothly blend the part the filtered and unfiltered segments with a gradual function such as a sinusoidal window function. That will make this a little more agnostic to the type of input function since it will retain the original, unprocessed function at the boundaries yet leave less ...
Only top voted, non community-wiki answers of a minimum length are eligible
