38
What you've implemented is a single-pole lowpass filter, sometimes called a leaky integrator. Your signal has the difference equation:
$$
y[n] = 0.8 y[n-1] + 0.2 x[n]
$$
where $x[n]$ is the input (the unsmoothed bin value) and $y[n]$ is the smoothed bin value. This is a common way of implementing a simple, low-complexity lowpass filter. I've written about ...
19
Warning: include some history, old papers (I love them) and punch cards!
You used, with $a=0.2$ the form:
$$y(n) = y(n–1) + a[x(n) – y(n–1)]\,,$$
sometimes written as:
$$y(n) = ax(n) + (1 – a)y(n–1)\,.$$
The first above version is less natural, but it avoids one multiply, and is somehow more efficient.
Both formulae yield a linear, causal and infinite ...
15
An image "should not be blurred using a Gaussian Kernel" in general. This can be a safe bet for a lot of basic image processing needs, and a smoothing is almost mandatory when you want to control the information lost by the downsampling.
Blurring is (often, not always) another word for low-pass filtering. When an image contains high-frequency ...
9
Are there better approaches or further study on solutions to this which I should look at?
The normal approach for audio meters is a "lossy peak detector".
if new_value > current_value
current_value = new_value;
else
current_value = current_value * decay;
This reacts immediately to any new or peak or transient in the signal but it lingers on for a ...
8
In more standard DSP terms, you have the following filter:
$$
y[n] = (1-a) x[n] + a y[n-1]
$$
where $x[n]$ and $y[n]$ are the input and output signals at time $n$ respectively.
The transfer function (which you didn't ask for) is:
$$
H(z) = \frac{1-a}{1 - az^{-1}}
$$
so here is your single pole, at $z=a$ in the complex plane. This filter is also known as ...
8
According to (digital) sampling theorem, signals should be properly bandlimited, before they are (down) sampled.
A digital filter limits the bandwidth of the signal and makes it suitable for downsampling without aliasing.
A Gausssian kernel is very suitable as a lowpass filter, as it has a number of nice features. The Gaussian function is mathematically ...
7
Around US DoD contractor circles, this particular filter is frequently called an "alpha filter", because it can be characterized with one parameter that is traditionally named "alpha".
It is directly analogous to a simpe analog RC low-pass filter.
They are extremely simple, have serious limitations, but they have the undeniable advantage over more complex (...
7
In the Total Variation framework we define 2 flavors:
$$ \text{Isotropic TV} \; {TV}_{ {L}_{2} } \left( X \right) = \sum_{ij} \sqrt{ { \left( {D}_{h} X \right) }_{ij}^{2} + { \left( {D}_{v} X \right) }_{ij}^{2} } $$
$$ \text{Anisotropic TV} \; {TV}_{ {L}_{1} } \left( X \right) = \sum_{ij} \sqrt{ { \left( {D}_{h} X \right) }_{ij}^{2} } + \sqrt{{ \left( {D}_{...
6
As techwinder did in C++, I used datageist's algorithm and implemented it in Python. Maybe this will help somebody in the future.
import numpy as np
def non_uniform_savgol(x, y, window, polynom):
"""
Applies a Savitzky-Golay filter to y with non-uniform spacing
as defined in x
This is based on https://dsp.stackexchange.com/questions/1676/...
6
The usual approach to change detection is the CUSUM algorithm.
I've done an implementation that just addresses the level (mean) change issue. It's included (in R) below.
The black line is the noise-free data, the red line is the noisy data and the blue bars are the detected breaks (for this realization).
This just addresses the level change; to address ...
6
Hmmmmmmmmm, interesting question.
Since you want to use the second derivative as your criteria, it would seem that you would want to have the maximum second derivative absolutie value for as short of a duration as possible. This would suggest piecing together parabolas, matching the first derivatives at the joints.
How to do this algorithmically will take ...
6
Not sure if this has a name, but it is a nonlinear low pass filter that uses different smoothing constants depending on the input signal deviation from the filtered output. Small deviations are typically assumed to indicate consistency with the smooth estimation and result in little adaption to the input, while large deviations indicate a relevant state ...
6
Since the discussion in the existing answers and comments has mainly focused on what Savitzky-Golay filters actually are (which was very useful), I will try to add to the existing answers by providing some information on how to actually choose a smoothing filter, which is, to my understanding, what the question is actually about.
First of all, I'd like to ...
5
Let's call the window length $M$, and $N$ is the order of the polynomial. One important thing to realize is that you use a polynomial of order $N$ to approximate $M$ data points. This means that you use $M$ points to compute $N+1$ polynomial coefficients (an $N^{th}$ order polynomial has $N+1$ coefficients). So if $N+1=M$ then you do not smooth at all but ...
5
Gaussian Blur is Spatially Invariant Linear Filter.
Hence it can be analyzed in the Frequency Domain which in fact shows its Low Pass properties.
Namely it attenuates High Frequency Energy.
In Image, Edges, which are abrupt change from one color to other, requires high frequencies in order to be local.
Gaussian Blur attenuates and smear it.
As written above ...
5
In the classic framework both the Smoothing and the Difference Filter are applied using Convolution.
Since it is done using convolution it implies the operation is Linear Spatially Invariant (LSI).
LSI operators can be applied in any order and the result will be the same. This is also a result of the commutativity property of the convolution operator.
Let's ...
5
Let's analyze it in 1D as the intuition is the same.
First, let's have a look on a few different Gaussian Kernels:
As expected, they are wider as the Standard Deviation (STD) increase.
It means that when the kernel is applied using the convolution, more information is aggregates from farther samples. On the other side it means data is spread.
Now, in your ...
4
In the Probabilistic settings we have many methods applied to the Stochastic Gradient Descent in order to decrease the variance of the Gradient Estimation (ADAM / RMS Prop / AdaDelta, etc...).
The nice thing is to utilize them in deterministic settings.
So for instance you can use Momentum which to Signal Processing guy will look just like applying IIR / AR ...
4
2 point discrete differentiation is bound to produce highly noisy results.
try the 5-points stencil. you can also generate coefficients (i.e. more points) yourself using derivation of Lagrange polynomials.
4
If I understand you correctly you want to smooth the data (Namely reduce "Noise") yet regular filters would ruin the data on discontinuities.
What you need is an Edge Preserving Filter.
You can try the Bilateral Filter or Anisotropic Filter.
I have an advanced implementation of the Anisotropic Filter - Fast Anisotropic Smoothing of Multi Valued Images Using ...
4
Looks like your data is virtually free of noise. That, combined with a very high sampling frequency would mean that at the jumps the data is exactly at the threshold between two quantized values. Set up nodes at the middle points of the vertical jumps and construct splines that connect the nodes. The easiest is to just draw straight lines between successive ...
4
It seems the amplitude is not scaled properly. Rather than (2*A/pi) using (A/atan(1/delta)) seems more appropriate. In other words I propose:
y = (A/atan(1/delta))*atan(sin(2*pi*t*f)/delta);
Below is a figure illustrating the difference between the two scaling approaches. For low delta values the difference is not clear but for high delta values the ...
4
I think least squares is going to be the best approach, and that's not going to be that computationally expensive (I think! Please correct me if I'm wrong).
The gradient can be estimated from a sliding window of your data using:
$$
\hat{k} = \frac{\sum (x_n - \bar{x})(y_n - \bar{y})}{\sum (x_n - \bar{x})^2 }
$$
where the sum over $n$ is taken over the ...
4
A number of features will return some estimate of the smoothness of a signal. In general, these are all measures of dispersion with slightly different takes on "dispersion".
The choice of the "right" metric, depends very much on the application and the characteristics of the system and its signals.
The simplest metric would be the variance or the standard ...
4
The best tool for this job is normalized convolution. It can deal with missing samples as well as uncertainty.
The paper describing the method is "Normalized and Differential Convolution -- Methods for Interpolation and Filtering of Incomplete and Uncertain Data" by Hans Knutsson Carl-Fredrik Westin. There is a PDF on Semantic Scholar.
Normalized ...
4
You can apply Gaussian Blurring on an image in many ways:
Using FIR Approximation by Convolution.
Using Approximation by Box Blur.
In the Fourier Domain by Multiplication by a Fourier Kernel.
Using IIR Approximation.
You may have a look on my project - Fast Gaussian Blur.
Update
Meaning of FIR
In this context FIR is the coefficients which are utilized ...
4
Have a look at my Fast Gaussian Blur Project at GitHub.
You will find there implementation of IIR Approximation of Gaussian Blur which implements the following papaers:
Recursive Gabor Filtering.
Recursive Implementation of the Gaussian Filter.
Boundary Conditions for Young - van Vliet Recursive Filtering.
The idea is pretty straight forward.
4
@Greyfrog. Here are the descriptions of four different kinds of averaging operations:
4
Assuming you meant to produce something similar to the green line:
What about
$$\text{output}[n] = \max\{\text{input}[n-k], \text{input}[n-k+1], \ldots ,\text{input}[n]\}$$
i.e. you just find the maximum along a sliding window over the last $k$ input values?
3
NeatImage probably uses Wavelets based Noise Reduction.
You can look for methods based on that.
Today you need methods which are "Edge Aware", namely they smooth yet keep edges in tact.
Have a look at Fast Anisotropic Curvature Preserving Smoothing.
Only top voted, non community-wiki answers of a minimum length are eligible