0
$\begingroup$

I'm relatively new to this material, so apologies in advance if I'm missing something obvious.

I'm trying to smooth some very noisy signals. What I observe is that if I apply an ewma, even with a large center of mass, the resulting signal is still relatively noisy (in the sense of jittering from step to step). If however, I apply a filter with relatively low weights at the front end then the result is much smoother.

This effect is illustrated in the examples below.

At the level of impulse response this sort of makes sense to me, as there is large difference in filter weight between lag 0 and lag 1. Is it possible to understand this effect at the level of frequency response? I would have expected to see the ewma crossover killing high frequency noise much more than the ewma filters. However, this does not appear to be the case.

Replicating code

import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
from scipy import signal

def apply_filter(X, f):
    result = []
    for i in range(len(X)):
        result += [(X.iloc[:i + 1] * f[:i + 1][::-1]).sum() ]
        
    return pd.Series(result, index=X.index)

X = pd.Series(np.random.normal(size=1200))  # .diff()

indicator = pd.Series([0] * 1000 + [1] + [0] * len(X)) 
filter = {'raw': indicator,
          'ewm20': indicator.ewm(20).mean(),
          'ewm100': indicator.ewm(100).mean(),
          'mean5ewm20': indicator.rolling(5).mean().ewm(20).mean(),
          '5-15crossover': indicator.cumsum().ewm(5).mean() - indicator.cumsum().ewm(15).mean(),
          'firwin': pd.Series([0] * 1000 + signal.firwin(80, 0.1, window=('kaiser', 8)).tolist() + [0] * len(X))}
filter = {k: (v / v.abs().sum()).iloc[1000:1001 + len(X)].values for k, v in filter.items()}
smoothed = {k: apply_filter(X, v) for k, v in filter.items()}
freq = {k: pd.Series(signal.freqz(v)[1], index=signal.freqz(v)[0]) for k, v in filter.items()}

fig, axs = plt.subplots(len(filter), 3, figsize=(16, 8))
pd.DataFrame(smoothed).loc[:, filter.keys()].plot(subplots=True, ax=axs[:, 0])
pd.DataFrame(filter).loc[:, filter.keys()].iloc[:400].plot(subplots=True, ax=axs[:, 1])
np.log10(pd.DataFrame(freq).abs().loc[:, filter.keys()]).plot(subplots=True, ax=axs[:, 2])
axs[0, 0].set_title('Data')
axs[0, 1].set_title('Impulse Response')
axs[0, 2].set_title('Frequency Response')
axs[-1, 0].set_xlabel('Time')
axs[-1, 1].set_xlabel('Lag')
axs[-1, 2].set_xlabel('Frequency [rad/sample]')
plt.tight_layout()

Example

$\endgroup$
6
  • 3
    $\begingroup$ Can you clarify what your question is? If I'm interpreting your figure correctly, you get the best result when you apply the best filter, so there's nothing surprising about that. $\endgroup$ – MBaz Nov 12 '20 at 15:48
  • $\begingroup$ Sorry it was unclear. My question is why doesn't an EWMA, even with a large center of mass produce a smoother output? If understand correctly, this jitter we observe is very high frequency noise. Given the EWMA is a low-pass filter, which we see from the frequency response, I'd expect it to remove this jitter. $\endgroup$ – WMycroft Nov 13 '20 at 8:11
  • 2
    $\begingroup$ Compare the response of the ewm filter with the firwin filter and you'll see that it actually does not have a large out-of-band rejection, which means that it is not a very good filter. $\endgroup$ – MBaz Nov 13 '20 at 14:41
  • $\begingroup$ That is very helpful - thank you. I see for the firwin it is very clear that it kills high frequencies very effectively. However, I don't see a clear difference between the 5-15 crossover and the ewm100, if anything the ewm100 appears to do a better job killing high frequencies. However the ewm100 plot looks much smoother to me. Am i missing something obvious here? $\endgroup$ – WMycroft Nov 14 '20 at 10:24
  • 1
    $\begingroup$ The ewm100 filter actually has a more pronounced slope in the low frequencies, so larger rejection in general. Plot both responses in the same figure to compare them more precisely. $\endgroup$ – MBaz Nov 14 '20 at 18:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.