I'm relatively new to this material, so apologies in advance if I'm missing something obvious.
I'm trying to smooth some very noisy signals. What I observe is that if I apply an ewma, even with a large center of mass, the resulting signal is still relatively noisy (in the sense of jittering from step to step). If however, I apply a filter with relatively low weights at the front end then the result is much smoother.
This effect is illustrated in the examples below.
At the level of impulse response this sort of makes sense to me, as there is large difference in filter weight between lag 0 and lag 1. Is it possible to understand this effect at the level of frequency response? I would have expected to see the ewma crossover killing high frequency noise much more than the ewma filters. However, this does not appear to be the case.
Replicating code
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
from scipy import signal
def apply_filter(X, f):
result = []
for i in range(len(X)):
result += [(X.iloc[:i + 1] * f[:i + 1][::-1]).sum() ]
return pd.Series(result, index=X.index)
X = pd.Series(np.random.normal(size=1200)) # .diff()
indicator = pd.Series([0] * 1000 + [1] + [0] * len(X))
filter = {'raw': indicator,
'ewm20': indicator.ewm(20).mean(),
'ewm100': indicator.ewm(100).mean(),
'mean5ewm20': indicator.rolling(5).mean().ewm(20).mean(),
'5-15crossover': indicator.cumsum().ewm(5).mean() - indicator.cumsum().ewm(15).mean(),
'firwin': pd.Series([0] * 1000 + signal.firwin(80, 0.1, window=('kaiser', 8)).tolist() + [0] * len(X))}
filter = {k: (v / v.abs().sum()).iloc[1000:1001 + len(X)].values for k, v in filter.items()}
smoothed = {k: apply_filter(X, v) for k, v in filter.items()}
freq = {k: pd.Series(signal.freqz(v)[1], index=signal.freqz(v)[0]) for k, v in filter.items()}
fig, axs = plt.subplots(len(filter), 3, figsize=(16, 8))
pd.DataFrame(smoothed).loc[:, filter.keys()].plot(subplots=True, ax=axs[:, 0])
pd.DataFrame(filter).loc[:, filter.keys()].iloc[:400].plot(subplots=True, ax=axs[:, 1])
np.log10(pd.DataFrame(freq).abs().loc[:, filter.keys()]).plot(subplots=True, ax=axs[:, 2])
axs[0, 0].set_title('Data')
axs[0, 1].set_title('Impulse Response')
axs[0, 2].set_title('Frequency Response')
axs[-1, 0].set_xlabel('Time')
axs[-1, 1].set_xlabel('Lag')
axs[-1, 2].set_xlabel('Frequency [rad/sample]')
plt.tight_layout()
