# Statistic that is responsive to changes in time series, yet not too volatile

Let's say I have a fairly volatile time series $$X_t$$ - it doesn't have any reason to show an upward / downward trend, but it does show drops and spikes from time to time. It can also change level (e.g. for a few weeks it will be at one level, but it will quickly drop down to a new level a few % points down and stay there).

I'd like to have a statistic (basically a time series Y_t) that signals a change in $$X_t$$ quickly. It should also converge to the same (or close to the same) level as $$X_t$$ with as little delay as possible. Finally, $$Y_t$$ should have as little volatility as possible (I'm aware there's a tradeoff between low volatility and responsiveness).

I've tried the rolling mean - if there's a short duration (say 1 period) drop in $$X_t$$, then the rolling mean reacts quickly to the drop, but shows a delay in reacting to the recovery in $$X_t$$ (i.e. when $$X_t$$ is restored to normal values). The delay is the same as the window size minus the drop duration.

At this stage, I'm considering two statistics. The exponential moving average with a window of $$5$$:

$$E_t=\frac{\sum_{i=0}^t(1-\alpha)^iX_{t-i}}{\sum_{i=0}^t(1-\alpha)^i}$$ where $$\alpha=\frac{2}{\text{window}+1}$$

The other statistic I'm considering is a combination of EMA and median. Let $$M_t$$ denote the rolling median with a window of $$9$$. Then $$C_t=aE_t+(1-a)M_t$$

To test their response to a drop, I did a quick simulation as follows:

a = pd.Series(np.hstack((np.ones(10)*0.81, np.ones(4)*0.77,
np.ones(4)*0.72, np.ones(5)*0.77, np.ones(10)*0.81)))
x = 0.5
b = a.ewm(span=5).mean()
c = x*a.ewm(span=4).mean() + (1-x)*a.rolling(7).median()

figure(num=None, figsize=(27, 9))
plt.plot(a, marker="o")
plt.plot(b, marker=".")
plt.plot(c, marker=".") By the time $$C_t$$ converges to the same values as $$X_t$$, the difference between $$C_t$$ and $$E_t$$ is really small (less than a percentage point).

$$C_t$$ has the feature that it won't drop as sharply as $$E_t$$ in response to a drop in $$X_t$$, but after a few periods it will abruptly drop down to the level of $$X_t$$. Similarly when $$X_t$$ recovers, $$C_t$$ initially shows a slower rise than $$E_t$$, but then suddenly shoots up, overtaking $$E_t$$ and catching up with $$X_t$$.

Now intuitively it seems that overall the EMA $$E_t$$ would be a better statistic, but I'm not sure. Is there any concrete, rigorous justification for the superiority of any one statistic over the other? For example, is it possible to show that maybe $$C_t$$ reacts similarly to two different scenarios, and is therefore unsuitable to differentiate between those scenarios?

Finally, given the requirements of quick response, quick convergence and low volatility, is there any other statistic I may have missed that's better than these two?

• Please explain exactly what you're trying to detect and what you're trying to surpress. — Another remark, regarding your attempt: consider thinking of 1st-order IIR lowpass filter instead of exponential moving average. It's basically the same thing in result, but IMO makes it much clearer what's going on. – leftaroundabout Oct 28 '19 at 13:56

Hi: I don't think that there are general statements that can be made because the median component used in the second approach is pretty ad-hoc.

You probably already know this but the $$\alpha$$ that you defined for the EWMA is such that your EWMA has the same weighted age as just taking a moving average of length "window". Therefore, if you an moving average of length window and an EWMA with parameter $$alpha = \frac{2}{window + 1)$$, they will look quite similar.

The way you look at things in terms of "reaction time" till convergence or catching up is interesting.

The standard approach for this ( and more systematized but I'm not critcizing your creative approach ) is to just use an ewma with parameter $$\alpha$$. The different values of $$\alpha$$ chosen will equate to the half life of an impulse response for $$S_t$$. So, just by plotting the impulse response, you can see what $$\alpha$$ you prefer.

So, for example, suppose you used the EWMA with parameter $$\alpha$$. ( $$\alpha$$ is between zero and one for invertibility purposes ) Therefore, your exponentially smoothed value, $$\hat{S}_{t}$$, at every time step, $$t$$ would be:

$$\hat{S}_t = \alpha \times \hat{S}_{t-1} + (1-\alpha) \times X_{t}$$.

Now, we assume that $$S_{0} =1$$ and $$X_{t} = 0$$ for $$t$$ greater than zero and that we start start smoothing $$S$$ at $$t=1$$. So, since the halflife corresponds to the period when $$\hat{S}_t$$ becomes half of what it was originally, this corresponds to

$$\hat{S}_{halflife} = \frac{1}{2}$$.

Therefore, we have $$\frac{1}{2} = \alpha^{halflife}$$.

Taking the log of both sides gives $$halflife = \frac{log(1/2)}{log(\alpha)}$$.

So, the point is that, given the impulse response at time $$t=0$$, then different $$\alpha$$ correspond to different halflives for the response and you can play around with the halflife to see what works best for your problem.

Finally, if this was obvious or you were already familiar with it, my apologies. Also, IMHO, the best textbook exposition of this is Brown's "Smoothing, Forecasting and Prediction of Discrete Time Series". There are used versions on Amazon that are pretty cheap.