ARMA Model

$$y_t = \sum_{k=1}^p a_{k}y_{t-k} + \sum_{k=0}^q b_{k}\epsilon_{t-k}$$

where $b_0$ is usually 1 and $y_t$ is observable.

General Linear Filter

$$x_t = \sum_{k=1}^p a_{k}x_{t-k} + \sum_{k=0}^q b_{k}y_{t-k}$$

$y_t$ is observable (input), $x_t$ is filter output, and there is a feedback loop via $x_{t-k}$.

ARMA is a model, but from filtering perspective it looks like a filter for $\epsilon_t$, by simply moving $y_t$ to the right side of the equation and $\epsilon_t$ to the left side of the equation, then normalizing by $b_0$ (usually 1 so no effect).

  1. Is a model always a filter at the same time?
  2. Given a linear filter can we directly infer the underlying process model by simply rearranging the terms?

For example given an FIR filter to recover $x_t$ from $y_t = x_t + \epsilon_t$

$$\hat{x_t} = \sum_{k=0}^3 b_{k}y_{t-k}$$

Can we say that the whole process is $AR(3)$ with Gaussian errors given the filter is optimal in MSE sense?

$$y_t = \frac{b_{1}}{-b_0}y_{t-1}+\frac{b_2}{-b_0}y_{t-2}+\frac{b_3}{-b_0}y_{t-3}+\epsilon_t$$

This doesn't look correct.

I think the following information is relevant, but didn't really clarify it for me.


Linear prediction and autoregressive modeling are two different problems that can yield the same numerical results. In both cases, the ultimate goal is to determine the parameters of a linear filter. However, the filter used in each problem is different...In the case of linear prediction, the intention is to determine an FIR filter that can optimally predict future samples of an autoregressive process based on a linear combination of past samples. The difference between the actual autoregressive signal and the predicted signal is called the prediction error. Ideally, this error is white noise. For the case of autoregressive modeling, the intention is to determine an all-pole IIR filter, that when excited with white noise produces a signal with the same statistics as the autoregressive process that we are trying to model.

So according to this text, the corresponding model should actually be $AR(4)$

$$y_{t+1} = \sum_{k=0}^3 b_{k}y_{t-k} + \epsilon_t$$

two different problems that can yield the same numerical results

Are there situations where they wouldn't?

  • $\begingroup$ are you asking about the two problems mentioned in your text, or problems in general $\endgroup$
    – user28715
    Jul 11 '19 at 21:10
  • $\begingroup$ Hi! Are you interested in the DSP point of view of the explanation of linear prediction and ARMA modeling and linear filtering? Our notation and statisticians' are not the same... $\endgroup$
    – Fat32
    Jul 11 '19 at 21:11
  • $\begingroup$ @StanleyPawlukiewicz about the specific questions $\endgroup$ Jul 12 '19 at 0:55

Based on your exploration of the topics I can make the following explanation, mostly following the approach in Statistical Digital Signal Processing and Modeling, HAYES.

Linear Time Invariant (LTI) filtering is associated with the following LCCDE (Linear Constant Coefficient Difference Equation) with zero initial conditions and assuming a causal solution; i.e., interested in the solution for $n \geq 0$ in general.

$$ \sum_{k=0}^{N} a_k y[n-k] = \sum_{k=0}^{M} b_k x[n-k] \tag{1} $$

where $a_k$ and $b_k$ are the coefficients of the LCCDE (mostly $a_0 = 1$), and $x[n]$ is the input, while $y[n]$ is the output of the filter (aka solution of LCCDE).

The filter is associated with an impulse response $h[n]$ (coefficients of the LTI filter) which is the solution of Eq(1) when $x[n] = \delta[n]$; $x[0] = 1$ and $x[n] = 0$ for all $n> 0$.

$$ x[n] \longrightarrow \boxed{ h[n] } \longrightarrow y[n] $$

Impulse response can be used to compute the solution $y[n]$ of the LCCDE in Eq(1) for a given input $x[n]$, through what's called a convolution sum :

$$ y[n] = h[n] \star x[n] = \sum_{k=-\infty}^{\infty} h[k] x[n-k] \tag{2} $$

Z-transform of the impulse response is the following and is useful as an algebraic transform representation of the LTI filter system.

$$ H(z) = \frac{ \sum_{k=0}^{M} b_k z^{-k} }{\sum_{k=0}^{M} a_k z^{-k}} \tag{3}$$

From which the output $y[n]$ can also be expressed as $$ Y(z) = H(z) X(z) \tag{4} $$

which is nothing but Z-transform applied to Eq(2).

ARMA-(p,q) modeling :

Consider an LTI filter with LCCDE coefficients $a_k$ and $b_k$, as given in Eq(1), then if the input of this filter is a white-noise $v[n]$ with unit variance, then the output of this filter is called (by definition) Auto-Regressive Moving Average process with orders (p,q), abb as ARMA-(p,q), the associated LCCDE is written like

$$ \sum_{k=0}^{p} a_k x[n-k] = \sum_{k=0}^{q} b_k v[n-k] \tag{5} $$

Note that with the filtering convention in Eq(1), $x[n]$ was the input and $y[n]$ was the output, but withing the ARMA-(p,q) convention in Eq(5), $v[n]$ is the input and $x[n]$ (the ARMA process) is the output. The AR order is $p$ (associated with the lefthand side, recursive part, of the LCCDE) and the MA order is $q$ (associated with the right hand side of the LCCDE).

Now given some WSS RP $x[n]$, then you can model this process as an ARMA-(p,q) by finding the set of coefficents $a_k$ and $b_k$ according to Eq(5), which would minimize the mean square error between the modeling ARMA-(p,q) filter output $\hat{x}[n]$ and the given process $x[n]$ to be modeled. The solution for $a_k$ are generated by Modified Yule-Walker Equations and $b_k$ are found in a more complex manner following MA modeling techniques etc.

Linear Prediction :

Given a WSS random process $x[n]$, we want to estimate its current value $x[n]$ by using its $p$ past values $x[n-1],x[n-2],...,x[n-p]$ via a linear combination :

$$ \hat{x}[n] = \sum_{k=1}^{p} c_k x[n-k] $$

I have used these $c_k$ as the linear prediction coefficients; it can be shown that these coefficients are related to all-pole signal model Prony coefficients, or an ARMA-(p,0) model coefficients as $c_k = -a_k$.

Rewriting Eq(5) with $q=0$,and $a_0 = 1$, as

$$ x[n] = - \sum_{k=1}^p a_k x[n-k] + b_0 v[n] $$

with a suitable $b_0$, reveals that, if the predicted process $x[n]$ was generated via an ARMA-(p,0) model using Eq(5), then the linear prediction coefficients $c_k$ which minimize the prediction error will be $c_k= -a_k$ of Eq(5).

  • $\begingroup$ CowboyTrader: I think you're not understanding what an MA(q) is. When you say "ad-hoc" filter, I think you are refereeing to a moving average filter which is NOT what an MA(q) is. So, there are two levels of your confusion. A moving average versus an MA(q) and then the time-series notation versus DSP. But you should iron out MA(q) versus moving average first. $\endgroup$
    – mark leeds
    Jul 12 '19 at 2:22
  • $\begingroup$ Also, as I told you on crossvalidated, the whole statement from MA(4) to AR(4) is not correct/doesn't make sense. I strongly recommend that you read the two references I suggested ( box-jenkins-reinsel book and the ARIMA chapter in hamilton's "time series analysis" text ) and then come back with questions. $\endgroup$
    – mark leeds
    Jul 12 '19 at 2:26
  • $\begingroup$ @CowboyTrader Actually in the comments I was not clear enough. We (DSP comm) do not categorise filters based on their inputs. IIR, FIR, Tapped Delay Line, Transversal all refer to SYSTEMS (aka filters), while the terms ARMA-(p,q), AR-p, MA-q all refer to SIGNALS; specifically a class of WSS RPs. Therefore MA alone is an FIR filter and refers to a SYSTEM, whereas MA-q refers to a SIGNAL and cannot be used to refer to a system (a filter). Linear Prediction (estimator) is neither a system nor a signal, but a function of random variables; yet it's implemented via a prediction filter... $\endgroup$
    – Fat32
    Jul 12 '19 at 16:29
  • $\begingroup$ @CowboyTrader btw there's also an Exponential Moving Average (EMA) which is IIR instead of FIR. There are many MA variations, from a DSP point of view they are just special subclasses of filters... $\endgroup$
    – Fat32
    Jul 12 '19 at 16:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.