I see that there are different ways to write an AR model into a state-space representation, so that we can apply Kalman filter to estimate the signal. See Example 1, 2 and 3 here.
I wonder what differences are between the different state-space representations on the estimation by Kalman filter?
Thanks!
I unfortunately don't know a whole lot about Kalman filters, but I think I can help you out with the state space stuff.
In Example 1, the AR model is exactly your good old DSP recursive definition of output:
$$ y_t = \alpha + \phi_1y_{t-1} + \phi_2y_{t-2} + \eta_t$$
In this case we write down the state-space model with direct correspondence with the above equation:
$$\begin{pmatrix}y_t \\ y_{t-1}\end{pmatrix} = \begin{pmatrix}\phi_1 & \phi_2 \\ 1 & 0\end{pmatrix}\begin{pmatrix}y_{t-1} \\ y_{t-2} \end{pmatrix} + \begin{pmatrix}\alpha \\ 0 \end{pmatrix} + \begin{pmatrix}1 \\ 0 \end{pmatrix}\eta_t$$
Note that in this case, the states of the system are current and previous values of the output.
In the second example, you're separating your states $c$ from your output values. This means that the states can now be anything, even though they still directly map onto output values. This way we get
$$y_t = \mu + c_t$$
$$ c_t = \phi_1c_{t-1} + \phi_2c_{t-2} + \eta_t$$
And therefore
$$\begin{pmatrix}c_t \\ c_{t-1}\end{pmatrix} = \begin{pmatrix}\phi_1 & \phi_2 \\ 1 & 0\end{pmatrix}\begin{pmatrix}c_{t-1} \\ c_{t-2} \end{pmatrix} + \begin{pmatrix}1 \\ 0 \end{pmatrix}\eta_t$$
You should also recognize this as the standard state-space representation of a linear system, because you equations for state evolution and state-dependent output are two different equation. This separation is trivial in case of an AR model, but this latter notation is how we think of all linear state-space models in general.
The third example is a curious one. If you multiply out all the coefficients you will realize that it is actually equivalent to the first and the second examples. So why do it? I turns out that example 2 (being the proper state-space representation of the system) is called the Controllable Canonical Form of this system. If you do some reading or simply analyze the system carefully, you will realize that we can put this system into any state we like provided well-behaved values for $\phi_1$ and $\phi_2$ with the single input $\alpha$. Therefore we call such systems controllable, and it's very easy to see from this form of the state-space equations.
You should notice that two linear systems can be identical up to a change of basis. This means that we can pick a different basis to represent the same linear system. You can convince yourself that that's exactly what we've done to go from second to third example. Particularly, we like this linear transformation to transpose the state transition matrix, so that we would get for some unknown state $\boldsymbol{s}$
$$y_t = \begin{pmatrix}1 & 0\end{pmatrix} \boldsymbol{\alpha_t}$$
$$\boldsymbol{\alpha_t} = \begin{pmatrix}s_t \\ s_{t-1}\end{pmatrix} = \begin{pmatrix}\phi_1 & \phi_2 \\ 1 & 0\end{pmatrix}\begin{pmatrix}s_{t-1} \\ s_{t-2} \end{pmatrix} + \begin{pmatrix}\alpha \\ 0 \end{pmatrix} + \begin{pmatrix}1 \\ 0 \end{pmatrix}\eta_t$$
Now we can use the change of basis to find out what this state $\boldsymbol{s}$ has to be with respect to state $\boldsymbol{y}$. And we can calculate it to be
$$\begin{pmatrix}s_t \\ s_{t-1}\end{pmatrix} = \begin{pmatrix}y_t \\ \phi_2 y_{t-1}\end{pmatrix}$$
This form (transpose of Controllability Canonical Form) is called the Observability Canonical Form because if we can put a system in this form, we can easily deduce which states of the system can be observed by simply looking at the output. For some description of the canonical forms, you can read this document, and of course look around on the web. Note that in the document the states are flipped upside down, which does not change anything about the system representation, simply reordering the rows / columns of the matrices.
In short, it all depends on what you're trying to estimate i.e. what you know about the signal and what you don't. The Kalman filter will try to estimate the state based on your definition of what that state is. The conventional problem is when we are trying to estimate the AR coefficients.
Let's take an example of an $AR(2)$ model with no constant term $\mu$.
$$ y_k = a_1y_{k-1} + a_2y_{k-2} + \eta_k$$
To estimate the system above, all you need to do is estimate the AR coefficients, $a_1$ and $a_2$.
General Kalman Filter State Space set-up:
$$ {\bf x}_{k} = {\bf F}_{k-1}{\bf x}_{k-1} + {\bf w}_k $$ $$ {\bf y}_{k} = {\bf H}_{k}{\bf x}_{k} + {\bf v}_k $$ $ {\bf w}_k= WGN(0, Q^s)$ and $ {\bf v}_k= WGN(0, Q^o)$
In this case, we need to estimate $a_1$ and $a_2$. So it is natural to set the state as these coefficients. ${\bf x}_k = [a_1, a_2]^T$ For this example, tthese coeffients are constant (${\bf F}_{k} ={\bf F}_{k-1} = {\bf I} $) and there is no noise in these coefficients either -> ${\bf w}_k = {\bf 0} \implies Q^s = {\bf 0}$.
Since all we observe is $y_k$, they become the measurements for our system. Since we have already defined what the state vector is, for our measurement equations to be equal to the AR model given, we replace our measurement noise $ {\bf v}_k$ with $\eta_k$ and ${\bf H}_{k} = [y_{k-1}, y_{k-2}]$.
$$ {\bf x}_{k} = {\bf x}_{k-1} = \begin{bmatrix} a_1 \\ a_2 \end{bmatrix}$$ $$ y_k = {\bf H}_{k}{\bf x}_{k} + \eta_k = \begin{bmatrix} y_{k-1} & y_{k-2} \end{bmatrix}\begin{bmatrix} a_1 \\ a_2 \end{bmatrix} + \eta_k $$
Now, you can use the Kalman filter to estimate your state and consequently your signal.
Note: The only odd thing here is your matrix ${\bf H}_k$ depends on your measurements $y_k$. Some people have the misconception that the Kalman Gains and State Covariance Matrix are always measurement independent and that they can be computed beforehand. This case clearly shows that this is not the case. Both the Kalman Gain and State Covariance Matrix are estimated with functions of ${\bf H}_k$, which in this case is measurement dependent.
