Notations to Use in Formulating of Maximum Likelihood Estimation

Question

The received noisy signal $y_n \in \mathbb{R}$ is expressed as: \begin{align} y_n = \mathbf{h}^\mathsf{T}\mathbf{u}_n + w_n. \tag{1} \end{align} $\mathbf{h} = [h_0,h_1,\ldots,h_{p-1}]^\mathsf{T} \in \mathbb{R}^{1 \times p}$ of length $p$ which represents the impulse response of length $p$, and $\mathbf{u}_{n} = [u_{n}, u_{n-1}, u_{n-2},\ldots,u_{n-p+1}]^\mathsf{T}$ having variance $\sigma^2$. $w \sim N(0,\sigma^2_w)$ is the Additive White Gaussian noise. $\mathbf{y} = [y_0,y_1,\ldots,y_N]$ and $\theta = [\mathbf{h},\sigma^2_w]$ I will be estimating the following terms of the model below using $y$. The estimation of the model is formulated as follows: \begin{align} z_n = {\mathbf{h}}^\mathsf{T}{\mathbf{u}_n}. \tag{2} \end{align}

I want to use a variable $z$ to separate what is known and unknown. Therefore, I have used $z$. But, if I use this new variable, the mathematical formulation should have the term $z$.
The condition pdf $\mathbf{y}$ given $\theta$ is given by $P(\mathbf{y}|\mathbf{\theta}) =\prod_{n=1}^{N}\frac{1}{\sqrt{2\pi \sigma^2_w}}\exp(-\frac{(y_n -z_n)(y_n -z_n)^T}{2\sigma^2_w})$

My confusions are:

1) What notations to use for probability density function is it the one below:

$\mathsf{P}_y(y_n|{\mathbf{u}_n})$ or $\mathsf{P}_z(z_n|{\mathbf{u}_n})$ what goes in the subscript if I want to use $z$?

2) Is $P(\mathbf{y}|\mathbf{\theta})$ correct or $P_y(\mathbf{y}|\mathbf{\theta})$ or $P_z(\mathbf{z}|\mathbf{\theta})$?

3) If I want to use expectation maximization, then would $\xi$ which denotes the complete data set consisting of the input signal and the received noisy signal be written as: \begin{align} \xi &= \{u_0,\ldots,u_{N-1},y_0,\ldots,y_{N-1} \} \tag{3} \end{align} or \begin{align} \xi &= \{\hat{u_0},\ldots,\hat{u_{N-1}},z_0,\ldots,z_{N-1} \} \tag{4} \end{align}

4) Would the posterior be written as \begin{align} p_{\theta}(y_{1:N}|u_{1:N})p_{\theta}(u_{1:N}) = \prod_{n=1}^{N} \mathsf{P}_u(u_n|{u}_{n+1}) \prod_{n=1}^{N} \mathsf{P}_y(y_n|\mathbf{u}_n). \tag{5} \end{align} or \begin{align} p_{\theta}(z_{1:N}|u_{1:N})p_{\theta}(u_{1:N}) = \prod_{n=1}^{N} \mathsf{P}_u(u_n|{u}_{n+1}) \prod_{n=1}^{N} \mathsf{P}_z(z_n|\mathbf{u}_n). \tag{6} \end{align} Please help what the correct notations are used to represent the pdf, likelihood, log-likelihood if I use the variable $z$ model.

Please point out any other mistakes as well. Thank you.

Answer 1

1) What notations to use for probability density function is it the one below: $\mathsf{P}_y(y_n|{\mathbf{u}_n})$ $\mathsf{P}_z(z_n|{\mathbf{u}_n})$ what goes in the subscript if I want to use $z$?

One way to write the density function is to subscripted upper case letters for the random variables. So $p_{Y_n}$ denotes the density of random variable $Y_n$ and $p_{Z_n}$ is the density of the random variable $Z_n$. Similarly, $p_\mathbf{Y}:=p_{Y_1,\ldots,Y_n}$ denotes the joint density of all the $Y_i$'s.

2) Is $P(\mathbf{y}|\mathbf{\theta})$ correct or $P_y(\mathbf{y}|\mathbf{\theta})$ or $P_z(\mathbf{z}|\mathbf{\theta})$?

See my answer to (1) above. In some cases it is ok to drop the subscript if it is understood which random variable the density $p$ belongs to. However, if you have lots of different random variables it is good to explicitly include the subscripts.

If I want to use expectation maximization, then would $\xi$ which denotes the complete data set consisting of the input signal and the received noisy signal be written as: $\xi = \{u_0,\ldots,u_{N-1},y_0,\ldots,y_{N-1} \}$ or $\xi = \{\hat u_0,\ldots,\hat u_{N-1},y_0,\ldots,y_{N-1} \}$

It is not clear what the hats mean.

4) Would the posterior be written as

You need to clarify the posterior of what conditioned on what. I assume you are interested in estimating the $u$'s so the posterior you are referring to is

\begin{eqnarray} p_{\mathbf{U}|\mathbf{Y}}(\mathbf u|\mathbf y) &\propto& p_{\mathbf{Y}|\mathbf{U}}(\mathbf y|\mathbf u) p_\mathbf{U}(\mathbf u) \\ &=& p_{Y_1,\ldots,Y_N|U_1,\ldots,U_{N}}(y_1,\ldots, y_N|u_1,\ldots u_N) \; p_{U_1,\ldots,U_{N}}(u_1,\ldots u_N)\\ &=& \left(\prod_{i=1}^N p_{Y_i | U_{i},\ldots U_{1 \wedge (i-p+1)}} (y_i|u_{i},\ldots u_{1\wedge (i-p+1)}) \right) \; p_{U_1,\ldots,U_{N}}(u_1,\ldots u_N). \end{eqnarray}

That $\wedge$ symbol denotes a min. operation and makes sure that the subscripts don't go below 1. If you have a probabilistic model for the $U_i$'s you can further factor out $p_{U_1,\ldots,U_{N}}$ into a product of conditional densities. For example, if it's a 1st order Markov model with a stationary initial distribution $p_{U_1}(\cdot)$ you can write, $$ p_{U_1,\ldots,U_{N}}(u_1,\ldots,u_N) = p_{U_1}(u_1)\;\prod_{i=2}^N p_{U_i|U_{i-1}}(u_i|u_{i-1}). $$

pdf, likelihood, log likelihood

Finally, I think, you may be interested in writing the "complete data" log-likelihood function i.e. the density of $\xi$ parameterized by $\theta$. This can be written as follows:

\begin{eqnarray} p_{\xi|\theta}(\xi|\theta) &=& p_{U_1,\ldots ,U_N,Y_1,\ldots ,Y_N|\theta}(u_1,\ldots u_N, y_1,\ldots y_N | \theta) \\ &=& p_{Y_1,\ldots ,Y_N | U_1,\ldots ,U_N,\theta}(y_1,\ldots y_N | u_1,\ldots u_N, \theta) \; p_{U_1,\ldots ,U_N|\theta}(u_1,\ldots u_N| \theta) \end{eqnarray}

and simplify this further using your observation model for $Y_i$'s and Markov transition model for $U_i$'s.

_{It seems that your confusion stems from being unfamiliar with how random variable notation works. I would recommend reviewing a textbook on probability and random processes for electrical engineers.}