I have random variables $X_1, X_2, \cdots, X_m $, which can take $n$ values and is distributed iid according to $\Theta=(\theta_1, \theta_2, \cdots, \theta_n)$. That is $X_k$ can take values $\{1,2,\cdots,n\}$ and $P(X_k=i)=\theta_i, \ 1\leq k \leq m, \ 1 \leq i \leq n$.

I observe a random variables $Y_1, Y_2, \cdots, Y_m$, where I have that $P(Y_k=j|X_k=i)=c_{j,i}$.

Let $C$ be the matrix $[C_{j,i}]_{1\leq j,i \leq n}$. I have that $C$ is a full rank matrix.

I want to get an ML estimate of $\Theta$, from observations $Y_1, Y_2, \cdots, Y_m$.

Let $\Psi=(\psi_1,\cdots, \psi_n)$, where ${\psi}_i=P(Y_k=i)$. The ML estimate of $\Psi$ is clearly the vector of empirical frequencies. That is, we have the ML estimate of $\psi_i$, is $\hat{\psi}_i = \sum_{\ell=1}^{k} I(Y_{\ell}=i)$, where $I$ is an indicator function. Let us call this $\hat{\Psi}$.

I wanted to know if the ML estimate of $\Theta$, $\hat{\Theta}= C^{-1}\hat{\Psi}$. I can prove it if $n=2$, but do not seem to be able to show this for a general $n$.

Another way to put this is that I have a discrete memory-less source, which has a support of $\{ 1,2, \cdots, n\}$. I can observe this though a discrete memoryless channel, with transition probability matrix $C$. I want to estimate the source symbol probabilities. An ML estimate the observed symbol probabilities can be easily derived to be the empirical frequencies. I wanted to know if it can be shown that the ML estimate of the source symbol probabilities is $C^{-1}$ times the ML estimate of the observed symbol probabilities.

  • $\begingroup$ I think this is an interesting question, please could you expand on the description as it's a little sparse at the moment. $\endgroup$
    – Tom Kealy
    Jul 15, 2014 at 15:11
  • $\begingroup$ I don't understand the distribution of X exactly. Could you explain it? Thanks. $\endgroup$
    – Royi
    Jul 15, 2014 at 20:37
  • $\begingroup$ @Drazick I have expanded it explaining the distribution more clearly. $\endgroup$
    – Devil
    Jul 16, 2014 at 3:08
  • $\begingroup$ @TomKealy I've expanded the question, giving more description of the problem. Let me know if anything is unclear here. $\endgroup$
    – Devil
    Jul 16, 2014 at 3:09

1 Answer 1


I'm inclined to think this is true, but so far I've only gotten a counterexample:

Consider the channel:

\begin{bmatrix}0.5 & 0.5 & 0 & 0\\ 0 & 0.5 & 0.5 & 0 \\ 0 & 0 & 0.5 & 0.5 \\ 0.5 & 0 & 0 & 0.5\end{bmatrix}

This a typewriter channel. This matrix is not invertible (it has zero determinant, and rank 2). So the ML estimate of $\Theta$ cannot be gotten from this channel.

However, for $p \neq 0.5$ I think it could well be true.

EDIT: In fact I think the noisy channel aspect of this may well be a red-herring. The ML estimator of $\Theta$ is as you describe, simply because you have a linear model. You don't need any conditions of $C$ other than it should be invertible (i.e. all rows are linearly independent).

  • $\begingroup$ Yes. I think it should hold when $C$ is invertible as well. However, I've not been able to show this for $n >2$, because the algebra gets messy. I think there must be a more elegant way to show this. $\endgroup$
    – Devil
    Jul 18, 2014 at 19:51
  • $\begingroup$ This is just a linear regression problem, and there's a well developed theory about all this. $\endgroup$
    – Tom Kealy
    Jul 18, 2014 at 19:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.