# Beginner level : Help in understanding from paper how the log-likelihood term has been obtained

The document explains probabilistic approach to principal component analysis using Maximum A - Posteriori (MAP) estimation. The way MAP is used is When we compnesate for the lack of information due to limited observations with an a priori preference on the parameters based on prior knowledge. The prior knowledge is given by the prior's distribution. Then,

$$\hat{\theta}_{MAP} = \arg \max p\left(D\vert\theta\right)p(\theta)$$

We maximize over the logarithm.

$p(\mathbf{x}_n)$ is normal with zero mean and unknown variance, and $\mathbf{y}_n$ is a linear function of $\mathbf{x}_n$ plus independent Gaussian noise where $n = 1...N$ represents the number of data vectors where, each vector $\mathbf{y}_n \in R^d$ and vector $\mathbf{x}_n \in R^q$. The missing/ latent variables are $\mathbf{x}_n$ and the complete data is formed from the observations $\mathbf{y}_n,\mathbf{x}_n$.

(please note that I have denoted $m_i$ as $\mu_i$)

• Problem 1: In Section 6.4 Eq(39) is the cost function for MAP is given.

\begin{align} C_{\rm MAP}=&\frac{1}{v_y}\sum_{ij\in O}\left(y_{ij}-\mathbf W_i^{\rm T}\mathbf x_j-m_i\right)^2+N\log 2\pi v_y+\frac{1}{v_m}\sum_{i=1}^dm_i^2+d\log 2\pi v_m\\ &+\sum_{k=1}^c\left[\frac{1}{v_{w,k}}\sum_{i=1}^d w_{ik}^2 +d\log 2\pi v_{w,k}+\sum_{j=1}^n x_{kj}^2 +n\log 2\pi \right].\qquad (39) \end{align}

Can somebody please help in making me understand how this cost function comes ? The priors for $X,m$ are from normal distribution (please refer Eq(19-20))

• Problem 2: I am unable to follow how the last term in Eq(39) has been obtained. The first 4 terms are obtained from taking the logarith of these 2 terms : $\prod p(y\vert x) \prod p(m)$

Also, There is another expression $p(d_{ij}) = 1/N \exp(-{|| d_{ij} - dist_{ij}||}^2/\sigma^2_d)$ where $d_{ij}$ is the estimated distance between latent points $\mathbf{\hat{x}}_{i}$ and $\mathbf{\hat{x}}_{j}$ and $dist_{ij}$ is the true known distance between the corresponding high dimensional points $\mathbf{y}_{i}, \mathbf{y}_{j}$ $dist_{ij}$ can be expressed as :

$$dist_{ij} = (\mathbf{y}_{i} - \mathbf{y}_{j}){(\mathbf{y}_{i} - \mathbf{y}_{j})}^T = (\mathbf{W}^T \mathbf{x}_{i} - \mathbf{W}^T \mathbf{x}_{j} ){(\mathbf{W}^T \mathbf{x}_{i} - \mathbf{W}^T \mathbf{x}_{j} )}^T$$

$$= {(\mathbf{x}_{i} - \mathbf{x}_{j} )}^T \mathbf{WW^T}(\mathbf{x}_{i} - \mathbf{x}_{j} )$$

So, $dist$ is a function of the latent variable. I was thinking if the prior $p(X)$ can be included with this but I don't know the rule for doing so.

How can I include this pdf as a prior or some other way into the likelihood expression?

Problem 1 The cost function is just the negative log posterior distribution of the unknowns $x$ conditioned on the data $y$ and hyperparameters. The optimization routine must minimize this cost function to obtain the MAP estimate.

Problem 2 As you correctly noted, the first four terms come from the $p(y|x)$ and $p(m)$ terms in the posterior. Next, note that there are priors on $x$ and $W$ as well defined to be Gaussian distributed (Eq. (13) and (20)). $x$ has a Gaussian distribution $N(0,I)$ and the $k^{th}$ column of $W$ follows $N(0, \nu_{wk}I)$. These lead to the last term in Eq. (39).

As for your final question, I could not find this method in the paper. Your definition of distance also seems wrong - perhaps you mean to say $(\mbox{dist}(\mathbf y_i, \mathbf y_j))^2 = ||\mathbf y_i - \mathbf y_j||^2 = (\mathbf y_i-\mathbf y_j)^T(\mathbf y_i-\mathbf y_j)$? Please clarify where you got the expression $p(d_{ij})$ from and how it is connected your original questions about MAP estimation.

Perhaps you are trying to impose a prior on the $\mathbf x$'s in such a way that if $\mathbf x_i$ and $\mathbf x_j$ are "close together," then $\mathbf y_i$ and $\mathbf y_j$ are also "close together." I would think that imposing a prior for this is not necessary because the first term in Eq. (39) is automatically taking care of this for you as explained below.

Assume for the moment that $\mathbf m=0$ Then the first term in Eq. (39) will force $\mathbf y_i$ to be close to $\mathbf W^T\mathbf x_i$ by choice of $\mathbf x_i$. So if $\mathbf y_i$ and $\mathbf y_j$ are nearby, so will $\mathbf W^T \mathbf x_i$ and $\mathbf W^T \mathbf x_j$. In other words, $\mathbf x_i$ and $\mathbf x_j$ are close together in the Mahalanobis distance sense i.e. $(\mathbf x_i-\mathbf x_j)^T\mathbf W^T \mathbf W (\mathbf x_i-\mathbf x_j)$ should be small.

The MAP estimation problem can be written as $$\mathbf X _{MAP} = \arg \max _{\mathbf X} p( \mathbf X | \mathbf Y, \mathbf W).$$ Note that the posterior is parametrized by $\mathbf W$. We can simplify this using Bayes' rule: $$p(\mathbf X | \mathbf Y, \mathbf W) \propto p(\mathbf Y | \mathbf X, \mathbf W) p(\mathbf X) p(\mathbf W).$$ Taking the negative logarithm, the MAP problem becomes a minimization problem: $$\mathbf X_{MAP} = \arg \min _\mathbf X [-\log p(\mathbf Y | \mathbf X, \mathbf W) -\log p(\mathbf X) -\log p(\mathbf W)]$$ This can be further simplified by using the pdf of the various Gaussian densities and it will start looking a lot like Eq.(39). For example, \begin{eqnarray*} -\log p(\mathbf W) &=& -\log \prod_{k=1}^c p(\mathbf W_{:,k}) \\ &=& -\sum_{k=1}^c \log \frac{1}{(2\pi v_{wk})^{d/2}} e^{-\frac{1}{2v_{wk}} w_{ik}^2} \\ &=& \frac{1}{2}\sum_{k=1}^c \left[ d\log(2\pi v_{wk}) +\frac{1}{v_{wk}} \sum_{i=1}^d w_{ik}^2 \right]. \end{eqnarray*}

The rest of the terms can be derived with similar algebraic manipulations.

• Thank you for your reply, I was now able to write out the likelihood expresson. The final Question about distance is based on the idea of including the condition of distance preservation between points that are close in high dimension must be close in lower dimension as well. But, I don't quite understand how to include this condition in the probabilistic formulation and in the likelihood expression. – SKM Aug 15 '16 at 18:42
• I got this idea from another paper. The details of this idea is posted as a Question stats.stackexchange.com/questions/229809/… If you may kindly have a look at this Question, then it seems that the model $p_B = Rp_A + t$ is based on a transformation although it does not deal with dimension reduction. However, I can apply the distance preserving condition over here but the likelihood formulation would change. I need help in how I can formulate the likelihood for probabilistic PCA with the distance condition. – SKM Aug 15 '16 at 18:46
• Please see updated answer. I suspect an explicit prior for what you are trying to do is unnecessary because the optimization in Eq. (39) is automatically taking care of this for you. – Atul Ingle Aug 15 '16 at 19:10
• I see, so just to reconfirm the likelihood expression is formed as $\prod P(y|x) p(x) p(w) p(m)$? However, I should ignore $p(m)$ based on your updated answer in order to make the points in low dimension close by – SKM Aug 15 '16 at 20:09
• The subscripts need run over all your $y_i$'s right? It looks like the paper has $n$ different $d$-dimensional vectors. So $1\leq i \leq n$. – Atul Ingle Aug 17 '16 at 1:05