Training a CNN-HMM model

Question

I am currently trying to train a CNN-HMM acoustic model for speech recognition.

The CNN model is able to detect a center monophone given a context window of x (limits has not been tested yet - but works with 50) frames from a spectogram. The CNN provides me with a posterior probabilities of all possible monophones for the center monophone, but I am not sure how i am supposed to combine it with a transistion model such as HMM (Hidden markov model), given the CNN (Convolutional neural network) already provides posterior probability of all possible phones.

How am I supposed to train a HMM? Since i am classifying monophone, i seem to have a hard time understanding where the use of HMM would be appropriate here, as each HMM would just have one state, and encode one monophone, which the posterior probability in itself provides.

Is it possible to combine the CNN and HMM in a meaning full way, I am using kaldi, and the dataset consist of utterances of yesno from one speaker. (Simple case).

Answer 1

CNN

Let's say you have $P$ phones $p_0,p_1,\ldots,p_{P-1}$ and the CNN generates the posterior probabilities $x_i = x(p_i)$ for $i=0,\ldots,P-1$.

HMM

A Hidden Markov Model is a system that has $S$ states $s_0,\ldots,s_{S-1}$ that can produce $O$ observations $o_0,\ldots,o_{O-1}$.

The probability of jumping from state $s_i$ to state $s_j$ is $a_{ij}$, and this is captured by the state transition (probability) matrix $\mathbf{A} = [a_{ij}]$.

The output $o_i$ is generated by the state $s_j$ with probability $b_{ij}$ to form the output matrix $\mathbf{B} = [b_{ij}]$.

How to connect them?

So let's say that your phones $p_i$ are the states of the HMM $s_i$ (so that $S=P$).

But after that, I can't really say. The $x_i$ aren't really the $a_{ij}$. Nor are they the $b_{ij}$.

This paper contains the image below... but their explanation isn't really clear to me.

I'll read the paper again in the morning and see if that sheds any light.