Training a CNN-HMM model

I am currently trying to train a CNN-HMM acoustic model for speech recognition.

The CNN model is able to detect a center monophone given a context window of x (limits has not been tested yet - but works with 50) frames from a spectogram. The CNN provides me with a posterior probabilities of all possible monophones for the center monophone, but I am not sure how i am supposed to combine it with a transistion model such as HMM (Hidden markov model), given the CNN (Convolutional neural network) already provides posterior probability of all possible phones.

How am I supposed to train a HMM? Since i am classifying monophone, i seem to have a hard time understanding where the use of HMM would be appropriate here, as each HMM would just have one state, and encode one monophone, which the posterior probability in itself provides.

Is it possible to combine the CNN and HMM in a meaning full way, I am using kaldi, and the dataset consist of utterances of yesno from one speaker. (Simple case).

• Welcome to SE. DSP! This question is all about modeling. Often, when modeling, you don't have all the answers (parameters), to you need to make some assumptions. What assumptions can you make?
– Peter K.
May 29, 2017 at 15:03
• the states I am detecting are monophones. So my HMM has to be Monophone, and I have 5 different phone states. May 29, 2017 at 20:14
• I am more concerned on how to implement it.. since the CNN only provides the posterior probability of a monophone state, given an image... My question is how how to add the HMM in a meaning full way... May 31, 2017 at 21:39

CNN

Let's say you have $P$ phones $p_0,p_1,\ldots,p_{P-1}$ and the CNN generates the posterior probabilities $x_i = x(p_i)$ for $i=0,\ldots,P-1$.

HMM

A Hidden Markov Model is a system that has $S$ states $s_0,\ldots,s_{S-1}$ that can produce $O$ observations $o_0,\ldots,o_{O-1}$.

The probability of jumping from state $s_i$ to state $s_j$ is $a_{ij}$, and this is captured by the state transition (probability) matrix $\mathbf{A} = [a_{ij}]$.

The output $o_i$ is generated by the state $s_j$ with probability $b_{ij}$ to form the output matrix $\mathbf{B} = [b_{ij}]$.

How to connect them?

So let's say that your phones $p_i$ are the states of the HMM $s_i$ (so that $S=P$).

But after that, I can't really say. The $x_i$ aren't really the $a_{ij}$. Nor are they the $b_{ij}$.

This paper contains the image below... but their explanation isn't really clear to me.

I'll read the paper again in the morning and see if that sheds any light.

• Oh... Thanks. I think i was concerned with HMM at phone-level that i forgot that it also could be applied at a word level... I am using Kaldi, which I think uses HMM at a phone level... Jun 1, 2017 at 8:28