Keyword spotting - constant phoneme length

Question

I wish to implement a keyword spotting algorithm (in speech), on the basis of what was published in this article (Apple's Machine Learning Journal).

The article describes a neural-network-based method for keyword spotting, in which a network is first trained to identify twenty phoneme-classes and is triggered if the the sequence of phonemes that were inferred by the network correspond to Apple's 'Hey-Siri'.

It states that:

The microphone in an iPhone or Apple Watch turns your voice into a stream of instantaneous waveform samples, at a rate of 16000 per second. A spectrum analysis stage converts the waveform sample stream to a sequence of frames, each describing the sound spectrum of approximately 0.01 sec. About twenty of these frames at a time (0.2 sec of audio) are fed to the acoustic model, a Deep Neural Network (DNN) which converts each of these acoustic patterns into a probability distribution over a set of speech sound classes: those used in the “Hey Siri” phrase, plus silence and other speech, for a total of about 20 sound classes

What is puzzling to me is the bolded line.
It seems that the network is fed a constant number of frames (MFCC features) without taking into account that phonemes are uttered differently (the length varies) even by the same speaker (let alone by different ones).

Do I understand it correctly? What would be (or is), a good way of dealing with the variations in length of phonemes?

Answer 1

There is no relation between fixed frame length and varying phoneme length. Since frames are just 0.01 seconds and average phoneme length is about 0.1 seconds, there would be a frame located around a center of the phone. And in that point of time DNN detector that spans 20 frames around central frame will detect the phone of interest even if it does not have any time-varying architecture. The fully connected DNN they use could do that pretty reliably.

Of course they could use something more advance and adapted to varying length sequences, for example, multilayer bidirectional LSTM, that would result in better detection accuracy but at the same time it would be much more computation-consuming and thus battery-consuming, it would also affect reaction time, thats why they prefer simpler architecture.

Their approach would not work for some significant phoneme length variations. For example you can test that "heeeeeeey siiiiiiiiri" does not work that reliably. At the same time one could argue that "heeeeeeey siiiiiiiiri" is not the same as "hey siri" for a human either and should not activate Siri. For a balance between energy consumption and detection accuracy such simple solutions are a good choice.