What do cloud-based speech-to-text services upload to the cloud?

Question

Services like Siri and Google Now perform speech recognition (speech-to-text) on mobile phones by doing part of the computation on a server in the cloud.

What do they upload to the cloud? Do they upload the raw audio? Or do they do some feature extraction and only upload features (e.g., MFCC coefficients or something) to the cloud? Is uploading the full audio important/useful for obtaining the best results (e.g., best accuracy of speech recognition)?

I'm curious about understanding the technical details of what is uploaded to Google/Apple, e.g., to better understand the privacy implications. Note that I'm not asking about their privacy policy, or what they do with the data, or whether I should be concerned about privacy, or any of that stuff. My interest is in the technical details of how speech recognition computation is split between your phone and their servers.

---

The research I've done, and what I've found:

• Based on public statements that Apple has made, it appears that Siri works by uploading full audio clips (not just features) to Apple's server. The article also discusses some speculation about whether doing local processing would be feasible. (See also here.)

• Back in 2011, Applidium reverse-engineered the Siri protocol. They reported that the phone is uploading a full audio clip to Apple's servers: the only computation done on the phone is to compress the audio using Speex. The scripts they used to help them are available on Github.

• I haven't been able to find anything on Google Now.

Answer 1

Its probably safe to assume that all of the major companies send enough information to reconstruct the audio. This is because having that audio for training is such a valuable resource. A certain percentage of the audio segments will be listened to and transcribed by a human annotator.

Also features in these systems are more complicated than MFCCs. You obviously want to offload computing to the cloud so that's another reason to expect that they send the whole audio.

Answer 2

To give more variants to Aaron's answer, speech recognition pipeline has multiple stages where you can cut the line between client and server. There are the following variants:

• Lossless audio (flac)
• Lossy audio (speex 8kb/s)
• MFCC features (about 6.3kb/s bitrate)
• Compressed MFCC features ETSI distributed speech recognition standard, 4.4 kb/s bitrate
• Phonetic or word lattice or other intermediate results from the first-pass decoding as described in Architecture of Scalable Speech Recognition and just rescore them on the server (less than 1kb/s bitrate)

Most of the services upload audio though it's not really necessary to do that. It's just that they need to collect some data for the future updates of the system. Once the accuracy will be stable there will services with reduced bandwidth.