Services like Siri and Google Now perform speech recognition (speech-to-text) on mobile phones by doing part of the computation on a server in the cloud.
What do they upload to the cloud? Do they upload the raw audio? Or do they do some feature extraction and only upload features (e.g., MFCC coefficients or something) to the cloud? Is uploading the full audio important/useful for obtaining the best results (e.g., best accuracy of speech recognition)?
I'm curious about understanding the technical details of what is uploaded to Google/Apple, e.g., to better understand the privacy implications. Note that I'm not asking about their privacy policy, or what they do with the data, or whether I should be concerned about privacy, or any of that stuff. My interest is in the technical details of how speech recognition computation is split between your phone and their servers.
The research I've done, and what I've found:
Based on public statements that Apple has made, it appears that Siri works by uploading full audio clips (not just features) to Apple's server. The article also discusses some speculation about whether doing local processing would be feasible. (See also here.)
Back in 2011, Applidium reverse-engineered the Siri protocol. They reported that the phone is uploading a full audio clip to Apple's servers: the only computation done on the phone is to compress the audio using Speex. The scripts they used to help them are available on Github.
I haven't been able to find anything on Google Now.