Speex has been in use for some time for real-time chat. It was specifically designed for voice. Some more advanced codecs for voice are based on it. The advantage of Speex is that it is free, source code readily available and fast in terms of encoding and decoding. The compression ratio is variable but your are most likely to use a setting of about 10:1 for quality reasons.
I haven't used Speex in some time and have no idea where the code currently stands. I recall having to use an older revision because a newer version used C++ new extensively and was unsuitable for use by the system's coprocessor environment it was running in.
You didn't mention whether the source you are streaming is live or pre-recorded. Whether you need to both encode and decode is important. It typically takes longer to encode than to decode.
Also important is whether you will always play from the beginning or can jump to any point ahead or behind. Many compression schemes rely on "state" from the previously decoded block which can make seeking a problem. (This would be a comment but I'm a lowly newb.)