In line with a previous similar question here are my suggestions:
There are so many nice books but I believe you should first have a look at the science of sound from Rossing for getting the most broad view on the subject.
Then you can look at the following books , each dealing with a particular dimension of the problem:
1- Acoustics_BERANEK
2- Elements of Acoustical Engineering_OLSON
3- Music, Physics and Engineering_OLSON
4- Art of Digital Audio_WATKINSON
5- Audio Engineering_SINCLAIR
6- Digital Audio Effects_ZÖLZER
7- Digital Audio Signal Processing_ZÖLZER
8- Fundamentals of Acoustic Signal Processing_TOHYAMA
9- Audio Signal Procesing and Coding_SPANIAS
10- Audio Programming Book_BOULANGER
11- PsychoAcoustics_ZWICKER
12- Acoustics and PsychoAcoustics_HOWARD
EDIT:
I'm sorry that I've forgotten to mention about books on speech processing, so here you can find a list:
1- Speech and Audio Signal Processing_GOLD
2- Speech and Hearing for Communication_FLETCHER
3- Discrete Time Processing of Speech Signals_PROAKIS
4- Fundamentals of Speech Recognition_RABINER
5- Introduction to Digital Speech Processing_RABINER