It is difficult to assign meaning to individual phase values in an FFT. If you wanted to reconstruct your original signal using each of the frequency components in the FFT, you would use the phase values to time align each of the sinusoids so you get the proper destructive and constructive interference to reproduce the original signal. Altering the phase relationship of the frequency components can dramatically change the time domain representation of the signal (peaks and troughs can be shifted around this way).
Phase is more meaningful when applied over a band of frequencies. The cumulative effect of phase shift over a band of frequencies can delay the attack and or decay of audio signals. This can be important in audio systems and in particular speaker design.
You can look up "Group delay and phase delay" (google or enter in wikipedia) for a discussion of how these concepts affect sound quality.
It would be very difficult to do meaningful matching of audio signals by comparing values of FFT results. You can do exact matches if things are properly time aligned, but even minor changes to the audio signals can cause major changes in FFT values. For identification or matching of sounds, more complex methods are used. For example speech recognition uses CEPSTRAL analysis which involves multiple transforms, filter banks and a log computation to create sets of "feature" coefficients (Mel frequency cepstral coefficients
) that can be compared for similarity.