You need to know the numerical requirements of your algorithm and choose the precision accordingly.
So let's do the math here: A 32-bit floating point has a 24 bit mantissa and an 8 bit exponent. This gives you about 150 dB signal to noise ratio over a dynamic range of about 1540 dB. That's plenty for most things audio. Double precision gives you roughly twice as much.
Each algorithm has certain requirements for numerical precision. If properly designed all audio algorithms that I know of do just a fine with 32-bit floating point. "properly designed" is the keyword here. For example a 6th order band pass from 40-200 Hz sampled at 44.1kHz implemented as a direct from II IIR bi-quad filter will indeed have some noise problems at 32-bit. However it works perfectly fine as transposed form II or direct form I filter.
If you attempt a partial fraction expansion of the same band pass filter using for example Matlab's residuez() function, you will get bad results even with double precision. Again the numerical requirements of the algorithm for that specific input data exceed what double precision has to offer. The key to fixing this is not to blindly up the precision, but to use a better algorithm instead.
Finally let's take a look at what makes floating (32 bit or 64 bit) vulnerable: You have enormous dynamic range, i.e. you can scale down the signal by 200dB, amplify by 500dB, reduce again by 300dB and you end up exactly where you started with almost no loss in precision whatsoever. So that's not it. Floating point has trouble adding numbers that are vastly different in size. There is a point where adding a small number just doesn't make any difference, i.e. you get 1 + dx =1. This number "dx" is about 1.2e-7 for 32-bit floating point and 2.2e-16 for 64 bit. If you algorithm includes adding or subtracting numbers that are that far apart in magnitude, you may run into problems.
A good example for this is the Direct Form II filter mentioned earlier: The direct From II filter (see e.g. https://ccrma.stanford.edu/~jos/fp/Direct_Form_II.html) basically computes the state variables by filtering the input with the pole-only transfer function first and then filtering with the zeros to create the output. Now if the poles are close to the unit circle, the pole-only transfer function gets very, very large. So the state variable can be much bigger than the input (80db to 100dB bigger) and summing state variables with the input creates a lot of noise.
The solution here is to go to a transposed Form II or direct Form I filter. Analysis shows that the state variables cannot be bigger than input/output then maybe 12dB or thereabouts, so the problem magnitude mismatch doesn't occur in the first place.