I want to transform my FFT output values into a dB scale, but I'm struggling to determine the function I should run each bin amplitude through. My understanding of the decibel scale is that a value representing an intensity must be divided by some reference intensity. The log of this ratio, times 20, will represent the dB scale value:
$$ \text{dB} = 20 \log_{10}\frac{\text{intensity}}{\text{intensity}_{\text{ref}}} $$
I should be able to pass each an FFT bin amplitude value into this equation as $\text{intensity}$, and the result will be the dB level of that bin.
Unfortunately, I'm not sure what $\text{intensity}_{\text{ref}}$ should be. Based on my reading, I believe it should represent the maximum possible intensity. If the readings were samples from a wave, this would be easy (the highest possible reading for 8-bit audio would be 255; for 16-bit it would be 65535, etc). However, I'm not sure that FFT outputs have such a predictable range or scale.
One option would be normalizing the amplitude values (dividing each value by the maximum amongst all bins for all FFT windows computed for the signal). This way, the bin with the highest amplitude would be at 0 dB and all others would be negative values relative to that maximum. However, this would only work if I pre-computed every FFT window for an entire signal. What if I was trying to compute dB values for an FFT of a just a single window?
So my question is: what is $\text{intensity}_{\text{ref}}$? I'm still relatively new to signal processing, so if there's an easy way to guarantee some range of FFT amplitude values or any other obvious solution I'm missing please let me know.