How to get Instantaneous Magnitude for a Instantaneous Frequency From FFT?

My understanding if a frequency in my signal doesn't line up exactly with a bin, it is smeared over a few bins to the right and left.

How can the instantaneous magnitude be determined? (do I have to worry about the windowing function?)

Frist of all you need compute the Magnitude:

windowed = framed .* hann(length(framed))
Fourier = fft(windowed)
Mag = abs(Fourier)

And you need build a Window Kernel based in your Window Function!

For a Hann Window the kernel can be:

Wk(k) = 0.5 * (sinc(k*(M/N)) / (1 - (k*(M/N))^2))

M=Window Size
N=FFT Size

Of course you can use the same size for both, N and M !

k = Bin Number from FFT / N
sinc = Normalized Sinc, visit http://en.wikipedia.org/wiki/Sinc_function

If you have already determined the Instantaneous Frequency, you need find the corresponding offsets for each Frequency and the Instantaneous Magnitude is found when you multiply the magnitude from FFT by your Hann Window Kernel index:

Offset = abs(InstFreqs(i) / (Fs/N) - i)
index = floor(abs(Offset) * N) + 1
InstMag = Mag(i) * Wk(index)

Instantaneous frequency is given by the derivative of the phase of the analytic signal. The analytic signal is

$$f_A(x) = f(x) + \mathrm{i} (h \ast f)(x)$$

where $h(x)$ is the Hilbert transform kernel. It can be written as

$$f_A(x) = A(x) e^{\mathrm{i} \phi(x)}$$

where $A(x) = |f_A(x)|$ and $\phi(x) = \arg(f_A(x))$. $A(x)$ is called the instantaneous amplitude and $\phi(x)$ is called the instantaneous phase. Instantaneous frequency in radians is then given by

$$\omega(x) = \frac{d}{dx} \phi(x)$$

The Hilbert transform has an infinite impulse response and therefore uses the entire signal to compute. Typically one would bandpass the signal to localise the response. This both smears the signal in the spatial domain and removes DC and high frequency components. The values calculated as above are then called local amplitude, phase and frequency. Since the Hilbert transform has a discontinuity at DC in the Fourier domain, it is good to remove this first.

In MATLAB to get analytic signal it is

f_A = hilbert(f);
A = abs(f_A);
phase = angle(f_A);

But now the tricky bit, how to differentiate the phase? This is discussed here: