I want to smooth the phase of a measured (transfer) spectrum without destroying unit-complexity of the phase factor. Suppose $$f:\mathbb{R}\to \mathbb{C}\qquad , \qquad f(\omega)=r(\omega)\cdot {\rm e}^{i\phi(\omega)}$$ is the spectrum, then it is well-known how to smooth the power spectrum (i.e. $f^*f=r^2$): just multiply the autocorrelation function (i.e. inverse Fourier transform of $f^*f$) with a window function $w(t)$ with limited width and use the convolution theorem to show that this is the same as filtering $f^*f$ with the Fourier transform of the window. For example, if $w(t)$ is a Gaussian, $w(\omega)$ will also be Gaussian (with reciprocal width with respect to the width of $w(t)$).
But what about smoothing the phase? If one simply smoothes the phase factor (which has unit modulus) in the same way, the result usually will not have unit modulus anymore. Moreover, phases that change rapidly but linearly will cause vast cancellation. But fast phases in a transfer function correspond to big translations/lags of wave groups, which I want to be represented faithfully in the transfer function. If I just cut them off (which is what happens if I filter the phase factor), bigger translations/lags will just disappear.
For example take $$f(\omega)=e^{i\omega t_0}$$ as a pure phase function, then averaging this over a frequency interval from $\omega=\omega_0-\pi/t_0\dots \omega_0+\pi/t_0$, the result will be zero (in complex) independent of $\omega_0$ because it always integrates over a whole unit circle: $$\bar{f}(\omega_0)=\frac{1}{2\pi/t_0}\int_{\omega_0-\pi/t_0}^{\omega_0+\pi/t_0} e^{i\omega t_0}d\omega = 0$$ By contrast, the average phase angle will be linearly increasing, i.e.: $$\bar{\phi}(\omega_0)=\frac{1}{2\pi/t_0}\int_{\omega_0-\pi/t_0}^{\omega_0+\pi/t_0} \omega t_0 d\omega = \omega_0 t_0 = \phi(\omega_0)$$
So what I actually want to smooth is the phase angle $\phi(\omega)$ itself! But joining phases continuously is very challenging for measurement data, so the ATAN2 function is not an option. What I have so far come up with is the following. Suppose we had a pure phase function $$f(\omega)={\rm e}^{i\phi(\omega)}$$ Then $$f^\prime(\omega) f^*(\omega)=i\phi^\prime(\omega)$$ So if $\phi(\omega)$ is a linearly increasing phase angle, $\phi^\prime(\omega)$ is constant, independent of how fast the angle changes. Hence, $\phi^\prime(\omega)$ is perfectly destined to be smoothed by convolution, in my opinion. I don't know yet how to properly reconstruct a (smoothed) phase factor from that, other than directly integrating it.
Does this make sense? I want make sure it does before I calculate any further. I haven't seen anything like that in the literature. Maybe this method is already known. Of course, what I would most enjoy in the end is a simple smoothing formula that does not depend on whether $f$ has unit modulus or not.
Edit: Upon further thinking, I have recognized that the computation of the phase gradient (aka group delay) via $f^\prime(\omega) f^*(\omega)=i\phi^\prime(\omega)$ is exact only for a continuous spectrum, while it can get very inaccurate when applied to a discrete spectrum. The derivative for the discrete spectrum becomes a finite difference and if the phase changes very rapidly, the angles between adjacent spectral points become very blunt, making the finite differences inaccurate. So there is no way around computing the phase explicitely and unwrapping it, like Dan Boschen has proposed in his solution.