I want to thank everybody for their answers and comments : each one of them actually gave me a new insight to the problem.
This is a very long post but I tried to make a comprehensive list of all the cases I learned while studying the answers with the most mathematical justification I could. Feel free to make any comment or suggestion.
Context and basic definitions :
In what follows :
- E[.] is the probability expectancy operator
- Finite length signals are starting at 0 and are of length T (continuous) and N (discrete)
- $\mathcal{F}$ is the right Fourier transformation adapted to the situation
- $\tau$ is the time difference that defines the autocorrelation. Wether it's real of integer is obvious in the context.
- Signals with integer subscripts like $h_i$ are discrete signals, signals with arguments like $h(t)$ are in continuous time signals
- Signals in uppercase are the results of the Fourier transformation of the lower case signal. Example : $H = \mathcal{F}(h)$
There seems to be three major dichotomies that superimpose on top of one another to answer the question at hand :
- Finite energy signals VS power signals
- Discrete signals VS continuous signals
- stochastic signals VS deterministic signals
I did not cover every cases but I covered what I believe to be the most commons.
About finite energy and power signals
Finite energy signals are signals x that verify :
$$\int_{-\infty}^{+\infty}\vert x(t)\vert^{2}dt <+\infty \quad \text{(resp. } \sum_{k=-\infty}^{+\infty}\vert x(k T_{s})\vert^{2} <+\infty )$$ The previous integral (resp. series) is called the energy.
Also, surprisingly, if you have finite energy, you actually don't need to verify :
$$\int_{-\infty}^{+\infty}\vert x(t)\vert dt <+\infty \quad \text{(resp. }\sum_{k=-\infty}^{+\infty}\vert x(kT{s})\vert <+\infty )$$ to take a Fourier transform, thanks to Plancherel's theorem (not Parseval's formula, the other Plancherel's theorem. A proof of both here and here)
In the finite energy context the Energy Spectral Density (ESD) is then naturally defined as :
$$ESD(x)(\omega) = \vert \mathcal{F}(x)(\omega)\vert^{2} $$
with $\mathcal{F}$ being the Fourier transformation that applies in the context (see below for more details on the various contexts)
It is called Energy spectral density because the sum/integral of $\vert \mathcal{F}(x)(\omega)\vert^{2}$ is called the energy.
Sometimes this sum/integral does not exist and to access frequency information we need to normalize/average with respect to time in order to evaluate a finite quantity.
=> This division time will lead us to talk about Power Spectral Density (PSD) instead of ESD.
That kind of signals are then also called power signals. They are treated at the end as a special case.
Autocorrelation definitions :
Finite energy signals are usually considered deterministic. Therefore their autocorrelation is usually defined as :
$$R_{xx}(\tau) = \int_{-\infty}^{+\infty} x(t)\overline{x(t-\tau)}dt$$
resp.
$$R_{xx}(\tau) = \sum_{k=-\infty}^{+\infty} x_k\overline{x_{k-\tau}}dt$$
with some adaptations for finite length signals (I'll give details when needed).
In a stochastic context, the autocorrelation is defined as :
$$R_{xx}(\tau) = E[x(t)\overline{x(t-\tau)}]$$
Now it's going to be a bit repetitive but let's show that this relationship $R_{xx}(\tau) = \mathcal{F}^{-1}(PSD)$ is true no matter the paradigm, starting with the finite energy signals.
Case finite energy - continuous time - continuous frequencies (i.e infinite signal):
in that context we have
$X(\omega) = \int_{-\infty}^{+\infty}x(t) e^{-i\omega t}dt$ and $x(t) = \int_{-\infty}^{+\infty}X(\omega) e^{i\omega t}d\omega$
So we have :
$$
\begin{split}
R_{xx}(\tau) &= \int_{-\infty}^{+\infty}x(t)\overline{x(t-\tau)}dt\\
&=\int_{-\infty}^{+\infty} [\int_{-\infty}^{+\infty}X(\omega)e^{i\omega t}d\omega]\overline{x(t-\tau)}dt\\
&=\int_{-\infty}^{+\infty} \int_{-\infty}^{+\infty}X(\omega)e^{i\omega\tau}\overline{x(t-\tau)}e^{i\omega (t-\tau)}d\omega dt\\
\end{split}
$$
Now in order to invert the two integrals we need first to show that $I(\omega) = \int_{-\infty}^{+\infty}X(\omega)e^{i\omega\tau}\overline{x(t-\tau)}e^{i\omega (t-\tau)}dt = X(\omega)\overline{X(\omega)}e^{i\omega\tau}$ exists.
It does, it's a constant multiplying a reverse Fourier transform that exists by Plancherel's theorem.
Now we also need to show that $\int_{-\infty}^{+\infty}\vert I(\omega)\vert dw = \int_{-\infty}^{+\infty} \vert X(\omega)\vert^{2}$ exists.
It does thanks to Plancherel's theorem (Parseval's formula version) and the fact that we assumed $x$ to be square integrable we know that $X(\omega)$ also is. Therefore we get :
$$\begin{split}
R_{xx}(\tau) &= \int_{-\infty}^{+\infty}\int_{-\infty}^{+\infty} X(\omega)e^{i\omega\tau}\overline{x(t-\tau)}e^{i\omega (t-\tau)}dt d\omega\\
&=\int_{-\infty}^{+\infty} \vert X(\omega)\vert^{2}e^{i\omega\tau}d\omega
\end{split}$$
Which is equivalent to $R_{xx} = \mathcal{F}^{-1}(ESD)$
Case finite energy - continuous time - discrete frequencies (i.e finite signal):
This one was tricky. I couldn't go around considering the periodised signal, $$x_p(t) = \sum_{k=-\infty}^{+\infty} x(t-kT)$$
It makes sense since the spectrum coefficients in that context are the Fourier coefficients of the periodised signal.
Also the definition of the autocorrelation in that case is :
$$R_{xx}(\tau) = \frac{1}{T}\int_{T}x_p(t)x_p(t-\tau)dt $$
In that context we have $$X_k = \frac{1}{T}\int_{0}^{T}x(t)e^{-i\frac{2\pi}{T}kt}dt = \frac{1}{T}\int_{T}x_p(t)e^{-i\frac{2\pi}{T}kt}dt$$
and
$$x(t) = \sum_{k=-\infty}^{+\infty} X_k e^{i\frac{2\pi}{T}k t}\quad \forall t\in [0, T]$$ and for the perdiodised function :
$$x_p(t) = \sum_{k=-\infty}^{+\infty} X_k e^{i\frac{2\pi}{T}k t} \quad \forall t\in \mathbb{R}$$.
Note the $\frac{1}{T}$ factor in the Fourier coefficient. This will allow us to talk about PSD (Power Spectral Density) instead of ESD (Energy Spectral Density). Note also how $\int_{-\infty}^{+\infty}\vert x_p(t)\vert^{2}dt$ does
not converge due to periodicity.
$$
\begin{split}
R_{xx}(\tau) &= \frac{1}{T}\int_{T} x(t)\overline{x(t-\tau)}dt\\
&= \frac{1}{T}\int_{T} \sum_{k=-\infty}^{+\infty} X_k e^{i\frac{2\pi}{T}k t}\overline{x(t-\tau)}dt \\
&=\frac{1}{T}\int_{T} \sum_{k=-\infty}^{+\infty} X_k e^{i\frac{2\pi}{T}k \tau}\overline{x(t-\tau)}e^{i\frac{2\pi}{T}k (t-\tau)}dt
\end{split}
$$
in order to switch the integral and the sum we first need to show that $\frac{1}{T}\int_{T} X_k e^{i\frac{2\pi}{T}k \tau}\overline{x(t-\tau)}e^{i\frac{2\pi}{T}k (t-\tau)}dt = X_k e^{i\frac{2\pi}{T}k \tau}\frac{1}{T}\int_{T}\overline{x(t-\tau)}e^{i\frac{2\pi}{T}k (t-\tau)}dt$ exists.
It does because it is a classical integral on a interval. To be specific if $x$ is piecewise continuous on $[0, T[$ then $x_p$ which is T-periodic and coincides with $x$ on $[0,T[$ has to be piecewise continuous by periodicity. I guess it's trickier if $x$ doesn't have a finite support but luckily we're not in that case.
We also need to show that $X_k e^{i\frac{2\pi}{T}k \tau}\frac{1}{T}\int_{T}\overline{x(t-\tau)}e^{i\frac{2\pi}{T}k (t-\tau)}dt= \vert X_k\vert^{2}e^{i\frac{2\pi}{T}k \tau} $ is summable.
It is since $\vert \vert X_k\vert^{2}e^{i\frac{2\pi}{T}k}\vert = \vert X_k\vert^{2} $ which is summable via Parseval's forumula applied on the Fourier coefficients of $x_p$.
Then we can conclude :
$$
\begin{split}
R_{xx}(\tau) &= \sum_{k=-\infty}^{+\infty} \frac{1}{T}\int_{T} X_k e^{i\frac{2\pi}{T}k \tau}\overline{x(t-\tau)}e^{i\frac{2\pi}{T}k (t-\tau)}dt\\
&= \sum_{k=-\infty}^{+\infty} X_k e^{i\frac{2\pi}{T}k \tau}\frac{1}{T}\int_{T}\overline{x(t-\tau)}e^{i\frac{2\pi}{T}k (t-\tau)}dt\\
&=\sum_{k=-\infty}^{+\infty} X_k e^{i\frac{2\pi}{T}k \tau}\overline{X_k}\\
&=\sum_{k=-\infty}^{+\infty} \vert X_k\vert^{2} e^{i\frac{2\pi}{T}k \tau}
\end{split}
$$
Which is equivalent in that context to $R_{xx} = \mathcal{F}^{-1}(PSD)$
Case finite energy - discrete time - continuous frequencies (i.e infinite signal):
In that context $X_{2\pi}(\omega) = \sum_{-\infty}^{+\infty} x_k e^{-i\omega k}$ and $x_k = \frac{1}{2\pi}\int_{2\pi}X_{2\pi}(w)e^{i\omega k}$
So we have :
$$
\begin{split}
R_{xx}(\tau) &= \sum_{k=-\infty}^{+\infty}x_k \overline{x_{k-\tau}} \\
&= \sum_{-\infty}^{+\infty}[\frac{1}{2\pi}\int_{2\pi}X_{2\pi}(w)e^{i\omega k}d\omega] \overline{x_{k-\tau}}\\
&= \frac{1}{2\pi}\sum_{k=-\infty}^{+\infty} \int_{2\pi}X_{2\pi}(w)e^{i\omega k} \overline{x_{k-\tau}}d\omega\\
&= \frac{1}{2\pi}\sum_{k=-\infty}^{+\infty} \int_{2\pi}X_{2\pi}(w)e^{i\omega \tau} \overline{x_{k-\tau}}e^{i\omega (k-\tau)}d\omega\\
\end{split}
$$
In order to exchange the integral and the sum we need first to show that $I(\omega) = \sum_{k=-\infty}^{+\infty}X_{2\pi}(w)e^{i\omega \tau} \overline{x_{k-\tau}}e^{i\omega (k-\tau)}= X_{2\pi}(w)e^{i\omega \tau}\sum_{k=-\infty}^{+\infty} \overline{x_{k-\tau}}e^{i\omega (k-\tau)}$ exists.
It does, it is a constant multiplying a Fourier transform (DTFT in that context) which exists because we assumed the signal to be square summable.
Now we need to show that $X_{2\pi}(w)e^{i\omega \tau}\sum_{k=-\infty}^{+\infty} \overline{x_{k-\tau}}e^{i\omega (k-\tau)} = X_{2\pi}(w)e^{i\omega \tau}\overline{X_{2\pi}(w)} = \vert X_{2\pi}(w)\vert^{2}e^{i\omega \tau}$ is piecewise continuous.
I actually don't know how to do that but I guess if we recognize $X(\omega)$ as a Fourier series it comes easy. We can therefore intervert the sum and the integral. We get :
$$\begin{split}
R_{xx}(\tau) &= \frac{1}{2\pi}\int_{2\pi}X_{2\pi}(\omega)e^{i\omega \tau}\sum_{k=-\infty}^{+\infty} \overline{x_{k-\tau}}e^{i\omega (k-\tau)}\\
&= \int_{2\pi}X_{2\pi}(w)e^{i\omega \tau}\overline{X_{2\pi}(w)} \\
&= \int_{2\pi}\vert X_{2\pi}(w)\vert^{2}e^{i\omega \tau}\\
\end{split}$$
Which is equivalent in that context to $R_{xx}(\tau) = \mathcal{F}^{-1}(ESD)$
Case finite energy - discrete time - discrete frequencies (i.e finite signal):
In that context we also need to periodise the signal with the periodised sequence $\tilde{x}_k$ being defined by $\forall s\in \mathbb{Z}\quad \tilde{x}_{s+N}=\tilde{x}_{s}$ and $\forall 0\leq t\leq N-1 \quad\tilde{x}_t=x_t$. The sequence $(\tilde{x}_t)_{t\in \mathbb{Z}}$ is then N-periodic.
In that context the Fourier transform is the DFT defined by :
$$X_k = \sum_{t=0}^{N-1}x_t e^{-i\frac{2\pi}{N}kt} = \sum_{N}\tilde{x}_t e^{-i\frac{2\pi}{N}kt}$$
also :
$$\forall 0\leq t \leq N-1 \quad x_t = \frac{1}{N}\sum_{k=0}^{N-1}X_k e^{i\frac{2\pi}{N}kt}$$
And finally :
$$\forall t \in \mathbb{Z} \quad \tilde{x}_t = \frac{1}{N}\sum_{k=0}^{N-1}X_k e^{i\frac{2\pi}{N}kt}$$
Note the $\frac{1}{N}$ factor in the Fourier coefficient. This will allow us to talk about PSD (Power Spectral Density) instead of ESD (Energy Spectral Density). Note also how $\sum_{t=-\infty}^{+\infty}\vert \tilde{x}_t(t)\vert^{2}$ does
not converge due to periodicity.
and the convolution on the periodised N-periodic sequences is called the circular convolution. For the autocorrelation we get :
$$R_{xx}(\tau) = \sum_{t=0}^{N-1}x_t\overline{\tilde{x}_{t-\tau}} = \sum_N \tilde{x}_{t}\overline{\tilde{x}_{t-\tau}}$$
Let us start the proof :
$$
\begin{split}
R_{xx}(\tau)
&= \sum_{t=0}^{N-1}x_t\overline{\tilde{x}_{t-\tau}}\\
&= \sum_{t=0}^{N-1} [\frac{1}{N}\sum_{k=0}^{N-1}X_ke^{i\frac{2\pi}{T}kt}] \overline{\tilde{x}_{t-\tau}}\\
&=\frac{1}{N}\sum_{k=0}^{N-1}\sum_{t=0}^{N-1}X_ke^{i\frac{2\pi}{T}kt} \overline{\tilde{x}_{t-\tau}}\\
&=\frac{1}{N}\sum_{k=0}^{N-1}X_ke^{i\frac{2\pi}{T}k\tau} \sum_{t=0}^{N-1}\overline{\tilde{x}_{t-\tau}}e^{i\frac{2\pi}{T}kt}\\
&=\frac{1}{N}\sum_{k=0}^{N-1}X_ke^{i\frac{2\pi}{T}k\tau} \overline{X_k}\\
&=\frac{1}{N}\sum_{k=0}^{N-1}\vert X_k\vert^{2} e^{i\frac{2\pi}{T}k\tau}\\
\end{split}
$$
Which is equivalent in that context to $R_{xx} = \mathcal{F}^{-1}(PSD)$
Case of discrete/continuous power signals (not square integrable/summable)
I imagine this context exists to study very long signals.
For power signals, we don't assume that $\int_{-\infty}^{+\infty}\vert f(t)\vert^{2}dt <+\infty $ (resp. $\sum_{k=-\infty}^{+\infty}\vert f(k T_{s})\vert^{2} <+\infty $ ). In fact they might have infinite energy, i.e :
$$\lim_{a,b \to +\infty}\int_{-a}^{b} \vert x(t)\vert^{2}dt = +\infty$$
resp.
$$\lim_{a,b \to +\infty}\sum_{k=-a}^{b} \vert x(kT_s)\vert^{2} = +\infty$$
which is very unfortunate, because then we can't apply the Fourier transform.
To get around that issue we consider the signals to be stochastic and, when the limit exists, we define the spectrum as :
$$S_{xx}(\omega) = \lim_{T\to +\infty}\frac{1}{T}E[\vert X_T(\omega)\vert^{2}]$$ with $$X_T(\omega) = \int_{-\frac{T}{2}}^{\frac{T}{2}} x(t)e^{i\omega t} dt$$
Note the normalized limit which suggests a hidden ergodic process assumption (however I am not sure of that).
Also note the $\frac{1}{T}$ normalization factor. As expected we talk about Power Spectral Density for power signals.
As we are in a stochastic context the autocorrelation is defined as :
$$R_{xx}(\tau) = E[x(t)\overline{x(t-\tau)}]$$
Now if we consider $x$ to be a wide-sense stationary signal (i.e the mean and the covariance of the signal are constant with respect to time) the Wiener-Khinchin theorem (a nice proof here) tells us that :
$$PSD(\omega) = \int_{-\infty}^{+\infty} R_{xx}{\tau}e^{-i\omega \tau}d\tau$$
I am pretty sure there is a version of the Wiener-Khinchin theorem for discrete power signals.
It is even possible to go further than wide-sense stationary signals and use quasi-stationary signals. However in that context I don't know if there is an extension of the Wiener-Khinchin theorem, everytime I read about it the spectrum was defined as the Fourier transform of the autocorrelation.
