Rayleigh's theorem says that if $x(t)$ is a _square-integrable signal (meaning that $\int_{-\infty}^\infty |x(t)|^2 \,\mathrm dt$, which is called the energy of the signal, is finite) then
$$\int_{-\infty}^\infty |x(t)|^2 \,\mathrm dt = \int_{-\infty}^\infty |X(f)|^2 \,\mathrm df \tag{1}$$ where $X(f)$ is the Fourier transform
$$X(f) = \int_{-\infty}^\infty x(t)\exp(-j2\pi ft) \,\mathrm dt\tag{2}$$
of $x(t)$. The finite-energy assumption suffices to assure us that $X(f)$ is properly defined, that is, the integral in $(2)$ exists (doesn't diverge or otherwise fail to converge), and everything is hunky-dory.
Now, if we perversely insist on using Raylegh's theorem on signals such as $\cos(t), -\infty < t < \infty$ which are not square-integrable, we have the problem of dealing with the left side of $(1)$ since the integral there is
\begin{align}
\int_{-\infty}^\infty |x(t)|^2 \,\mathrm dt &=
\lim_{T_1\to -\infty}\lim_{T_2\to \infty}\int_{T_1}^{T_2}|x(t)|^2 \,\mathrm dt\\
&= \lim_{T_1\to -\infty}\lim_{T_2\to \infty}\int_{T_1}^{T_2}\cos^2(t) \,\mathrm dt\\
&= \lim_{T_1\to -\infty}\lim_{T_2\to \infty}\int_{T_1}^{T_2}\frac{1+\cos(2t)+1}{2} \,\mathrm dt\\
&=\lim_{T_1\to -\infty}\lim_{T_2\to \infty} \left.\frac t2+\frac{\sin(2t)}{4} \right|_{T_1}^{T_2}\\
&= \lim_{T_1\to -\infty}\lim_{T_2\to \infty} \frac{T_2-T_1}{2} + \frac{\sin(2T_2)-\sin(2T_1)}{4}
\end{align} in which the inner limit doesn't exist since the limitand diverges to $\infty$ as $T_2 \to \infty$. So, at best we can say with a straight face (after all, we are DSP engineers, not mathematicians!) that the left side of $(1)$ has "value" $\infty$. But matters are even worse for the right side of $(1)$ because $X(f)$ is even properly defined! Note that
\begin{align}X\left(\frac{1}{2\pi}\right) &= \int_{-\infty}^\infty x(t)\exp(-jt) \,\mathrm dt\\
&= \int_{-\infty}^\infty \frac{\exp(jt)+\exp(-jt)}{2}\exp(-jt) \,\mathrm dt\\
&= \int_{-\infty}^\infty \frac{1+\exp(-2jt)}{2} \,\mathrm dt\\
\end{align}
which diverges to $\infty$ (as does the integral for $X\left(\frac{1}{2\pi}\right)$) while for $f \neq \frac{1}{2\pi}$,
$$X(f) = \int_{-\infty}^\infty \frac{\exp(jt)+\exp(-jt)}{2}\exp(-j2\pi ft) \,\mathrm dt$$ doesn't converge in that the limit
$$\lim_{T_1\to -\infty}\lim_{T_2\to \infty}\int_{T_1}^{T_2}\frac{\exp(jt)+\exp(-jt)}{2}\exp(-j2\pi ft) \,\mathrm dt$$ doesn't exist because the integral oscillates between two fixed values regardless of how large $T_1$ or $T_2$ gets. In short, we can't even get started on the right side of $(1)$.
