BOTTOM LINE UP FRONT: I think the exponential decay growth in $\left<|x(t)|^2\right>$ can be shown in the frequency domain only if the "boundary terms" are nonzero when we compute the Fourier transform of $dx(t)$ from the original SDE.
I provide only a start in the work below.
Since these processes seem like they could possibly be complex-valued, I will consider
$\left<x(t)\overline{x(t')}\right>$ and
$\left<y(t)\overline{y(t')}\right>$, where the
$\texttt{overline}$ indicates complex conjugation.
I will write out some steps, because I need to see some things that you did not write explicitly.
\begin{equation}
dx(t) = ax(t)dt + by(t)dt.
\end{equation}
Fourier transform:
\begin{equation}
\begin{split}
\int e^{-i\omega t}dx(t) &=~ \int e^{-i\omega t}ax(t)dt + \int e^{-i\omega t} by(t)dt\\
\int\left[d\left(e^{-i\omega t}x(t)\right) - x(t)d(e^{-i\omega t})\right] &=~ a\widehat{x}(\omega) + b\widehat{y}(\omega)\\
\underbrace{\int d\left(e^{-i\omega t}x(t)\right)}_{\textrm{Boundary terms: $-c$, a constant}} - \int x(t)(-i\omega)e^{-i\omega t}dt &=~ a\widehat{x}(\omega) + b\widehat{y}(\omega)\\
i\omega\widehat{x}(\omega) &=~ a\widehat{x}(\omega) + b\widehat{y}(\omega) + c
\end{split}
\end{equation}
Result:
\begin{equation}
\widehat{x}(\omega) = \frac{b\widehat{y}(\omega) + c}{i\omega - a}
\end{equation}
\begin{equation}
\begin{split}
x(t) &=~ \frac{1}{2\pi}\int \widehat{x}(\omega) e^{it\omega}d\omega\\
&=~ \frac{1}{2\pi}\int \frac{b\widehat{y}(\omega) + c}{i\omega - a} e^{it\omega}d\omega\\
& \\
x(t') &=~ \frac{1}{2\pi}\int \widehat{x}(\omega') e^{it'\omega'}d\omega'\\
&=~ \frac{1}{2\pi}\int \frac{b\widehat{y}(\omega') + c}{i\omega' - a} e^{it'\omega'}d\omega'
\end{split}
\end{equation}
\begin{equation}
\end{equation}
\begin{equation}
\begin{split}
\left<x(t)\overline{x(t')}\right> &=~ \left<\frac{1}{2\pi}\int \frac{b\widehat{y}(\omega) + c}{i\omega - a} e^{it\omega}d\omega\frac{1}{2\pi}\int \frac{\overline{b\widehat{y}(\omega') + c}}{-i\omega' - a} e^{-it'\omega'}d\omega'\right>\\
&=~ \frac{1}{4\pi^2}\int\int\frac{b^2\left<\widehat{y}(\omega)\overline{\widehat{y}(\omega')}\right> + bc\left<\overline{\widehat{y}(\omega')}\right> + b\overline{c}\left<\widehat{y}(\omega)\right> + |c|^2}{(i\omega - a)(-i\omega' - a)}e^{it\omega}e^{-it'\omega'}d\omega d\omega'
\end{split}
\end{equation}
\begin{equation}
\begin{split}
\left<\widehat{y}(\omega)\overline{\widehat{y}(\omega')}\right>
&=~
\left<\int y(\tau)e^{-i\omega\tau}d\tau\overline{\int y(\tau')e^{-i\omega'\tau'}d\tau'}\right>\\
&=~
\int\int\left<y(\tau)\overline{y(\tau')}\right>e^{-i\omega\tau}e^{i\omega'\tau'}d\tau d\tau'\\
&=~ \int\int\delta(\tau-\tau')e^{-i\omega\tau}e^{i\omega'\tau'}d\tau d\tau'\\
&=~ \int e^{-i(\omega - \omega')\tau}d\tau,
\end{split}
\end{equation}
after integrating in
$\tau'$. This integral does not converge, but
in the sense of distributions,
\begin{equation}
\int e^{-i(\omega - \omega')\tau}d\tau = 2\pi\delta(\omega - \omega').
\end{equation}
What to do about the cross-terms? We must consider the integrals in
$\omega$ and
$\omega'$ separately for those. Consider the one for
$\omega$:
\begin{equation}
\int\frac{\left<\widehat{y}(\omega)\right>}{i\omega - a}e^{i t\omega}d\omega
\end{equation}
If we assume analyticity of
$\left<\widehat{y}(\omega)\right>$ (as a function of complex-valued
$\omega$) and assume that it decays rapidly enough for large
$|\omega|$ in the complex-
$\omega$ plane, then we can appeal to
contour integration and
Cauchy's integral formula. The only pole in the complex-
$\omega$ plane is found at
$\omega = -ia$, where I assume
$a$ is real. The integral is
\begin{equation}
\int\frac{\left<\widehat{y}(\omega)\right>}{i\omega - a}e^{i t\omega}d\omega
~=~
\lim_{N\to\infty}\oint_{\gamma_N}\frac{\left<\widehat{y}(\omega)\right>}{i\omega - a}e^{i t\omega}d\omega
~=~ 2\pi i\left<\widehat{y}(-ia)\right>e^{at},
\end{equation}
where
$\gamma_N$ is a path that includes the real interval
$[-N,N]$ and a semi-circular arc connecting
$N$ and
$-N$. You can find examples of this path in almost any undergraduate complex analysis book. We assume that the integral on this arc decays to zero as
$N\to\infty$.
I think this is how we pick up the exponentially
decaying growing behavior in
$\left<|x(t)|^2\right>$.
