From Karhunen–Loève theorem, when talking about stochastic processes:
In the theory of stochastic processes, the Karhunen–Loève theorem
(named after Kari Karhunen and Michel Loève), also known as the
Kosambi–Karhunen–Loève theorem is a representation of a
stochastic process as an infinite linear combination of orthogonal
functions, analogous to a Fourier series representation of a function
on a bounded interval.
Basically, K.-L. yields an adaptive representation, while Fourier provides a fixed representation with sinusoidal functions.
There are many flavors of Fourier tools, so since you are talking about "fast" Fourier and PCA, I assume for now you are dealing with discrete data, thus the empirical version of the Karhunen–Loève transform (PCA) and the discrete Fourier transform (DFT). The idea of a continuous set of frequencies does not fit gracefully here.
Both PCA et discrete Fourier can be cast into linear and orthogonal transforms, so they are applicable to any data, and may preserve all information.
From Principal Component Analysis (p. 44 sq.), you can find that you can derive DFT bases from PCA, when you are studying a process that follows a "correlated" Markov model.
Finally, when you talk about infinite sines with infinitesimal bandwidth, you are more in the context of continuous functions (or distributions) and standard Fourier analysis, which might be a cause for confusion. A solid book on the topic could help, like Mathematical principles of signal processing: Fourier and wavelet analysis, by P. Brémaud.