So I'll get this answer started by saying again that what you are describing reminds me a lot of Singular Spectrum Analysis. The process is pretty much exactly how you describe: you have some observed signal, $Y[n]$, which is some sort of mixture, and we take $Y[n]$ and create a Hankel matrix from it using some desired frame length M (this will effectively determine your sub-space resolution).
Now that we have our Hankel matrix, we can either compute $R_Y$ directly and then do an eigendecomposition, or we could go with a Singular Value Decomposition of the trajectory matrix. You'll find them to essentially be equivalent if we use the left singular vectors; either method is perfectly valid.
So once we get our matrix of eigenvalues, $U$, we need to project them using your typical Karhunen-Loeve method to get what we'll call principal components (this is sort of classic SSA notation, I dislike the naming but I digress):
$P = Y^H U$.
Now that we're here, we can use the singular values to determine amplitude of the signals themselves, and the principal components essentially determine the "parts" of the signal, i.e. $X[n]$ and $\epsilon[n]$.
Let's say we have N principal components, and we know that we only "need" the first one: from our principal components, we perform what is typically called Eigentriple grouping via the following multiplication:
$C = UP^H$
This new matrix, C, is what we call the reconstructive component(s) of $Y$, and is based on which principal components we selected. We're not quite done yet, becuase $C$ is the same dimension as our Hankel matrix, $Y$, so do get back to our signal, we'll need an additional step. To reconstruct the series itself, we'll perform what's called diagonal averaging (a bit of a misnomer I think). The goal here is that we'll finally map back to a single signal, let's call it $Y_out[n]$. It's in this step that we'll be able to utilize that one-to-one relationship of the Hankel to the series to extract out signals. This step is a bit long winded, so I've omitted it for brevity, but you can consult Wikipedia or any of the many papers/texts on SSA for more information.
So now that we've briefly gone through it, you might find yourself asking "when will this method actually work". SSA, like anything, is a tool and it has its limits. From a signal processing standpoint, it's great at determining individual signals which are fairly narrowband (i.e. they don't cover the entire spectrum) and are "separable". In the context of SSA, "separable" means that the signals are non-overlapping in frequency. For a simple example, let's say I have two sinusoids, one at 50Hz, and another at 500Hz. SSA will be able to easily separate the two signals, and we will be able to isolate the signals pretty easily; we would call these signals separable. However, let's say I change the second signal to be 51Hz instead of 500Hz. In this case, SSA will struggle, as the signals are very, very close to one another; we would call these signals non-separable.
How do we determine which signals we care about? That's the difficulty with this method. If you have some appreciable signal-to-noise ratio (SNR), you can perhaps use that to your advantage and identify which signals are of interest and which signals are just purely noise by inspecting the eigenspectrum; true signals as represented by the principal components will likely be associated with large eigenvalues, whereas noise will be relatively low power, and spread.
So, in short, if you have some time series observation, you can use SSA to attempt to discern individual signals/trends. It's a really powerful tool, and as long as the signals are separable, you should have some reasonable success. Hope that helps!