I have a two periodic signals $x(t)$ and $y(t)$ which are related to each other by a periodic modulation function $\alpha(t)$: $$y(t) = \alpha(t) x(t)$$ If $x(t)$ is a sinusoid, is it possible to determine $\alpha(t)$ from the spectrum of $y(t)$? How might I do this?
The Fourier transform is reversible, so the same amount of information is contained in the spectrum (not in the power spectrum, that doesn't contain the phase, and hence, only half of the information)! Just convert your spectrum back to time domain.
However, if you just got the power spectrum plot of a signal, you'll find that many modulations have "characteristic" spectral shapes. With a bit of practice, you can tell binary GMSK from an 4-FSK pretty easily.
Now, in general, the problem of detecting a signal and its modulation is pretty hard – governments pay loads of money for well-working signal classificators, because in many cases, you're looking at a very weak $y(t)$ within a mixture of hundreds, and it's still hard to guess whether you're looking at an 16-QAM or an 8-PSK if you don't have timing information.
Thus: without a lot of additional info on what you're looking at, guessing the modulation class of $\alpha$ from $|Y(f)|$ is very hard. And it's harder to actually estimate $\alpha(t)$ from that.