I want to calculate the loudness of a given sound over time (like, in a graph) from its spectrogram. Can I do this from the spectrogram or is it better to do from the raw sound?
Yes, in fact it's not a bad idea at all to use some kind of spectrogram representation to calculate a loudness envelope. Depending on what exactly you need you can either just sum the spectral magnitude over all frequencies at a given time, or first apply some kind of perception based weighting functions to make your loudness measure more related to actual auditory perception.
So if you give some more detail about the application you have in mind you will likely receive a more accurate answer.