However, since I have taken the modulus, it must be impossible to go from spec back to audio correct?
You need to resort to some approximations like the Griffin-Lim algorithm or various vocoders (WaveNet,WaveGlow,etc.).
So does that mean that librosa.istft does NOT convert a spectrogram to a wav file?
It's a bit more complicated than that. An FFT ideally ...