However, since I have taken the modulus, it must be impossible to go from spec back to audio correct?
You need to resort to some approximations like the Griffin-Lim algorithm or various vocoders (WaveNet,WaveGlow,etc.).
So does that mean that librosa.istft does NOT convert a spectrogram to a wav file?
It's a bit more complicated than that. An FFT ideally ...
You may want to try training your model on 2D arrays containing entire datasets (all 75 MFCC frames of each audio file). You can try recurrent or convolutional early layers in your DNN to handle the 2 dimensional input shape. You may need a far larger amount of training data.