What you are trying to do is time-variant frequency domain processing. Unfortunately this is quite tricky and mathematically complicated.
The main problem you will have to solve is time-domain aliasing. The modification of the spectrum in each frame is basically a filter, which has a corresponding impulse response in time. If you do nothing to constrain the length of this filter you get the following situation:
Let's assume you have a frame length of $N$ and with zero padding an FFT size of $2N$. You create a spectral mask to implement your filtering. The inverse FFT of this mask is the impulse response of the filter. This will in general have a length of $2N$ samples since the mask has also $2N$ samples. The length of the convolution of signal and filter than will have a length of $3N-1$ samples. However your FFT frame is only $2N$ samples long. Since multiplication in the frequency domain implements circular convolution, your extra $N-1$ samples will wrap around and be added to the first half of your output frame. That's time domain aliasing and it sounds really, really bad.
There are multiple ways to deal with this, but there is no "one size fits all" solution and typically requires a lot of tweaking to get to the optimum set of trade offs for your specific requirements.
Things that can be tweaked:
- Frame size
- FFT length and/or amount of zero padding. Longer FFTs give the filter impulse response a better chance to decay a the edges.
- Hop size
- Double windowing: apply both an analysis and a reconstruction window
- Choice of window: makes sure that the product of both windows overlapped and summed as given by your frame and hop size actually adds to 1. Preferably to EXACTLY one (which a 50% overlapped Hanning window doesn't).
- Frequency domain smoothing. The narrower the frequency domain features are, the longer the impulse response will be (in the sense of more energy far away from the center). There are multiple ways of doing this. Either apply a sliding window in the frequency domain or simply do an inverse FFT, window the impulse response making sure that N samples are zero and do an FFT again. Of course this will somewhat change the frequency response of your filter. It's a trade-off.
- Consider making the filter minimum phase instead of zero phase. That makes it causal and also tends to reduce the amount of aliasing since the impulse response has more time to decay.
- Slow down how fast any frequency bin can move over time. Apply a first order lowpass to the "gain vs. time" function.