Assuming we're looking at plots of the time domain I&Q, I'll guess that your multiplication operation is of limited bit width and is causing the quantization.
If you look at the Hamming Window in the time domain, only the central sample(s) are weighted at 1.0, with all the other weights down along a relatively smooth curve. https://en.wikipedia.org/wiki/File:Window_function_and_frequency_response_-Hamming(alpha_%3D_0.53836).svg
If you look at the Tukey Window most of the samples are weighted at 1.0, with some samples on the edges of the window weighted along a curve. https://en.wikipedia.org/wiki/File:Window_function_and_frequency_response_-Tukey(alpha_%3D_0.5).svg