staying faithful to the subject "removing window effect in freq domain via convolution" (despite the OP perhaps wanted to achieve something else or something similar), I feel to add my comment having personal experience with this specific topic.
Often I have the necessity to remove a Hann window in frequency domain, working in a STFT framework which uses Hann windowed frames as default, to carry advanced spectral processing where the input spectrum is expected as NON windowed instead (eg. overlap-save convolution or filtering).
In a word: yes you can. Despite mathematically removing a window (either in time or frequency domain) implies reconstructing data which are lost forever, in practice you can with a likely minimal loss.
Let' s take a Hann (rised cosine) window. Its time domain formula is y=(1-cos(pi*x))/2 with x ranging from zero to and excluding one thru the frame. Its corresponding frequency domain representation is bin0=(0.5,0i), bin1=(-0.5,0i). To remove its effect in time domain you may simply want to divide the signal by the above said window function. To do the same in frequency domain, you can simply convolve the spectrum to be un-windowed with the spectrum of the reciprocal of said function. Since this function is zero at both ends (actually it is mathematically zero only at the first point, unless of rounding errors), to avoid infinity you can simply exchange infinity with a large value like 10000 or so. The result of such convolution is the un-windowed spectrum. By converting it to back to time domain, you can of course detect a slight degradation affecting the very first and very last points, which is expectable, but in practice the process can be done.
Perhaps you can' t remove a rectangular window though, because the amount of data lost by multiplying large areas of the signal by zero is impossible to recover in theory. But I think this depends on the spectrum content. For example, if it is the spectrum of a plain sinusoid, by removing the rectangular window pattern by convolving with the spectrum of a function which is a high value where the rectangle was zero, and one where it was one (i.e its reciprocal), you can still perhaps obtain (substantially) the spectrum of a sinusoid to reconstruct the whole signal.
UPDATE: I am adding this in case somebody interested may end up here in the future by a google search. Also, this is something very specific about advanced spectral manipulation which I came up with by my efforts and is NOT covered anywhere as far as I know.
A Hann window CAN BE REMOVED WITH SOME VERY SIMPLE OPERATIONS IN FREQUENCY DOMAIN losing only the first sample, without resorting to a convolution or to conversion to time domain to divide by the reciprocal window function, both processes which involve unnecessary loss of precision.
As a premise, from a mere information perspective, when you apply a Hann window
(1-cos(x*PI2/z))/2 0<=x<z
to a block of z samples, you just lose ONE sample: the first sample, because only at x=0 the window function is zero, so that sample is lost forever.
Therefore, if you accept that you will never be able to recover the first sample, the following pseudocode will perfectly remove a previously applied Hann window in frequency domain.
S[] is the half-complex spectrum (positive frequencies only, from bin zero = DC to bins z/2 = Nyquist) to "de-window".
s and k are real variables.
M,P and TMP are complex variables.
z is the frame size.
nb is the number of bins which is equal to z/2+1
s=0
k=1
iterate cnt from nb-2 to 1
{
s = s + im(S[cnt]) * k
k=k+1
}
im(S[0]) = -2 * s / z
P=0
M=0
iterate cnt from 0 to nb-1
{
TMP = S[cnt] * 4
P = P + M
S[cnt] = P
M = M-TMP
}
S[0] = S[0]/2
S[nb-1] = S[nb-1]/2
The first time-domain sample will remain undefined as stated previously. If needed, its value can be discovered and changed easily still remaining in frequency domain, keeping in mind that the value of the first time domain sample is equal to the sum of all the real parts of the spectrum bins.
Such a trick can only be applied to a Hann window since its spectrum is simply made of two bins, a DC and an unitary frequency component