Here is another partial answer, mostly about designing custom windows. I came up with this while doing something that (as I know now but didn't then) is called "windowing in the frequency domain." Then, after reading some original papers on windowing, I figured that it was probably the way that some windows were conceived in the first place, but I don't have any real background knowledge.
Start with a rectangular window and look at its Fourier transform, the sinc function:
Now, scale and (frequency-)shift two of them so the sides lobes tend to cancel each other out when added together:
(Result in green; sorry for the bad quality and useless legend.)
As you can see, side lobes are not only reduced in general, they also roll off much more quickly.
With "windowing in the frequency domain," the shifting and scaling is closely related to what actually happens in practice. But you are probably more interested in a time-domain representation, which is easy to obtain by applying appropriate formulas for the frequency shifts. It simplifies to $\cos(\pi t)$.
Repeat this process, and you get better and better roll-off, at the cost of a wider main lobe:
This simplifies to $(\cos(\pi t))^2$ in the time domain, which is precisely a Hann window. In general, repeating this $n$ times yields $(\cos(\pi t))^n$. $n=4$ is a special case of a Blackman window, and all even $n$ belong to the Blackman-Harris family.
Among Blackman-Harris windows, these yield the fastest side lobe roll-off. (I started writing down a proof of this, but didn't even finish it because how to calculate roll-off and other parameters seems to be common knowledge among experts.)
If you want to optimize something else than roll-off, you can start with a window that has sufficient roll-off, then do something similar to the above, but scale and shift in a different way (usually using three terms instead of two). This will keep the roll-off exactly the same, but it allows you to reduce the first side lobes, for example.
Hope this helps. Have fun.