I have a sum of periodic signals that I am trying to untangle using time-frequency analysis. I seem to get wildly different results depending on the window length and shape. This is a problem because I want to develop an automated, and hopefully sequential algorithm to do the job.
Window functions have an inherent tradeoff between two of their frequency-domain properties:
Main lobe width: Any tapered window function will cause some "smearing" in the frequency domain. This is visualized by the width of the center lobe in the window function's frequency response. The wider the main lobe, the more difficult it is to resolve two tones that are close in frequency (if they are closer to one another than the main lobe width, they will tend to smear together). So ideally, you would like to have a window function that has a very narrow main lobe.
Maximum sidelobe height: Many window functions have frequency responses that consist of a single main lobe surrounded by repeated sidelobes that decay at some window-specific rate. The height of these sidelobes can make it difficult to resolve two tones that are separated in frequency, but differ greatly in amplitude. So ideally, you would like to have a window function that has very low sidelobes.
The problem: if you decrease the main lobe width of a window function, the sidelobes will grow, and vice versa. So, you need to strike an application-specific balance when choosing a window, based upon the distances in frequency and amplitude that you expect between your signals of interest. Given specific parameters of your system, it's possible to choose a window that (hopefully) meets your requirements.
As far as choosing the length of your window (which is equivalent to choosing the length of the DFT), you're best served with making your observation as long as possible within the constraints that your application might impose (e.g. latency requirements, how long the signals of interest can be considered stationary, computational resources, etc.). Your ability to resolve in frequency is directly proportional to the observation length (measured in time, not necessarily based on the FFT length, which can be zero-padded with no improvement in frequency resolution).
The window length should depend on the variation in frequency of your signal. You should adjust a window short enough to appproximately capture a constant spectra of your signal in that window.
If you want to know until what extent your signal is similar to a shape, you should use a Wavelet Transform (CWT).
For what it's worth, from a practical stand point, I found that Kaiser windows are quite useful. There is a single parameter that allows you to dial in the main lobe width vs. side lobe attenuation and in regards to most metrics a properly tweaked Kaiser window is as good or better than any of its cousins.
As a (very unscientific) rule of thumb you can determine the "beta" parameter as 0.133 times the desired side lobe attenuation in dB. This can be used to get a quick starting point and tweak from there.