There are three main characteristics of the filters that are affected by the filter length
- Passband Ripple
- Transition Width (from passband to stop band, or stop band to passband)
- Stopband attenuation and roll-off of sidelobes
Unfortunately, there aren't any formulas for how the length of the window and the type of window used affect all of these 3 aspects of the filter. So you may need to increase the filter length to meet your passband ripple requirement, but it may not meet your required attenuation and/or roll-off requirement, so you'll need an even longer filter. This will need to be done by trial and error.
I believe there were some articles in the IEEE that gave some length formulas when using the Kaiser window, but I don't have the references at hand.
Even the length formulas for using the Parks–McClellan design (Remez exchange) are just a heuristic and were developed after performing a lot of experiments. Even these formulas can fail, often when a corner frequency is near 0 Hz or Fs/2. Most of the time these formulas are a good first estimate you may need a few extra coefficients to meet your exact requirements. Using the estimate of the filter order for the PM algorithm is a good first estimate for the minimum length of your filter using a windowing filter design technique.