When designing an FIR filter using the windowing method, how can one estimate the filter order ? It's obvious that the type of window and the transitions band's width has some effect on the order.
There are only heuristic formulas for estimating the filter order. For a Kaiser window (which is probably the most frequently used window for filter design) the required filter order can be estimated from 
where $A=-20\log\delta$ ($\delta$ is the maximum deviation from the desired response), and $\Delta\omega$ is the (smallest) transition bandwidth. This formula is of course only valid for the approximation of ideal frequency selective filters (low pass, high pass, etc.). This formula is implemented in Matlab's
 Oppenheim, Schafer, Buck, Discrete-Time Signal Processing, 2nd ed., p. 476