There is no one perfect order "N". You can always get a better (or at least as good) filter by increasing N, at the cost of more computational load. So usually the question is "what is the lowest N that will be barely good enough?"
A relatively easy way to determine that is experimentally. Guess what you think the optimal N will be, design a filter, see if it meets the filter criteria or not. If it does with large margins, reduce N by a lot. If it barely makes it either stop or reduce N by a little. If it doesn't make the criteria, increase N. Rinse and repeat.
frederic harris (for some reason he likes to have his name uncapitalized) gave the following formula in his book "Multirate Signal Processing for Communication Systems":
$ N \approx \frac{f_s}{\Delta f} \frac{Atten(dB) - 8}{14}$
where $N$ is the estimated filter order, $f_s$ is the sample frequency, $\Delta f$ is the transition band width, and $Atten(dB)$ is the amount of attenuation you want in the stop band, in dB. I would not use the formula to get the optimal $N$, rather I would use it as a starting place for the iterative process described above.