In your plot of nb vs. image it appears that there is some high-frequency "noise" (no surprise there), and that the windowed mean is surprisingly stable. For the purposes of this answer I am going to assume that the high-frequency noise is fairly representative of the general case, but that there will also be some slow-to-medium trends as well. It seems like there would have to be if the images are changing. The frame rate should generally be faster than the changes, though, so the trends should be relatively slow.
Normally I would suggest a low-frequency filter to get rid of the high-frequency "noise", but in this case I don't think that would be wise. For any given image the next image is very likely to be highly correlated, so I think it makes sense to adjust somewhat if an image has a large change in nb. I suggest two different solutions.
1) Use a simple moving average of the past n nb values. Performance is obviously important in this application, and a moving average is computationally cheap. A moving average gives some stability, which is important given the high-frequency noise, but also is responsive to changes in the data. You can trade responsiveness for stability by decreasing or increasing n.
Once you have the nb moving average you use that to produce the "error" which is your target nb minus the moving average nb. Thus, if the average nb is too low, the error is positive, if the average nb is too high, the error is negative. You use the error to adjust the threshold. Typically the new threshold would be
$threshold = threshold - \alpha * error$
$\alpha$ needs to be determined either by analysis of your problem or by empirically testing. You substract the error result from the threshold because a positive error means you need nb to go up, which means that you need the threshold to go down.
2) Use a PID controller. The PID controller would produce a threshold value for every image, and use the target nb vs. actual nb error (similar to the moving average error except it's an instantaneous error, not an average) to update its threshold for the next image.
The "integral" portion of the controller will give you stability, while the proportional/derivative part will give you responsiveness. You would need to make sure that the loop itself is stable, and test the constants on different videos, but this would be a very powerful way of adjusting the threshold. It also offers the possibility of making your app smarter by changing the loop constants dynamically, if needed.
Solution 1 is easier to implement and is a little cheaper computationally, solution 2 is more powerful but also more complicated. I would see if the moving average is good enough. If it is, great, if not, implement the PID controller.
EDIT: Tried to make the two solutions a little clearer.