The "matrix factorization" technique will NOT help you do your job! The paper referred by @mirror2image is about the background subtraction but NOT based on "matrix factorization".
Using running video to detect moving objects (be it human or vehicles) is an active area of research.
As a basic principle the system estimates a typical static background by sampling over multiple pictures and take a difference of energy between incoming image to the background. If the energy is significant the pixel is classified as foreground. Such set of foreground tells you if there is an entry of the object in the system.
The best reference to your research paper (and also relatively simpler if you want to really implement) would be - W4 System find it here and see Picardi paper here as a more detailed survey for other techniques in the system.
There are many challenges that applies to the problem:
Presence of noise creates the issues of major ambiguity. The approach here is to apply efficient temporal filtering and considering variance of noise to make it immune to threshold.
Presence of shadow creates ambiguity of neither being a foreground nor. There are papers who model the color vs. intensity distinction to distinguish shadow vs. real foreground.
The background can be complex like waving trees or sea etc.
The background can have slow or sudden variation of lighting where earlier "learned" background is then adapted to the new one.
One of the most referred landmark paper is called Wall flower algorithm shows the best way to combine various such scenarios to produce robust moving object detection.