Detecting & tracking an arbitrary object in a video

Question

I have a video from a fixed (stationary) camera. Sometimes there are slight movements in the frames (e.g. tree moving a bit in the wind). Other times, some might walk into the view of the camera, a bird might fly past, etc.

I could build a classifier to detect birds and people... but I don't know what will be picked up on camera. I am aware of models such as imageNet and YOLO, but I don't want to make assumptions (color, shape, size, type, etc.) about what might appear in the frames.

As such, I am looking for a way to detect arbitrary objects (i.e. anything moving relative to the rest of the image) in the video and track them. Classification can happen later.

What approaches exist for this problem? I will be writing in Python.

Answer 1

If all you want to do is to isolate* and track objects that have some contrast against the background, then the key phrase you want to search on is "object tracking", "video tracking", "video object tracking", or those three with "target" replacing "object".

You do not need neural networks to do this -- video trackers based on plain old rules-based image processing can work just fine.
I said "will" in the previous paragraph, then changed it to "can" because it depends on a lot of variables -- if you have one object with high contrast against a boring background it's easy. Trying to track multiple birds against a bunch of trees -- that could be hard. Doing it with neural nets won't necessarily make it easier.

If you start from first principles you do need to spend some time on it -- video tracking isn't easy, and you need to start with a good foundation in image processing and in dynamic systems theory (to do the 'tracking' part).

OpenCV has a Python interface, and a quick search on "OpenCV video object tracking" got me a lot of promising hits on what appears to be an object tracking module for OpenCV. I think that's where I'd start, if I were you. Especially if you're more interested in having a video tracker than making a video tracker, just using what OpenCV gives you should be a good start.

* "isolate" as opposed to "identify" or "classify" -- i.e., if your classification starts and ends at "that's a thing" then you're fine.