I am able to find out the velocities of pixels using the Lucas-Kanade method. Now I want to use it for tracking. I am using a Kalman filter taking $x$, $y$, $v_x$ and $v_y$ ($(x,y)$ is position of a particular object pixel and $(v_x,v_y)$ is velocity of a pixel) as state variables of the Kalman filter. Now the problem I am facing is how will I get to select the pixels that are representing the object to be tracked?


2 Answers 2


Tracking objects can be simple to complex depending on what type of background you have, whether the background is static or moving, whether the object is clearly distinguishable or can share similar properties with background there by blending in the background, and whether we have unique object at hand or we are tracking multiple objects which may cross the trajectories.

The good news is, the solution to all of these exists. Depending on the nature of task at hand, you may choose simple to complex solution.

What exactly we need?

As you implement any tracking algorithm, you have an estimate of where the new position of the object is likely be. Now you need to verify that indeed the object is at the same position or different. The difference is the error that you feedback to your model. For this you need to identify/segment the object in the vicinity where your projected value. In other words you need to detect that object with in a given window.

If the object's appearance or key behaviour doesn't change quite drastically you can capture feature vectors from original reference and compare the same around the vicinity where your target is expected to be. This is search of feature vector within a window based on some distance criteria.

So the question boils down to - what should be the feature vector and what should be the distance matrix?

There are many possible answers. Writing down some of them (from simple to complex) with remark on when they might work.

  1. Color based features: This include identifying mean or dominant color, color histogram over a local window etc. If the object color is unique and doesn't blend with background then you can identify object with this manner. See [Ref 1] and [Ref 2] below

  2. Texture based features: There is a whole range of features here. The famous Gabor filter banks, then there is co-occurrence matrix, you can take gradient of images or find edges and have edge histograms over local regions, or you can identify Local binary patterns near the regions. See [Ref 3]. Also, many approaches combines Color and Texture information for more effectiveness. See [Ref 4] and [Ref 5]

  3. Shape based features: Shape based technique works when the object you are tracking has a particular shapes that distinguishes from other objects and from background. This includes many techniques. See [Ref 6], [Ref 7] and [Ref 8] below.

More complex features:

If the above features doesn't work out, specifically if the object is complex and the background as well -then we can use these features:

  1. SIFT - The scale-invariant feature transform allows to compute key points which characterises the object and hence you can compare the same object from past frame against a new frame. See [Ref 9]

  2. SURF, FAST, ORB: - If SIFT turns out to be blocked because of patent issue, and due to it's speed, then you can use it's alternatives which are SURF, FAST, ORB. See [Ref 10] for a good comparison.

  3. HoG and Haar: Finally you can use HoG - which is a good generic object detection framework. [Ref 11] and Haar based classifiers when the problems are really complex. [Ref 12]

I am not prescribing any specific solution, and none of the above could be your direct solution - yet this will help you thinking through the problem at hand. You can pick whichever feature works for you irrespective of tracking methods that are used in those respective papers.

Go through the paper Good Features to Track thoroughly and it will give you much better perspective on the same.

What about distance criteria?

For each different feature typically different distance measure should make sense. But, distance matrix is usually dependent on your feature vector itself.

[Ref 1]: Object Tracking Based on Color Information Employing Particle Filter Algorithm Budi Sugandi, Hyoungseop Kim et. al.

[Ref 2]: Applying a New Spatial Color Histogram in Mean-Shift Based Tracking Algorithm Dong Xu, Yimin Wang, and Jinwen An

[Ref 3]: Texture Features for Browsing and Retrieval of Image Data B.S. Manjunathi and W.Y. Ma


[Ref 5]: Multi-Object Tracking Using Color, Texture and Motion Valtteri Takala and Matti Pietikainen

[Ref 6] : A Survey on Feature Extraction Techniques for Shape based Object Recognition Mitisha Patel, Purvi Tandel


[Ref 8]: An Evaluation of Local Shape-Based Features for Pedestrian Detection Edgar Seemann, Bastian Leibe, Krystian Mikolajczyk, Bernt Schiele

[Ref 9]: Object Tracking using SIFT Features and Mean Shift Huiyu Zhou, Yuan Yuan, and Chunmei Shi

[Ref 10]: Image Matching Using SIFT, SURF, BRIEF and ORB: Performance Comparison for Distorted Images Ebrahim Karami, Siva Prasad, and Mohamed Shehata

[Ref 11]: Part-based Multiple-Person Tracking with Partial Occlusion Handling Guang Shu, Afshin Dehghan et. al.

[Ref 12]: Real-time hand tracking using a set of cooperative classifiers based on Haar-like features Andre L. C. Barczak and Farhad Dadgostar


If the only thing that moves is the object, you might threshold the magnitudes of the flow vectors. Magnitudes which are greater than some threshold simply represent the optical flow components of the moving object, and nothing else.

  • $\begingroup$ ok. it means this method of tracking will be suitable for the video scenes where background should be static $\endgroup$
    – Amit_DSP
    Jan 8, 2014 at 9:06
  • $\begingroup$ Yes, as you are only using motion information, and no other means of object detection. If you incorporate some texture knowledge (such as detection by SIFT), you might then use it to narrow down your area of tracking to the object boundaries only. Also check out background subtraction. OpenCV has many methods. There are other more complex ways to do it as well. $\endgroup$ Jan 8, 2014 at 9:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.