How the select the representative frame when detecting vehicles

Question

I am detecting vehicles from the video (taken with a camera), and the vehicle detection part works fine.

My problem is that I should always extract only one representative frame per vehicle, and not the whole sequence.

As an example, if I have a few seconds of video (and possibly multiple vehicles), my approach currently extracts multiple frames (for every vehicle). Instead, as an output I would like one representative frame in which the vehicle is whole, i.e. well visible.

I have already implemented an entropy technique on it which makes it better but am still getting too much frames of the same vehicle. Could you suggest a technique which I can use to extract only one frame which contains the whole vehicle and ignore all other frames which contain that (same) vehicle.

I use the following entropy technique in my problem

double entropy = 0.0;
for (int i=0; i<histNorm.rows; i++)
{
    float binEntry = histNorm.at<float>(i,0);
    if (binEntry != 0.0)
        entropy -= binEntry * log2(binEntry);
}

Answer 1

I can not give you a full approach, but I might have an interesting suggestion. As soon as you detect a vehicle, start tracking it.

For every detection, you could "remember" the first and last frame of the video where you tracked that vehicle. You associate every vehicle detection with a series of consecutive frames. The beginning frame would be the one when the detection was made, and the ending would be the first frame when the tracking algorithm fails.

Then, you could select only a single frame for every distinct vehicle you tracked during your video sequence, with more or less simple approaches:

• select the middle frame of the tracking sequence, as there is a high chance that in the middle of the tracking, the vehicle was well visible
• do matching between several points in the sequence, or just base your decision on the number of features you can still track at a certain frame: selecting a frame with a large number of trackable/matchable features is probably a good choice
• remember the "quality" of your detection for every frame in the sequence, and select the frame where the detection quality was the best
• depending on the preferences, be mindful to either select one representative frame if the sequences for two vehicles overlap, or purposefully select different frames (possibly from non-overlapping parts of the sequences).