Video: Detecting a digital timer and when it ticks

Question

Input: A video with a digital timer, possibly several hours long, such as this video. The timer might use any font or color, and might be located anywhere on the video, but it probably won't fill the entire screen.

Output: A list of time indices when the timer starts and stops, it might start and stop multiple times. For example:

Timer 1: Starts off paused, starts at 0h20m32s, stops at 2h30m04s, ends up being paused.
Timer 2: Starts off ticking, stops at 0h04m00s, starts at 0h06m38s, stops at 0h32m01s, end up being paused.

Possible frames from different input video files:

How might this be achieved? In particular, what algorithms could be used to categorize the data? How would could you detect a timer that might be everywhere, of any color or font size?

Answer 1

To OCR or not?

If the timer is about $h$ pixels height, you could segment each frame into lines of height $h/2$, run an OCR on each line, and scan the result with a regexp for anything that looks like a timer, eg. \d+[\.:,;-]\d+[\.:,;-]. This gives you $x=time_{real}$ vs. $y=time_{video}$ data pairs. Sure, most of them are wrong, but imposing the restriction $y=1*x+n$ (unless the video speed was changed) eliminates most of them. Just group all data points according to whether they lie on a line of slope m=1.

However, this works only if there are at least some good data points. If we cannot make any assumptions about the color, font, or size (or even script!), OCR doesn't perform well.

Instead, making use of some intrinsic properties of all timers works much better and is agnostic to the exact font or script used. Let us make the following basic assumptions about a timer:

• overlayed on a static background, ie. there's at least one bounding box that contains only the timer and no other changing objects
• it remains on a fixed position for the entire duration of the video
• a ticking timer is changing periodically
• it contains a seconds digit with a frequency of 0.1 Hz
• it is not animated, ie. the seconds digit remains static and changes only every second

We can use these assumptions to recognize the timer automatically.

Downsampling

Scanning every frame, pixel, and every color channel is a lot of work, takes time, and also completely unnecessary. Let's say we've got a 2 hour video, 720p60, yuv420p. That's 2*3600 seconds * 60fps * 1280*720 pixels/frame * 3 channels/pixel =~ 1200 billion data values to examine.

• Grayscale. The color channels are down-sampled already anyway.
• 1-0.1 fps is usually enough, the timer's frequency is only 0.1Hz.
• A resolution of 320x180 pixels is more than enough, unless the timer had been really small to begin with, that still leaves us with ~20-60 pixels to analyze.

This reduces the number of data points by about 3*600*4*4~=30000, for a total of about 40 million data points.

Locating the timer(s)

This is the hard part.

We employ several methods to check each pixel whether it might belong to the timer, then we combine them to larger areas. The video is split into intervals of length dt, and for each interval,n frames get analyzed. 60 frames per interval works pretty nicely.

• Check for changes. At the very least, the timer must be changing constantly. We could use the standard deviation of each pixel, but I found that randomizing the sample order and taking the average difference between adjacent pixels gives better results sometimes. The standard deviation tends to be lower.

• Simple check for 10-seconds periodicity. Group pixels into those that are pixels 10 seconds apart of each other, take the average difference for each group, then average all groups. Basically the auto-correlation for $\tau=10s$ This is simple, fast, and tends to work well.

• Compute the full auto-correlation (via FFT for speed). Sounds better, but you need to find a way to analyze it. What I found works is checking at 10 seconds, 20 seconds etc. whether there is a peak (higher value than at nearby samples), and checking that there is no higher value between two adjacent peaks.

• Check step-behavior. Take samples every 0.2 or 0.1 seconds and check when the pixel changes. As each digit remains on screen 1 second, it should do so every 5 or 10 frames. Some digits may look alike, but the difference between two changes should be an integer multiple of 5 or 10.

I found that for the most part, combing the first two methods yields quick, fast, and accurate results.

Now we've got a mask of pixels that possibly belong to the timer. Apply a close operation on them (with a threshold of ~0.5), label connected areas, compute the bounding boxes, eliminate very small or very large ones, and areas that haven't got many pixels that were classified as belonging to the timer.

After doing this for all intervals, group bounding boxes together that are close to each other, eliminate groups with only a few members, and take only those groups with the most members. These are most likely the bounding boxes for the seconds digit of the timer. Average each group and return the result.

(Left: Mask, Right: Image with Mask)

(Top: Raw image, 2nd: checked for changes, 3rd: simple check for 10s periodicity, 4th: auto correlation diagram analyzed, 5th: all masks combinded, Bottom: Bounding box computed)

Note that the close step is not shown above.

Scanning for activity (start/stop)

Now that we found a timer, or rather the seconds digit, we scan the video to analyze when it is ticking and when not. If it is ticking, the digit will keep changing.

• extract n frames at an interval of dt , crop to the timer rectangular area
• for each pixel, take the squared difference between adjacent frames, and average over all pixels
• threshold to 0 (stopped) or 1 (ticking), a good value is about 1.0-1.5
• apply a gaussian smooth of about 2-4 samples against rapid changes, and threshold again to 0.5
• take difference between adjacent frames again to get the changes between ticking and paused

Also, the timer may not be visible at some parts of the video, add the following step(s) before applying the gaussian smooth

• for each timer region found during the scanning process, crop all scanned frames to the timer region
• group these images by similarity (squared difference), and select only the ~10-15 groups with the highest frequency, these are most likely the actual digits
• average all images of each group to get an idealized sample for each digit
• if the timer does is not rectangular (eg. italic text), save the original mask for the timer region and compare only these pixels

Now, after taking the difference between adjacent frames, set all values to 0 (stopped) where the image does not match any of these digits (again, squared difference).

(left: all images from intervals classified as containing the timer extracted from the timer area; right: images grouped by sum of squared difference)

Considerations

There are a few things to consider when applying these methods.

• Acquiring frames. Sounds simple, but you don't want to dump them all uncompressed to your HDD. I'm using ffmpeg with fast seeking -ss time -i input.mp4, which seeks to the nearest key frame and decodes frames until it reaches the requested time. Except it won't work some times, especially for bad video files that are common when you are saving a network stream. After transcoding the video file (with ffmpeg), most files that didn't work before were recognized perfectly. If you scale with ffmpeg, encode to 320x180, set a low key frame interval and set -movflags faststart, reading frames from the resulting file is accurate and fast. And at 320x180, it doesn't use much space either. Avoid downsampling the frame rate, as that might destroy your time stamps.

• Choosing the right threshold values, time intervals and other parameters. This is what took by far most of my time, especially since testing takes some time.

• In particular, it helps to choose a time step such that each digit appears. Choosing 5 seconds for example is a bad choice, as it will give you only 0's and 5's. Choosing a time step lower than 1 second won't work well when trying to analyze how much each pixel changes, as the consecutive digits will be the same.

• Related to this, each interval needs to contain enough frames. If the period is 10s and you choose 60 frames for the interval length and a time step of 1.2 seconds, the length after which the digit repeats is $lcm(1.2s,10s)=60s$, ie. 50 frames out of 60. And that is not enough. I found that, in order to eliminate unwanted noise, the number of frames should be least 3-6 times as high as the period length in frames. So a time step of, say 3s, works quite well, as lcm(3,10)=30, which is 10 frames, so you get 6 periods.

Does it work, though?

I made a simple ruby script that implements the method described above.

It's probably full of bugs and the code is a mess, but it seems to work. I tested it on 200 videos of different length (5 minutes to 6 hours, most around 30 minutes). They all contained the same kind of timer (color, font), but at possibly different sizes and positions. After transcoding those video files where ffmpeg didn't manage to seek accurately, the script got the correct result for pretty much all of them, it got only 4 wrong (detected stopped when it had been ticking).

It's possible I might have fine-tuned the parameters for that specific testing set, but at least we can say it does work rather well if you can make some general assumptions about the timer. Also, I tested a few other videos with different font sizes/colors, and they worked as well. I also tested it on a video with two timers (one that kept ticking the entire video), and the script recognized them as well.

As far is accuracy is concerned, it usually determines the starting/ending point at $\pm 1...3$ seconds, (because of some smoothing and averaging to eliminate noise), which is more than enough for me. I guess you could get an even better estimate if, after determining there is a starting/ending point at a certain frame interval, you check exactly when the digit starts/stops changing.

Sample output:

Transcoding video (110.89 min) file before analysis...

ffmpeg -y -loglevel quiet -i "dump_000000038.mp4" -c:v libx264 -an -vf scale=320:180 -preset ultrafast -sws_flags bilinear -g 20 -movflags faststart -crf 12 -f mp4 ./transcode.mp4 < /dev/null
Transcoding took 423.545512367s, 16x real-time.

Analyzing <transcode.mp4>, duration 1:50:53.684...

Scanning for timers...

Trying timestep 11.0, method simple...
Analyzing 0:00:00.000-0:11:00.000...
 - got 0 connected area(s)
Found 0 candidates.

Analyzing 0:11:00.000-0:22:00.000...
 - got 1 connected area(s)
Found 1 candidates.

Analyzing 0:22:00.000-0:33:00.000...
 - got 1 connected area(s)
Found 1 candidates.

Analyzing 0:33:00.000-0:44:00.000...
 - got 0 connected area(s)
Found 0 candidates.

Analyzing 0:44:00.000-0:55:00.000...
 - got 1 connected area(s)
Found 1 candidates.

Analyzing 0:55:00.000-1:06:00.000...
 - got 1 connected area(s)
Found 1 candidates.

Analyzing 1:06:00.000-1:17:00.000...
 - got 1 connected area(s)
Found 1 candidates.

Analyzing 1:17:00.000-1:28:00.000...
 - got 1 connected area(s)
Found 1 candidates.

Analyzing 1:28:00.000-1:39:00.000...
 - got 1 connected area(s)
Found 1 candidates.

Analyzing 1:39:00.000-1:50:00.000...
 - got 0 connected area(s)
Found 0 candidates.

Extracting timer digit images...
Grouping timer digit images...


Timer(s) found:
Timer 1: (176,150), size 5x10


Analyzing timer(s) for event(s)...

Trying coarse timestep 11.0, fine 1.75...
Scanning timers for activity...
Timer 0: Found 4 event(s).
Checking at the beginning of the video at a higher resolution for the timer 0...
Checking at the end of the video at higher a resolution for the timer 0...
Refining estimate for timer 0, event 0...
Refining estimate for timer 0, event 1...
Refining estimate for timer 0, event 2...
Refining estimate for timer 0, event 3...
Reading frame 50/50...

This is what they're doing:
Timer 1:
  - starts off paused.
  - starts at 0:04:53.000
  - stops at 0:37:32.000
  - starts at 0:38:57.000
  - stops at 1:45:26.000
  - ends up paused.

Scanning took 25s, 16265 fps.