There are a number of videos that I'm looking to process of different video games to detect various "states" in them.
The first game that I am tackling is any edition of Super Street Fighter 4.
In it, I'd like to detect when the "vs" character screen comes up. Here's an example of one frame of the video:
If I could detect the "vs", then I'd be able to detect that frame of video is indeed the "vs" screen, which would allow me to look for other information (for now, let's say I'll use it to detect the timestamp in the video where the match is about to start).
That said, here is what can be assumed about the frames from the videos that I will be processing (this is not the only video, there are thousands, if not tens or hundreds of thousands of videos, but the issue of scale in processing that many videos is a completely different domain):
- I'd prefer (but it's not necessary) to process the lowest resolution image possible with reliable results (lower resolutions = faster processing time). The image above is 480 x 270 pixels (taken from a YouTube video with a
fmt18) but they may come in different sizes (I've gotten YouTube videos with
fmt18 but with dimensions 640 x 360 pixels).
- Most videos will be direct-feed
- Most videos will be 16:9 aspect ratio
- The reddish background will be animated, but generally be within that orange-red color (it's flames)
- Sometimes there will be a badge fading in and out over the lower part of the "vs" to indicate a version (that will be important, but not right now), which might obfuscate the "vs", like so:
- The size and position of the "vs" is going to be roughly the same (I haven't verified this yet but I know it doesn't move) in proportion to other direct-feed videos
- The characters will be chosen from a pool of more than 30 on each side (in other words, those areas of the frame will vary)
- The videos will generally be anywhere from two to four minutes long, with somewhere between 4,000 and 6,00 frames. However, there might be longer videos (maybe a two hours) which have various other games and live action cut in. These videos are not as important, but if a solution tells me where a certain game pops up in the larger overall video, great
- The native resolution of the captures is 720p, so a baseline image of the "vs" can be taken at what would be considered a "native" size.
Ultimately, I'm looking to code this pipeline in .NET, but that's not super important, the proof-of-concept is more important here as well as an understanding of the techniques involved so that I can translate and optimize it for .NET as well as for other videos of other games in the same genre (if I can pick out the significant discriminators, and videos of say, Ultimate Marvel vs. Capcom 3, Street Fighter x Tekken, BlazBlue: Continuum Shift, etc.).
I'm also dipping my toes in Mathematica and have home version 8.0, so a proof-of-concepts in that environment is more than welcome as well.