How to approach detection of anomalies in a video feed?

Question

As part of a project I have a video from a quadrotor that captures the road and the environment around it and I want to identify and detect potholes and speedbumps on the road using image processing. What is the best approach for this?

Here is what I've done until now. The video feed is converted into grey scale and I use one particular portion of it(marked by the blue rectangle) for processing. In that window, I again get smaller windows of and I calculated their corresponding Grey Level co-occurrence Matrix (GLCM henceforth) of all those windows. I calculated the textural features from the GLCMs like ASM, erratic homogeneity etc and I plotted them as well.

I did this with the hope that there will be some kind of obvious change in the features when the blue rectangle reaches the speed bump(which is slightly ahead in the image). But the results erratic and I cannot make a definite decision.

If there is any other approach, or if there's any change I could implement to get better results in the same method do let me know.

Answer 1

As part of a project I have a video from a quadrotor that captures the road and the environment around it and I want to identify and detect potholes and speedbumps on the road using image processing. What is the best approach for this?

There are two approaches that can be taken at this. One operates in two dimensions, the other in three.

Neither is going to be "easy" but the application is interesting.

Working in two dimensions:

That is, try to resolve potholes and speed bumps using the video stream from a single camera on the quadro-copter.

This is a simple pattern recognition problem where a pothole is defined as "a patch of asphalt with a definite shape that is not as smooth as its surrounding" and a speed bump as "a band of asphalt, usually vertical to the direction of the road that has a checker-board pattern on it".

For this to work well, the drone will need to fly higher, with a down facing stabilised camera and auto-pilot on the height (with respect to the ground). This will provide a clear video stream that is also "slow" enough to avoid motion-blur artifacts and makes it easier for simple algorithms to process the video on a frame-by-frame basis.

A "training" video will need to be recorded which will be used to isolate representative pothole and speed-bump patches that will be used to train a very simple classifier.

The features mentioned in the original question would probably work well, especially GLCM, as it makes it easy to discriminate different textures. Information on colour can be used as well to easily reject things that could get in the way (for example leafs, branches, people etc).

In other words, fly with the quad, collect the video, mark the areas of interest, calculate feature values for 4 (possible) classes of ROAD, NOT ROAD, POTHOLE and SPEEDBUMP and use them to build the decision boundaries. For more information about this, please see this link.

This is the simpler approach.

Working in three dimensions:

That is, try to resolve potholes and speed bumps using either successive frames from a single camera on the quadrocopter or fitting a stereo-camera on the quadrocopter.

(The same comment for the flight path applies here as well, fly higher with a down-facing camera).

Either of these options poses further difficulties. Both of these solutions are trying to take advantage of parallax to derive the third dimension (depth). For more information about parallax please see this link and this link.

In this case, a pothole is defined as "a hole in the average level of what is perceived as ground" and a "speed bump" is defined as "a bump in the average level of what is perceived as ground".

Essentially, successive frames of the video feed ($n$, $n+1$ where $n$ is the frame number) would need to be processed to derive a "height" map of the scene which would then be used to detect very low and very high areas (with very low and very high depending on the application).

In addition, for this solution to work effectively, the information from the GPS (and ideally gyroscopes too) would be required to be able to tell how far has the camera travel between two frames and how much it rotated towards a direction. All of these parameters need to be taken into account to derive a reliable "height" map.

The task becomes a little bit easier if a stereo-camera was to be used, some of which already support extracting the "height" map via their APIs (for example this one).

However, because of reasons to do with how parallax works, there would be limits in how deep a hole or how high a bump you can detect as a function of drone flight level. Furthermore, the lower the drone flies, the more motion-blur the images would be getting which would make recognition even more difficult.

Hope this helps.

Answer 2

Here is some brainstorming:

You have two interesting variables here, I would say:

• spatial changes in y-direction (left to right)
• temporal changes (t-direction).

I would not directly include the x-direction (up-down direction) in the first try, since this is correlated to the t-direction.

Try the following:

• Calculate the y-gradient
• Calculate the t-gradient
• Create a binary mask from both using two threshold values (you probably have to figure them out in a test run), so that the whole image is zero where there is no change and one where there is a change.
• To suppress spurious noise in the binary mask, you could median-filter it with a 5-by-5 or 7-by-7 median filter (try out the best size)
• Use a morphologic dilate filter to close "holes" in the binary mask and also to dilate the masks so that they do have a broader overlap

This way you are directly using the visual features of the image - changes in the gray values. Now a few things can happen with those two masks

• Areas of the t and y masks overlap: You have a change in the image in y direction and there is also a change between to consecutive images - you found a localized change that could be a hole or a speedbump

• There is a change in y direction, but no or just a small area in t direction: That could be a road marking (the algorithm might fail if your drone tumbles quite strongly and your camera has a low number of frames per second - then the t-gradient might be quite high in this case. To suppress those signal changes from lane markings, you could suppress lines in the binary mask by a Hough-transformation of the mask and kicking out pairs of dots)

• There is change in t but not in y - probably a lane mark that started/stopped

If this really works with your data, I do not know. However, starting with simple things can lead you into a quite robust and reasonably fast direction for your image processing task.