# How to approach detection of anomalies in a video feed?

As part of a project I have a video from a quadrotor that captures the road and the environment around it and I want to identify and detect potholes and speedbumps on the road using image processing. What is the best approach for this?

Here is what I've done until now. The video feed is converted into grey scale and I use one particular portion of it(marked by the blue rectangle) for processing. In that window, I again get smaller windows of and I calculated their corresponding Grey Level co-occurrence Matrix (GLCM henceforth) of all those windows. I calculated the textural features from the GLCMs like ASM, erratic homogeneity etc and I plotted them as well.

I did this with the hope that there will be some kind of obvious change in the features when the blue rectangle reaches the speed bump(which is slightly ahead in the image). But the results erratic and I cannot make a definite decision.

If there is any other approach, or if there's any change I could implement to get better results in the same method do let me know.

• Welcome to SE..DSP. Your request is quite broad and the word "anomalies" calls for precisions. Can you detail some of your initial steps, and the performance your are aiming at, Why for instance do you consider you should detect anomalies – Laurent Duval Mar 6 '16 at 8:19
• Thank you for your reply, Laurent. It's a sub-part of my final year engineering project. I can't indulge in too many details about what I'm doing I hope you understand. This is what my approach is currently. I am using python's sci-kit image module and OpenCV. I obtained the feed, found the Grey level co-occurrence matrix and calculated the textural features and plotted them. I was hoping there would be some obvious observable change in the features when there was a pothole or a speed bump as compared to the plain road. But, no. The results are way too erratic. – kauDaOtha Mar 6 '16 at 8:45
• Providing such details can help answering your question (and an example of images with road only and with "anomaly" can provide us with some context). I suggest you edit your question with the stuff you have done, and where they fail – Laurent Duval Mar 6 '16 at 8:52

As part of a project I have a video from a quadrotor that captures the road and the environment around it and I want to identify and detect potholes and speedbumps on the road using image processing. What is the best approach for this?

There are two approaches that can be taken at this. One operates in two dimensions, the other in three.

Neither is going to be "easy" but the application is interesting.

Working in two dimensions:

That is, try to resolve potholes and speed bumps using the video stream from a single camera on the quadro-copter.

This is a simple pattern recognition problem where a pothole is defined as "a patch of asphalt with a definite shape that is not as smooth as its surrounding" and a speed bump as "a band of asphalt, usually vertical to the direction of the road that has a checker-board pattern on it".

For this to work well, the drone will need to fly higher, with a down facing stabilised camera and auto-pilot on the height (with respect to the ground). This will provide a clear video stream that is also "slow" enough to avoid motion-blur artifacts and makes it easier for simple algorithms to process the video on a frame-by-frame basis.

A "training" video will need to be recorded which will be used to isolate representative pothole and speed-bump patches that will be used to train a very simple classifier.

The features mentioned in the original question would probably work well, especially GLCM, as it makes it easy to discriminate different textures. Information on colour can be used as well to easily reject things that could get in the way (for example leafs, branches, people etc).

This is the simpler approach.

Working in three dimensions:

That is, try to resolve potholes and speed bumps using either successive frames from a single camera on the quadrocopter or fitting a stereo-camera on the quadrocopter.

(The same comment for the flight path applies here as well, fly higher with a down-facing camera).

In this case, a pothole is defined as "a hole in the average level of what is perceived as ground" and a "speed bump" is defined as "a bump in the average level of what is perceived as ground".

Essentially, successive frames of the video feed ($n$, $n+1$ where $n$ is the frame number) would need to be processed to derive a "height" map of the scene which would then be used to detect very low and very high areas (with very low and very high depending on the application).

In addition, for this solution to work effectively, the information from the GPS (and ideally gyroscopes too) would be required to be able to tell how far has the camera travel between two frames and how much it rotated towards a direction. All of these parameters need to be taken into account to derive a reliable "height" map.

The task becomes a little bit easier if a stereo-camera was to be used, some of which already support extracting the "height" map via their APIs (for example this one).

However, because of reasons to do with how parallax works, there would be limits in how deep a hole or how high a bump you can detect as a function of drone flight level. Furthermore, the lower the drone flies, the more motion-blur the images would be getting which would make recognition even more difficult.

Hope this helps.

Here is some brainstorming:

You have two interesting variables here, I would say:

• spatial changes in y-direction (left to right)
• temporal changes (t-direction).

I would not directly include the x-direction (up-down direction) in the first try, since this is correlated to the t-direction.

Try the following:

• Calculate the y-gradient
• Calculate the t-gradient
• Create a binary mask from both using two threshold values (you probably have to figure them out in a test run), so that the whole image is zero where there is no change and one where there is a change.
• To suppress spurious noise in the binary mask, you could median-filter it with a 5-by-5 or 7-by-7 median filter (try out the best size)
• Use a morphologic dilate filter to close "holes" in the binary mask and also to dilate the masks so that they do have a broader overlap

This way you are directly using the visual features of the image - changes in the gray values. Now a few things can happen with those two masks

• Areas of the t and y masks overlap: You have a change in the image in y direction and there is also a change between to consecutive images - you found a localized change that could be a hole or a speedbump

• There is a change in y direction, but no or just a small area in t direction: That could be a road marking (the algorithm might fail if your drone tumbles quite strongly and your camera has a low number of frames per second - then the t-gradient might be quite high in this case. To suppress those signal changes from lane markings, you could suppress lines in the binary mask by a Hough-transformation of the mask and kicking out pairs of dots)

• There is change in t but not in y - probably a lane mark that started/stopped

If this really works with your data, I do not know. However, starting with simple things can lead you into a quite robust and reasonably fast direction for your image processing task.