Detecting Trail in Forest Images

Question

Is anyone aware of any research/papers/software for identifying a trail (as a line or point-to-point curve) in an image of a forest scene (from the perspective of the camera standing somewhere along the trail)?

I'm trying to find an algorithm that could take an image like:

and produce a mask, identifying a likely "trail", such as:

As you can see, the original image is a bit blurry, which is purposeful. The image source can't guarantee perfect focus, so I need to be able to handle a reasonable amount of noise and blurriness.

My first thought was to apply a Gaussian blur, and segment the image into blocks, comparing adjacent blocks looking for sharp color differences (indicating a trail "edge"). However, I quickly realized that shadows and other changes in lighting easily throws that off.

I was thinking about extracting SURF features, but I've only had success with SURF/SIFT when the image is perfectly clear and with consistent lighting.

I've also tried scaling the images and masks down to much smaller sizes (e.g. 100x75), converting them into 1xN vectors, and using them to train a FANN-based neural network (where the image is the input and the mask is the desired output). Even at such a small size, with 1 hidden layer with 75% the size of the input vector, it took 6 hours to train, and still couldn't predict any masks in the testing set.

Can anyone suggest any other methods or papers on the subject?

Answer 1

I don't believe you have enough information in the source image to produce the mask image. You might start by segmenting on color, i.e. green is not trail, gray/brown is. However, there are gray/brown regions on the "trail borders" that are not represented in your mask. (See the lower left quadrant of your source image.)

The mask you provide implies structural constraints not evident in the source image: for example, perhaps your trails are of fixed width - then you can use that information to constrain the preliminary mask returned by your pattern recognizer.

Continuing the topic of structure: Do trails merge with others? Are trails delineated with certain soil/gravel features? As a human (that is reasonably good at pattern recognition!), I'm challenged by the features shown in the lower left quadrant: I see gray/brown regions that I cannot discount as "trail". Perhaps I could do so conclusively if I had more information: a map and a coarsely-known location, personal experience on this trail, or perhaps a sequence of images leading to this point - perhaps this view is not so ambiguous if the recognizer "knows" what led to this scene.

A collection of images is the most interesting approach in my opinion. Continuing that line of thought: one image might not provide enough data, but a panoramic view might disambiguate the scene.

Answer 2

There's no single algorithm that will magically detect trails in a random image. You will need to implement a machine learning based routine and "train" it to detect trails. Without going into too many details, here's a rough outline of what you would do in a supervised learning approach.

1. You will need a set of "training examples", by which I mean several pictures of trails in different environments, in which you (the supervisor) have labeled what counts as "trail" and what's the background "forest". You break the images up into smaller sections (typically 8x8) and transform it to a "feature space" by taking the DCT (discrete cosine transform) of the blocks. The DCT of each block in this case gives you a 64 point "feature vector".

2. Defining a feature space $\mathcal{X}$, set of features $\mathbf{x}$ (a subset of your 64 point feature vector), and a class space $\mathcal{Y}$ with classes $y_1=trail$ and $y_2=forest$, you calculate from your training sets:

   • the class conditional distributions
     • $\mathcal{P}_\mathcal{X|Y}(\mathbf{x}|trail)$, the conditional density for the features when the class is $trail$.
     • $\mathcal{P}_\mathcal{X|Y}(\mathbf{x}|forest)$, the conditional density for the features when the class is $forest$.
   • the class probabilities or the prior
     • $\mathcal{P}_\mathcal{Y}(trail)$, probability of finding a $trail$ in a block
     • $\mathcal{P}_\mathcal{Y}(forest)$, probability of finding a $forest$ in a block

3. With this, you test your image (again, breaking it up into smaller pieces) and calculate the posterior probability. Using Bayes' decision theory, you'd define your binary (in this case) selection criteria something like

   $$\widetilde{y}_i(\mathbf{x})=\arg \max_{y_i}\quad \mathcal{P}_\mathcal{X|Y}(\mathbf{x}|y_i)\ \mathcal{P}_\mathcal{Y}(y_i)$$ where you assign each block to that class which has the highest posterior probability. This will result in your binary mask.

Note that this is a very simplified overview of the approach. There are several things to take into consideration and the most important of them is choosing the right set of features for your problem. You can also do more complicated things like use mixture models and kernel based density estimations, but all of that is too detailed and time consuming to write in an answer.

For a motivation and confirmation that this approach is worth trying, here's an example from something I did a long time ago as a course homework, which is very similar to what you're trying to achieve. The objective was to detect the animal from the background vegetation (left image). The figure on the right shows the binary mask obtained after "learning" to distinguish between the foreground and the background.

To learn more about machine learning, you might want to look at a few text books. One of the well known and often recommended textbooks in the field is:

T. Hastie, R. Tibshirani and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2nd Ed., Springer-Verlag (2008)

which is now available as a free PDF at the link provided. Another decent book is:

R. O. Duda, P. E. Hart and D. G. Stork, Pattern Classification, 2nd Ed., John Wiley & Sons (2001)

Answer 3

It may not be sufficient by itself, but since one of the problems has to do with lighting variations, a shadow removal pre-processing step may help. The technique I'm thinking of is described in the following paper:

"On the Removal of Shadows From Images", G. D. Finlayson, S. D. Hordley, C. Lu, and M. S. Drew, IEEE Pattern Analysis and Machine Intelligence (PAMI), Vol.28, No.1, Jan, 2006, pp.59-68. http://www.cs.sfu.ca/~mark/ftp/Pami06/pami06.pdf

The first part of the process produces an illumination invariant grayscale image, which is probably what you want in this case. Here's a shot of the example they give in the paper:

_{(source: datageist.com)}

In fact, taking the process one step further to produce a "chromaticity" image may be what you need to cleanly distinguish between the live and dead leaves. Once again, an example from the paper:

_{(source: datageist.com)}

The catch, however, is that the camera needs to be calibrated first. Assuming that's possible, some combination of the representations they describe will probably make the other methods you're using more effective.

Answer 4

Is this of any interest?

Real-time traversable surface detection by colour space fusion and temporal analysis

Answer 5

It looks like a problem for texture segmentation (not a color segmentation)

There are many methods:

• Gabor wavelets.
• Superpixels based segmentation.
• Graph Cut Segmentation.

Wikipedia - Segmentation has a great overview of the subject.

Answer 6

If you need a simple real-time approach then I suggest to use color (limited range on HSV domain) and texture features (such as GLCM). Then, by evaluation the discontinuity in horizontal imaginary lines you will be able to find points that delimitate the way.

For a more complex approach, I believe that you can divide the image into big squares and then using a simple deep-learning method (such as mobilenet) will allow you to distinguish grass from ground. For a more refined result you can try U-Net, dedicated to segmentation.