# Image segmentation issue of different materials

Hi CV/Pattern Recognition Community,

I've got a serious problem regarding the segmentation of an image. The scenario is an atmosphere within a furnace which makes my head go insane. And I need to detect object contours of different materials (glass, ceramics, Al, Ir,..) in a short period of time (<10 seconds) and not just for one special case. I also need the contour in a sequential row of pixels for the code. Therefore a chain code or so called border/contour following is also needed, so open holes aren't good. In the background are non linear noises, approximately of dust, particles or somewhat else, that are appearing from time to time.

Matlab or OpenCV suggestions are welcome.

To make it more clear, I've posted another image of my goal and a half-transparent object, which needs also to be detected. Also further examples which need to be aware of.

As you can see in the Image #1, there are particles in the right part of the image and nearby the outer contour of the star, which is the object. Also the overall contrast is not very good. The object itself stands on an underground, which is not relevant for the contour detection. The image #2 shows a halftransparent object, which is also possible.

I want to find contour/perimeter of that object, like on the next screen (red line). The two rectangles (yellow) are marking the starting (left) and the ending point (right). The blue line is ignorable.

At first I thought that I could solve the problem of that filthy atmosphere with just filters. But after a honourable ammount of investing time, I just realized, that I have to elimate or reduce the noises significantly in order to increase contrast of foreground and background. I'd tried a lot of methods, like histogram equalization, Otsu-adaptive equalization, linear filters (e.g. gauss), nonlinear filters (median, diffusion), Active Contours, k-Means, Fuzzy-c-means and also Canny for pure Edge Detection in combination with morphological operators.

• Canny: The particles and the atmosphere are causing holes, but I need a complete contour of the object. Still with closing, dilatating of morphological operators it is not well enough. Canny has still the best results of all methods I've studied because of hysteresis.
• Active Contours: They work on edges/gradients as well, they act completely crazy after initializing inside the object, which maybe is caused by the edge map resulting the 'open' object. As far as I know the contour has to be closed. Tried it with different derivates (GVF/VFC/Classic Snake).
• k-Means: Results include the furnace atmosphere, because of foggy background. Same for fuzzy-c-means. I chose two clusters, because of separating the object from the background. More clusters lead to weaker results.
• Histogram/Otsu: Because of the very close gray intensities (imho!), it's merging the object with the background. Tried it with local and global methods.
• Filters: Especially GLPF or other LPF are smearing the edges, which is not so good and doesn't even reduce the foggy atmosphere.
• Non-Linear Filters are preserving the edges. Most of them take too long for calculating the large images. Took a fast bilateral filter for now. Results see below.

Therefore not a single method is good enough for post-processing steps, because the gained results of the object segment are poor competed to an existing algorithm. That existing algorithm is very local and therefore it works for this very special scenario.

So I am asking you, if I have missed something completely... I have no further idea how to process and how I should get good contour results, without having gaps or holes.. Is it possible without making a lot of changes on the CCD and the physical environment? Thanks in advance!

# Last Approach so far (after a long night of experiments with MOs):

• Bilateral Filter (edge preserving, but smoothing homogenous areas)
• Canny (Sigma = 2, Threshold = [0.04 0.08])
• Morphological Operations (MO): bwareopen,closing,remove & bridge
• bwlabel for selecting only the perimeter of the contour, which removes unwanted noises. no updated screenshots yet, but it works for the star. the glass gots an inner-contour which is connected to the outter contour, which can also be seen on the screenshot below.

So I'm afraid that I need a special algorithm for the traversal of the outter contour. It will be some clockwise/counterclockwise lookup of neighbourhood. That clockwise/counterclockwise step can switch, if there is a corner point. if there is a gap, increase the radius and look again. if there are two or more possible following points, take the one who got the same direction as the previous. Do you think, that contour following algorithm makes sense?

• Have you tried adaptive threshold? You don't seem to mention it. I think OTSU should work somehow after removing noise but maybe adaptive threshold is better. Oct 12 '12 at 17:17
• Hi Rui, I tried adaptive thresholding with this Matlab Extension: Adaptive Thresholding Here are the results for playing around with the window parameters: 1st Try 2nd Try 3rd Try As you can see, with increasing parameters the middle part goes from white to black (what is good, imho) but the rest of the background also turns into black, what is bad. Oct 12 '12 at 18:09
• Have you tried some basic image reduction techniques? Such as subtracting a flat-field correction to get rid of that darkening on the top right (en.wikipedia.org/wiki/Flat-field_correction). Also, if the particles are static, they would instantly be removed. Then, you could use any edge detection method you want... Oct 12 '12 at 18:25
• Hi Phil, as far as I am concerned and as I know, there is a series of pictures who are shot before an object gets into the furnace. Kind of calibration is therefore here. I'll talk to the physicist who is responsible for the CCD and the environment on monday. But thanks for the advice, I'll give it a shot! Oct 12 '12 at 19:03
• The second image you added looks totally different. Can you post all possible images? Oct 14 '12 at 13:04

You can try the following:

• Hi oli, regarding sparse methods: could you be more specific of what methods of that code I should use? I'm not very deep into that section and I didn't find something helpful in the docs regarding de-noising or blurring... Thank you in advance. Oct 12 '12 at 17:28
• You can find an "easier to use" version, there: lear.inrialpes.fr/people/mairal/denoise_ICCV09.tar.gz
– oli
Oct 12 '12 at 17:36
• sorry to complain another time ;-) ... do you have win32 sources as well? thank you again! Oct 12 '12 at 17:54
• I am affraid I do not...
– oli
Oct 12 '12 at 17:58

I think that you gave up on threshold techniques too early. Take a look at your histogram, it is clearly tri-modal: (I removed the white columns to the right of your image manually, I assume that they are not part of the image - please take this image before running my code)

Take a look at all values in the first group:

In order to find modes in tri-modal histogram, it is possible to use K-means clustering with K=3 on intensity. The following Matlab code finds th1=67 on your code. The idea is to assume that you have the 3 sets, and calculate the weighted centroid on each one. Then, each intensity level is assigned to its own cluster. You stop when the weighted centroids cease moving. Here is the result of finding two thresholds on your image, shown on the histogram.

function [th1,th2]=SegmentHistTo3()
h = imhist(im(:,:,1)); %# Calculate histogram

th1new = round(256/3); %# Initial thresholds
th2new = round(256*2/3);
th1 = 0;
th2 = 0;

while (th1~=th1new) || (th2~=th2new) %# While the centroids keep on moving
th1 = th1new;
th2 = th2new;

wa1 = WeightedAverage(h,1,th1);  %# Calculate 3 weighted averages
wa2 = WeightedAverage(h,th1+1,th2);
wa3 = WeightedAverage(h,th2,numel(h));

th1new = round( (wa1+wa2)/2 );  %# The thresholds are middle points between the averages
th2new = round( (wa2+wa3)/2 );
end

figure; hist( double( reshape(im(:,:,1),1,[]) ),256);
hold on;
plot( [th1 th1],[0 max(h)],'r','LineWidth',2);
plot( [th2 th2],[0 max(h)],'r','LineWidth',2);

figure;imshow( im(:,:,1)<th1);
end

function wa = WeightedAverage(region,th1,th2)
regionNonEmpty(th1:th2) = region(th1:th2);
wa = sum( regionNonEmpty .* (1:numel(regionNonEmpty))) / sum(regionNonEmpty);
end


Solving the problem afterwards is a piece of cake, simply do some simple morphological operations, like opening.

• Hi Andrey, but how should I do a generalization of that thresholding you mentioned? I got several cases, not just that one and I still need automation. And the Otsu Thresholding (function in matlab) gave me no good results. Any further hints? Kind regards Oct 13 '12 at 15:06
• Hi again, thanks so far, but the code doesn't work. Blank figure screen appears. Tried it with my original data (bitmaps) and the PNG you posted above. I am debugging meanwhile... Oct 13 '12 at 15:54
• @mchlfchr, do you have image processing toolbox? If you don't, it is possible to change imhist to hist Oct 13 '12 at 15:56
• @mchlfchr, please see updated version Oct 13 '12 at 15:58
• Andrey, if I insert the original bitmap file, the results are as I mentioned in my origin post. Maybe this is happening regarding the resolution? Source image is 576x768 pixels and grayscale (256). Here are the results, if I use your function with my original image: i.imgur.com/UXALJ.png histogram-figure of your function: i.imgur.com/7RiPP.png Thanks for your help! regards Oct 13 '12 at 17:43

As suggested above, thresholding can be very effective on this image, which is essentially binary, except that a constant threshold will not do because of uneven lighting. You need adaptive thresholding.

My advice would be to do background reconstruction with a simple model (possibly planar [3 DOF] or quadradic [6 DOF]), by sampling a small number of values in the light regions. The best is to use small ROIs to average away the noise. Then correct the shading by subtracting (or dividing by) the background values.

If human interaction is not an option, you can automate the search for background areas by first straight Otsu and considering uniform ROIs (low variance) well under the threshold. After a first background reconstruction, you can probably improve by applying this process to the flat-corrected image.

The whole process can be implemented to run well below a second.

• Hi Yves, an automated processing is prefered. The DOF aspect is interesting, but I'm not sure about the Otsu method, because Otsu itself is not working well. Am I understanding it right that you want to pick random areas of the image and then you would threshold after mean value over all picked areas? Kind regards Oct 13 '12 at 12:52

I think best way is to use Active contours. If you are not aware what active contours are have a look at this video on youtube http://www.youtube.com/watch?v=ijNe7f3QVdA

Basically, u need to give an initialize segmentation and it will improve the shape. My suggestion is to one of the methods discussed on this post and use active contours as a 2nd step ie. as an improvement step.

Here is an implementation of active contours you could use http://www.mathworks.com/matlabcentral/fileexchange/19567

• Welcome to dsp.se :) Thanks for contributing, you provided a nice answer. If you want to make it even better, I think it would be interesting to provide answers to some of these questions: Why do you think this is the best approach (e.g. do you have personal experience with the tecnhique)? Which of the already suggested approaches do you think would work well in combination with your suggestion? Offer a short explanation of the technique, or, if you have time, try to provide experimental results using the technique on the example images provided. And have fun on dsp! Oct 26 '12 at 16:13
• @mkuse, as you may have read the initial post, i already had tried the active contours, combined with noise reduction and edge maps. the results were bad and had a bad run-time for large images. Oct 27 '12 at 6:50
• how about you have a look at noise reduction techniques. You can find a summary of those here : lnmiitdip.files.wordpress.com/2011/12/… Oct 27 '12 at 11:04
• @mkuse, I already mentioned the mechanics you have posted in your PPT file in my initial post. I edited my initial post, to make it more clear, what kind of filters I've used. Oct 27 '12 at 13:08

You clearly know what you're about but you haven't mentioned using thresholding, in particular have you tried applying a global threshold using Otsu to calculate the right level, then finding contours and selecting the largest ?

[Edit to clarify]

Global threshold obviously won't work due to the visible graident across the image.

I had a quick play with this and find that if you break the image into 6 chunks (2 rows of 3 columns equal sized) , then perform thresholding using Otsu on each one and then reassemble, it does a pretty god job at cleaning up the image.

There are still some minor artifacts on the upper right portion of the star.

It occurs to me that since the object has straight line boundaries, you might want to consider a Hough transform to extract these edges, intersect them to locate vertices and use the result as your object contour.

• Hi Dave, I tried Otsu, but it brings the effect that the right upper background is merging with the object, which is absolutely inacceptable. Oct 12 '12 at 15:16
• Hi Dave, Hough is not an option, because of run-time requirements and as far as I have knowledge about the HT, it's very time consuming for large images. Oct 13 '12 at 15:55

Are the outlines always straight lines or known curves?

If so then rather than trying to get each pixel along the edge correct I would use Hough transforms to get the equations of the lines and then recreate the countours from the lines and itnersections

• As I already mentioned: I need nearby real-time approaches. And as far as I know the HT, it's very time consuming. Another aspect is, that I don't know the curves and the lines aren't always straight. The contour depends on the material, which is in the furnace (for more information, see my origin post). Oct 13 '12 at 17:27
• For straight lines it's pretty fast, and if you know roughly where the lines are (eg from a previous frame) you can search only that paramter space Oct 13 '12 at 18:36