Can I use standard computer vision techniques for images taken in the NIR spectral range?

Question

I'm relatively new in Computer Vision and I need some advice with respect to the general possibilities and shortcomings of images taken in the near infrared (NIR) range. I passed a great deal of time researching about that matter but I rather found specific papers in areas like pedestrian detection and face recognition which not completely answered my question.

The background is that I have to chose camera and lens for a stereo-vision system which will be used for object detection tasks.To make the system robust against variations in illumination I thought about the feasibility of employing active illumination with NIR-light and a NIR-bandpass filter. Thus, working exclusively in the NIR spectral range.

I found publications about stereo matching in the far-infrared range, which come to the conclusion that it is not possible to obtain a dense depth field. However, I could not find anything related to the NIR range. It may be because it is too obvious but unfortunately not to me at this point.

Thats why I would like to know in particular:

• if it is possible to employ standard stereo matching algorithms like block-matching to images taken in the NIR spectrum and if so would it be possible (of course depending on the conditions) to obtain a dense depth map.
• if standard feature detection and description techniques like SIFT, HOG,etc produces similar results in the NIR spectrum.
• if a neural network which was trained with visible-light images would be also able to detect an object in an NIR image. (I am aware that this is somehow connected to the prior question)

I would be grateful for any advice, personal experience, book/paper references you may could share with me.

Thank you!

Answer 1

I have a collection of 170 reflectance spectra of various "materials" (such as ripe brown banana and asphalt) attributed to Ron Gershon of Eastman Kodak. I believe they are diffuse reflectance, which would be zero for a perfect mirror. Here is a plot of all the spectra in the dataset:

Figure 1. All reflectance spectra

Here is a plot of the mean and standard deviation of reflectance at each wavelength across the whole dataset:

Figure 2. Mean and standard deviation of reflectance spectra

On average, reflectance increases going towards infrared, and there is plenty of standard deviation in reds but the standard deviation seems to decrease going to near infrared. If the trend continues as such, it would be better not to go too far into infrared as then there might not be enough deviation in reflectances to see patterning in an object if the patterning arises from a mosaic of materials.

Longer wavelengths also have more blur from subsurface scattering.

Answer 2

if it is possible to employ standard stereo matching algorithms like block-matching to images taken in the NIR spectrum and if so would it be possible (of course depending on the conditions) to obtain a dense depth map.

Yes, it is possible. The Leap sensor operates at 850 nm and it is composed of two wide angle cameras that track hands (specifically) in 3D space. The geometry of the cameras is of course fixed but still, the device performs a lot of image processing (on the NIR images) to locate and relate a number of key points from the physical hand to the its internal model.

The Leap is using reflected light from 3 IR LEDs which makes its operation closer to a typical visible light camera. Using the stereoscopic vision principle to infer depth, it must at least perform pixel matching along a scanline to create the correspondence of the pixels between frames.

The Kinect 1 operates at 780nm but it uses a different principle. It projects a known pattern and senses depth via tracking the deformation of that pattern. Having said that, here is an analysis of its depth resolution at different distances using different methods. (The Kinect 2 is at 850 nm and uses yet another technique (section 3))

But, the question is a little bit incomplete. The key question to ask is "What sort of resolution (or discriminating capability) is required at some distance $d$ from the camera?" and then use that figure to focus your technology search better.

if standard feature detection and description techniques like SIFT, HOG,etc produces similar results in the NIR spectrum.

Yes they do. As mentioned by @Tomtana, if there is texture, then there will be features to lock on. Having said that, the performance of these algorithms might be changing depending on the distance from the camera.

If a neural network which was trained with visible-light images would be also able to detect an object in an NIR image. (I am aware that this is somehow connected to the prior question)

Depending on how this is done, the answer would be (mostly) yes. I am assuming training on grayscale images and recognising IRs through a set of features. So, if the features were to be selected carefully, then it should be possible to obtain robust matches. There are obvious problems here with the information content because IR sees things that visible does not (and vice versa), so, we are already, in a way, accepting a lower level of performance, simply because the images are not the same.

If you shoot a portrait in IR, the eyes are likely to come out as two bright disks and with minimal detail of the iris. The typical visible image is of course entirely different. So, the choice of the features will have to be done carefully to balance this difference in information content in the two types of images.

Hope this helps.

Answer 3

As far as I know, it is a common practice to use regular image processing techniques in this field. In fact I have recently finished a project in this field.

if it is possible to employ standard stereo matching algorithms like block-matching to images taken in the NIR spectrum and if so would it be possible (of course depending on the conditions) to obtain a dense depth map.

Yes, it is, I think, since NIR images as opposed to Far-Infra red Images has a lot of texture which you probability can (It is very very similar to RED band of the image), as an example people doing these:

repository.upenn.edu/cgi/viewcontent.cgi?article=1115&context=cis_reports

if standard feature detection and description techniques like SIFT, HOG,etc produces similar results in the NIR spectrum.

I think so, since again it has lots of texture and wherever there is texture is corners, edges and all types of feature points,which you can extract for further porcessing, check this out:

enter link description here

I think you are better to also take a look at reference books such as:

Machine Vision and Advanced Image Processing in Remote Sensing:

https://books.google.com/books?isbn=3642601057

Remote Sensing, Models, and Methods for Image Processing:

https://books.google.com/books?isbn=0126289816

Hope you've got your answer.