3
$\begingroup$

In the paper Fast R-CNN available here, I am trying to understand the relationship between the region proposals and the convolutional feature map.

So from what I understand, Fast R-CNN takes in an image and a set of region proposals for object detection, with the goal of taking each proposal and giving it a classification from a set of classes, and a bounding box (or none in the case of background regions). More processing can be done after with the region proposals to merge them etc. but in general this is how it works.

What I don't understand is how the region proposals relate to the convolutional feature map. In section 2.2 the authors state that they test their architectures by pre-training networks with some image net architectures, which we will modify by replacing the max pooling layer by a RoI pooling layer, followed by some fully connected sibling layers for regression and classification. My question is what is the relationship between a region proposal, say a region $(x,y, h ,w)$ given by the top left corner and a height and width, and the convolutional feature map, which may have spatial dimensions (and definitely more channels) that differ from the original image, (see the image below)? It would make sense certainly if the feature map had the same spatial dimensions as the original image, but otherwise I'm not sure. enter image description here

$\endgroup$
1
+50
$\begingroup$

Notation

  • Lets say you have an input image I with size WxHx3
  • ROI proposal x0,y0,w0,h0.
  • When you forward the image through the feature extractor you get a feature map F with size WfxHfXC size.

The relation is a spatial relation

The ROI doesn't affect the number of output channels from the feature map. The relation is a spatial relation which maps an input ROI to an equivalent spatial patch on the feature maps which has the same number of channels has the feature map. So the ROI in the output feature map will be of size UxVxC the same number of channels the feature map has.

Tracking an input pixel throgh a CNN

Now lets track a the pixel x0,y0 between successive layers of a CNN.

For simplicity i will consider only the following layers types (The same has in VGG 16 which is the feature extractor used in the article, for other layers the same logic can be applied with small variations):

  • Convolution layer with "SAME" padding, kernel = 3, stride = 1 and denoted by Conv
  • Pooling layer with some stride = 2 and denoted by Pool

Now what happens when we insert an image to Convolution layer? In other words to which pixel an input pixel x0,y0 is mapped to in the output of the Convolution?

The output size of a convolutional layer is calculated using the following formula: Output size calculation so in our case we will get that the output size is equal to the input size and that the pixel in the convolution response for pixel x0,y0 in the input is mapped to pixel x0,y0 in the output of the convolution.

For a polling layer with stride 2 we will get that if the input size is WxH the output size will be W/2xH/2 so pixel x0,y0 in the input is mapped to floor(x0/2), floor(y0/2)

Summery of tracking (Given our simplified case)

  • Convolution x0,y0 in the input will be mapped to x0,y0 in the output
  • Pooling x0,y0 will be mapped to x0/2,y0/2

Converting x0,y0 in image to x0^,y0^ in the output feature map

So if we have a CNN with Conv->Conv->Pool->Conv->Conv->Pool .... The pixel x0,y0 in the input will be mapped to the pixel floor(x0/2^(num of pool layers)),y0/2^(num of pool layers)

An ROI mapping

Lets take the case in which the feature extractor is VGG16, which the network used in the article. All the Convolution layers give output size is the same has the input size and the polling size is half the input size. The input size is 224X224 and the feature map is 7X7 -> the pixel x0,y0 in the input image is mapped to x0/32,y0/32 in the feature map.

The roi x0,y0,w,h is mapped to x0/32,y0/32,h/32,w/32

| improve this answer | |
$\endgroup$
  • $\begingroup$ "When you forward the image through the feature extractor you get a feature map F with size WfxHf size", that's not true in general. In general, as the OP says, the feature map volume can have a different depth than the depth of the image. That sentence suggests that the feature map will have depth 1, which isn't true in general, unless it's true in the case of mask R-CNN, but I doubt it. Also, you can use MathJax/latex on this site. Your post is quite unreadable $\endgroup$ – nbro Jun 17 at 17:21
  • 1
    $\begingroup$ @nbro Agreed i didn't write the number of channels and also it isn't relevant to the question $\endgroup$ – Amitay Nachmani Jun 17 at 17:22
  • $\begingroup$ It's actually very relevant. It's actually the question. Read the question! If it's not relevant, you should explain why. That's exactly the question: what is the relationship between the dimensions of the feature map and the region proposals, which are four numbers that represent the bounding boxes. $\endgroup$ – nbro Jun 17 at 17:23
  • 1
    $\begingroup$ No it is not. The question is about the relation between the ROI in the input image and to the roi in the output feature map not the number of channels. You could have any number of channels you want at the output feature map there is no relation between the ROI and the number of output features. $\endgroup$ – Amitay Nachmani Jun 17 at 17:25
  • 1
    $\begingroup$ Yes. The mapping can be done also from the feature map to the image space $\endgroup$ – Amitay Nachmani Jun 18 at 3:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.