Understanding the relationship between the convolutional feature map and the region proposals in Fast R-CNN

Question

In the paper Fast R-CNN available here, I am trying to understand the relationship between the region proposals and the convolutional feature map.

So from what I understand, Fast R-CNN takes in an image and a set of region proposals for object detection, with the goal of taking each proposal and giving it a classification from a set of classes, and a bounding box (or none in the case of background regions). More processing can be done after with the region proposals to merge them etc. but in general this is how it works.

What I don't understand is how the region proposals relate to the convolutional feature map. In section 2.2 the authors state that they test their architectures by pre-training networks with some image net architectures, which we will modify by replacing the max pooling layer by a RoI pooling layer, followed by some fully connected sibling layers for regression and classification. My question is what is the relationship between a region proposal, say a region $(x,y, h ,w)$ given by the top left corner and a height and width, and the convolutional feature map, which may have spatial dimensions (and definitely more channels) that differ from the original image, (see the image below)? It would make sense certainly if the feature map had the same spatial dimensions as the original image, but otherwise I'm not sure.

Answer 1

Notation

• Lets say you have an input image I with size WxHx3
• ROI proposal x0,y0,w0,h0.
• When you forward the image through the feature extractor you get a feature map F with size WfxHfXC size.

The relation is a spatial relation

The ROI doesn't affect the number of output channels from the feature map. The relation is a spatial relation which maps an input ROI to an equivalent spatial patch on the feature maps which has the same number of channels has the feature map. So the ROI in the output feature map will be of size UxVxC the same number of channels the feature map has.

Tracking an input pixel throgh a CNN

Now lets track a the pixel x0,y0 between successive layers of a CNN.

For simplicity i will consider only the following layers types (The same has in VGG 16 which is the feature extractor used in the article, for other layers the same logic can be applied with small variations):

• Convolution layer with "SAME" padding, kernel = 3, stride = 1 and denoted by Conv
• Pooling layer with some stride = 2 and denoted by Pool

Now what happens when we insert an image to Convolution layer? In other words to which pixel an input pixel x0,y0 is mapped to in the output of the Convolution?

The output size of a convolutional layer is calculated using the following formula: Output size calculation so in our case we will get that the output size is equal to the input size and that the pixel in the convolution response for pixel x0,y0 in the input is mapped to pixel x0,y0 in the output of the convolution.

For a polling layer with stride 2 we will get that if the input size is WxH the output size will be W/2xH/2 so pixel x0,y0 in the input is mapped to floor(x0/2), floor(y0/2)

Summery of tracking (Given our simplified case)

• Convolution x0,y0 in the input will be mapped to x0,y0 in the output
• Pooling x0,y0 will be mapped to x0/2,y0/2

Converting x0,y0 in image to x0^,y0^ in the output feature map

So if we have a CNN with Conv->Conv->Pool->Conv->Conv->Pool .... The pixel x0,y0 in the input will be mapped to the pixel floor(x0/2^(num of pool layers)),y0/2^(num of pool layers)

An ROI mapping

Lets take the case in which the feature extractor is VGG16, which the network used in the article. All the Convolution layers give output size is the same has the input size and the polling size is half the input size. The input size is 224X224 and the feature map is 7X7 -> the pixel x0,y0 in the input image is mapped to x0/32,y0/32 in the feature map.

The roi x0,y0,w,h is mapped to x0/32,y0/32,h/32,w/32