# Faster R-CNN and 4-step Alternating Training

I am reading the paper Faster R-CNN available here. In section 3.2, page 6 bottom left, the authors describe what they refer to as a 4-step Alternating Training between the Region Proposal Network, used to obtain region proposals used in the Fast R-CNN method for object detection. They write:

In this paper, we adopt a pragmatic 4-step training algorithm to learn shared features via alternating optimization. In the first step, we train the RPN as described in Section 3.1.3. This network is initialized with an ImageNet-pre-trained model and fine-tuned end-to-end for the region proposal task. In the second step, we train a separate detection network by Fast R-CNN using the proposals generated by the step-1 RPN. This detection network is also initialized by the ImageNet-pre-trained model. At this point the two networks do not share convolutional layers. In the third step, we use the detector network to initialize RPN training, but we fix the shared convolutional layers and only fine-tune the layers unique to RPN. Now the two networks share convolutional layers. Finally, keeping the shared convolutional layers fixed, we fine-tune the unique layers of Fast R-CNN. As such, both networks share the same convolutional layers and form a unified network.

What I do not understand is the third step. They say they use the detector network to initalize RPN training. I am not sure what this means. The detector network outputs bounding boxes and class labels for objects within a given image.

Consider the layers specific to RPN. We have a certain number of convolutional layers, followed by sliding windows with anchor boxes being fed to regression and classification (class label layers). Are the authors saying that the initially trained convolutional layers are now discarded for RPN and replaced with those trained by Fast R-CNN in step 2? Or do they mean something else.

Since no one answered this, I will post my attempt after further research.

Here is my understanding after having thought about it more. Lets say we are using VGG-16 as the backbone.

Step 1) Train a region proposal network with VGG-16 trained on imagenet (minus one layer popped off, and not including the network head). The important weights learned are those after the backbone. Only the layers Conv3_1 and up are trained.

Step 2) Using the object proposals generated from the network obtained in step 1), train a second network with the VGG-16 backbone trained on imagenet, and a network head as in fast R-CNN. Then as in fast R-CNN we train for object classification of a region, as well as regression to learn the displacement from the proposal to a bounding box around the object (if the highest activation in the softmax does not correspond to a "background" class, i.e. no object).

Step 3) Use the backbone and learned layers before the ROI projection layer from step 2) to initialize RPN training, with the layers not part of backbone learned in Step 1) attached as the head of a network to be trained again.

Step 4) Finally keeping the backbone layers (before ROI projection layer from step 3), the detection network is trained yet again as in Fast R-CNN with region proposals generated from the RPN in step 3)

At test time, the anchor boxes are sampled randomly, and using the feature map generated from the backbone of step 4, a sample of anchor boxes is fed through for classification and regression. The N (say N= 10) anchor boxes scoring the highest probability values are then post-processed (independently for each class) using non-maximum suppression to get the final region proposals. These region proposals are then fed through the network for classification and regression where the final RPN and the fast R-CNN networks share convolutional layers up until and including the output of the feature map.

Keep in mind there are actually 9 softmax layers and 9 regression layers, and for a given input, you only look at the output of the layer corresponding to the scale/aspect ratio of the anchor box linked to the input image. At test time, you can feed input values of the form [Image, [array of anchor box samples]] and so you only need to perform convolution on the image through the backbone once, and the subsequent convolutional layers are performed on the result of the ROI projection layer.