Faster R-CNN and 4-step Alternating Training

Question

I am reading the paper Faster R-CNN available here. In section 3.2, page 6 bottom left, the authors describe what they refer to as a 4-step Alternating Training between the Region Proposal Network, used to obtain region proposals used in the Fast R-CNN method for object detection. They write:

In this paper, we adopt a pragmatic 4-step training algorithm to learn shared features via alternating optimization. In the first step, we train the RPN as described in Section 3.1.3. This network is initialized with an ImageNet-pre-trained model and fine-tuned end-to-end for the region proposal task. In the second step, we train a separate detection network by Fast R-CNN using the proposals generated by the step-1 RPN. This detection network is also initialized by the ImageNet-pre-trained model. At this point the two networks do not share convolutional layers. In the third step, we use the detector network to initialize RPN training, but we fix the shared convolutional layers and only fine-tune the layers unique to RPN. Now the two networks share convolutional layers. Finally, keeping the shared convolutional layers fixed, we fine-tune the unique layers of Fast R-CNN. As such, both networks share the same convolutional layers and form a unified network.

What I do not understand is the third step. They say they use the detector network to initalize RPN training. I am not sure what this means. The detector network outputs bounding boxes and class labels for objects within a given image.

Consider the layers specific to RPN. We have a certain number of convolutional layers, followed by sliding windows with anchor boxes being fed to regression and classification (class label layers). Are the authors saying that the initially trained convolutional layers are now discarded for RPN and replaced with those trained by Fast R-CNN in step 2? Or do they mean something else.

Answer 1

Since no one answered this, I will post my attempt after further research.

Here is my understanding after having thought about it more. Lets say we are using VGG-16 as the backbone.

There are three fundamental components, the backbone, the RPN heads (objectness and RPN regression), and the ROI heads (classification and regression). Using the RPN heads, we can sample positive and negative ROIs.

Step 1) Train a region proposal network (backbone + RPN head) with the VGG-16 backbone pre-trained on imagenet with the head popped off. Some layers of the backbone will be trainable.

Step 2) Using the object proposals (300 or so per image) generated from the network obtained in step 1) (the backbone trained with the rpn), train a second network with another separate VGG-16 backbone pre-trained on imagenet, and a network head as in fast R-CNN (we will call ROI head). Then as in fast R-CNN we train for object classification of a region, as well as regression to learn the displacement from the proposal to a bounding box around the object (if the highest activation in the softmax does not correspond to a "background" class, i.e. no object).

Step 3) Use the backbone from step 2) for RPN training, which will be fixed (= not trainable), with the RPN head trained from step 1), and fine tune the RPN head.

Step 4) Finally keeping the backbone from step 2) fixed, and the RPN heads from step 3) fixed, fine tune the ROI Heads from step 2) to train bounding box regression and classification once more on the region proposals generated from the backbone and RPN head.

Edit: Original answer had many errors so I came back to correct them, also original question doesn't really make sense now that I read it (I was new to this topic). Please just read this answer.