Pose Invariant Face Detection and Recognition

Question

I would like to detect faces and later apply recognition algorithms to surveillance videos. Ideally, the application should run in real-time (or near real-time) and transfer the data first to an algorithm deployed on cloud (or on a powerful server) to perform face detection/recognition and give a list of likely/best-matching candidates.

Here is an example image which is similar to the images from the videos on which I would like to run the detection and recognition algorithm.

Of course, it is assumed that the person's face will (at least partially) turned towards one of the deployed cameras. However, unlike the usual benchmarks and datasets used in face detection/recognition papers, under real-world conditions, one rarely has a full frontal view of a person's face. In the best case, there are slight tilts in vertical direction (pitch), and in most cases there is also a tilt in the horizontal direction (yaw). Another challenge is the bad lighting conditions and variations in lighting (plays a role when one wants to do matching among faces detected in different environments).

What kind of robust algorithms exist in the literature for face detection and recognition that can operate on faces that do not necessarily have a frontal pose. Are deep learning based algorithms robust in terms of dealing with pose variations for the same identity? I can imagine that this problem is easier to solve for face detection but could be very challenging for face recognition/verification. Does anyone have practical experience with such algorithms and use them on real-world data such as surveillance videos?

Answer 1

The path you should go is deep learning and the robustness of your model depends on the training phase and the data set used.

There are many capable models which allow real time inference.
You may use face detection dedicated models or just train a general model (Like YOLO v3 / v4 / v5) on this task.

For instance a good starting point would be the Retina Face model.

Make sure you do a transfer learning and extend the data set with your own kind of images. Also pay attention and be creative with the data augmentation in order to make the output model as robust as it cam be.