# Image preprocessing for facial detection->embedding->clustering pipeline

I am trying to implement an end to end pipeline for facial clustering so that it can group people with the same faces. This will be quote a long post as I know that this is a very broad topic, so I have listed out what I have done step by step to show the pipeline that I currently have.

So far this is what I have done:

Detection

I used the dlib library to detect the bounding boxes for faces in the image which was created by Vahid Kazemi and Josephine Sullivan using an ensemble of regression trees to detect faces.

# Setup the models
face_detector = dlib.get_frontal_face_detector()
predictor_model = "shape_predictor_68_face_landmarks.dat"
face_pose_predictor = dlib.shape_predictor(predictor_model)

# Define the variables needed for the affine transform
w, h = 160, 160
eye_corner_dst = [[np.int(0.3 * w), np.int(h / 3)],\
[np.int(0.7 * w), np.int(h / 3)]]

# Get an image and find the faces
image = mpimg.imread(image_path)[:,:,0:3].astype(np.uint8) # RGBA -> RGB
detected_face = face_detector(image)
faces = []


Alignment: Part 1

After the boxes containing the faces are found I looped through all the detected boxes, and found the facial landmarks that are used to align the faces. The transform to align the faces is done in similiarty_transform, in this function two points are created in such a way that each point creates an equilateral triangle with each of the 2 input points eye_corner_src and eye_corner_dst. Afterwards, an affine transform can be estimated using cv2.estimateRigidTransform, the resulting transform is stored in the transform variable. Finally, I transform all detected landmarks using an affine transform determined by the transform matrix, which is stored in landmarks_t and created a Face object to store the relevant data which is appended to faces.

for face_rect in detected_face:

# First crop the face
left = max(0, face_rect.left())
top = max(0, face_rect.top())
right = min(image.shape[1], face_rect.right())
bottom = min(image.shape[0], face_rect.bottom())

new_face_rect = dlib.rectangle(0, 0, right-left, bottom-top)
cropped_image = copy.deepcopy(image)[top:bottom, left:right, :].astype(np.uint8)

# Get the the landmarks for the face
pose_landmarks = face_pose_predictor(cropped_image, new_face_rect)

# This function just returns an array of an array of points for the landmarks
landmarks = get_landmark_points(pose_landmarks, dlib_point=False)

# Get source points
left_eye = landmarks[36]
right_eye = landmarks[45]
eye_corner_src = [left_eye, right_eye]

# Compute similarity transform, returns
transform = similarity_transform(eye_corner_src, eye_corner_dst)

# Apply similarity transform on image
cropped_image_t = cv2.warpAffine(cropped_image, transform, (w, h))

# Apply similarity transform points
# Note here rs stands for reshape, t for transformed
landmarks_rs = np.reshape(np.array(landmarks), (68, 1, 2))
landmarks_t = cv2.transform(landmarks_rs, transform)

# Append boundary points. Which will be used in Delaunay triangulation
landmarks_t = np.float32(np.reshape(landmarks_t, (68, 2)))

faces.append(Face(cropped_image_t, landmarks_t))


This is the result at the end of the first part of the alignment stage

Alignment: Part 2

Now that the landmarks are found, the goal now is to align the face using Delaunay Triangulation but to do this I need to have destination points for the landmark points. Therefore, I simply chose a front facing picture and found the landmark points for that picture.

Here is a result of that:

Alignment: Part 3

In this final part of the alignment, I use Delaunay Triangulation (created by the landmarks found in Part 1) to align the affine transformed face (found in Part 1) with the "ideal" face (found in Part 2) triangle by triangle. This is done so that all the facial features appear in the same place.

# Get the delaunayTriangles
dt = calculateDelaunayTriangles(rect, np.array(landmarks_dst))

# Get the transformed landmarks
landmarks_t = face.landmarks

# Get the transformed image of the face
face_image = face.face_image.copy()

# Output image
output = np.zeros((h, w, 3), np.float32())

# Transform the triangles one by one
for j in range(len(dt)):
triangleIn = []
triangleOut = []

# So here we are getting the jth triangle (remember this this
# is a 3-tuple with each tuple corresponding to an index which
# can be used to find a point in landmarks_t or dst_landmarks)
for k in range(3):
pointIn = landmarks_t[dt[j][k]]
pointIn = constrainPoint(pointIn, w, h)

pointOut = landmarks_dst[dt[j][k]] # replace this with the set landmark coordinates
pointOut = constrainPoint(pointOut, w, h)

triangleIn.append(pointIn)
triangleOut.append(pointOut)

# Now draw on the new image
warpTriangle(face_image, output, triangleIn, triangleOut)


This is the output image after all the triangles are warped:

This is the image that is used to create the facial features.

Feature Extraction

I used facenet to extract the facial features. All I did here was to load up the model and run my image through the network to receive the extracted feature which is a unit sphere in 128 dimensional space.

# Now utilize facenet to find the embeddings of the faces
MODEL_DIR = "20170216-091149"

# Get the save files for the models
meta_file, ckpt_file = facenet.get_model_filenames(MODEL_DIR)
with tf.Graph().as_default():
with tf.Session().as_default() as sess:
embedding_layer = facenet.custom_load_model(sess, MODEL_DIR, meta_file, ckpt_file)

# Find the embedding for the face
face_embedding = embedding_layer(output)


Clustering Before the clustering step I perform PCA (although my results show that it does not increase the accuracy by much) as clustering doesn't work that well in high dimensions.

Note here that I have assumed a variable face_embeddings which consists of multiple face_embedding.

pca = PCA(n_components = 10) # captures ~91% of variance
face_embeddings_reduced = pca.fit_transform(face_embeddings)


I then cluster with K-means:

clusterer = KMeans(n_clusters = 2, random_state = 10)
cluster_labels = clusterer.fit_predict(face_embeddings)


The result that I got was good, but not that good as I manually determined the number of clusters, and I only tested images from 2 different people.

Here are the results:

Cluster 0 has individuals: {'Abdullah_Gul': 6, 'Bill_Gates': 8}
Cluster 1 has individuals: {'Abdullah_Gul': 10, 'Bill_Gates': 9}


I am wondering where can my pipeline by improved, and what is the cause of this low accuracy. Is it during face alignment? Is the method that I'm using outdated? I have read about using two CNNs to create a pipeline. The first network being the MTCNN for creating the bounding boxes for faces, and the second network from here which has a Spatial Transformer module in the first layer to automatically align the faces. If someone could provide me with some resources to read or some pointers it would be great!