# How to relocate face points in opencv / face distortion?

I have to simulate facial expressions on a face image ( say open mouth ). For this I first extracted the facial feature points and found the corners of the lips. Now I need to deform the image by moving the points down.

In the above image I need to move the points ( 2 ) and ( 3 ) to some distance left and right respectively. And point ( 18 ) and ( 19 ) littele bit down. So that I will get an expression like opened mouth.

My Questions:

1) Is this the above way right to proceed to simulate facial expression?

2) If it is right how to move the points and create a new image in opencv?

-

## 1 Answer

Sure, that is one way of accomplishing your task so long as you do blending/interpolation to smooth the pixels in between. See these slides (esp. the last slide) or this survey for some literature on image warping and registration.

Thin-plate spline or "surface spline" problem
One reasonable approach would be establish point correspondences $r_1, \ldots, r_N$ and $w_1, \ldots, w_N$ as you have done already then solve the following problem:

Find the 2D transformation model $\mathbf{p}=\{c_1,...,c_N,A,t\}$ s.t. $$w_i=f(r_i)=Ar_i+t+\sum_{k=1}^{N}c_k\phi(d_i),$$ where $\phi(d_i)$ are radially symmetric basis functions (RBF). An example RBF choice is $\phi(d_i) = d_i^2\ln[d_i^2]$, where $d_i$ is a distance of your choice, e.g. Euclidean $d_i=\|r_i-r_k\|_2$.

Under certain thin-plate spline conditions not detailed here, this problem can be cast as a linear system $\mathbf{L}\mathbf{p}=\mathbf{w}$: $$\begin{bmatrix}\mathbf{K} & \mathbf{P}\\\mathbf{P}^T & \mathbf{0}\end{bmatrix}\begin{bmatrix}\mathbf{c}_{N\times2}\\\mathbf{A}_{3\times2}\end{bmatrix}=\begin{bmatrix}\mathbf{r}_{N\times2}\\\mathbf{0}_{3\times2}\end{bmatrix},$$ where $\mathbf{K}_{ij\,\,[N\times N]}=\phi\left(\left\|(r_i^x,r_j^y)-(r_j^x,r_j^y)\right\|_2\right)$ and $\mathbf{P}_{N\times3}=\begin{bmatrix}\mathbf{1}_{N\times1}\left|\mathbf{w}_{N\times2}\right.\end{bmatrix}$, and then solve for $\mathbf{p}$.

You can then apply the transformation model $\mathbf{p}$ to every pixel in the original/control image, and use 2D interpolation to get the final result.

And to answer your second question, assuming you are using OpenCV's CvPoint structure, you can just do the following:

cv::Point newPoint( oldPoint.x + xoffset, oldPoint.y + yoffset )

-