I have to simulate facial expressions on a face image ( say open mouth ). For this I first extracted the facial feature points and found the corners of the lips. Now I need to deform the image by moving the points down.

enter image description here

In the above image I need to move the points ( 2 ) and ( 3 ) to some distance left and right respectively. And point ( 18 ) and ( 19 ) littele bit down. So that I will get an expression like opened mouth.

My Questions:

1) Is this the above way right to proceed to simulate facial expression?

2) If it is right how to move the points and create a new image in opencv?


1 Answer 1


Sure, that is one way of accomplishing your task so long as you do blending/interpolation to smooth the pixels in between. See these slides (esp. the last slide) or this survey for some literature on image warping and registration.

Thin-plate spline or "surface spline" problem
One reasonable approach would be establish point correspondences $r_1, \ldots, r_N$ and $w_1, \ldots, w_N$ as you have done already then solve the following problem:

Find the 2D transformation model $\mathbf{p}=\{c_1,...,c_N,A,t\}$ s.t. $$w_i=f(r_i)=Ar_i+t+\sum_{k=1}^{N}c_k\phi(d_i),$$ where $\phi(d_i)$ are radially symmetric basis functions (RBF). An example RBF choice is $\phi(d_i) = d_i^2\ln[d_i^2]$, where $d_i$ is a distance of your choice, e.g. Euclidean $d_i=\|r_i-r_k\|_2$.

Under certain thin-plate spline conditions not detailed here, this problem can be cast as a linear system $\mathbf{L}\mathbf{p}=\mathbf{w}$: $$\begin{bmatrix}\mathbf{K} & \mathbf{P}\\\mathbf{P}^T & \mathbf{0}\end{bmatrix}\begin{bmatrix}\mathbf{c}_{N\times2}\\\mathbf{A}_{3\times2}\end{bmatrix}=\begin{bmatrix}\mathbf{r}_{N\times2}\\\mathbf{0}_{3\times2}\end{bmatrix},$$ where $\mathbf{K}_{ij\,\,[N\times N]}=\phi\left(\left\|(r_i^x,r_j^y)-(r_j^x,r_j^y)\right\|_2\right)$ and $\mathbf{P}_{N\times3}=\begin{bmatrix}\mathbf{1}_{N\times1}\left|\mathbf{w}_{N\times2}\right.\end{bmatrix}$, and then solve for $\mathbf{p}$.

You can then apply the transformation model $\mathbf{p}$ to every pixel in the original/control image, and use 2D interpolation to get the final result.

And to answer your second question, assuming you are using OpenCV's CvPoint structure, you can just do the following:

cv::Point newPoint( oldPoint.x + xoffset, oldPoint.y + yoffset )

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.