high resolution facial performance capture and animation alex ma, i-chen lin, ming ouhyoung...
TRANSCRIPT
High Resolution Facial Performance Capture and
Animation
Alex Ma, I-Chen Lin, Ming Ouhyoung
馬萬鈞, 林奕成, 歐陽明CMLab, Dept. of CSIE,
National Taiwan University
10/27/2008
My Face Generation Dreams我的十年大夢
虛擬實境 (Virtual Reality) 的極致金庸的小說,主角換成自己的朋友?熱門遊戲的角色中有自己的參與?
My Face Generation DreamsPhase 1: Talking Head
A Speech Driven Talking Head System Based on a Single Face Image, pp. 43-49, Proc. Pacific Graphics'99
Phase 2: Human Face Expression
Realistic 3D Facial Animation Parameters from Mirror-reflected Multi-view Video, IEEE CG&A, 2002
Phase 3: Try to pass the human face Turing Test , Facial Performance Synthesis using Deformation-Driven Polynomial Displacement Maps, SIGGRAPH ASIA 2008, ACM Trans. Graphics
VR Talking Head,TalkingShow (CL product)
Realistic 3D Facial Animation Parameters from Mirror-reflected Multi-view Video
Facial Performance Synthesis using Deformation-Driven Polynomial Displacement
Maps
Emily project: with Image Metrics Inc.
Emily demo: Turing test?
Demo video
Outline
Estimation of 3D facial motion trajectories
Synthesis of facial animation Results and conclusion
Introduction (cont.)
Estimating 3D facial motion parameters from mirror-reflected multi-view video.
Fig 1. (a) the capture equipment. (b) a mirror-reflected multi-view video clip. (55 markers: 10 for the lip contour, 12 for lip surfaces, 10 for mouths, 8 for cheeks, and 10 for the forehead)
What’s new
Previous tracking approach from mirror-reflected video:• E.C. Patterson et al.(Computer
Animation’91): Simply assumed that the mirror was vertical, and
the camera was also vertical.• S. Basu et al. (ICCV’98):
Position evaluation via R, t estimation of virtual camera.
The proposed method:• Taking advantages of nice properties of
mirrored images.• More robust, accurate, and simpler.
FAPs extraction
Demo1: a captured video and synthesized result (viseme “o”).
Estimation of 3D facial motion
The proposed procedure:
1)Semi-automatic tracking the markers’ projected 2D trajectories in videos.
2)Estimation of markers’ 3D trajectories.
3)Head motion removal.
Marker tracking in video
Fig 2. The flow chart of semi-automatic markers’ projected positions tracking in video.
Manually designate markers’ positions
(1st frame)
Manually designate markers’ positions
(1st frame)
Search the position with lowest cost(ith frame)
Search the position with lowest cost(ith frame)
(for i = 2nd~nth frame)
Adaptive Kalman filtering
Adaptive Kalman filtering
Markers’ 2D projected trajectories
Markers’ 2D projected trajectoriesestimated
positions
(ith frame)
Predicted positions in the (i+1)th frame Kalman
predictor
Kalman predictorDelayDelay GUI EditorGUI Editor
3D trajectory estimation Known: pi, pi’, f.
Unknown: mi, mi’, u, d. Calculating 3D positions
via mirror plane estimation.
Properties:– ax+by+cz=d, u=(a,b,c)t, |
u|=1. (1)– mi
’=mi + ku.(2)
– (mi’-)=Hu(mj - ), where Hu
is the Householder matrix.(3)
Fig 3. The geometric representation of physical point m, reflected point m’, and the projected point p and p’.
3D trajectory estimation (cont.)
Deduction from eq. (2),
(4) Calculate the mirror plane from
projected point p and p’ by the least square method.
(5)
3D trajectory estimation (cont.)
Depth of each marker i (zmi and zmi’)can be estimated by the least square method.
(6)
The 3D motion trajectories are reconstructed now.
Extraction of facial motion
marker’s motion = facial motion + head motion.
Head motion (R,t) estimation from specified rigid markers.
ri (j+1) = Rri j + t Remove the head motion.
dispi = R-1(vi(j+1) - t) – vij, where vij is the estimated 3D position of marker i at
time j.
Discussion Unknown: u (4 DOF) R,t (6 DOF) The proposed method can reach an
accuracy with much fewer calibration points.
Our proposed method is more robust in a noisy situation.
0
100
200
300
0 1 2 3
variance of noise (pixels)
the
abso
lute
err
or
proposed method
general purposedmethod via R,testimation
0
100
200
300
10 15 20 25 30
number of calibration points
the a
bso
lute
err
or
proposed method
general purposedmethod via R,testimation
Fig 3. The situation where normal-distributed noise perturbed the estimation of marker motion in video.
Synthetic face Deform a generic model to fit scanned
range image. Radial-basis scatter data interpolation
function. Animation: Morph the head according to
scaled facial motion trajectories.
Synthetic Face
Demo 2: Synthetic face.
Facial Animation
Demo 3: Facial Animation
Seems to be OK, but, how to improve the face
colors?
What’s wrong?
三種膚色: Henrik Jensen 的研究 (2006 EGSR)
皮膚色由:表皮, 真皮層 黑色素, 血紅素,油脂,決定
非洲人含有較多的 Eumelanin 暗沈黑色素,白種人則含有較多的 Pheomelanin 明亮的黑色素 , 而
亞洲人含有兩種相似等量的黑色素
Cm: 暗沈黑色素, Bm: 暗沈VS 明亮黑色素 , Ch: 血紅素
半透明材質: Translucent ,活人與死人的差異
鼻子附近皮膚的毛孔 : (a) 中性表情, (b)氣憤(拉扯皮膚), (c) 真的照片。
Capture procedure flow diagram
Use a human face as a test 江任遠的臉
散射反射 VS. 似鏡面反射散射反射 (Diffuse Reflection)
與觀測位置無關 :COS (theta)
theta : 光源與物體平面法向量的夾角
似鏡面反射 (Specular reflection)
與觀測位置有關, COSn(alpha),
alpha: 光源反射角與觀測向量的夾角 , n = 1 to 10000
Use a faked orange as a testleft to right: diffuse, specular,
hybrid, real photo
(a) 散射打光 (Diffuse only) (b) 似鏡面反射 (Specular)
不均勻反射( Anisotropic reflection ) (a)(b)(c) 三種方向打光效果,注意下排的花
色變化 (I3D2006, 馬萬鈞 )
如何量測人臉的細微毛孔及縐摺?
粗燥皮膚:散射反射 ,
與觀測位置無關 :COS (theta)
theta : 光源與物體平面法向量的夾角
油脂皮膚:似鏡面反射 (Specular),
與觀測位置有關, COSn(alpha),
alpha: 光源反射角與觀測向量的夾角
從 X , Y , Z 三方向產生漸進式亮度 (gradient) 的光源 , 及全
亮度
This would potentially allow to quickly compute the normals by taking only 4 photographs, divide the gradient images by the constant image, and then directly map the range to a normal direction.
Observed reflectance r(x) in X direction vs. normal n(x)
Observed reflectance r(x) in X-direction:The normalized halfway vector between w (light
reflected direction) and v (viewing direction) corresponds to the normal direction.
左側:純散射對法向量的估計產生的結果右側: Specular 反射對法向量的估計產生的結果
高度反光物體也可用此法
Future games? Turing Test
Demo video
The End
Q&A