real-time performance-based facial...
TRANSCRIPT
![Page 1: Real-time Performance-Based Facial Animationresources.mpi-inf.mpg.de/...RealTimeFacialAnimation... · Real-time Performance-Based Facial Animation Mark Pauly. Eurographics 2012, Cagliari,](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5f9d246976330e7f5b26c0/html5/thumbnails/1.jpg)
Real-time Performance-BasedFacial Animation
Mark Pauly
![Page 2: Real-time Performance-Based Facial Animationresources.mpi-inf.mpg.de/...RealTimeFacialAnimation... · Real-time Performance-Based Facial Animation Mark Pauly. Eurographics 2012, Cagliari,](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5f9d246976330e7f5b26c0/html5/thumbnails/2.jpg)
Eurographics 2012, Cagliari, Italy
Introduction
2
![Page 3: Real-time Performance-Based Facial Animationresources.mpi-inf.mpg.de/...RealTimeFacialAnimation... · Real-time Performance-Based Facial Animation Mark Pauly. Eurographics 2012, Cagliari,](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5f9d246976330e7f5b26c0/html5/thumbnails/3.jpg)
Eurographics 2012, Cagliari, Italy
Overview
Facial Tracking using Blendshape Model• rigid transformation + blendshape weights per frame
Personalized Blendshape Model• static face capture• example based facial rigging
Tracking• rigid & nonrigid registration• animation prior
3
![Page 4: Real-time Performance-Based Facial Animationresources.mpi-inf.mpg.de/...RealTimeFacialAnimation... · Real-time Performance-Based Facial Animation Mark Pauly. Eurographics 2012, Cagliari,](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5f9d246976330e7f5b26c0/html5/thumbnails/4.jpg)
Eurographics 2012, Cagliari, ItalyEurographics 2012, Cagliari, Italy
Animation Pipeline
![Page 5: Real-time Performance-Based Facial Animationresources.mpi-inf.mpg.de/...RealTimeFacialAnimation... · Real-time Performance-Based Facial Animation Mark Pauly. Eurographics 2012, Cagliari,](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5f9d246976330e7f5b26c0/html5/thumbnails/5.jpg)
Eurographics 2012, Cagliari, Italy
Facial Expression Model
Static Face Capture
[Weise et al, 2009]
5
![Page 6: Real-time Performance-Based Facial Animationresources.mpi-inf.mpg.de/...RealTimeFacialAnimation... · Real-time Performance-Based Facial Animation Mark Pauly. Eurographics 2012, Cagliari,](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5f9d246976330e7f5b26c0/html5/thumbnails/6.jpg)
Eurographics 2012, Cagliari, Italy
Facial Expression Model
Face Fitting
[Li et al, 2009]
6
![Page 7: Real-time Performance-Based Facial Animationresources.mpi-inf.mpg.de/...RealTimeFacialAnimation... · Real-time Performance-Based Facial Animation Mark Pauly. Eurographics 2012, Cagliari,](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5f9d246976330e7f5b26c0/html5/thumbnails/7.jpg)
Eurographics 2012, Cagliari, Italy
Facial Expression Model
Texture Reconstruction
Poisson Image Editing[Pérez et al, 2003]
7
![Page 8: Real-time Performance-Based Facial Animationresources.mpi-inf.mpg.de/...RealTimeFacialAnimation... · Real-time Performance-Based Facial Animation Mark Pauly. Eurographics 2012, Cagliari,](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5f9d246976330e7f5b26c0/html5/thumbnails/8.jpg)
Eurographics 2012, Cagliari, Italy
Facial Expression Model
Example-based Facial Rigging
8
[Li et al, 2010]generic template personalized expression model
![Page 9: Real-time Performance-Based Facial Animationresources.mpi-inf.mpg.de/...RealTimeFacialAnimation... · Real-time Performance-Based Facial Animation Mark Pauly. Eurographics 2012, Cagliari,](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5f9d246976330e7f5b26c0/html5/thumbnails/9.jpg)
Eurographics 2012, Cagliari, Italy
Rigid Tracking
9
![Page 10: Real-time Performance-Based Facial Animationresources.mpi-inf.mpg.de/...RealTimeFacialAnimation... · Real-time Performance-Based Facial Animation Mark Pauly. Eurographics 2012, Cagliari,](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5f9d246976330e7f5b26c0/html5/thumbnails/10.jpg)
Estimate blendshape weights• similar to scans & color image• plausible animation
MAP estimation• non-rigid registration based on geometry & texture• animation prior from existing animation sequences
Eurographics 2012, Cagliari, Italy
Non-rigid Tracking
10
![Page 11: Real-time Performance-Based Facial Animationresources.mpi-inf.mpg.de/...RealTimeFacialAnimation... · Real-time Performance-Based Facial Animation Mark Pauly. Eurographics 2012, Cagliari,](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5f9d246976330e7f5b26c0/html5/thumbnails/11.jpg)
Eurographics 2012, Cagliari, ItalyEurographics 2012, Cagliari, Italy
N-Dim Expression Space
![Page 12: Real-time Performance-Based Facial Animationresources.mpi-inf.mpg.de/...RealTimeFacialAnimation... · Real-time Performance-Based Facial Animation Mark Pauly. Eurographics 2012, Cagliari,](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5f9d246976330e7f5b26c0/html5/thumbnails/12.jpg)
Eurographics 2012, Cagliari, ItalyEurographics 2012, Cagliari, Italy
N-Dim Expression Space
![Page 13: Real-time Performance-Based Facial Animationresources.mpi-inf.mpg.de/...RealTimeFacialAnimation... · Real-time Performance-Based Facial Animation Mark Pauly. Eurographics 2012, Cagliari,](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5f9d246976330e7f5b26c0/html5/thumbnails/13.jpg)
Eurographics 2012, Cagliari, ItalyEurographics 2012, Cagliari, Italy
N-Dim Expression Space
![Page 14: Real-time Performance-Based Facial Animationresources.mpi-inf.mpg.de/...RealTimeFacialAnimation... · Real-time Performance-Based Facial Animation Mark Pauly. Eurographics 2012, Cagliari,](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5f9d246976330e7f5b26c0/html5/thumbnails/14.jpg)
Eurographics 2012, Cagliari, ItalyEurographics 2012, Cagliari, Italy
Animation Manifold
![Page 15: Real-time Performance-Based Facial Animationresources.mpi-inf.mpg.de/...RealTimeFacialAnimation... · Real-time Performance-Based Facial Animation Mark Pauly. Eurographics 2012, Cagliari,](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5f9d246976330e7f5b26c0/html5/thumbnails/15.jpg)
Eurographics 2012, Cagliari, ItalyEurographics 2012, Cagliari, Italy
Probabilistic Expression Prior
![Page 16: Real-time Performance-Based Facial Animationresources.mpi-inf.mpg.de/...RealTimeFacialAnimation... · Real-time Performance-Based Facial Animation Mark Pauly. Eurographics 2012, Cagliari,](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5f9d246976330e7f5b26c0/html5/thumbnails/16.jpg)
Eurographics 2012, Cagliari, ItalyEurographics 2012, Cagliari, Italy
Probabilistic Animation Prior
![Page 17: Real-time Performance-Based Facial Animationresources.mpi-inf.mpg.de/...RealTimeFacialAnimation... · Real-time Performance-Based Facial Animation Mark Pauly. Eurographics 2012, Cagliari,](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5f9d246976330e7f5b26c0/html5/thumbnails/17.jpg)
Eurographics 2012, Cagliari, ItalyEurographics 2012, Cagliari, Italy
Temporal Joint Probabilistic Distribution
![Page 18: Real-time Performance-Based Facial Animationresources.mpi-inf.mpg.de/...RealTimeFacialAnimation... · Real-time Performance-Based Facial Animation Mark Pauly. Eurographics 2012, Cagliari,](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5f9d246976330e7f5b26c0/html5/thumbnails/18.jpg)
Eurographics 2012, Cagliari, ItalyEurographics 2012, Cagliari, Italy
MAP Estimation
![Page 19: Real-time Performance-Based Facial Animationresources.mpi-inf.mpg.de/...RealTimeFacialAnimation... · Real-time Performance-Based Facial Animation Mark Pauly. Eurographics 2012, Cagliari,](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5f9d246976330e7f5b26c0/html5/thumbnails/19.jpg)
Eurographics 2012, Cagliari, Italy
Demo
www.faceshift.com