chinese-speaking 3d talking head

1

Chinese-Chinese-SpeakingSpeaking

3D 3D Talking HeadTalking Head

Project No: H08040Project No: H08040

Sang Siew HoonSang Siew Hoon

Supervisor: Dr Ng Teck KhimSupervisor: Dr Ng Teck Khim

2

System ObjectivesSystem Objectives create a realistic Chinese-create a realistic Chinese-

speaking 3D talking model speaking 3D talking model

-- automaticautomatic

-- accurate and realisticaccurate and realistic

-- portableportable

3

ProblemsProblems Lack of references on Lack of references on

ChineseChinese Involves various fields Involves various fields

-- Face modelingFace modeling

-- Text-to-speechText-to-speech

-- Face Animation (Focus)Face Animation (Focus)

4

System OverviewSystem Overview5 modules5 modules

1.1. Face Model PreparationFace Model Preparation

2.2. Chinese VisemesChinese Visemes

3.3. Text-to-Speech (TTS)Text-to-Speech (TTS)

4.4. AnimationAnimation

5.5. RenderingRendering

5

6

Face Model Face Model PreparationPreparation

7

Face Model Face Model PreparationPreparation

Face Modeling 3DS MaxFace Modeling 3DS Max

8

Face Model Face Model PreparationPreparation

Defining MPEG-4 feature pointsDefining MPEG-4 feature points

9

Face Model Face Model PreparationPreparation

Assigning lip verticesAssigning lip vertices

10

Face Model Face Model PreparationPreparation

Cloning morph targetsCloning morph targets

FAP#3 open_jaw

FAP#4 lower_t_midlip

11

Chinese VisemesChinese VisemesVisemesVisemes- the visual equivalent of a the visual equivalent of a

phonemephoneme

1.1. Classification of Chinese Classification of Chinese SyllablesSyllables

2.2. Definition of Chinese VisemesDefinition of Chinese Visemes

3.3. Refining Dynamic VisemesRefining Dynamic Visemes

12

Chinese VisemesChinese Visemes Definition of Chinese VisemesDefinition of Chinese Visemes

13

Chinese VisemesChinese Visemes Definition of Chinese VisemesDefinition of Chinese Visemes

14

Chinese VisemesChinese Visemes Definition of Chinese VisemesDefinition of Chinese Visemes

15

Chinese VisemesChinese Visemes Refining Dynamic VisemesRefining Dynamic Visemes1.1. <i> +<i> + e only e only replace <f> erreplace <f> er2.2. zh, z, y + I zh, z, y + I drop i drop i3.3. d, l, g, j, zh, z, y, w + u | ü d, l, g, j, zh, z, y, w + u | ü drop <i> drop <i>4.4. j, y + <f> ending with an j, y + <f> ending with an replace the an by en replace the an by en5.5. j + complex finals headed by i and followed by more than one <f> j + complex finals headed by i and followed by more than one <f> drop i in complex finals drop i in complex finals

16

Text-to-SpeechText-to-Speech

17

Text-to-SpeechText-to-Speech

18

AnimationAnimation1.1. CoarticulationCoarticulation

2.2. Automatic Generation of Automatic Generation of Face AnimationFace Animation

3.3. Enhancing RealismEnhancing Realism

19

AnimationAnimation Enhancing realismEnhancing realism

- Eye blinks- Eye blinks

- Eyebrow raising- Eyebrow raising

- Gaze- Gaze

- Head Rotation- Head Rotation

20

ResultsResults Accurate and realisticAccurate and realistic No discrepancies for new No discrepancies for new

face modelsface models Automated results need Automated results need

slight manual interventionslight manual intervention

http://www.slamstudio.com/HYP/videoshttp://www.slamstudio.com/HYP/videos

21

ConclusionConclusion ContributionsContributions1.1. own system to define Chinese visemes own system to define Chinese visemes

- adds to the current research works of- adds to the current research works of

Chinese talking heads (currently Chinese talking heads (currently limited). limited).

2.2. an automatic Chinese lip-an automatic Chinese lip-synchronization systemsynchronization system

- saves an animator much time and - saves an animator much time and efforteffort

22

ConclusionConclusion System LimitationsSystem Limitations

1.1. TTS limitationTTS limitation

2.2. FAPs conflictsFAPs conflicts

3.3. Not integrated Not integrated

– – inconvenient for untrained inconvenient for untrained usersusers

23

ConclusionConclusion Future WorksFuture Works1.1. Automatic audio signal processing Automatic audio signal processing

2.2. Resolve FAPs values conflicts Resolve FAPs values conflicts

3.3. Automatic feature points Automatic feature points assignment system assignment system

4.4. Integration of systemIntegration of system