2 facial sketch tutorial - porto interactive center

FACIAL SKETCH PLUGIN FOR MAYA 2010 PIC – Porto Interactive Center José Carlos Miranda, Xenxo Alvarez Blanco, Verónica Costa Orvalho

2 FACIAL SKETCH TUTORIAL

FACIAL SKETCH TUTORIAL José Carlos Miranda, Xenxo Alvarez Blanco, Verónica Costa Orvalho

2

SUMMARY

We have developed a sketching system that allows easy, rapid and interactive prototyping of facial

expressions by drawing strokes. The sketching process is intuitive and similar to how we draw by hand.

The system allows to create facial poses on the fly. It supports non-hierarchy rigs, which make it very

convenient to control 3D models with complex structures, such as a face. We have implemented two

plugins for Autodesk Maya 2010: 2D SKETCH PLUGIN, which allows sketching on predefined and fixed

2D canvases; 3D SKETCH PLUGIN, which allows sketching directly over the 3D mesh and on a 2.5D

dynamic canvas, from any camera orientation;

SYSTEM FEATURES 2D SKETCH PLUGIN 3D SKETCH PLUGIN

DRAW CURVES: draw curves in fixed and predefined canvas draw curves directly on 3D mesh and on

dynamic canvas

CHANGE CURVES: modify the curves drawn

REFERENCE POINTS: show/hide curve control points

REFERENCE CURVES: show/hide reference curves (curves created from joints

initial position: reset curves – blue color) Not implemented;

3D DEFORMATION

WEIGHT:

to increase/decrease the “3D deformation window” (used to exaggerate certain facial

regions)

CREATE POSE: create a pose based on curves drawn

MODIFY POSE: modify selected pose based on curves drawn

RESET POSE: reset selected pose (to its initial position)

DELETE POSE: delete selected pose

SAVE POSES: save all poses in xml file

OPEN POSES: open xml file with poses previously recorded.

RETARGETING POSES: reuse the same pose (curves) in different characters Not implemented;

KEY RIG: put a key in the controls of the rig

SELECT RIG select the controls of the rig to work with the keys

RESET REGIONS: delete all curves and reset all joints to their initial position

RESET ALL: delete all curves, reset all joints to their initial position and delete all poses

BLEND SHAPE: blend shape interpolation using the Maya blendshape deform


3

SYSTEM OVERVIEW

DRAWING PANEL (A). This panel only exists in the 2D Sketch Plugin. Panel with 11 fixed and predefined

canvases to draw curves. By default the curves are painted in reset position (blue color). To paint a new curve, you

just need to select the tool icon and draw a stroke (red color) in each canvas. To reset a canvas click one time with

the mouse button in the canvas. To modify the curve change the reference points position (circle points). You can

hide the reference curves and reference points at any time (see the Sketching Tools on the Tool Settings window)

MODEL PANEL (B). In the 2D Sketch plugin, this panel shows the model deformation based on the strokes

drawn in the predefined canvases. In the 3D Sketch plugin, this panel shows the model deformation based on

the strokes drawn directly on the 3D mesh or drawn on dynamic canvases.

TOOL SETTINGS WINDOW (C). To open the tool settings window you need to press twice in the sketch tool in

the Maya tool box.


4

TOOL SETTINGS WINDOW DESCRIPTION

SKETCHING TOOLS:

1 Start Clock: to start the clock before sketching the

pose

2 Reset All: reset canvas, joints and poses created with

Pose Tools.

3, 4, 5 Deformation Weight to increase/decrease

the deformation (can be set for each region).

6 Reference Points Show/Hide curve reference points.

7 Reference Curves Show/Hide reference curves. The

reference curves in blue represent the reset position of the rig controls (only works in the 2D sketch plugin)

POSE TOOLS RIG TOOLS

8 Load Load XML file with poses.

9 Save Save the poses in a XML File

10 Create Create a pose and show the Poses

Window. Also, stops the clock when finish the pose.

11 Modify Modify the pose selected in the Poses

Window.

12 Reset Reset the selected pose.

13 Delete Delete the selected pose.

14 Key Rig Put a key in the controls of the rig.

15 Select Rig Select the controls of the rig to work

with the keys.

16 Reset Canvas Reset only the canvas curves.

KNOWN BUGS

Some errors displayed on the Maya status line don’t affect the system functionality.


5

BASIC TUTORIAL – 2D SKETCH PLUGIN

To test the plugin you need to follow the next steps:

1. Load Maya;

2. Load Model; 3. Load the plugin facialSketch2D.mll;

4. Run in the command line the mel script sketching;

Now, the system is ready to sketch facial expressions…..

DEFINITIONS

This is a canvas This is a reference curve (blue) This is a stroke in a canvas (red)

A Canvas is a drawing region. We

defined 11 facial regions: brow

left, brow right, eye left up, eye left down, eye right up, eye right down,

cheek left, cheek right, nose, mouth up, mouth down.

Reference curves are curves

generated automatically by the

system from the joints initial position: reset curves

A Stroke is a free-form curve

drawn on a canvas created by

the artist

The user can draw strokes on the canvas to deform the correspondent facial region. See the video 2Dsketch.mp4

(00:01:38).

SKETCHING A POSE

(i) Sketch the left BROW

Draw a stroke in correspondent canvas

and adjust the curve points to the

desired position


6

(ii) Close the left EYE

Change the reference curves to close

the left eye

(iii) Increase eye deformation

Increase eye deformation by editing the

Y Amount value on the Tool Settings

Sketching Tools > Deformation Weight Y drag slider to increase/decrease the deformation

(iv) Pull the left CHEEK

Sketch left cheek and adjust the control

points to the desired position

(v) Increase cheek deformation

Increase cheek deformation by editing

the X Amount value on the Tool Settings

Sketching Tools > Deformation Weight X drag slider to increase/decrease the deformation


7

(vi) Reset the cheek to the neutral position

(delete stroke)

One click inside the canvas (outside

the stroke) to delete the stroke and

reset this region.

(vii) Sketch the MOUTH

Draw a stroke on the upper canvas to deform

the upper part of the mouth (S1) and another stroke on the lower canvas to deform the lower

part of the mouth (S2)

(viii) CREATE the pose

Pose Tools > Create Display the poses in the shapes Manager

S1

S2


8

(ix) SAVE the pose in a

XML file

Pose Tools > SAVE > pose1.xml

NOTE: Data that is crucial for the experiment evaluation, like the time to create a pose, curve points,

number of clicks, number of curves created, modified and deleted are saved in the XML file.

Please, make sure each participant follows the next steps for each pose he will create:

1. RESET ALL

(to get the neutral pose)

2. START CLOCK (to start the clock before sketching the pose)

3. ….sketching the pose….

4. CREATE POSE (to stop the clock after finishing the pose)

5. SAVE POSE

(to save pose in xml file)


9

BASIC TUTORIAL – 3D SKETCH PLUGIN

To test the plugin you need to follow the next steps:

1. Load Maya; 2. Load Model;

3. Load the plugin facialSketch3D.mll; 4. Run in the command line the mel script sketching;

Now, the system is ready to sketch facial expressions…..

The user can draw strokes directly on the 3D mesh to deform a facial region. The region deformed will be the region closest to the first point of the stroke (where you started to draw).

Whenever a stroke comes out the mesh, a canvas is dynamically created to give the user the chance to continue

sketching. See the video 3Dsketch.mp4 (00:02:35).

SKETCHING A POSE

(i) Sketch the BROW

Draw a stroke from A to B;

Adjust the curve points 1, 2 and 3 to the

desired position

(ii) Close the upper part of the EYE


Adjust the curve point 1 to the desired

position

(iii) Close the lower part of the EYE



position

A

B

1

3 2

A

B

1

A B 1


10

(iv) Pull the left CHEEK


(notice that a dynamic canvas is created when the stroke comes out the mesh)

Adjust the curve points 1 and 2 to the

desired position

(v)

Sketch the NOSE


Adjust the curve points 1 and 3 to the

desired position

(vi) Stretch out the NOSE

Rotate the camera

adjust the middle curve point (2) to

come out the mesh to the desired

position

(vii) Reset the nose to the neutral position

(delete canvas and stroke)

One click inside the canvas (outside

the stroke) to delete it and reset the

nose.

1 3

2

2 A B

2

1 A

B 2


11

(viii) Stretch out the NOSE

(a different way)


(notice that a dynamic canvas is created when the stroke comes out the mesh)

adjust the curve point 1 to the desired

position

(ix) Sketch the MOUTH

Draw a stroke directly on the 3D mesh to

deform the upper part of the mouth (S1) and another stroke to deform the lower part of the

mouth (S2)

(x) Reset the mouth to the neutral position

(delete strokes)

One click on the upper part of the

mouth’s region (outside the stroke S1)

to reset it.

One click on the lower part of the

mouth’s region (outside the stroke S2)

to reset it.

(xi) Pull the MOUTH

To deform all joints of the same region at the same time, hold the <SHIFT> key while:

Draw a stroke from A (on the mesh) to B

(outside the mesh);


position

S1

1 A

B

S2

1 A

B


12

(xii) CREATE the pose

Pose Tools > Create Display the poses in the shapes Manager

(xiii) SAVE the pose in a

XML file

Pose Tools > Save > pose1.xml


13

NOTE: Data that is crucial for the experiment evaluation, like the time to create a pose, curve points,

joints position, number of clicks, number of curves created, modified and deleted are saved in the XML file.

Please, make sure each participant follows the next steps for each pose he will create:

1. RESET ALL

(to get the neutral pose)

2. START CLOCK (to start the clock before sketching the pose)

3. ….sketching the pose….

4. CREATE POSE (to stop the clock after finishing the pose)

5. SAVE POSE

(to save pose in xml file)

2 facial sketch tutorial - porto interactive center

Documents