a look at avatar technology and signing mediums
TRANSCRIPT
![Page 1: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/1.jpg)
A LOOK AT AVATAR TECHNOLOGY AND SIGNING MEDIUMS
Guest Lecture for Language technology for accessibility, Institute of Computational Linguistics, University of Zurich
Robert SmithSchool of Informatics and Engineering
Institute of Technology, BlanchardstownIrish Sign Language Special Interest Group
Computational and Functional Linguistics Group
![Page 2: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/2.jpg)
The “why” question
• Why do we need sign language?• 1st language of a nations deaf community
• Why do we need signing video or avatars?• Cannot “write” sign language like we do with spoken languages• Records• Information (TV, web, classroom, kiosk, etc…)
• Can’t Deaf people read subtitles, pamphlets etc.?• Yes but the average reading age of Deaf school leavers is comparable to
that of an 8-9 year old hearing child (Conrad, 1979)• Sign language for most Deaf people, is their 1st language
• Lip reading?• Really? Can you do it?
10/21/14 Robert Smith 2014 2
![Page 3: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/3.jpg)
Signing mediums
•When the signer is present•Live signing
•When the signer is not present•Signing video•Traditional animation•Motion capture signing•Synthesised signing
10/21/14 Robert Smith 2014 3
![Page 4: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/4.jpg)
The problems with video
10/21/14 Robert Smith 2014 4
![Page 5: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/5.jpg)
Signing avatars
•Signing video
•Traditional animation•Motion capture signing•Synthesised signing
10/21/14 Robert Smith 2014 5
These methods require knowledge of animation
The reminder of this lecture is dedicated to understanding avatar animation
![Page 6: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/6.jpg)
Origin of AvatarsAvatar (Original, Hindu term), manifestation or
appearance of a Hindu deity
Avatar (computing term), the graphical representation of a user
Avatar (Sign Language term), a 3D animated character used to deliver Sign Language utterances
Robert Smith 201410/21/14 6
![Page 7: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/7.jpg)
ANIMATION BASICS
10/21/14 Robert Smith 2014 7
![Page 8: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/8.jpg)
Frame on movie reel
Robert Smith 201410/21/14 8
![Page 9: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/9.jpg)
10/21/14 Robert Smith 2014 9
Zoopraxiscope
![Page 10: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/10.jpg)
Animated ball
Animation Frame
Robert Smith 201410/21/14 10
![Page 11: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/11.jpg)
2D animation
Robert Smith 201410/21/14 11
![Page 12: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/12.jpg)
2D Evolution, Pencil-to-Pencil
Robert Smith 201410/21/14 12
![Page 13: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/13.jpg)
3D ANIMATION
10/21/14 Robert Smith 2014 13
![Page 14: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/14.jpg)
2D - 3D animation
1994, and the first completely computer-generated animated movie was Toy Story
Robert Smith 201410/21/14 14
![Page 15: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/15.jpg)
Concept drawing
Body shape, sizes, colours, textures etc…
Robert Smith 201410/21/14 15
![Page 16: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/16.jpg)
Analogy (Papier-mâché)
1. Structure 2. Mesh3. Texture4. Colour
10/21/14 Robert Smith 2014 16
![Page 17: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/17.jpg)
Analogy (Papier-mâché)
1. Structure 2. Mesh3. Texture4. Colour
10/21/14 Robert Smith 2014 17
We will look at this last
![Page 18: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/18.jpg)
Analogy (Papier-mâché)
1. Structure 2. Mesh3. Texture4. Colour
10/21/14 Robert Smith 2014 18
This 1st
![Page 19: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/19.jpg)
Body Shape - 3D polygon mesh
Robert Smith 201410/21/14 19
![Page 20: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/20.jpg)
Mesh structure
Robert Smith 201410/21/14 20
![Page 21: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/21.jpg)
Number of polygons is important
Robert Smith 201410/21/14 21
![Page 22: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/22.jpg)
Adding skin detail
Normal maps (skin pores etc)
Specular and diffuse lighting mapsBump maps / texture maps
Robert Smith 201410/21/14 22
![Page 23: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/23.jpg)
Bump/Specular/Normal maps on a 3D mesh
Robert Smith 201410/21/14 23
![Page 25: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/25.jpg)
Analogy (Papier-mâché)
1. Structure 2. Mesh3. Texture4. Colour
10/21/14 Robert Smith 2014 25
![Page 26: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/26.jpg)
Structure/Movement (armatures)
Robert Smith 201410/21/14 26
![Page 27: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/27.jpg)
Body rigging
Robert Smith 201410/21/14 27
![Page 28: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/28.jpg)
Basic facial rigging
Robert Smith 201410/21/14 28
![Page 29: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/29.jpg)
Robert Smith 201410/21/14 29
![Page 30: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/30.jpg)
Robert Smith 201410/21/14 30
![Page 31: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/31.jpg)
3D Rendering
Robert Smith 201410/21/14 31
![Page 32: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/32.jpg)
Hardware vs time
10/21/14 Robert Smith 2014 32
![Page 33: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/33.jpg)
Real-time rendering
3D Rendering - examples
Robert Smith 201410/21/14 33
![Page 34: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/34.jpg)
3D animation methods
Up to now I’ve described how 3D characters are created…now lets talk about how we can use them…
1. ‘Traditional’ Animation using animators
2. Motion capture (mocap)
3. Full synthesis
Robert Smith 201410/21/14 34
![Page 35: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/35.jpg)
3D animation methods
Up to now I’ve described how 3D characters are created…now lets talk about how we can use them…
1. ‘Traditional’ Animation using animators
2. Motion capture (mocap)
3. Full synthesis
Robert Smith 201410/21/14 35
![Page 36: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/36.jpg)
Using Animators
• 3 year production
• Teams of artists and animators
• 3500 Sun Microsystems processors, compared with 1400 for Toy Story 2 and only 200 for Toy Story
Robert Smith 201410/21/14 36
![Page 38: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/38.jpg)
3D animation methods
Up to now I’ve described how 3D characters are created…now lets talk about how we can use them…
1. ‘Traditional’ Animation using animators
2. Motion capture (mocap)
3. Full synthesis
Robert Smith 201410/21/14 38
![Page 39: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/39.jpg)
Motion captureThe most commonly used technique for sign language animation on the web is motion capture
Robert Smith 201410/21/14 39
![Page 40: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/40.jpg)
Motion capture setup
Robert Smith 201410/21/14 40
![Page 41: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/41.jpg)
Facial emotion on these projects was good
Motion/Performance capture - examples
Robert Smith 201410/21/14 41
![Page 42: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/42.jpg)
“Avatar” movie• 190 inferred cameras + Dozens of HD cameras
Robert Smith 201410/21/14 42
![Page 43: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/43.jpg)
Facial mapping
Robert Smith 201410/21/14 43
![Page 44: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/44.jpg)
360˚ HD photo sphere
Robert Smith 201410/21/14 44
![Page 45: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/45.jpg)
Capturing emotion
Robert Smith 201410/21/14 45
![Page 46: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/46.jpg)
MoCap drawbacks
• Expensive equipment.
• Requires labour-intensive calibration of the equipment.
• …and post-editing of the captured data.
• Good for the movie industry• Realism, faster production, cheaper
• Bad for Sign Language• Finite data set, Signed Exact English
Robert Smith 201410/21/14 46
![Page 47: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/47.jpg)
3D animation methods
Up to now I’ve described how 3D characters are created…now lets talk about how we can use them…
1. ‘Traditional’ Animation using animators
2. Motion capture (mocap)
3. Full synthesis
Robert Smith 201410/21/14 47
![Page 48: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/48.jpg)
SL Synthesis
• Sign synthesis is not new (The concept is over 10 years old).
• Less costly to implement.
• Has the flexibility of a writing system.
• Real-time rendering.
Robert Smith 201410/21/14 48
![Page 49: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/49.jpg)
SL Synthesis2005-2006, ASL project,
University of Pennsylvania
2006-2007, The South African SL Project,
University of Stellenbosch
2004-2007, The Spanish SL project,
University of Madrid
Robert Smith 201410/21/14 49
![Page 50: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/50.jpg)
Issues with synthesis
• Focus is often narrowed i.e.• Machine Translation• Specific task• Specific linguistic attribute
• Low processing requirements for rendering• Mesh polygons• Armature bones• Texture/Specular/Normal mapping
• Little work to date on NMF
Robert Smith 201410/21/14 50
![Page 51: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/51.jpg)
State-of-the-art system ( )
HamNoSys-to-SiGML
AnimGenAnimation data
SiG
ML
HamNoSys
<sigml> <hns_sign gloss="$PROD:Morning"> …al> </hns_sign></sigml>
Robert Smith 201410/21/14 51
![Page 52: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/52.jpg)
SummaryMoCap
Synthesis
Concept Mesh Skin Rigging Render
HamNoSysAdd animation
parametersMachine readable
Render
Concept Mesh Skin Rigging Render
Capture Motion Data
SearchData
Machine readable
Assemble utterance
Add animation parameters Render
Robert Smith 201410/21/14 52
![Page 53: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/53.jpg)
Summary
•When the signer is present•Live signing
•When the signer is not present•Signing video•Traditional animation•Motion capture signing•Synthesised signing
10/21/14 Robert Smith 2014 53
![Page 54: A look at Avatar technology and Signing mediums](https://reader033.vdocuments.site/reader033/viewer/2022051617/55a38f2b1a28ab60028b45ea/html5/thumbnails/54.jpg)
Questions
?
Robert Smith 201410/21/14 54