Project Overview

• A new immersive 3D learning environment to increase math/science skills of deaf and hearing children

• The application teaches math/science concepts and ASL (American Sign Language) math terminology through user interaction with fantasy 3D virtual signers and environments

• It can be displayed in spatial immersive devices (SID) such as the FLEX [1] or CAVE [2], and includes a gesture control system

• [1] Fakespace Systems, FLEX http://www.fakespace.com/flexReflex.htm

• [2] The CAVE: a VR theater http://www.evl.uic.edu/pape/CAVE/oldCAVE/CAVE.html

S M I LE © (Science and Math in an Immersive Learning Environment)

Project Sponsors

National Science Foundation -NSF-RDE. "Software for Math Education of the Deaf". (grant#0622900)

PHS-NIH Nat. Inst. Deafness and Comm. Disorders. “Modeling the non-manuals of American Sign Language” (grant#511 1419 -1641)

The College of Technology at Purdue University -- I3 grant

(grant #00006585)

The Envision Center For Data Perceptualization

S M I LE © (Science and Math in an Immersive Learning Environment)

Research TeamPIs

Prof. Nicoletta Adamo-Villani, Department of Computer Graphics Technology Prof. Ronnie Wilbur, Department of Audiology and Speech Sciences and Chair of Linguistics

DEVELOPMENTRobin Wright, ASL signer David Jones, graduate student in Computer Graphics Technology Edward Carpenter, graduate student in Computer Graphics Technology Kelly Wright, senior in CGT Jim Gilchrist, senior in CGT Melissa Walsh, senior in CGTJac Piette, senior in CGT

COLLABORATORSThe students, teachers, and parents at The Indiana School for the Deaf (ISD) Dr. Laura Arns, Associate Director of the Envision Center for Data Perceptualization

S M I LE © (Science and Math in an Immersive Learning Environment)

Why this project ?• Deaf individuals are very underrepresented in the fields of

science and engineering. Several factors contribute to this disparity:

1. Difficulty in acquiring math concepts early in life

2. Significant delay in deaf children’s reading comprehension

3. Inaccessibility to incidental learning (exposure to media in which mathematical concepts are practiced and reinforced)

4 . Difficulty of hearing parents (73% deaf children have hearing parents) in conveying basic science and mathematical concepts in sign language

S M I LE © (Science and Math in an Immersive Learning Environment)

Why VR as the technology of choice ?

• Existing data suggest that VR technology offers significant, positive support for education in general

• Research shows that VR is particularly suitable to math and science education

• In regard to disabilities education, literature findings suggest that VR has advantages over other teaching technologies because it can fulfill the majority of the learning requirements of students with disabilities

S M I LE © (Science and Math in an Immersive Learning Environment)

Existing VR applications for the Hearing Impaired

• ‘Virtual Supermarket’ developed at the University of Nottingham, England

• VREAL (Virtual Reality Education for Assisted Living) project, funded by the U.S. Department of Education

S M I LE © (Science and Math in an Immersive Learning Environment)

User Interface Presentation Components

S M I LE © (Science and Math in an Immersive Learning Environment)

User Interface Input Mechanisms

1. A gesture control system, comprised of a pair of pinch gloves and a wrist tracker, allows the user to: (1) grasp and release virtual objects; (2) input a limited number of ASL hand-shapes; and (3) navigate the virtual environment.

or

2. A wand allows the user to: (1) grasp and release virtual objects, and (2) navigate the virtual environment.

S M I LE © (Science and Math in an Immersive Learning Environment)

Software and System Architecture• Maya 7.0

• 3D Studio Max

• Motion Builder

• OpenGL

• OpenSceneGraph

• VRJuggler toolkit

• OsgCal

• Cal3D

S M I LE © (Science and Math in an Immersive Learning Environment)

Innovative Features

• Seamless polygonal characters• Realistic deformation

• Organic appearance

• Realistic and fluid signing motion Motion is captured directly from non-hearing signers and

smoothly blended in real time

• Complex interactivity

S M I LE © (Science and Math in an Immersive Learning Environment)

Character designS M I LE © (Science and Math in an Immersive Learning Environment)

Character Rigging

Rigs

S M I LE © (Science and Math in an Immersive Learning Environment)

Signing Animation

Animation

S M I LE © (Science and Math in an Immersive Learning Environment)

Environment Design: the Clock Store

S M I LE © (Science and Math in an Immersive Learning Environment)

Environment Design: the Bakery

S M I LE © (Science and Math in an Immersive Learning Environment)

S M I LE © (Science and Math in an Immersive Learning Environment)

Environment Design: the Candy Store

Interactive content

S M I LE © (Science and Math in an Immersive Learning Environment)

Evaluation Methodology

• User task analysis• Expert panel evaluation

(1) usability of the program

(2) overall quality of the virtual world

(3) quality of the signing motion

• Formative evaluation (1) usability of the program

(2) appeal/fun

• Summative evaluation (ISD, Fall 2007)(1) Learning Outcomes

S M I LE © (Science and Math in an Immersive Learning Environment)

Evaluation Results

• Minor Usability Problems

• Need to improve the gesture control system

• Need to include alternative travel methods

• Need to include way-finding techniques

• Suggested use of narrative as a key element to increase the users’ level of engagement

S M I LE © (Science and Math in an Immersive Learning Environment)

Q U E S T I O N S ?

S M I LE © (Science and Math in an Immersive Learning Environment)

Visit the project website at: http://www2.tech.purdue.edu/cgt/i3/smile/

Read about SMILE in the Animation World Magazine: http://mag.awn.com/?article_no=3015