building the design studio of the future aaron adler jacob eisenstein michael oltmans lisa guttentag...

Building the Design Studio of the Future

Aaron AdlerJacob EisensteinMichael OltmansLisa GuttentagRandall Davis

October 23, 2004

AARON ADLER Building the Design Studio of the Future October 23, 2004

Sketching Is Not the Whole Picture


Speech and Gestures

• Express more precise spatial and structural relationships

• Relate sketches that are drawn from different perspectives

• Describe temporal events or paths of motion

• Disambiguate using multiple modalities


Roadmap

Exploratory Studies

Multimodal Communication

DomainModeling

DialogueManagement

Recognition Components

WoZ Dialogue Studies

DomainReasoning

Multimodal Understanding

Multimodal Intelligent

Design Studio

Usability

Human Dialogue

- Out-of-vocabulary terms

- Illuminate unclear parts- Possible design mistakes


Outline

• Sketching is not the whole picture

• Roadmap

• Trimodal Studies

• Preliminary Components– Speech and Sketching– Gesture and Speech


Interesting Questions to Study

• Sketch– What information is best communicated with

sketching?– How do people sketch when they’re

explaining things?– How do they sketch when they’re taking

notes?



• Speech– What information is best communicated with

speech?– How do people talk when they’re drawing?

How can ASR better support speech in the context of other modalities?

– How should computers generate and understand back-channel feedback?



• Gesture– What information is best communicated with gesture?– Which features of gesture are idiosyncratic? Which

are universal?– What does gesture tell us about discourse structure?– Are gestures cues for disfluencies? Sentence

breaks?– How can we build embodied agents with realistic body

language?


Trimodal Interaction Study

• How are speech, gesture, and sketching used together?– 15 pairs of participants– Two video cameras, a Mimio, and a Tablet PC– 4.5 hours of data


Looking Ahead to Recognition

• Captured digital video

• Colored gloves used for future hand tracking

• Headset microphones

• Time-stamped sketch data


Diagrams and objects


Trimodal Setup

Listener

Speaker


Trimodal Setup

Listener SetupSpeaker Setup


Details to Observe

• Structure then function• Back channel communication• Collaboration between speaker and

listener


Movie Clip I


Movie Clip II


Movie Clip III


Early Results

• We identified two phases in the explanations:– Structure phase– Function phase

• Participants used back channel communication during the explanations

• The amount of sketching the listener did varied


Outline

• Sketching is not the whole picture

• Roadmap

• Trimodal Studies

• Preliminary Components– Speech and Sketching– Gesture and Speech


Results: Speech and Sketching

• Filled pauses important: “ahh”, “umm”

• Key phrases: “there are”, “and”


Multimodal Understanding and Integration

• Combining speech and sketching we can make it easier to perform some operations

“There are three identical pendulums.”

“There are three touching pendulums.”


Speech-Gesture Integration

• Align referential pronouns and gestures to the diagram

“ this piece is made of wood”

• “it moves back and forth like this”

deictic tracing trajectory

Time

this piece goes here this…andso


Results: Speech and Gesture

• Corpus-based approach– Feature set based on linguistic knowledge

• Improves performance– 84% F-measure for baseline system– 95% F-measure for optimization system


Conclusion

• Intelligent Design Studio of the Future• Trimodal Corpus• Empirical Results• Initial Components

building the design studio of the future aaron adler jacob eisenstein michael oltmans lisa guttentag...

Documents

future speech

aaron adlerbuilding

future sketching

speech slide

big picture design studio

picture slide

future outline sketching

future dia