user interface design tools for the future multimodal ui research in the hcc james a. landay jason...

User Interface Design Toolsfor the Future

Multimodal UI Research in the HCC

James A. Landay

Jason Hong, Scott Klemmer,Jimmy Lin, Mark Newman, & Anoop Sinha

HCC RetreatJuly 5, 2000

7/5/2000 2

Several Projects on Design UIs of the Future

Designer’s Outpost SUEDE Multimodal Design Assistant

7/5/2000 3

The Future: Information-centric UIs

7/5/2000 4

Designer’s Outpost: Tangible Tools for Information Design

7/5/2000 5

Designer’s Outpost: Virtual Feedback on Physical Layouts

7/5/2000 6

Designer’s Outpost: Virtual Feedback on Physical Layouts

7/5/2000 7

Designer’s Outpost: Virtual Linking of Related Information

7/5/2000 8

Designer’s Outpost: Export to Next Stage Tools (DENIM)

7/5/2000 9

Designer’s Outpost: Infrastructure for Next Prototype

Two video/still cameras* still camera above desk/wall

+captures ink / IDs* video camera below desk/wall

+occlusion free+captures structure

SmartBoard* 1280 x 1024 resolution

+looking for a projector donation* direct pen input

Cross iPen tablet* high-res ink capture

7/5/2000 10

One Future: Our Access to Information will be via Speech

Read my important email

7/5/2000 11

SUEDE: Low-fidelity Prototyping for Speech-based UIs

7/5/2000 12

A Better Future: Our Access to Info will be via Multimodal UIs

Take advantage of the best mode of input/output for the task/context at hand

How do we design so that these new devices will not seem impoverished relative to desktop computers?

7/5/2000 13

A Better Future: Our Access to Info will be via Multimodal UIs

How do we combine speech, gesture, etc. into a UI design?* rapid production of “rough cuts”

+ informal (sketching / “Wizard of Oz”)+ iterative design (user testing/fast mods)

* generate initial code+ UIs for multiple devices+ designer adds detail / improve UI

Study* uses of these novel modes alone

+ e.g., SUEDE* construction of multimodal apps

+ e.g., a multimedia notebook

Early stages so far* based on inferring models

Model

More information on these design tools at

http://guir.berkeley.edu