björn hartmann loren yu, abel allison, yeonsoo yang scott r. klemmer

stanford hci group oct 21, 2008

Björn HartmannLoren Yu, Abel Allison, Yeonsoo

YangScott R. Klemmer

Design as ExplorationCreating Interface Alternatives through Parallel Authoring and Runtime Tuning

2

Exploration of alternatives is central to design.

3

Prototypes for the Microsoft mouseFrom Moggridge, Designing Interactions, Ch2

MAP THEDESIGN SPACE

4From Design Secrets: Products 2

CommunicatE

5

Tohidi et al, CHI 2006

TEST

6

How can tools support the creation of user interface alternatives?

Color picker #1

Color picker #2 Color picker #3

7

“[D]esigners frequently wanted to have multiple designs side-by-side [...] However […] there is no built-in way in today’s implementation tools to have two versions of a behavior operating side-by-side.”

Myers et al., VL/HCC 2008

Adobe Flash Arduino / Processing

Adobe Dreamweaver Xcode (iPhone)

Interaction Designers Write Code

8

9

Programming requires working with two representations.

Editor: author behavior

… …

………

Runtime: observe behavior

10

Outline Requirements for Alternatives of

Programmed Interactions Juxtapose for Desktop Interactions:

System & Evaluation Alternatives off the Desktop Related & Future Work

11

3 Requirements for Programmed Interactions

Manage parameter variations Manage code alternatives Access variations & alternatives at

runtime

(based on 3 interviews & code inspection)

12

Manage parameter variations Manage code alternatives Access variations & alternatives at

runtime

Processing code brought in by an interviewee

3 Requirements for Programmed Interactions

13

Manage parameter variations Manage code alternatives Access variations & alternatives at

runtime

3 Requirements for Programmed Interactions

14

Juxtapose: Source alternatives…

15

Juxtapose: Source alternatives…

16

… are executed in parallel,

17

and tuned through an generated UI.

18

Parallel Editing

Juxtapose editor forActionScript 2.Generates Flash files.

19

Parallel Editing

20

Parallel Editing

Linked Editing: Toomim 2004

21

Parallel Editing

Linked Editing: Toomim 2004

22

Example: Rethinking the Progress Bar Harrison et al., UIST 2007

Progress profilesProgress bar test application

23

24

Parallel Input: Event Echoing

25

Implementing Parallel Editing: The Cursor Correspondence Problem

Straightforward solution:Longest common subsequence algorithmdiff(A,B,…,N) Run during idle times

Doc. A Doc. B Doc. C

26

Limits of Parallel Editing & Execution

Level of granularity for editing:Extension to multi-file projects is not trivial.

Can the user make sense of behaviors running in parallel?

27

Limits of Event Echoing

Yes No

Delete this file?

Ok

Delete this file?

Yes No

28

Parameter Tuning

29

Example: Tuning Phosphor Baudisch, UIST 2006

30

Example: Tuning Phosphor Baudisch, UIST 2006

31

Generating Tuning Interfaces

Juxtapose User Library

User’s Application://…

Number A = 10;Boolean B = true;

User ProcessJuxtapose Process

Inspect

Send variable info

X B

A

32

Generating Tuning Interfaces

Juxtapose User Library

User’s Application://…

Number A = 10;Boolean B = true;

User ProcessJuxtapose Process

Update

Send tuning message

X B

A

33

What parameter ranges should be chosen?0.0-1.0?

0-100?0-255?Number.minValue –

Number.maxValue?

Any nontrivial program property is undecidable.

34

Take a guess, then give control to the user.

At authoring time: source annotations

35

Limits of Tuning Which variables matter?

Juxtapose extracts only globals

36

Limits of Tuning Which variables matter?

Juxtapose extracts only globals Are variables read again?

Juxtapose uses callbacks

01 var tunable = 5; //@RANGE 0..100 03 var callback= function(varName,oldVal,newVal){ 04 redraw(); 05 return newVal; 06 } 07 this.watch(′tunable′,callback);

Understanding User Strategies & Measuring Benefits Questions:

How does Juxtapose fit into interaction prototyping practice?

Can we quantify the benefits? Method:

Lab evaluation, N=18, students 75-90 minute sessions with design

tasks & survey

37

38

Mapping TaskGiven ActionScript code that loads a

multilayered map, develop navigation options for a handheld GPS prototype for pedestrians and bicyclists.

39

Alternatives for MapNavigation & Rendering

40

Participants’ Strategies Linked vs. unlinked editing with

multiple alternatives applied by all participants.

Alternatives used both for different scenarios, as well as “scratch space”

Snapshots important to save state

41

Suggested improvements Automate code restructuring for

callbacks Introduce direct manipulation for

parameters in user’s application (when possible)

42

Tree Matching TaskSearch in 4D parameter space. Given recursive drawing code for this:

Produce these:

43

44

0

10

20

30

40

50

60

70

2.60

64.26Mean Parameter Changes

Tested per Minute

ControlJuxtapose

Chan

ges/

min

ute

45

Tree 1 Tree 2 Tree 3 Tree 40

50100150200250300350

Tree Matching Task:Mean Completion Times by Tree

ControlJuxtapose

seco

nds

p<0.01 p~0.01not significant

not significant

46

Outline Requirements for Alternatives of

Programmed Interactions Juxtapose for Desktop Interactions:

System & Evaluation Alternatives off the Desktop Related & Future Work

47

Juxtapose Mobile

Alternative 1

Alternative 2

Alternative3

48

49

JuxtaposeRuntime on PC

Juxtapose Mobile

Juxtapose WrapperUser’s FlashApplication

Juxtapose WrapperUser’s FlashApplication

TCP/IPvar1

Alt. 1 Alt. 2

var2

50

Juxtapose for Microcontrollers

Arduino:8bit Atmel AVR RISC chip

programmed in C with gcc

51

How might one implement variable tuning in a language without reflection?

52

Parse code and build symbol table

Var. Name Type Pointer“dly” int &dly“blink” boolean &blink

int dly = 25;boolean blink=false;void setup() {...}//…

53

Emit table into code & compile

void initVarTable(void) { varTable[1].varName = PSTR("dly"); varTable[1].varType = VAR_TYPE_INT; varTable[1].varPtr = &dly;

varTable[2].varName = PSTR("blink"); varTable[2].varType = VAR_TYPE_BOOLEAN; varTable[2].varPtr = &blink;}

// plus a bunch of communication code...

Auto-generatedtable

54

At Runtime

Serial message:

tune “dly” value 100

*(varTable[“dly”].varPtr) = 100;X

dly blink

55

So What’s Different?Target Platform How are

alternatives executed? Why?

How do users interact with alternatives?

In parallel Sequentially or in parallel

In parallel, because commodity hardware is (relatively) cheap

Sequentially(parallelism is possible, but less appealing)

Sequentially, because custom hardware is expensive to build

Sequentially

56

What’s Important?A structured approach to alternatives eliminates cruft and enables more exploration.

57

What’s Important?A structured approach to alternatives eliminates cruft and enables more exploration.Tool support requires connecting authoring and execution environments.3 Techniques:

Linked editing to manage code alternatives

Execute set of programs side-by-side Auto-generate tuning interface

58

Related Work

Design Galleries, Marks, SIGGRAPH 97Spreadsheets for Images, Levoy, SIGGRAPH 94Spreadsheets for Visualization, Chi, IEEE CGA 98Parameter Spectrums & Parallel Pies, Terry, UIST 02 & CHI 04Subjunctive Interfaces, Lunzer, TOCHI 08TEAM STORM, Hailpern, CHI 07

(only first authors listed)

Partials, Terry, PhD Thesis 05Amulet Inspector, Myers, IEEE ToSE 97JPie, Goldman,Sci. of Comp. Prog. 98ChucK, Wang, NIME 04 Adobe Image Foundation Tookit, 08

Alternatives in Other DomainsAugmented Development Tools

59

Related Work

Language-level support

Partials, Terry, PhD Thesis 05Amulet Inspector, Myers, IEEE ToSE 97JPie, Goldman,Sci. of Comp. Prog. 98ChucK, Wang, NIME 04 Adobe Image Foundation Tookit, 08

Augmented Development Tools

60

Related Work

Alternatives are generated automatically from formal specification

Design Galleries, Marks, SIGGRAPH 97Spreadsheets for Images, Levoy, SIGGRAPH 94Spreadsheets for Visualization, Chi, IEEE CGA 98Parameter Spectrums & Parallel Pies, Terry, UIST 02 & CHI 04Subjunctive Interfaces, Lunzer, TOCHI 08TEAM STORM, Hailpern, CHI 07

Alternatives in Other Domains

61

Related Work

Exploit spatial syntax

Design Galleries, Marks, SIGGRAPH 97Spreadsheets for Images, Levoy, SIGGRAPH 94Spreadsheets for Visualization, Chi, IEEE CGA 98Parameter Spectrums & Parallel Pies, Terry, UIST 02 & CHI 04Subjunctive Interfaces, Lunzer, TOCHI 08TEAM STORM, Hailpern, CHI 07

Alternatives in Other Domains

62

Current Work: Revisions for Interaction Design

Revisions & commentsfor text in Microsoft Word

Revisions & comments for interactionsin d.tools

stanford hci group oct 21, 2008

http://hci.stanford.edu

Björn HartmannLoren Yu, Abel

Allison, Yeonsoo Yang

Scott R. Klemmer

Supported by:NSF grant IIS- 0745320Equipment from Intel & NokiaSAP Stanford Graduate Fellowship

64

At runtime: editable min/max values

Take a guess, then give control to the user.

65

Implementing Parallel Editing: Incremental Structure Tracking

Doc. A

User inserts

textEmpty blockcreated

Blockssub-divided

User deletes

text

Doc. B

Doc. A Doc. B

Doc. A Doc. B

Doc. A Doc. B

INSERT DELETE

66

Future Work

Graphics + Code?

67

68

FRAME DEMONSTRATE UNDERSTAND

69

FRAME DEMONSTRATE UNDERSTAND

70

FRAME DEMONSTRATE UNDERSTAND

71

Tangible Control

72

Switching alternatives

Load

Alternative2.hex

Alternative 2

Bootloader

73

0

50

100

150

200

250

300257.81

161.81

Tree Matching Task:Mean Completion Times

ControlJuxtapose

seco

nds

p<0.001 (paired two-tailed Student’s t)

Buxton, Sketching User Experiences

75

How can interaction design toolssupport creation and management of user interface alternatives?

76

JuxtaposeRuntime on PC

Juxtapose Mobile

Juxtapose WrapperUser’s FlashApplication

Open socket, Send variable info

Juxtapose WrapperUser’s FlashApplication

TCP/IP

Open socket, send variable info

var1

Alt. 1 Alt. 2

var2

77

Juxtapose Mobile

Juxtapose WrapperUser’s FlashApplication

Tuning messagesFor Alternative 2

Juxtapose WrapperUser’s FlashApplication

Tuning messagesFor Alternative 1

TCP/IPJuxtaposeRuntime on PC

var1

Alt. 1 Alt. 2

var2

78

Adding Tuning to the Arduino IDE

79

Tuning without ReflectionParse source to extract global variable declarations

Emit variable table + wrapper code into source

Compile modified source

80

Related WorkDesign Galleries, Marks, SIGGRPAH 97Parameter Spectrums & Parallel Pies, Terry, UIST 02 & CHI 04Subjunctive Interfaces, Lunzer, TOCHI 08Spreadsheets for Images, Levoy, SIGGRAPH 94Spreadsheets for Visualization, Chi, IEEE CGA 98TEAM STORM, Hailpern, CHI 07Partials, Terry, PhD Thesis 05Amulet Inspector, Myers, IEEE ToSE 97Jpie, Goldman,Sci. of Comp. Prog. 98ChucK, Wang, NIME 04 Adobe Image Foundation Tookit, 08(only first authors listed)

Mixed initiative

Redrawn fromBuxton, Sketching User Experiences

Generate Options, Select. Repeat.

82

Limits of Parallel Editing Cognitive challenge:

Changes to alternatives are not visible during editing

Level of granularity for editing:Extension to multi-file projects not trivial

83

Beyond the DesktopMOBILE PHONES MICRO

CONTROLLERS