andruid kerne - tamu college of engineering...© 2006 by andruid kerne, eunyee koh, ross graeber,...

© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.

expressive interfacesandruid kerne

interface ecology labdepartment of computer sciencetexas a&m university

temporal composition

Media Workbenchediting, annotation

locative mediacollection server

contextsemantics

media presentation

GP

S

mobile media

capture

site-specificmobile

locative


expanding the roles of digital systems in human life

support a broad range of activities

emphasize human expression

ecosystems approach: combine methods (art, science)

interface design, knowledge representation, conceptual art

information visualization, pattern recognition, information retrieval

new hybrid methods of evaluationqualitative data (ethnographic)

quantitative data


research areas

combinFormation

represent collections through mixed-initiative composition of image and text surrogates

embodied interfaces

choreographic buttons

location-aware gaming

locative media collectiontemporal

composition

Media Workbenchediting, annotation

locative mediacollection server

contextsemantics

media presentation

GP

S

mobile media

capture

site-specificmobile

locative


combinFormation

represent collections through mixed-initiative composition of image and text surrogates


information discovery

not just finding information

having ideas in the context of the stimulus of new information

creativity

collecting new sets of existing materials

seeing them in new ways

search 1 search 2 search n

User:forming goals ,finding, restructuring,understanding

composition ofhybrid

surrogates

...


library science

a replacement for an original item, which gives some description of the

item, and how it can be obtained

surrogate


digital surrogate(aka hypersign)

represents aninformation resource

special type of hyperlink

formed from metadata

tightly linked to the original

the surrogate is the object!?

amazon catalogue entry

google gists


prior representations of collections

textual list

spatial hypertext


evidence for image + text

working memory

phonological loopprocesses auditory & text information

visuo-spatial sketchpad

dual coding strategy for representation of surrogates

marchionini et al 1999-2003

“combined” surrogates for video -> better comprehension, reduced processing time


composition

“putting together”

assemble collection elements to form a connected whole

visual design principles represent relationships

spatial organization

compositing

fading

expressive


combinFormationmixed-initiative composition of

surrogates

initiatives

user

agent

processes

information finding / choosing / collecting

visual composition

semantic model

Generative Information Collecting

GenerativeTemporal Visual

Compositioncomposition space

Direct Manipulation Information Collecting

Directing the Agent

Direct Manipulation Composition

Seeding

Agent InitiativesUser Initiatives


direct manipulation

information collecting

visual composition


direct manipulationinformation collecting

drag and drop

Firefox Extension


metadata system

labels

values

types

details on demand


direct manipulationcomposition

rearrange, remove, edit, color, resize

navigate


generative agents

model

information collecting

temporal visual composition

semantic model

Generative Information Collecting

GenerativeTemporal Visual

Compositioncomposition space

Direct Manipulation Information Collecting

Directing the Agent

Direct Manipulation Composition

Seeding

Agent InitiativesUser Initiatives


interface design to direct the agent


example

paper thin displays

wearable computing


experiment 1direct manipulation only

source format only

create a 3-level collection of psychology resources in 3 formats

formats of surrogates

text

images+text

formats of collection

linear list

spatial composition


results:subjective measures

liked best [X2(2) = 34.88, p < .001]

easiest to use [X2 (2) = 16.12, p < .001]

0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80

like the most easiest to use most helpful Surrogate Format

Prop

ortio

n of

resp

onse

s

Spatial Labeled Images Spatial Text

Linear Text


experiment 1 addressed

format for presenting information collectionto users

but: for information discovery questions, the answer is also a collection!


4 divergent thinking questions

input surrogate collection format

linear text

composition of image+text

output surrogate collection format

linear text

spatial image+text

experiment 2direct manipulation only

source format + answer format


experiment 2 results

initial participants ratings

23

18

0

5

0

5

10

15

20

25

Which answer format did youlike better?

Which answer format wasbest for expressing your

ideas?

# of

Res

pons

es

Image+TextComposition(cF)

Linear text

Answer format

3.02 2.772.602.33 2.952.98

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

expressive communicative informative

text lists

image+textcomposition


experiment 3 - mixed-initiativesfield study - initial deployment

ENDS 101/200 The Design Process

Hybrid and Invention Assignments

182 students

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

prior work Hybrid

scor

es

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

prior work Inventionsc

ores


embodied interfacechoreographic buttons

interactive art installation

body-based interface

promoting social interaction

collaborative art creation


a tale of two grids

two 3x3 grids

physical

visualized

direct spatial mapping: grid cells

choreography gridphysical point of interaction

movements captured via cameras

imagery gridvisualization of image collection (collage)

cross fading transition between images


interaction

recognizedmovements

jumpingcrouching

system actionsselection

locking image (persistence)fast forwarding

(finding specific images)

mappingsposition = selection

jump = lockcrouch = fast forward


affordances

yellow border = selection

state indicator iconsappear during selection

each has two visible states (i.e. open and closed lock)spatially positioned to afford activating movements

images in collection represent literal and metaphorical up and down movements

Icon for locked or unlocked

Icon for fast forwarding


evaluation

art exhibition /party / user study

food and musicpost-questionnaire

video-taped

movement-based vs. mouse-basedin same room, projected on two different walls

observing engagement and social interaction


movement-based versionmore socially engaging

“It's funny to see people jumping and concentrated in looking at the screen. It seems more natural to talk and interrupt people in the

movement-based than in the mouse-based.”

evaluation

Which was more entertaining to watch others use?

0

2

4

6

8

10

12

14

16

18

20

movement-based mouse-based equal

Responses

Num

ber o

f Par

ticip

ants

Which involved more social interactions?

0

2

4

6

8

10

12

14

16

movement-based mouse-based

Response

Num

ber o

f Par

ticip

ants


embodied interfacelocative media collection

collect images, soundsw GPS identifiers

additional metadata

retrieval by location

visualizations including temporal composition

andruid kerne - tamu college of engineering...© 2006 by andruid kerne, eunyee koh, ross graeber,...

Documents