drag-and-popa technique for accessing remote screencontent on touch- and pen-operated systems

patrick baudisch& ed cutrell, dan robbins, mary czerwinski,peter tandler, ben bederson, and alex zierlinger

microsoft researchvisualization and interaction research

goals

• mixing touch screens/pen-input with large screens creates interesting new interface challenges

• bringing target icons to the user (“drag-and-pop”) allows users to complete drag interactions faster

• general theme: limiting interaction space to the display space at the user’s location can solve problems

touch/pen input breaks

touch/pen-input + multimon

• touch and pen input renaissance• PDAs• Tablets• Liveboards /

Smartboards

• multi-display systems• DynaWall, iRoom

Smartboard wall• connect tablet to external

screen• …

scenario 1: tables + screen

tablet users scribble with pen… but filing iconsinto folder on external monitor requires mouse

scenario 2: dragging + bezels

dragging across bezels in display wall is no problem for the mouse…

…but a big problem when using pen/touch input

scenario 3: long distances

dragging is designed for small screens…… but becomes time-consuming on large screens

drag-and-pop• users starts

dragging icon towards a distant folder or application

• icons of compatible type come towards mouse cursor

• user drops icon with minimal motion

• targets retractdrag-and-pop works across bezels

demo…

bringing target icons to the user allows users tocomplete drag interaction at the user’s current locations

scenario 1: tablet + screen

filing icons into folder on external monitor

scenario 2: dragging + bezelsdragging acrossbezels in display wall

scenario 3: long distances

dragging on large screens

video

related work• techniques for transferring information

• drag-and-drop avoids hidden clipboard (e.g. Xerox Star)• hyper dragging (Rekimoto, 1999)• pick-and-drop (Rekimoto, 1997)+ take-and-put (Streitz et al., 2001)

• overcome large distances• magic pointing (Zhai et al., 1999) requires an indirect input device• gesture input techniques (Rubine, 1991)• throwing (Geißler, 1998) for reliable target acquisition?• laser pointers to acquire targets on a Smartboard (Myers et al. 2002)

• mouse-based interaction techniques• lodestones and lay lines (Jul, 2002)• flick (Dulberg et al., 1999)• sticky icons (Swaminathan and Sato 1997)

design

selecting candidates

• initialize• all icons are candidates

• filter• eliminate icons with non-matching file types• eliminate icons that are too close• eliminate icons outside target angle• if necessary, restrict to some hard limit

preserving layout

• snap to grid• eliminate empty

rows and columns

• translate back• place center of

bounding box in front of user

• closer for experts

the rubber band

• animationdid not work

• “frozen”motion blur

• narrow midriff• suggests elasticity• clue for distance

• simplified version

getting it out of the way

• to rearrange icons on the desktop (overloaded):• any mouse motion moving away from the

“popped-up” icons de-activates drag-and-pop• introduce flick gesture into mouse motion

study

pre-study

• 3 layouts for study: sparse (11), frame (28), cluttered (35)

• 15 single, 6 dual, and4 triple monitor users

• overall resolutions 800,000 pixels to 3,900,000 pixels

• (= 66% more than the display wall used in the experiment).

user study

• participants: 2 female, 5 male• dynaWall

• 3 Smartboard• 15’ long (4.5m)• 3 x 1024x768 pixels

• native code not stable enoughà Macromedia Flash version

• task: drag icons into matching folder• highlighting disappeared when started• each desktop: 11-35 icons + 10 icons to be filed

results

• faster with drag-and-pop• error rate higher with drag-and-pop• most of the effect caused by the bezels

0 1 20

2

4

6

8

10

12

14

Number of Bezels Crossed

Drop

Pop

Control

Drag-and-pop

3.7 times3.7 timesspeedupspeedup

0

10

20

30

40

0 500 1000 1500 2000 2500Target Distance (pixels)

Drop

Pop

Control

Drag-and-pop

subjective satisfaction

• > 6 (out of 7)• “I liked using drag-and-pop”• “I always understood what was happening when drag-and-

pop was on”,• “I would use drag-and-pop for large displays.”

• < 3 for• “It took a long time to get used to drag-and-pop”• “It was hard to control what the targets did when drag-and-

pop was on.”

• drag-and-pop interface causes less manual stress and fatigue than the control interface

lesson learned

• the shortest connection between two points on a display wall is not a straight line

• (we fixed this by opening target sector towards top of display)

the larger picture

general theme

• WIMP metaphor can breakon large screens with pen/touch input

• drag-and-pop generalizes direct manipulation• bring content to the user• let the user interact with it• send content back

interaction space is not the same as display space anymore

drag-and-pickproblem• launch app or open

file

drag-and-pick• user drags

“background”• all icons in that

direction move to the cursor

• user drags % releases mouse over it

• target is activated

what about dialog boxes?

inside applications…

• drag-and-pop workseven if target is• occluded• clipped• closed (folder)

• use the concept tofile emails?

goals revisited - conclusions

• mixing touch screens/pen-input with large screens creates interesting new interface challenges

• bringing target icons to the user (“drag-and-pop”) allows users to complete drag interactions faster

• general theme: limiting interaction space to the display space at the user’s location can solve problems

thank you!

try it out: Google drag-and-pop

thanks to: the VIBE research group(mary czerwinski, george robertson…)diane kelly, dieter boecker,lance good, amanda williams,