interacting with large screens
DESCRIPTION
interacting with large screens. patrick baudisch microsoft research visualization and interaction research. large screens. building a large display focus plus context screens interacting using the mouse high-density cursor mouse ether interacting using pen & touch drag-and-pop - PowerPoint PPT PresentationTRANSCRIPT
interacting with
large screens
patrick baudisch
microsoft researchvisualization and interaction research
large screensbuilding a large display
focus plus context screens
interacting using the mouse
high-density cursormouse ether
interacting using pen & touch
drag-and-poptablecloth
(halo) return
large screens and multimon
... are coming
[Jon Peddie ResearchDec, 2002 N=6652]
No Multimon30%
Plan to Use Multimon
38%
Use Multimon32%
• information mural[Guimbretière, Winograd]
• on large screens optical flow helps navigation [Tan 2001]
• large screens help productivity tasks [Czerwinski 2003]
• multi-monitor setups: access palette windows in Photoshop, CAD… [Grudin 2001]
building one
mouse acrossbezels
reach icons
reach anything
mouse acrossdistances
focus plus context screens
conclusions
• Hardware– At least one hi-res
display– At least one larger low-
res display
• Software– scaling of the display
content is preserved– resolution varies
skip to dnp
setup
application scenarios
video
field study: users & tasksSubject’s task Document/view
Sta
tic do
cum
en
ts
Web designer Page: 800 pixel
Mechanical engineer Polybot segment: 5cm
Graphic designer Poster: 1m
Architect in remodeling Building: 50m
Photogrammetry (2) Highway 2 miles
Geographic info. system County: 80km
Chip designers (2) Wafer: 12cm
Dyn
am
ic
Air traffic ctrl. tool builder Zone: 50km
Ego shooter gamer Surrounding: 360º
Submarine ROV op. Surrounding: 360º
Strategy gamers (2) Map: 30k pixel
Smallest detail Ratio
Table detail: 1 pixel 800
Clearance: 0.03mm 2,000
Align: 0.5mm 2,000
Accuracy: 1cm 5,000
Accuracy: 1 inch 100,000
Land boundaries: 0.5m 160,000
Grid: 0.5m 240,000
Plane distance in 25m steps 2,000
Aiming: 0.1º 3,000
Use arms: 1mm/0.05º 8,000
Aiming: 1 pixel 30,000
focus plus context screen VisualizationSame # of pixels
fisheye
55
overview plus detail
44
Display technology
homogeneous resolution
44wall-size, hi-res display
44
What participants
used
What participants
used
Available to½ of participants
Available to½ of participants
… andcurrentsolutions
experiment 1:• 3 interfaces:
• focus plus context screen• overview + detail• homogeneous
• 2 tasks
• 12 subjects from Xerox PARC• Within subjects, counter-balanced• Same number of pixels
task 1: closest hotel
8 maps per interface
F+C screen and O+D use same magnification factor
task 2: verify connections
Verify a different set of 24 connections on the board
0
100
200
300
400
500
600
700
Map task Board task
zooming + panning
overview+detail
focus+context screen
resultsA
vera
ge
ta
sk c
om
ple
tion
tim
es
in s
eco
nd
s
21% faster21% faster 36% faster36% faster
manually zooming
takes time
manually zooming
takes time
visually switching reorientation
visually switching reorientation
visually more ambiguous
visually more ambiguous
experiment 2:driving simulation
120 sec sequence100 fields of nails; 30 rocks; tradeoff
resultsM
ea
n n
um
be
r o
f co
llisi
on
s su
bje
cts
cau
sed
• Sweet spot:flight simulation, unmanned vehicles…
0
5
10
15
20
25
run-over nails rocks hit
overview+detailfocus+context screen
Error rate only 1/3 of two-
monitor setup
Error rate only 1/3 of two-
monitor setup
Subjects preferred thef+c interface
Subjects preferred thef+c interface
observations
• low-res periphery is often ok
• on a wall-size display: If periphery was hi-res I still could not read it due to distance
focus-plus-context interaction
toc
• mouse • pointer moves too fast, users lose track of it• pointer behaves in unpredicted ways when
crossing screen boundaries
• touch/pen input• users cannot reach their content anymore
ktop?ktop?we observed: lots of input problems
conclude
building one
mouse acrossbezels
reach icons
reach anything
mouse acrossdistances
high density cursor
conclusions
problem: mouse across long distances
• large screen longer distances
higher mouse acceleration
• temporal aliasing: 500 pixels jumps
• lack of visual continuity & weak stimulus users lose track of the pointer
the problem will get worse
• cursor update is limited by screen refresh rate
• screen refresh rate has actually decreased (LCDs)• larger screens + lower refresh rate status quo• future: even larger screens problem will get worse
how it works
previous cursorposition
current cursorposition
mouse
motion fill-in cursors
current framefill-in cursors
previous frame
inserts cursor image between actual cursor positions the mouse cursor appear more continuous
this is not the mouse trailthe windows
mouse trail…• makes mouse
trail last longer• drawback: cursor
images lag behind
...is not high-density cursor
• hd cursor makes mouse trail denser
• lag-free: mouse stops => cursor stops
video
benefitsprevious cursor position
mouse
motion
current cursor position
fill-in cursors
current framefill-in cursors
previous frame
mouse
motion
1. mouse cursor appear more continuous easier to track the cursor
2. higher “visual weight” easier to re-acquire the cursor
designs alternatives
acceleration
• reference: exponential acceleration
a
b
d
e
f
c
h
g
frame
designs alternativesa
b
d
e
f
c
h
g
frame acceleration
• motion blur with higher weight
designs alternativesa
b
d
e
f
c
h
g
frame acceleration
• temporal super-sampling vs. motion blur
chose discreet version1. latest cursor position is always shown blur-free and in full opacity2. appearance that users are familiar with today3. computationally less expensive
chose discreet version1. latest cursor position is always shown blur-free and in full opacity2. appearance that users are familiar with today3. computationally less expensive
designs alternativesa
b
d
e
f
c
h
g
frameacceleration
• density = detectability vs. intrusiveness
transfer function
mouse speed
distancebetweencursorimages
onset threshold(configurable)
cursor trail provides no speed cues
hd c
urso
r has
no
effe
ct
transfer function(configurable)
designs alternativesa
b
d
e
f
c
h
g
frameacceleration
• optional cursor growth
user study• conducted pre-study to define interface candidates• interfaces: control vs.
high-density cursor (conservative, tripleDensity, plusScaling) • fitts’ law task• triple-mon: button located at 5” to 40” distance• participants: 7 external participants, 5 coworkers
• hypotheses• high-density cursor faster• the greater the distance the
greater the effect• tripleDensity and plusScaling
faster than conservative
resultsti
me %
rela
tive t
o r
eg
ula
r cu
rsor
90
92
94
96
98
100
102
target distance (mm)125 250 500 750 1000
regular mouse cursor
high-density cursors
speedupup to 7%
conservative
+ scale
+3-dense
short distance
subjective satisfaction
Condition Liked Most Liked Least
control 0 7
HD_conservative 2 3
HD_tripleDensity 2 0
HD_plusScaling 6 0
• most participants did not notice that cursor was different!“did that condition use a different mouse acceleration?”…
• lesson we learned:display frame rate is not a hard limit
toc
building one
mouse acrossbezels
reach icons
reach anything
mouse acrossdistances
mouse ether
conclusions
mouse ether
target
start
tasks with incorrecttasks with incorrectvisual feedbackvisual feedback
mouse across multimonproblem 1: different dpi
targetstart
low dpihigh dpi
mouse across multimonproblem 2: the gap
target
start
contents
• problem: why the cursor gets warped
• eliminating warping with mouse ether
• calibration
• user study: up to 28% faster in acquisition task
• conclusions: need to test fitts’ law
why the cursor gets warped
targetstart
low dpihigh dpi
target
start
system oblivious of• dpi differences
system oblivious of• gaps
mouse ether
target
start
targetstart
low dpihigh dpi
target
start
targetstart
low dpihigh dpi
with mouse ether
now users can acquire targets on direct path
now users can acquire targets on direct path
calibration
step 1step 2step 3
the math
screen i
screen j
(cx,cy)
(ex,ey,)
on in
it
+(dx,dy)every frame
(cx,cy)
(ex,ey,)’ ’
’ ’
off-screen travel
target
start
with mouse ether
more off-screen travel
target
start
blocked
same dpi
user’s perspective system’sperspective
target
blocked
high dpilow dpi
start
start
target
target
start
same dpi
with mouse ether
target
high dpilow dpi
start
start
target
name ether
taming off-screen travel
• iteration 1:restrict to convex hull
• iteration 2:allow entering ether only for the purpose of transit• towards another screen• at sufficient speed
• otherwise screen boundaries “hard”
start
target
user study
• 2x 18” screens• 1280x1024 and
800x600 resolution• eight participants• WinFitts software• 72 movements
(=4 trials x 9 paths x 2 directions)
results
High-Low Low-High0
200400600
8001000
120014001600
1800
Mov
emen
t Tim
e (m
s ±S
EM
)
Direction of Movement
Control
MouseEther
with mouse ether up to 28% faster
with mouse ether up to 28% faster
discussion
• multimon offers incorrect visual cues• this hinders successful target acquisition
• mouse ether improves acquisition speed by matching pointer behavior to visual cues
• off-screen travel allows for direct target acquisition
• next steps: investigate target acquisition under contradictory visual feedback ( Fitts’ law?)
toc
target
start
pen/touch input on walls-size displays
• mouse input can be mapped to any screen geometry
• not so for “direct” input technique, such as touch and pen input
building one
mouse acrossbezels
reach icons
reach anything
mouse acrossdistances
drag-and-pop
conclusions
a technique for people with limited reach…
• user may not be able to reach content
• user may not be able to read content
drag-and-pop…
• let’s focus on a specific case for a moment: extend basic actions drag-and-drop and picking
scenario 1: long distances
dragging is designed for small screens…… but becomes time-consuming on large screens
scenario 2: dragging + bezels
dragging across bezels in display wall is no problem for the mouse…
…but a big problem when using pen/touch input
drag-and-pop: demo• users starts
dragging icon towards a distant folder or application
• icons of compatible type come towards mouse cursor
• user drops icon with minimal motion
• targets retractdrag-and-pop works across bezels
• drag-and-pop generalizes direct manipulation• bring content to the user• let the user interact with it• send content back
scenario 1: long distances
dragging on large screens
scenario 2: dragging + bezelsdragging acrossbezels in display wall
the displays we used…
design
selecting candidates
• initialize• all icons are candidates
• filter• eliminate icons with non-matching file types• eliminate icons that are too close• eliminate icons outside target angle• if necessary, restrict to some hard limit
preserving layout
• snap to grid• eliminate empty
rows and columns
• translate back• place center of
bounding box in front of user
• closer for experts
the rubber band
• animationdid not work
• “frozen”motion blur
• narrow midriff• suggests elasticity• clue for distance
• simplified version
getting it out of the way
• to rearrange icons on the desktop (overloaded):• any mouse motion moving away from the
“popped-up” icons de-activates drag-and-pop• introduce flick gesture into mouse motion
pre-study
• 3 layouts for study: sparse (11), frame (28), cluttered (35)
• 15 single, 6 dual, and4 triple monitor users
• overall resolutions 800,000 pixels to 3,900,000 pixels
• (= 66% more than the display wall used in the experiment).
user study
• participants: 2 female, 5 male• dynaWall
• 3 Smartboard• 15’ long (4.5m)• 3 x 1024x768 pixels
• native code not stable enough Macromedia Flash version
• task: drag icons into matching folder• highlighting disappeared when started• each desktop: 11-35 icons + 10 icons to be filed
results
• faster with drag-and-pop• error rate higher with drag-and-pop• most of the effect caused by the bezels
0 1 20
2
4
6
8
10
12
14
Number of Bezels Crossed
Drop
Pop
Control
Drag-and-pop
3.7 times3.7 timesspeedupspeedup
0
10
20
30
40
0 500 1000 1500 2000 2500Target Distance (pixels)
Drop
Pop
Control
Drag-and-pop
subjective satisfaction
• > 6 (out of 7)• “I liked using drag-and-pop”• “I always understood what was happening when drag-and-pop
was on”,• “I would use drag-and-pop for large displays.”
• < 3 for• “It took a long time to get used to drag-and-pop”• “It was hard to control what the targets did when drag-and-pop
was on.”
• drag-and-pop interface causes less manual stress and fatigue than the control interface
• lesson learned: the shortest connection between two points on a display wall is not a straight line (fixed)
drag-and-pick
problem• launch app or open file
drag-and-pick• user drags “background”• all icons in that direction
move to the cursor• user drags % releases
mouse over it• target is activated
current work: push-and-pop
outperforms pantograph-based techniques
0
1
2
3
4
5
6
7
8
9
10
11
12
198 455 480 504 920 1075 1123 1173 1651 1695 1850 1897Distance (pixels)
Tim
e (
s)
drag & drop
push & throw
pick & drop
acc. push & throw
drag & pop
push & pop
0
1
2
3
4
5
6
7
8
9
10
260 586 678 712 1363 1592 1624 1660 2476 2505 2734 2766
Distance (pixels)
Tim
e (s
)
drag & drop acc. push & throw
push & throw pick & drop
drag & pop push & pop
insert halo
outperforms pantograph-based techniques
building one
mouse acrossbezels
reach icons
reach anything
mouse acrossdistances
tablecloth
conclusions
tableclothgeneralizing drag-and-pop to arbitrary interactions
for users withouttable manners
object I would like to interact with
window panning endswhen window edgereaches edge of screen
tablecloth panningallows panning any screenobject to any screen location
design challenge
• how to invoke tablecloth?• while dragging an object
1st design: crossing menu
dragging
2nd design: dual touch
jumpback
panning
pan
jump
dragging
video
focus-plus-context interaction
the distinction of screen space into focusand context regions is always there(focus plus context screens only emphasize it)
• focus-plus-context interaction• bring contents to user• interact• send contents back
toc
halo
discussion
• distance to peripheral space and human peripheral vision make peripheral content hard to read
• making periphery all hi-res does not solve this problem
• we need to support users using appropriate interaction techniques
building one
mouse acrossbezels
reach icons
reach anything
mouse acrossdistances
read more
UIST 2001CHI 2002
INTERACT2003
CG&A 2005
INTERACT 2003GI 2005
CHI 2004
patrickbaudisch.com/projects
thanks!• try it out www.patrickbaudisch.com
• thanks to: • focus plus context screens: victoria
bellotti, nathan good, paul stewart, pamela schraedley
• high-density cursor: ed cutrell, george robertson
• mouse ether: ed cutrell, ken hinckley, rob gruen
• drag-and-pop: & ed cutrell, dan robbins, peter tandler, ben bederson…
• tablecloth: george robertson
can we push this further?
• (unless the app has fixed focus of attention) bring content to the user on demand
• that’s great—so maybe the periphery does not need to be full resolution after all
• can we push this even further? can we create a visual periphery without any peripheral screen space?
halo
store content in periphery
<view halo section is the small screen deck>
access off-screen content
• make halo arcs (or city lights) clickable to pan display to that location
vs.
• preserve users’ spatial memory by using drag-and-pop instead: bring peripheral content to PDA screen
Seamless integration of displays
a b
Context
No reflections on focus screen
Focus
If I had to commercialize today…
• Build all-analog immersive video link• Immersive telepresence,• Remote operated vehicles, drones• Immersive VR• Remote medicine
ContextWall
Extra
(a) locate (b) closest
(d) avoid(c) traverse
How does it work?
The scaling software
• Display image on two display units of different resolution• Similar to two-headed display• but display units are overlapping• and one of them has to be scaled down
• (Related work “Flux capacitor”[Dr. Emmett Brown, 1985])
Linux/VNC
app
focus
contextinput
server
clip
scale
viewer
viewer
Image viewer
context
Photoshop
scale
ACDsee
ACDsee
.gif
.gif
mousefork
input
focus
related work• acquiring distant targets
• move cursor with eye gaze (Sibert and Jacob, 2000), Magic pointing (Zhai et al., 1999)• flick snaps cursor to target (Dulberg et al. (1999)• sticky icons capture cursor (Swaminathan and Sato, 1997)• throwing gets across long distances (Geißler, 1998) • expanding targets save space on screen (McGuffin and Balakrishnan, 2002)• drag-and-pop (baudisch et al 2003)
• enhance detectability of the mouse cursor• <ctrl> for radar animation (Microsoft, Steve Bathiche)• cursor growth (Kensington Mouseworks 2001)• mouse trail for slow response LCDs (e.g. MS Windows)• liveCursor points in the direction of its motion (Ben Bederson)
• motion blur and temporal supersampling• reduce temporal aliasing, such as stroboscope, e.g. wheel spokes• rendering a scene multiple times (Dachille and Kaufman, 2000)• improve the perceived responsiveness of graphics apps (Conner and Holden, 1997)• help users anticipate motion (Chang, 1993; Thomas & Calder, 2001)
design goals
• for users who track the cursorenhance the predictability of the cursor path• enhanced trail density/continuous blur• smooth interpolation of the cursor path• preservation of trail density as a cue for cursor speed.
• for users who reacquire the cursorincrease the detectability of the cursor (visual weight)• enhanced trail density• enhanced cursor opacity• and cursor scaling.
• preserve responsiveness
designs alternativesa
b
d
e
f
c
h
g
frameacceleration
• distance between cursor images as cue for mouse speed
designs alternativesa
b
d
e
f
c
h
g
frameacceleration
• smooth interpolation
bezier interpolation
cursor position
1. linear interpolation
2. attraction point
3. interpolate
pre-study
• goal: define interfaces for user study• participants: 14 coworkers
• informal procedure• try out high-density cursor• try out different settings (density, onset…)• choose “favorite” setting
• resulting interface parameters• 12-17 pixels/frame vs. 35 pixels/frame• distance = sqrt(n)• cursor growth on or off
touch/pen input breaks
touch/pen-input + multimon
touch and pen input renaissance• PDAs• Tablets• Liveboards /
Smartboards
multi-display systems• DynaWall, iRoom
Smartboard wall• connect tablet to
external screen• …
scenario 1: tables + screen
tablet users scribble with pen… but filing iconsinto folder on external monitor requires mouse
scenario 1: tablet + screen
filing icons into folder on external monitor
related work• techniques for transferring information
• drag-and-drop avoids hidden clipboard (e.g. Xerox Star)• hyper dragging (Rekimoto, 1999)• pick-and-drop (Rekimoto, 1997)+ take-and-put (Streitz et al., 2001)
• overcome large distances• magic pointing (Zhai et al., 1999) requires an indirect input device• gesture input techniques (Rubine, 1991)• throwing (Geißler, 1998) for reliable target acquisition?• laser pointers to acquire targets on a Smartboard (Myers et al. 2002)
• mouse-based interaction techniques• lodestones and lay lines (Jul, 2002)• flick (Dulberg et al., 1999)• sticky icons (Swaminathan and Sato 1997)
related work
• driving directionsvs. route planning aids
• overview-plus-detail• focus-plus-context
• pointing into off-screen space
inside applications…
• drag-and-pop workseven if target is• occluded• clipped• closed (folder)
• use the concept tofile emails?
intrusion border
handle
space for arcs…
and for corner arcs
reserve space for content
arc length = distance
handling many objects
• find best (restaurant): relevance cut-off
• see all (dangers): merge arcs
a first attempt… city lights