3d interaction mark green school of creative media
TRANSCRIPT
3D Interaction
Mark Green
School of Creative Media
Introduction
• Investigate 3D interaction• how do we want to interact in 3D• look at different interaction techniques• try to develop a set of interaction primitives
that can be used to construct interaction techniques
• use SE to prototype interaction techniques
Interaction Tasks
• Earlier we looked at standard interaction tasks
• the main ones are: navigation selection manipulation combination
Interaction Tasks
• Are there other things that we would like to do in a virtual environment?
• In some sense the standard tasks are quite general so this is hard to answer
• if we look at some of the standard types of environments we might get a better idea
Viewing
• In simple viewing environments the main interaction technique is navigation
• the user moves about the data, but doesn’t interact with it
• not much here to add to our list
Indirect Manipulation
• User doesn’t directly interact with objects• instead sends commands to application,
which could change objects• use 3D versions of 2D IT to interact with
application, examples: menus, sliders, control panels, etc
• this is a start, gives us some ideas
2D Applications in 3D
• This leads us to 2D applications in 3D• does this make sense?• I’m not sure, I’ve change my mind on this
several times over the past 10 or so years• when we went from 1D text UI to 2D GUI
UI many of the applications transferred as well, helped spread 2D UI
2D Applications in 3D
• If we want to have a 3D desk top we need to support standard 2D applications
• need things like word processing, spread sheets, email, web, etc
• do these applications transfer to 3D? Does this make sense?
• How would we do this?
2D Applications in 3D
• There are two things to consider: output interaction
• many 2D applications are heavily text oriented, plus some simple graphics, tend to be page or screen oriented
• simple solution: texture map 2D screen onto a 3D polygon
2D Applications in 3D
• This is a quick and dirty solution, but somewhat works
• experimented with a 3D version of Netscape
• each page texture mapped onto a polygon, several linked pages visible at the same time
• 3D navigation used to move between pages
2D Applications in 3D
• It was interesting, but not very usable• need a better way of presenting the
information• on MS systems can extract data from many
applications using COM, could use this to get the data so it could be displayed in a different way, maintain compatibility with 2D version
2D Applications in 3D
• COM also allows us to modify data in office applications
• develop our own ITs and UI for these applications
• what would this type of UI look like?• Could we just transfer 2D ITs to 3D, a
quick and dirty translation
2D Applications in 3D
• We produced a set of 2D like ITs for 3D• based on the Motif widget library used in
X11• had many of the standard 2D techniques,
floated on a panel in 3D space• used in several applications, nice bridge
from 2D to 3D
2D Applications in 3D
• Do we want to do more than this?• How would we edit text in 3D?• Cut and paste may be possible, but selection
may be difficult, restrict motion to 2D surface of the document
• text entry still requires a keyboard, could speech recognition be an alternative?
2D Applications in 3D
• Provide a new 3D UI for these applications, be able to work with the data in 3D
• what would be the advantage? Integrated view of 2D and 3D data possibly easier to work with multiple
documents possibly better overviews of document structure
Direct 3D Interaction
• The next step is applications that support direct interaction with objects
• be able to select objects and perform operations on them
• what types of operations do we want to perform?
• To some extent this depends on the application
Direct 3D Interaction
• A general manipulation is geometrical transformation, ability to move the object, change size or orientation
• another operation is changing object properties, colour or application properties
• this could involve some type of property sheet or a 3D equivalent
Direct 3D Interaction
• Other operations involve editing the shape of objects, possibly producing new objects
• objects can be combined• in a vehicle simulator, the user needs to be
able to operate controls• grab a control and move it or maybe twist a
dial
Art and Entertainment
• Do these applications have their own types of interaction?
• This is an interesting question that is worth exploring
• in a sports game the user controls one of the players
• this is more suggestive than absolute in terms of control
Art and Entertainment
• The player has certain physical limitations and their own model of performance
• the user can suggest a direction of travel, but the player may not be able to move as fast as the user would like him to
• this is a co-operative form of interaction between the user and the application
Art and Entertainment
• What about the user controlling multiple characters?
• This is possible in some strategy programs, the characters follow the general directions of the user, but have some autonomy
• hard to have detailed control over more than one character
Art and Entertainment
• Are there things we would like to do in a purely artistic environment that wouldn’t be found in other types of environments?
• Most artistic environments have concentrated on visual appearance and exploration, there hasn’t been a lot of interaction
• I’m not sure why this is the case
Interaction Techniques
• Now we turn our attention to a detailed study of a particular interaction task
• selection, particularly command selection is an important interaction task that appears in most user interfaces
• the common way of doing this is with a menu, a well known set of interaction techniques
Menus
• Before menus, commands were entered by typing the name of the command and its parameters
• problem: need to remember the names of all of the commands
• menus provide a list of commands, don’t need to remember name, just know the right command to select
Footnote
• Menus remove part of the cognitive burden of using a program, don’t need to remember how to spell the commands
• menus provide a list of what there is, easy selection
• still need to know what each command does, temptation to keep adding commands, increases the complexity of the UI
Menus: 2D
• Start by looking at the important components of 2D menus
• many types of menus, vary along several dimensions
• menu type: static: always on screen in the same place dynamic: only appears when needed, can
change position (pull down vs. pop up menus)
Menus: 2D
• Standard menu shape is rectangular, with rectangular menu items
• can have circular menus, called pie menus, with each menu item having a wedge shape
• each menu item can have a text string or icon as a label
• this covers the visual elements of 2D menus
Menus: 2D
• The basic selection operation is a click within a menu item
• for pop-up and pull-down menus extra actions are required to display the menu
• only a few simple actions, either a click or a press-drag-release action
Menus: 3D
• How does this translate to 3D?• We need both a visual representation and
the actions required to select an item• the visual representation of menu items
transfers reasonably well: 3D text strings icons: simple 3D shapes
Menus: 3D
• Layout is more difficult• the menu must appear in 3D space where
the user can see it• this isn’t a problem in 2D, the menu is
always visible when displayed• in 3D, if the menu is placed behind the user
it won’t be visible
Menus: 3D
• 2D static menus have a fixed position on the screen
• in 3D there are two options for static menus: a fixed position within the environment,
possibly attached to an object a fixed position with respect to the user, travels
with the user
Menus: 3D
• In the case of pop-up menus there are two options: appear at the current cursor position, good for
selection appear a fixed distance in front of the user,
easier to find
• what is the visual appearance of a 3D menu?
Menus: 3D
• 2D menus now have sophisticated appearance, doesn’t always work well in 3D
• a fancy menu will hide all the objects behind it, take resources
• for ease of selection we want to menu close and relatively large, but this could hide a significant part of the environment
Menus: 3D
• For menus that float in space (not attached to an object) they should: have no background, the objects float in space if they have a background, it should be largely
transparent so users can see through it
• in this way the menu hides as little of the environment as possible
Menus: 3D
• Why do we care about this?• The user is concentrating on the application,
not the menu• if we hide the application data, the user
looses concentration, looses the context of the operation
• want to interfere as little as possible with the user’s task
Menus: 3D
• The shape of the menu depends upon how the items are selected
• want to use as little effort as possible• what are the selection actions?• This depends on the input device and the
interaction style, there are a range of possibilities
Menus: 3D
• The most direct approach imitates 2D selection, place a cursor in the menu item and press a button
• a 3D cursor is used for this, its position is controlled by one of the input devices
• this works well if the cursor is close to the menu, no need to navigate to the menu
Menus: 3D
• If a joystick or puck is used, cursor based selection works well
• if an absolute device, such as a Polhemus or Intersense is used, this may be more difficult, the menu must be within reach
• for these devices a pointing approach might be better
Menus: 3D
• In pointing a ray or line is controlled by the device: start of the ray at the device’s location direction of ray controlled by device’s
orientation
• by changing the hand orientation can sweep the ray through space
• the menu item hit by the ray is selected
Menus: 3D
• Ray based selected works for menus that can’t be reached, faster in this situation
• it can be difficult to aim at menus some distance away, ray can be replaced by a cone
• works well when rays are used to select objects, only one selection mechanism, less user confusion
Menus: 3D
• Both cursor and ray are physically based, use multiple degrees of freedom
• menu is just a list, one dimensional• just using one degree of freedom might be
easier• a good example of this is a ring menu, the
menu objects are distributed around a horizontal ring
Ring Menu
Menus: 3D
• Selection is based on rotating the ring• the object that is closest to the user is
selected• there is a selection zone at the front of the
menu, the item in this zone is selected if a button is pressed
• z rotation (or any axis) on a puck or joystick used to rotate the menu
Menus: 3D
• Ring menus work well as pop-ups• when a button is pressed the menu appears
at the current cursor position• a simple action can be used to move the
desired item to the selection zone• another button press selects the object• as easy as a 2D pop-up menu
Menus: 3D
• This last selection mechanism introduces the idea of a constraint
• we have a 6 DOF device, but we only use 1 DOF, we ignore the rest
• this makes it easier to perform the task, don’t need to rotate the menu about all three axis, just makes the task harder
Daisy Menu
Menus: 3D
• The last thing to deal with is feedback• before we press the button, need to know
we are getting the right item• with a ring menu this is part of the display,
the item in the selection zone• with the other selection techniques it’s not
near as obvious
Menus: 3D
• For cursor and pointing based techniques, highlight the item that would be selected if the button was pressed
• as the cursor moves into an item it is highlighted, when it moves out of the item the highlight is removed
• this is important since we can’t see the cursor when its inside an item
Menus: Summary
• We have looked at menus in detail, a technique from 2D UI
• we are now ready to dissect this technique, find the individual components
• find the pieces we need to build interaction techniques
• trace the actions required for a menu selection
Menus: The Components
• An event triggers the menu: for pop-ups: a button press that causes the
menu to appear for static menus: moving the cursor or pointer
into the menu region
• next a sequence of actions to select menu items
Menus: The Components
• for cursor need to know when inside an item
• for pointer, ray intersects the menu item• when this occurs we highlight the item, may
need to un-highlight a previous item• next we have the select action, which is
usually a button press
Menus: The Components
• We also need an escape route• what happens if the user doesn’t want to
select from the menu?• With a static menu just move outside of the
menu region• for a pop-up, do the select trigger without a
menu item selected
Menus: The Components
• The components: triggers cursor in menu item ray pointing at menu item highlight and un-highlight cursor or ray in menu menu display
Building Menus
• How do we know if the cursor is in a menu item?
• Need a general solution, can’t have a different technique for each menu
• this can be done by using bounding box techniques
• a bounding box is the smallest box that contains a graphical object
Bounding Boxes
• Most common form is axis aligned, the edges of the box are parallel to the coordinate axis
• find the smallest x, y and z coordinates, they form one corner of the box
• find the largest x, y and z coordinates, they form the other corner of the box
Bounding Boxes
• Easy to compute from object vertices• can also easily compute:
if a point is inside the bounding box if a ray intersects the bounding box
• these are the things we need• compute bounding box for each item, and
for the whole menu
Building Menus
• To trigger a static menu, check if cursor is in menu’s bounding box, or pointer intersects the menu’s bounding box
• use bounding box tests to determine the menu item that is selected
• draw bounding box as a way of highlighting a menu item, can use different colours
Interaction Techniques in SE
• Now that we have some idea of how ITs work, we will look at how we can construct them in SE
• The first thing we need is position and orientation data from input devices
• We need this to move cursors, make selections, move and orient objects, etc
Interaction in SE
• The attach and follow actions retrieve position data from an input device
• The format of these actions:attach device, x, y, z;follow device, x, y, z;
• The device parameter is the name of the device, this should be a joystick, tracker or some other device that produces 3D data
Interaction in SE
• The device name comes from the se.map file
• A device that produces 3D data is called a value device, its entry is:value internal_name device_name
• The internal_name is the name used inside our program or script
Interaction in SE
• The device_name is the name of the actual piece of hardware, for example:console: keyboard and mouse
joystick1: the first joystick
interd: the Intersense head tracker
interd2: the Intersense wand
• You may need to edit the se.map file to add a new device
Interaction in SE
• The last three parameters are variable names
• In the case of attach, these variables get the position of the device
• In the case of follow these variables get the relative position of the device, how far it moved on the last update
Interaction in SE
• We can use attach in the following way• we add the following line to se.map
value hand joystick1• Now we can do the following:
on tickattach hand, x, y, z;translate x, y, z;
end
Interaction in SE
• This small event handler can be used to move a cursor in response to the joystick
• This gives us a way of moving things, but not much interaction
• Need to be able to select objects as well• We know the position of the cursor, can use
bounding box test to see if its in an object
Interaction in SE
• The in action performs bounding box tests:in x, y, z, object;
• The first three parameters are the position of the cursor, or any other object
• The last parameter is a variable: If the cursor is inside an object, it will be the
name of the object Otherwise it will have no value
Interaction in SE
• Note that we can use select to determine if an object was selected, how can we use this?
• If we have a select trigger, can have
on select
in x, y, z, obj;
select obj;
trigger obj, “selected”;
end
Interaction in SE
• This event handler is placed in our cursor object, so x, y and z are the cursor position
• If the cursor is inside an object, the obj variable will contain the object’s name
• The select action will exit the event handler if nothing selected
• Trigger will send an event to the selected object
Interaction in SE
• So now we can tell an object its been selected
• There are two things left: Feedback How the object responds to selection
Interaction in SE
• Use a simple form of feedback based on bounding box
• Draw bounding box in wire frame when cursor is inside the object
• We will use white as bounding box colour• Could use different colours for different
types of feedback
Bounding Box Feedback
Interaction in SE
• Two actions are used for highlighting:highlight object, r, g, b;
unhighlight object;
• The first parameter is a variable, contains name of object to highlight or unhighlight
• r, g, and b give the colour of the highlight
Interaction in SE
• How do we use this?• When the cursor enters an object we want
to highlight the object• When the cursor leaves the object, we want
to turn off the highlighting• Problem: how do we know when the cursor
leaves the object?
Interaction in SE
• in tells us when we are inside of an object, but it doesn’t tell us if we have left an object
• Need to remember last object we were in, unhighlight this object
• Then test to see if we are inside an object, and highlight that object
Interaction in SE
on tick
attach hand, x, y, z;
translate x, y, z;
unhighlight obj;
in x, y, z, obj;
select obj;
highlight obj, 1, 1, 1;
end
Interaction in SE
• If the cursor is inside an object, we first unhighlight the object, then highlight it again
• But when we move out of the object, we will unhighlight it, but it won’t be highlighted again
• This does what we want
Interaction in SE
• Now that we can select an object, what do we do with it?
• There are two possibilities: The user has selected a command, this
command applies a one shot change to the object
This is the start of a continuous manipulation, such as changing the position or orientation of the object
Interaction in SE
• In the first case, the cursor object sends an event to the selected object, which then changes in some way
• The cursor object needs to know the event to send, based on a state variable
• Cursor object received an event (from a menu?) that sets this state variable
Interaction in SE
• In the second case, the selected object will respond to the input device
• On each tick it reads the device and performs some action until it’s told to quit
• The action to perform is again determined by a state variable, in the selected object this time
Interaction in SE
• We need a way of examining state variables and then determining what to do
• This can be done with the triggerif action, a combination of the if and trigger actions:triggerif conditional expression, object, event;
• The first part looks like an if action, there is an expression that evaluates to zero or non-zero, will usually involve a comparison operator
Interaction in SE
• The second part contains the event to be sent if the condition is true
• The object is a variable that contains the name of the object the event is sent to
• The variable me contains the name of the current object
• Now lets see how this works
Interaction in SE
• Produce a simple environment, an object that can be selected, then translated or rotated
• use the following button as triggers: one: translate the selected object two: rotate the selected object three: stop translating or rotating
Interaction in SE
• The object has a variable, state, determines which action it’s performing
• on each tick examine state, then transfer to appropriate event handler for current action
• we can use triggerif for this, testing the value of state
Interaction in SE on tick
triggerif state == 1, me, “move”;triggerif state == 2, me, “turn”;
endon move
follow hand, x, y, z;move x, y, z;
endon turn
dorient hand, rx, ry, rz;spinx rx;spiny ry;spinz rz;
end
Interaction in SE
on select1set state, 1;
endon select2
set state, 2;endon stop
set state, 0;endon first
set state, 0;end
Interaction in SE
• The cursor object sends the select1, select2 and stop event depending upon the button the user presses
• for the first two buttons, check to see if the cursor is inside an object before the event is sent
• for the last event, need to remember the object performing the interaction
Interaction in SE on select
in x, y, z, obj;select obj;set last, obj;trigger obj, “select1”;
endon twist
in x, y, z, obj;select obj;set last, obj;trigger obj, “select2”;
endon three
trigger last, “stop”;end
Interaction in SE
• The tick handler for the cursor retrieves the cursor position, stores in variables x, y and z
• it also performs the highlighting and unhighlighting
• these event handlers only need to respond to button presses
Menus in SE
• We can also do standard menus in SE• To see how this works we will do a simple
pop-up menu that has three menu items• The user presses button two to pop up the
menu• The cursor is moved into an item, button 1
is pressed, the menu then disappears
Pop-up Menu
Text
• The first thing that we need for this example is some text
• We could use a modeler to produce the text, export it as a text file, …
• This is a lot of work for just a text string, so we have a text command that is used to produce 3D text
Text
• The text command starts with the word text, followed by the name of the text object, ends with a line containing end
• Number of options that define the text object
• The main option is text followed by the text string to be displayed, in quotes (“)
Text
• Other options include:position x y z
direction dx dy dx
up dx dy dz
size s
• The position option gives the location of the text string, can be transformed in an SE object
Text
• The direction of the characters in the string is given by the direction option
• The up option gives the direction of the up vector for the string (the vertical direction for the characters)
• The size option is used to scale the text string
Menustext text1
text "item one"
position 0.5 0.2 0.8
direction 0.2 1 0
size 0.3
end
object menu1
add text text1
on select
“do something”
end
end
Menus
• The add text option inside of an object command is used to add a text string to it
• All three items are basically the same. Only the z coordinate of the text position changes
• Now we need to arrange some way of popping the menu up and down
• We use a scene for this
Menusscene first
on entryactivate “cursor”;activate “xyz”
end
on twistactivate “menu1”;activate “menu2”;activate “menu3”;
endon select
deactivate “menu1”;deactivate “menu2”;deactivate “menu3”;
endend
Scenes
• When a script file has scenes none of the objects are active when the environment starts
• If an object isn’t active it isn’t displayed and it doesn’t respond to events
• The entry event handler makes the cursor and light active using the activate action
Menus
• When the user presses button 2, the twist event is generated
• The event handler for twist activates the three menu items
• When this happens the menu items become visible and they can be selected
Menus
• Each object has an event handler for the select event
• This event handler processes the command, more on this later
• The select event handler in the scene deactivates all three menu items, so they are no longer visible on the screen
Menus
• The scene object handles the popping up and down of the menu
• Done in response to button presses• Each menu item is responsible for its
command, whatever that is• But how is this done? All three menu items
get the select event
Menus
• Menu items need to know if they have been selected
• In gives us the object selected, but no way of telling if it’s the current object
• We use the inObject action to determine if the menu item has been selected
• This is a simple version of in
Menus
• The format of the inObject action is:inObject expression, x, y, z, variable;
• The first is the name of the object, either a text string or a variable
• If the point (x,y,z) is inside this object, the second variable is set to one, otherwise it is set to zero
Menus
object menu1add text text1on select
attach hand, x, y, z;inObject me, x, y, z, flag;select flag;trigger “menu1”, “selected”;
endon selected
print "Item one selected”;end
end
Menus
• The select event handler gets the cursor position
• Uses inObject with the me variable to test if the cursor is in this menu item
• The select action tests the return from inObject
• If selected the selected event is sent