Structure of Artist Virtual Environments

Mark Green

School of Creative Media

Introduction

We have examined some of the techniques for building virtual environments

time to look at some real environments: how they are structured techniques used to construct them what we need to support

then look at narrative structure and some of the other things we need

Artistic Virtual Environments

Very few virtual environments are well documented, lack of good visual and written descriptions

without good documentation can’t analyze the techniques used, not enough information

start with one project: Art and Virtual Environments, Banff Canada, 1991-94

Art and Virtual Environments

Around a dozen pieces produced over a three year period

hardware: SGI Onyx1 RE (4Mb texture, 64 Mb main

memory, R4000 processors) mainly HMD, some projection Fastrak and Dataglove for input

Bar Code Hotel: Perry Hoberman

Bar Code Hotel

Bar Code Hotel

Features: non-standard device configurations autonomous object behaviors users suggest behaviors, don’t control detailed

motion, commands like: chase, drift, wall flower and punch

richness comes from the interactions between multiple users

Archeology of a Mother TongueToni Dove and Michael Mackenzie

Takes place in a theatre, large screen projector, a guide runs the piece navigating with a Dataglove

approximately 40 minutes for the entire piece

strong emotional responses from the audience, not like a typical VR piece

Archeology of a Mother TongueToni Dove and Michael Mackenzie

Three main features: strong visual effect minimal interaction, mainly navigation strong narrative component

the piece tells a story, the guide moves the audience between the different parts of the story

Archeology of a Mother TongueToni Dove and Michael Mackenzie

Visual Richness

Narrative

Narrative structure consists of scenes and transitions

the story is told in the scenes, each scene contains part of the story

the transitions are used to move between scenes, change the setting for the next part of the story, usually short: 10 seconds

PlaceholderBrenda Laurel and Rachael Strickland

A shared environment, two participants

has a much looser narrative structure, more concentration on interaction

the interaction drives the story

intended to be an experience, participants take turns playing the four characters in the story

PlaceholderBrenda Laurel and Rachael Strickland

PlaceholderBrenda Laurel and Rachael Strickland

Features: role playing, users take on characteristics of

other characters, modify their view and behavior

voiceholders: ability to leave marks on the environment

interaction between participants spatial sound

Voiceholders

Artifacts in environment, used to record user’s voice

user grabs an empty voiceholder to record a message

grab a full voiceholder to hear the message

build up narrative by leaving messages for future users

Scenes and Transitions

Three scenes: cave, waterfall and hoodoos

each scene has a different visual presentation and characteristics, not story driven

users control scene transitions by entering a portal to another scene

10 second transition for user orientation

The Bush Soul

A more recent piece developed by Rebecca Allen at UCLA

under development for a number of years, with a good sized team

shown in 1998 and 1999

PC based multi-screen system with joystick as the main input device

The Bush Soul

This piece has a linear narrative structure, the user visits four sites over a virtual day

environment populated by creatures

these creature are autonomous, they react to each other, the user and actions that occur in the environment

user can enter the body of one of the creatures

The Bush Soul

The Bush Soul

Summary

Features that should be supported object behavior

autonomous user controlled

non-standard device configurations interaction

with user between objects between users

Summary

Features (continued): visual richness multiple media: graphics, sound, force, … scenes and transitions, support for narrative

structure control of user

how they experience the environment where they can move

More Behavior

The behavior we have done so far has been mechanical, it doesn’t appear intelligent

we will now look at ways of doing more intelligent behavior

what do we mean by intelligent behavior?

What a real organism might do?

Something that doesn’t look stupid?

More Behavior

The features of intelligent behavior: goal directed react to environment avoid collisions or other accidents social actions: move towards objects we like,

away from objects we don’t like do what we might do??

Goal Directed Behavior

Often over rated, many assume that all behaviors are goal directed

this makes the theory and programming much easier

we can select the goals, and then produce the appropriate actions

can string goals together to achieve larger goals

Goal Directed Behavior

Goals can have hierarchies

short term goals are short lived, solve an immediate problem

longer term goals take longer to reach, require more planning and time

long term goals usually divided into a sequence of shorter term goals

Goal Directed Behavior

Many AI and Alife techniques have been developed based on this approach

lots of algorithms to draw on

But, are we really goal directed?

Some of the time we definitely are, when we have a problem to solve, or something that needs to be done

Goal Directed Behavior

Other time assigning goals to our actions is quite difficult: watching cartoons on TV going for a walk browsing in a store (or the web)

we can try to assign goals to these activities, but they sound more like excuses

Goal Directed Behavior

Quite often we assign goals to people’s actions when we really have no idea of what they are doing

this makes life seem more rational

Example, a large crowd of people walking, we assume they all know where they are going, this may not be the case

Goal Directed Behavior

Doing good goal directed behavior is quite hard, often looks more like a robot than a living object

it’s not clear that it buys us anything, better off spending our time developing other behaviors

if it looks like it could have a goal, its probably good enough

Realistic Behavior

Actions should look like they have a purpose, object doesn’t do anything stupid

we want our objects to move around the environment: mainly travel in a straight line, so they look like

they know where they are going avoid running into other objects

start by looking at a simple object, mover

Mover

The mover moves in a simple environment: floor four walls

both the floor and walls are made from the same plane

want to make the mover look like it has a purpose, not just a random walk

the mover is a small cube

Mover

Two observations: we walk in the direction we are facing we walk in straight lines as long as possible

in each tick the mover moves a small amount in the direction its facing

the push action does this:push distance;

to change direction use rotz

Mover

Mover

Direction of motion

Mover

When we reach an obstacle (a wall) we need to do something

one solution is to turn so we are no longer facing the obstacle

two questions: how do we know when we are about to hit

something? How much do we turn?

Mover

The hit action can assist with obstacle avoidance

hit shoots a ray, given a starting point and direction, returns any object hit and the distance to the object:hit x, y, z, rx, ry, rz, obj;

first three parameters are start of ray, next three are the ray direction

Mover

The last parameter is a variable, name of hit object stored in this variable

the variable “distance” is set to the distance to the object

what direction and position should we use?

The position and direction of the object!

This information is stored in a set of variables

Mover

Object position: Locx, Locy, Locz

Object direction: Anglex, Angley, Anglez

hit functions as eyes, looks to see what is in front of the object

since the environment is enclosed, we will always hit something, so should we always turn?

Only if the obstacle is close

Mover

The environment is 12x12, we are moving at 6 units/second

start avoiding collision when we are 2 units from obstacle

the basic motion is done in the tick event handler

the turn event handler is used for basic obstacle avoidance

Mover

on tickeval v = 6*dt;push v;hit Locx, Locy, Locz, Anglex, Angley, Anglez, obj;select obj;if distance < 2;

trigger “mover”, “turn”;endif

end

Mover

Now we need to turn

the basic idea is to turn a bit (0.4 radian), and see if we are still going to collide

this is repeated until the collision is avoided

if for some reason the mover gets too close to the obstacle (less than 0.2 units) we have an emergency action, the back event handler

Mover

In the back event handler we move back two steps and then make a large turn (1.5 radians, about 90 degrees)

we then continue the motion

this should avoid any potential disaster

Mover on turn

spinz 0.4;hit Locx, Locy, Locz, Anglex, Angley, Anglez, obj;select obj;set flag 0;if distance < 2 ;

set flag, 1;endifselect flag;triggerif distance < 0.2, me, “back”;triggerif distance > 0.2, me, “turn”;

endon back

eval dx = -2*v;push dx;spinz -1.5;

end

Mover

Mover

What we’ve done so far gives us basically circular motion

the mover travels in a circle just inside the walls

this looks kind of stupid

periodically change the objects direction, a random change in direction, every 4 to 10 seconds

Mover

on changerandom dz, 1.20, 3.14;spinz dz;random t, 4, 10;triggerAt “mover”, “change”, t;

endon first

translate 0, 0, 0.15;triggerAt “mover”, “change”, 5;

end

Mover

Gives relatively good basic motion

can introduce a number of obstacles into the environment, a set of cubes

the mover can successfully avoid them

this gives us one motion unit, later we will add more behavior to the mover

Mover

The Thing

The mover can move around the environment, but it only reacts to obstacles

the thing is a plant-like creature that reacts to the user

the thing normally bends back and forth at a fairly slow rate (think about a plant slowly blowing in the wind)

The Thing

The Thing

The thing has a stem and two leaves, more leaves can be added by cut and paste

the leaves are separate objects with their own event handlers

the main tick event handler produces the waving motion

this is done by a rotation about the y axis

The Thingobject thing

add mesh stemadd object leaf1add object leaf2on tick

eval angle = angle+sign*da*dt;roty angle;if angle > 0.5;

eval sign = -1;endifif angle < -0.5;

eval sign = 1;endif

end

The Thing

Three variables control the motion: angle: the current rotation angle da: the rate that the angle changes sign: the direction of motion

two if actions are used to keep the angle within a reasonable range, between -0.5 and 0.5 radians

The Thing

on firstset angle, 0;set da, 0.2;set sign, 1;translate -3, 3, 0;

end

The Thing

The first event handler initializes the variables and positions the thing

the tick and first event handlers in the leaf1 and leaf2 objects are basically the same

their rotation is about the z axis

don’t need to position the leaf objects in the first event handler

The Thing

The thing reacts to the user, as the user moves closer it moves faster

when the user is within 2 units it moves 5 times faster, when the user is within 1 unit it moves 10 times faster

this is handled by a second tick event handler, the thing has two tick event handlers that operate in parallel

The Thing on tick

where “cursor”, dx, dy, dz;if distance < 2;

set da, 1.0;endifif distance < 1;

set da, 2.0;endifif distance > 2;

set da, 0.2;endifput “leaf1”, da, da;put “leaf2”, da, da;

end

The Thing

The where action is used to determine the location of a particular object:where object, dx, dy, dz;

the first parameter is the name of the object, and the next three parameters are variables that contain the distance to the object along the x, y and z axis

the distance variable is set to the distance to the object

The Thing

The cursor object represents the user, so we use where to find out how far away it is

next three if actions are used to set the rotation rate based on the distance to the user

the rotation rate for the leaves must also change, we could do the same thing in the leaf objects, but that’s a waste of time

The Thing

Since the rotation rate is the same in the leaf object, we just use the value we computed

the put action is used to change the value of a variable in another object:put object, variable, value;

the first two parameters are the object and the variable, the last parameter is the new value for the variable

Back to the Mover

What happens if we have more than one mover?

Should they interact with each other?

Should they treat other movers differently from other objects in the environment?

In the next example movers will try to move together

Movers

The basic idea is that movers want to move as a group

if there is another mover nearby, a mover will attempt to move in the same direction as that mover

it will adjust its direction of movement so its close to that of the other mover

Movers

How does this work?

First we change the mover object so none of the event handlers use the mover name, they use the me variable instead

next we duplicate the mover description two times, give the new moves slightly different names

now have three independent movers

Movers

Next each mover needs to detect other movers in the environment

they can only group together if they are close, two movers at opposite ends of the environment don’t group

we use the closest action to find the closest object to a mover, then we check to see if its another mover

Movers

The closest action has the following form:closest dx, dy, dz, object;

the first three variables get the distance to the object along the x, y and z axis, the last variable gets the name of the object

the distance variable is also set to the distance to the object

how do we detect another mover?

Movers

All mover objects have a type variable, and this variable is set to 1 (this is done in the first event handler)

we use get to get the value of this variable:get object, variable, variable;

the first two parameters are the object and variable, the last parameter gets the value

Movers

on tickclosest dx, dy, dz, obj;select obj;get obj, type, t;triggerif t == 1, me, “flock”;

endon flock

get obj, Anglez, angle;eval da = 0.3*(angle-Anglez);spinz da;

end

Movers

We use a triggerif to produce a flock event if the type is 1 (another mover)

in the flock event we get the heading of the other mover (the Anglez variable)

find the difference with our heading, scale it, and rotate in that direction

this will bring the headings together, fairly quickly

Movers

Discussion

Movers tend to pair up, sometime a group of three is formed

may get more grouping if there are more

they tend to move parallel to each other, but sometimes they follow each other

instead of changing the heading we could have moved them so they are closer to each other

Discussion

Changing heading works well in this case because they are moving fast relative to the size of the environment

could introduce other types of objects

could have a predator/prey scenario where one object chases another

instead of moving closer, prey will try to flee

Problems

If we watch this environment long enough we will notice a problem

sometimes a mover will escape from the environment

what’s happening here?

The two behaviors are interfering with each other, they have cross purposes that produce an undesirable effect

Problems

The movement and collision behavior is executed first, it changes the heading of the object so it won’t go through a wall

the grouping behavior then executes, it also changes the heading, but in the opposite direction

since it executes last, it overrides the collision avoidance behavior

Problems

Whenever two behaviors control the same variable, in this case the direction of the object, this problem can occur

when we start producing realistic behaviors this problem is inevitable

there are numerous possible solutions, we will look at a few of them

Behavior Conflicts

In this case a simple solution is to change the order of the event handlers

do grouping first, then collision avoidance

this works because collision avoidance is always more important

we can perform a strict ordering of the behaviors based on their importance

Behavior Conflicts

What happens if we can’t order the behaviors?

Either they are equally important, or in some cases one is more important, and in others the other is more important

one way of handling this is to have each behavior calculate how important it is, and only execute the most important one

Behavior Conflicts

Another approach is to allow a behavior to stop other behaviors

in our example, when a collision is close, the collision behavior stops all the other behaviors until the problem is solved

these two solutions select one of the behaviors to use, they ignore all the other behaviors

Behavior Conflicts

Another approach is to allow all the behaviors to contribute

in our case, the two behaviors produce a new direction, an angle value, the two angle values are then combined to produce the final angle

we could weight each value by the importance of the behavior

Behavior Conflicts

We could also use the high level goals of the object to determine the behavior to use, or how their outputs are weighted

but this returns us to the idea of goals

maybe this is a good use for them, as a way of avoiding conflicts

Goals, Again

We could interpret what we see in terms of goals

one of the movers gets pushed out of the environment, leaving two movers

we could interpret this as some form of mating ritual, one mover gets rid of its competition

we know this isn’t the case, but ...

Narrative Structure

Most of the artistic pieces have some structure, they are divided into scenes

the user doesn’t spend all of their time in one place, they move from place to place

each of these places has a different set of objects and different behaviors

there is usually a transition process

Narrative Structure

Narrative isn’t a new idea, its used in many media has been developed over the centuries

most media use a linear narrative structure, there is only one path through the story

the story teller guides you on this path

most media really only allow one path

Narrative Structure

Digital media introduced the idea of more complex narrative structures

this really isn’t a new idea, but most people doing digital media think it is

interactive fiction is based on a branching or tree structure

at key points in the story, the user decides which branch to follow

Narrative Structure

Besides branching, there are also loop structures and hyper links

some narratives allow branching, but bring the branches together before the next phase of the story

common in games, each level has many branches, but at the end they all join before going to the next level

Narrative Structure

We can think, or model this structure in the following way

there are two main components to our model: scenes transitions

scenes provide the location for the action, transitions move us between scenes

Narrative Structure

This is a simplified model, but it works okay for digital media

each scene has a set of active objects, the objects visible in the scene, user interacts with these objects

set of active objects changes from scene to scene, way of organizing the content

Narrative Structure

Transitions used to move from scene to scene

can be instantaneous or require some time

time can be used to: re-orient the user to a new location read in new content provide structure to the story, inform the user

that things are changing

Narrative Structure

There is some debate about non-linear narrative in digital media

interactive character development more important than branching structure

non-linear structure leads to shallow stories, many disjointed pieces

waste of time developing all of the branches

Narrative Structure in SE

The main unit of narrative structure in SE is the scene

like an object, but has no geometry

has a set of event handlers that control the overall flow of events within the scene

most scenes have entry and exit event handlers, executed when scene is entered and left

Scenes in SE

Following actions used to control set of active objects:activate object;

deactivate object;

deactivateAll;

activate makes the object active, visible and interactive

usually done in the entry event handler

Scenes in SE

Deactivate removes an object from a scene

deactivateAll deactivates all of the objects

these actions are usually used in exit event handlers

objects can be activated and deactivated at any point in a scene

works on objects, lights, sounds other scenes

Scenes in SE

If an environment has scenes, SE automatically activates the first scene and sends the entry event to it

can also specify the initial scene on the command line

can have more than one active scene at a time, not sure if this is useful

Example

Build an environment with two scenes and a transition effect

start in scene first, button press moves us to scene second

on entry there is a transition effect, more on this later

button press goes back to scene first

First scene

scene firston entry

activate “cursor”;trigger “cursor”, “first”;activate “xyz”;

endon select

triggerAt “first”, “exit”, 0.1;endon exit

deactivate “cursor”;deactivate “first”;activate “second”;trigger “second” “entry”;

endend

First Scene

On entry we activate the cursor and the spot light

when button is pressed, schedule the exit event (note same event is used for second scene transition)

the exit event deactivates all of the objects, activates the next scene and send it an entry event

Second Scenescene second

on entryactivate “cursor”;activate “test”;activate “xyz”;activate “light”;trigger “second”, “transition”;

endon exit

deactivateAll;activate “first”;trigger “first”, “entry”;

end

Second Scene

Start by activating the objects in the scene, the cursor, the sphere, the spot light and the light used for the transition effect, then trigger the transition effect

the exit event handler deactivates all of the objects, activates the first scene, and then sends and entry event to that scene

Second Scene

on selecttriggerAt “second” “exit”, 0.1;

endon transition

defaultLights 0;background 0, 0, 0;ambient 1.5, 1.5, 1.5;trigger “light”, “fadeup”;

endend

Second Scene

The select event trigger the exit from the scene

the transition event handler does the transition effect

this starts with the scene being dark, and fades up the lights over a short period of time

need to explain lighting environment

Lighting in SE

For transition effects we need to know more about how lighting is handled

SE has two default lights, an ambient and a point light

we can see things without defining a light in our scripts

if we want a fade we need to get rid of the default lights

Lighting in SE

Without default lights we can have a dark environment

the defaultLights action controls the default lighting:defaultLights flag;

if flag is zero the default lights are turned off, otherwise they are turned on

Lighting in SE

The background action can be used to change the background colour:background r, g, b;

the ambient action is used to control the ambient light level in the environment:ambient r, g, b;

the default ambient level is (0.2,0.2,0.2)

Transition Effect

The transition event handler turns off the default lights, sets the background to black and then sets a relatively high ambient light level

if then tells the light source light to fade up

this is a point light source located at the origin, its initial colour is (0,0,0), its basically turned off

Light Fade

light lightcolour 0 0 0position 0 0 0type pointon fadeup

set r, 0.02;set g, 0.02;set b, 0.02;set dr, 0.02;set dg, 0.02;set db, 0.02;triggerAt “light”, “fade”, 0.05;

end

Light Fade

The fadeup event handler initialises the variables used in the fade

the variables r, g, b are the colour of the light source

the variables dr, dg, db are the amount the colour changes each time its changed

the colour is updated 20 times per second

Light Fade

on fadeeval r = r+dr;eval g = g+dg;eval b = b+db;colour r, g, b, 1;eval r1 = r/2;eval g1 = g/2;eval b1 = b/2;background r1, g,1 b1;if r > 0.99;

set dr, 0;endifif g > 0.99;

set dg, 0;endif

if b > 0.99;set db, 0;

endifselect dr;triggerAt “light”, “fade”, 0.05;

end

Light Fade

Fade starts by computing new r,g,b colour of light and setting the light colour

the colour action is used to change the colour of a light or object:colour r, g, b, alpha;

first three components are colour, the fourth is the alpha, alpha not used for lights

Light Fade

Background also changes with light level, but at half the rate

next compute colour of background, r1, g1, and b1 and set background

next check for the end of the fade, occurs when light colour is (0.99,0.99,0.99)

Light Fade

Use if to detect when component reaches 0.99, and then set the increment to zero

this allows different fade rates and maximum values for each colour

a select on dr controls the trigger that causes the next colour update

a triggerAt with 0.05 gives 20 colour updates per second