Physical Presence in Virtual Worlds using PhysX

University of Michigan 3D Lab http://um3d.dc.umich.edu

Gabriel Cirio

Eric Maslowski

One of the biggest problems with interactive applications is how to suck the user into the experience, suspending their sense of disbelief so that they are completely immersed in the virtual world created for them. After receiving the extremely powerful SDK and accelerators from Ageia, we aimed to use this technology to bring the user one step closer to total immersion.

Virtual Reality and the CAVE

Since we wanted to create the most immersive and believable experience possible, we decided to initially de-velop for the CAVE. The CAVE is an advanced virtual reality system which consists of a 10’x10’ room where each wall is a stereoscopic 3D screen simi-lar to an IMAX theatre. Each wall is driven by its own dedicated computer with one master computer that tells each node what to draw and when. Since the CAVE is driven by many in-dependent computers each with their own internal clocks and timings, we encountered the early challenge of making the physical responses consistent and predictable across the entire cluster. The slight-est discontinuity would be immediately noticeable and jarring for the user breaking the whole experience. Without a perfectly deterministic system, one wall of the CAVE would show one interpretation of the scene, while another wall may show something entirely different. With the stability of PhysX, and the help of OpenSG, we were able to build a system that is com-pletely deterministic and consistent under the highest application load. With the foundation in place, we started to look at other uses of PhysX in a virtual world and how it could be used with the CAVE...

(break down of different walls of CAVE in actual demo)

Keeping Track of the User

Often when people talk about immersive displays such as the CAVE, the question of tracking comes up. Since the walls of the CAVE oriented at 90 degree angles, it’s necessary to know where the user is at any point in time so that the images created for each wall compensate for this.

photo taken from user’s perspective showing how images for each wall is adjusted for user’s perspective (use anaglyph (red/cyan) glasses to see in stereo)

To accomplish this, we use a Vicon motion capture system with eight cameras placed around the CAVE for complete coverage. Placing special reflective markers on the user’s glasses we can tell the computer exactly where the user is and what they are looking at. Having this ability to track anything inside the CAVE (not just the users head) created endless possibilities with PhysX but also exposed how lim-ited traditional systems are. One of the biggest problems with physics-centric worlds is how the user interacts with it. First, you have the user walk-ing through a scene with physical objects, and then you have the situation where the user wants to interact with a physical object. To address the problems found with a user navigating through a scene, combined with the fact that the user can physical walk around in the CAVE, we’ve introduced a new form of character controller which offers a great sense of immersion and freedom...

One of the CAVE’s strengths is the perceived freedom given to the user. Since the images for each wall are constantly updated for the users point-of-view, the user can walk around virtual objects, kneel down and look underneath them, and generally see the world as it would be in real life. We’ve taken these same principles and extended it to the physical presence of the user. The char-acter controller we’ve implemented addresses the short-comings often found in traditional sys-tems and keeps the system stable and predictable. One of the key features of the controller is the

use of real mass and physical properties to define the user. This introduces a great sense of realism by giving the sensation that an object is light or heavy according to how easy it is to move. Imagine knocking over a virtual cabinet that pushes you to the side, or standing in a PhysX river and being taken away by the current in a life-size virtual world. The controller brings the experi-ence one step closer to full immersion by allowing the user to act and react to the virtual world all while freely moving inside the CAVE. To achieve this, we developed a system based on dy-namic bodies using a series of joints that gave us com-plete control over the user’s changing height and loca-tion. The system is comprised of multiple capsules that adjust their position based on the user’s head height al-lowing for a very stable, resizable, character controller. The system has also been designed for easy expansion to include distinct arms, legs, etc. which match the user’s actual body allowing for incredible realism inside the virtual world...

Character Controller

In addition to the user moving through the virtual world, how they inter-act with it is equally important. Most applications rely on a joystick or mouse for the user’s input. Since we are able to track arbitrary objects inside the CAVE, we used this flexibility to track the user’s hands in ad-dition to their head. Wearing special gloves and attaching a kinematic actor to the user’s hand position allows us to give the user a method for natural interaction with the virtual world. By using kinematic actors, every motion was automatically translated by PhysX into a force. There-fore, the faster one swings their arm, the stronger the force applied to objects, adding more realism to the experience. Users can grab objects, move them, throw them, all by using what comes naturally. Users don't pass through virtual objects any more, and actually have unconscious awareness of their virtual presence in the world that surround them.

One possible use that we will be exploring for this is in our training simulators for first responders. When an officer approaches what could be the site of massive casualties, they have to be aware of many things. Are there any victims and who needs immediate attention? Are there additional immediate risks to bystanders/victims such as structural instabilities or secondary devices? Previ-ously, we were only able to train the officer on visual cues, but now can test their meth-ods as well. We can have buildings that are structurally unsound. We can have convinc-ing smoke using particles that billows and

folds around existing objects hindering the officers vision. We can place victims and secondary de-vices under debris forcing the officer to go through actual techniques used in the field. All of this helps bring the officer deeper into the simulation, immersing their senses and preparing them for when lives are actually at risk.

lifting panels in the virtual world - use anaglyph (red/cyan) glasses to see in stereo

To further extend our system and to completely remove the need for traditional input devices, we developed a series of controls and interfaces within the virtual world for the user to interact with. Since the main goal of the CAVE is com-plete immersion, we wanted the user to forget where the walls are and believe they were in the virtual world, even if only for a moment. Tradi-tional, screen-aligned GUI methods would im-mediately highlight where the screens are, thus breaking the entire illusion. Thus, we decided to bring the user interface into the virtual world as true 3D objects. This Virtual Graphical User Interface (VGUI) was presented as a platform surrounding the virtual body of the user. Over the platform are located several objects (the buttons), with distinct shapes, distributed in a layout comfortable for the user and within reach of their virtual hands. By using unique properties of the character controller, we are able to keep the VGUI at a comfortable dis-tance to the user regardless of their position or orientation in the CAVE. Whether we crouch, fall or jump, the VGUI will follow us. The buttons rely on PhysX triggers for their functionality keeping the VGUI consistent with other physical objects in the virtual world. Each button can only be activated by the user’s hand(s), with the possibility to use both hands for some advanced interactions. This provides a very natural re-sponse for the user without the need of external devices. While we chose to represent the VGUI as a platform around the user, it could be anything the de-veloper wants as the VGUI uses the same descriptions as other physics objects in the engine.

Virtual Graphic User Interface

(Spraying particles in the scene. Left hand used to start/stop spray. Right hand used for direction of spray)

With such a complex system at our disposal, then came the question of generating content. The 3D Studio Max scripts developed by Ageia and Marcus Storm were phenomenal. They allowed us to build an environment that would have been near impossible otherwise. We never expected to create a full environment with full physical response in such a short time, but the scripts stability and it’s power to preview the simulation in Max made such a project a pleasure. Therefore, we designed our system so that creating and navigating new scenes didn't require pro-gramming, and any new feature or specific interaction required few lines of code. Our system is art path centric, geometry and physics-wise. Even the VGUI follows these rules. This allows art-ists to create content and use the system quickly and efficiently.

3D Studio Max Pipeline

Outside environment and associated physical bodies. (red = static, blue = dynamic, green = trimesh)

Cellar environment and associated physical bodies. (red = static, blue = dynamic)

Our main focus will be to keep working on the integra-tion of features of the SDK into our system. Along with rigid bodies and joints, we have fluids working across the cluster. Our next goal is the integration of soft bod-ies and cloth. We are keeping an eye open and are very excited for any new features that Ageia will release in their SDK. We also plan to extend our character control-ler to get a full virtual body, with full tracking and physical interaction.

To increase our physics computation power, and improve the immersive capability of our system, we will experiment with the hardware cards in SLI mode. Hardware accelerated computations al-lowed us to run physics in parallel to other subsystems and before the actual rendering, therefore getting rid of the "one-frame delay" issue inherent to a system where physics are computed while rendering is done. This is crucial for a real-time immersive environment like the CAVE: a one-frame delay is noticeable, even at high frame rates, and is a huge drawback for realism and user immersion. No other physics system would have provided such a clean, simple and scalable solu-tion for this problem. For real world applications, we will be improving our Virtual Disaster Simulator and other applica-tions related to training first responders. However, the possibilities are endless, and our ideas run far beyond what has been included here including ap-plications to the entertainment industry. We look forward what Ageia comes up with next and hope to continue our close relationship for many years. Special Acknowledgements • Greg Stoner and Bob Whitecotton for the donation of Ageia boards • Marcus Storm for the great 3D Studio Max scripts • Andrew Hamilton who donated portions of the cellar scene for the demo.

Future Work and Credits