Much work is needed in this new field, but that is

IS FOR REAL

John A. Adam Senior Associate Editor

“Go for it!” we shouted. A visitor at the University of North Carolina’s “G lab” was teetering at the edge of make-believe stairs descending into a dark basement. The problem was he could not see his own legs.

“Uh-uh. No way. No way!” He took off his helmet and returned to the real world. Later he said he’d felt he was being sucked into a bottomless well.

When my turn came, I enclosed my head in the helmet and entered that fearsome ersatz world-a simulated household kitchen. Beacons on the lab’s ceiling allowed my motions to be tracked by helmet sensors and then computers calculated and (usually) rendered images at a rate of about 20 frames a second, fast enough to create and sustain the illusion.

There were no preplanned routes. I moved within this fake space at whim. Mischievously, I picked up a weightless black frying pan with my disembodied pink hand icon, then dropped it onto a blue tiled floor. Then I dragged some chairs through wood-grain cabinets. The experience, while impressive, hardly seemed real-until I reached the stairs. They looked steep.

I paused. Fear short-circuited reason. I knew, of course, that physically I was on a level floor in the lab. Yet my eyes were telling me otherwise. My leg trembled un- controllably. I finally went forward, my hands groping reflexivety for the stair walls, preparing for a tumble that never came.

That was my most gut-wretching expe- rience with VR. I also rode into canyons on a pterodactyl swooped off a skyscraper with a hang glider, and stared down the tailpipes of an F-15 fighter. These experiences were fun and impressive but (done on commercial systems and not on the university’s special renderer) did not approach realism.

“The current technology is a t the boundary of usability,” said James Helman of the advanced graphics division of Silicon Graphics Inc., Mountain View, CA, a leader in this field, “but a long way from reality.”

Virtual environments have been around for decades, mainly in expensive flight simu- lators and military displays. What’s new about “l reality“ is that the necessary computing power and displays are more af- fordable and new peripherals such as data gloves have become available, generating

22 001&9235/93/$3.00W393 IEEE

still further dimensions of immersion and interactivity.

‘‘Before, you wouldn’t spend $6 million for a truck simulation because using real truck prototypes was much cheaper,” said Ron Toupal, president of Paradigm Simulation Inc., Dallas, TX. Now, with the price of some very capable VR systems less than US $100 OOO, all that is changing.

In prosaic terms, VR endeavors to create a very natural interface between humans and computers, where communication can take the form of moving 3-D imagery, spatial sounds, and even physical forces- from motion to crude touch. (Smell is also being examined.) The better systems can be so responsive and render imagery so fast that the user can feel immersed in the syn- thetic environment.

This has all sorts of entertainment possi- bilities, like immersive video games and films, and many practical ones, too. For in- stance, the ability to do “soft designs” of cars or space stations that engineers can literally enter (rather than view from the window of a computer) might result in better products at lower cost.

There are many mutations of virtual en- vironments. One type, such as the North Carolina kitchen [see picture, opposite], em- phasizes immersion. The computer-gen- erates all that is seen by the user. Another approach, called augmented reality, en- hances the real world by computer-gen- erated overlays. For instance, a worker might ask the computer to display the correct electrical connection while he is looking at the actual aircraft wiring with see-through goggles. And some “virtual presence” work goes beyond videoconfer- encing to sense and manipulate objects from afar. Such applications include moni- toring a robot on Mars or diagnosing casu- alties near hazardous front lines.

In the mid-l980s, researchers a t the National Aeronautics and Space Administration (NASA) Ames Research Center had to use stick graphics to get 2F30 frames per second in architectural modeling. Now much richer scenes are available and it gets better every year. A small VR industry has sprung up, offering a variety of mainly first-generation methods of creating synthetic environments.

IEEE SPECTRUM OCTOBER 1993

head-coupled, 3-D glasses, and multiscreen projections-which are priced from several thousand dollars to roughly a d o n dollars [see “Tools of the trade,” belowl.

a virtual reality (VR) system consists of a display, a tracking a computer image generator, a three-dimensional

n soflware. The several types of displays available are suited to & i t tasks.

The most basic and economical, costing several thousand dollars and not incbding the computer, is the computer monitor approach, with liquid-crystal &otter g lass and trackr. The monitor dispkay changes according to the user‘s head movements as tracked by a device a m the monitor. Glasses enable 3-0 vievling of imagery as in the w of the Hubble Space Telescope’s corrective

ew of this system prevents a sense of immersion but, on s k r frwne rates and h m sharper imagery. Also, the compared to more immersive displays. Leading com- glasses and other equipment include Stereographics ,and TWwix Inc., Beaverton, OR.

ense of immersion is provided by the head-inomted display ks out the real world and wen allows the user to assume a

such as that of a fish. There are also see-through HMOs that ination or the virtual and real worlds, known as augmented

new HMO with 1-inch CRT screens was recently introduced by are still u n c o m f o ~ ~ l e ~ i t h the bulkiness of the unit,

though, must wait for &ses with built-in displays. A heabcoupled dwky (HCD) is like a huge pair ot binoculars supported

by a “ a b l e robot-like arm. The image sources and optical components are end of anarm, which also has a buitt-in W n g sy&m with no lag.

solution and a wider field of view beGause weight HMDs. They also offer immersion with the added t, which is why they are a favorite at trade shows ion is the Boam (binocular ~ n i ~ i ~ i ~ mm-

o Park, CA. A disadvantage in some applications manipulate the display and audio is not standard. use multiple large-screen projection displays,

ulf groups of viewers . Examples are the M1 icrosystems Computer Corp., Mountain View,

factors. Nausea, dizziness, or other ailments may result if a user is immersed for more than several minutes. But the potential is so great, and improvements are coming so

rapidly, that industries from manufacturing to health care to defense are already inves- tigating VR applications.

One of the most enthusiastic industrial

CA, and the Cave, created by the University of Illinois, Chicago [see p. 301. Liqui6crystal shutter glasses, such as CrystalEyes by Stereographies, must be worn. The Cave can cost as much as US $1 OOOOOO, but several are being built in industry and government.

High-end graphics workstations or specialized computers usually gene&? the virtual scene from a 3-D database that is often derived from some unique modeling packagelike computer-aiM design software andlor a finite- element analysis program. The rendering component transforms the abstract numerical representations into easily interpreted images, usually composed of colored polygons. The final result is hwo stereo images that are transmitted to the display.

Most of today3 tracking systems, which may sit atop workstatiis or be sus- pended from ceilings, use either migrtetic fields or ultrasound to measure the position and orientation of objects like HMDs o transmitter emits a continous signal that hits the translated into six-degreedfreedom meeso for the computer. Leaders in the hadring area in and Polhemus Inc., Colchester, VT.

Of the various input d e v i i used in VR, the wired glove is often the most useful. Its user can touch or grasp both virtual and real ob@s without undue diff i&y. Information from sxnsors W measure the Wing of finger and harKl joints is related to the position of the glove, which is tracked separately. In im- rnersive VR, the wired glove is represented in the vittuai scene, typically as a disembodted, floating hand.

The computer processes the trackiig information and OW inputs from&

4 A A .

A d a m - V i reality is for real 23

7 - - - I

users of VR so far is Caterpillar Inc., Peoria, IL. The $10 billion company is the world’s largest manufacturer of earth-moving and construction equipment. Its operators do much more than drive-controlling huge hydraulic shovels, for instance-so visibility from all sorts of angles is critical to effi- ciency and safety. Assessing interior visibility on new prototypes is the purpose of Caterpillar’s first endeavor in the VR field, something beyond its regular computer- aided design work.

‘The perspective that you get in [existing] 3-D visualization packages isn’t really true,” said design engineer Dave Stevenson of the Wheel Loader and Excavator Business Unit. Although you can rotate the image and even go inside it, you can’t do what you’ d do in the vehicle,” he said, such as lean a few cen- timeters to the side to get a better view of what is happening, as well as a better dynamic sense of the design model. FAKE TRACTORS. With VR, that is a cinch. To test a design, Stevenson sits in a “ l i s t physical mock-up of a cab. On his right are two levers that lift and tilt a huge ”virtual” bucket. In front are the steering wheel and gas and brake pedals. On his left is the gear shifter. These are actual physical controls, but they are also rendered and registered in the VR world so that, when a user reaches for the gear shifter in the fake world, contact is made with the actual shift in the real world.

When Stevenson dons the head-mounted display, the lab disappears and he finds himself seated inside a futuristic wheel loader at a construction site. He works the controls, and the bucket scoops up dirt. He leans forward and gazes down at the action. Then he backs up, makes a tight turn, and goes toward a dump truck to empty his load. All the while, he cranes his neck, checking visibility.

Except for the digging, nothing is canned. Stevenson can maneuver wherever he wants. Even while the vehicle is moving, he

can look out any window, although for complex scenes the frame rate dips below the 20 frames per second needed to create the illusion of fluid movement.

The imagery is deliberately kept simple enough to be rendered in close to real time, but also sufficient for the design engineers examining visibility. It takes just 2 to 3 minutes in VR immersion to evaluate a design change, Stevenson told me, so nausea and other human factors issues have not been a big problem.

The first VR-influenced designs will be seen in back-hoe loaders (contraptions with a bucket up front and in back a ditch digger articulated arm) to be introduced around 1996. John Ekttner, a Caterpillar engineer in the Building Construction Products Division, told Spectrum they had three distinct designs in concept, all ostensibly equally feasible.

But in the VR world, Ekttner said, they found that with two of the designs, ”when you operate the back-hoe linkage, you can get into a position that occurs dynamically, where you can’t see what you’re doing.” Without VR, he said, this would not have been determined until the first prototype was built. The result: Caterpillar chose a design nine months earlier than would oth- erwise have been possible and saved the tremendous cost of prototyping the two bad designs in iron.

The company produces hundreds of products, many of them, such as heaving mining equipment and log loaders, in small production runs. Consequently, if a profit is to be made on a line, money cannot be wasted on generations of iron prototypes, some of which may cost millions.

Virtual reality could change Caterpillar’s whole design approach. ‘We’ve been able to look at new ideas and either scrap them or develop them,” Stevenson said. “It’s ex- panded by orders of magnitude what we can try. In a few days we can probably crank out a dozen wssibilities.”

r I Cate&illar’s VR work is beine done at the National Center for Supercomputing I Applications, a government-private sector I consortium at the University of Illinois, UrbanaChampaign. Stevenson credited the consortium, as well as work begun three years ago with the University of Iowa, Ames, for the company‘s success with VR. The technology was too unproved and risky, he said, for Caterpillar to pursue on its own.

All the equipment used was commercially available: Silicon Graphics’ Skywnter for the graphic rendering, a Hewlett-Packard 735 for calculating the dynamics of new scenes, and Sense8’s WorldToolKit software. In ad- dition, an IBM PC drives the hardware inputs from the platform, like the gearbox controls, and an Apple Macintosh generates sounds. For displays, there is a choice of a head-mounted display or 3-D glasses and a 1.8-meter screen. Recently, Caterpillar began working with a Cave display with multiple projection screens.

24

The toughest part of Caterpillar’s VR work probably was converting the parts fdes. Now that some automatic translators have been written, it takes about three days to convert a parts file-a ProEngineer computer-aided design (CAD) file of a cab, a tire, a frame, a sheet metal part, and so on-for VR use, according to Stevenson. (One reason for the delay is that the CAD data, already in three dimensions, has a res- olution that is too high for today’s virtual en- vironments.)

Right now, Stevenson would never show a virtual prototype to a potential customer. “He wouldn’t buy it,” he said. ”It would be the VR flaws that he’d notice first.” For in- stance, mirrors are absent from the ren- dering because the computational com- plexity they require would lower frame rates. An eventual goal, of course, is to in- corporate customer input into VR pro- totypes that make use of sound and motion as well as nice imagery. AUTO MAKERS. Detroit has dabbled in VR, too. The Big Three among US. automobile makers have found the current VR tools not yet worth widespread adoption, but of such great potential that they are joining with a US. Army vehicle center, United Technologies’ automotive division, the University of Michigan, Ann Arbor, and a dozen other smaller companies to form a consortium for industrial VR.

“VR has the potential of revolutionizing design and manufacturing,” the industrial VR group stated in a July proposal to the Pentagon’s Advanced Research Projects Agency. It went on to predict savings in time and money, better market response, and better products.

Hyperbole or not, other companies cor- roborated many of the proposal’s claims, specifically:

Virtual prototyping will reduce or elim- inate the need for costly physical mock-ups.

Engineering analysis will become more ef- ficient through the integration of simulation results with virtual prototypes. For example, air-conditioning flows or even crash tests can be visualized in 3-D. Eventually, it will be possible to alter designs and see the im- mediate effects.

Operational simulations will permit the direct involvement of human beings in per- formance and ergonomic studies. For in- stance, passengers of various sizes will be able to comment on the convenience and look of a virtual car’s interior. Immediate design modifications will get immediate feedback. More iterations will be feasible than with physical prototypes.

Virtual simulation of assembly, production, and maintenance tasks will reveal possible problems at an early stage of the design process.

Support of assembly and maintenance through augmented reality will reduce job training and increase workforce flexibility. For instance, pages of a maintenance manual or diagram might be projected while

IEEE SPECTRUM OCTOBER 1993

A d a n - V i reahty IS for real

. -_

B A available, is the point of entry for simulating early prototype designs, Socks said.

k i n g ' s new airliner, the 777, to be rolled out next year, is its first to be designed without the use of a full-scale physical mock- up. The aircraft design is being specified dig- itally, using Catia software on IBM main- frames and CAD workstations.

It is a tribute to what can be done in soft design without VR. Current VR systems are orders of magnitude away from being able to render the complexity and detail nec- essary for such a design at acceptable frame rates. Representations capable of being dis- played in VR of the part geometries for the whole aircraft would consist of between 5 billion and 10 billion polygons, estimated David Mizell, virtual systems manager at W i g Computer Services, Seattle, WA.

But the allure is great. In certain situ-

ations where an engineer is trying to develop a mental image of a very complex collection of parts, sticking his head into the design and looking around should be more effective, in Mizell's opinion, "than merely looking at it 'through the window' of a work- station."

The Boeing team has written software that can import Catia geometry into their Silicon Graphics-based VR system. They are experimenting with a variety of algo- rithmic techniques aimed at rendering virtual worlds with the complexity and detail of CAD models at the frame rates required for VR. Currently, they can view subsets of the CAD design, such as a passenger com- partment interior or some of the hydraulics. But the frame rates are not yet high enough for production use.

'When you're in the virtual interior and

you look over to the side at the windows, the frame rate is really nice," Mizell told Spectrum. "But when you look straight down the aisle, [it1 drops quite noticeably." This is because most of the aircraft then comes within the user's field of view. The graphics hardware has to process several million polygons, representing all the objects in front of the viewer, whether or not they get displayed on the screen.

Mizell does not foresee each of k i n g ' s thousands of engineers getting VR capa- bility at all soon. He believes that conven- tional CAD workstations are generally ad- equate for designing individual parts. VR would be desirable to visualize a part in its installed context, among many other parts, and determine, say, whether it could be ma- neuvered around the other parts in order to be installed or removed for maintenance.

26 E E E SPECTRUM OCTOBER 199

Mizell sees modification of the CAD design while inside the virtual environment as farther off than visualization only. “We’re making improvements, but we’ve got a ways to go, particularly with the rendering per- formance, before we can make the decision to use VR in the design of the next plane,” he said.

Another Eking project, on augmented reality, begun in 1990, might come to fruition first-at least, it is less computationally de- manding. The goal, said Mizell, is to give more information to aircraft factory workers, many still doing hand labor. For in- stance, they might be able to consult a manual without having to stop work, or as- semble par t s marked with a virtual template.

One possible application would be hand layup of composite cloth. Aircraft manufac- tu rers a re building some parts out of graphite composites instead of aluminum. Parts with complicated, compound cur- vature are often laid up by hand. Currently, expensive templates or other marking systems are employed to mark on the previous layer where the next layer of cloth is to be placed.

A worker using augmented reality (AR) could simply look down at the layup tool and see a computer-generated rectangle seemingly drawn on the previous layer to mark where the next layer was to be placed. “They would perceive the graphic as painted on the workpiece,” Mizell said, even as they moved around. After laying this cloth, the user could push a button or say “Next” into a simple speech recognition system, and the AR system would then display the outline of the subsequent layer of composite cloth.

Another potential application the Boeing team is experimenting with is wire bundling assembly. An aircraft’s electrical wiring is assembled into bundles on “formboards,” from one to eight 0.9-by-2.5-meter (3-by-8- foot) particle board sheets that have a circuit diagram of the wire bundle glued to them. This makes each formboard unique to a wire bundle, incurring storage and con- struction costs. With augmented reality, Mizell said, a worker could assemble a wire bundle on a blank formboard, with one or two wires at a time being shown in the worker’s AR display, appearing to be drawn on the board.

All this could be done in a self-contained unit-a see-through display on the worker’s head and a fanny pack with memory and batteries around the waist. “We’re just drawing a couple of lines out there, not ren- dering everything the user sees,” said Mizell, so it is generally easier to do, with one exception.

Tracking the user’s head position and ori- entation presents a challenge because of the need for high accuracies @or example, 0.1 inch or better positional and 0.1 degree angular) over relatively long distances. Moreover, most commercially available head trackers are inadequate for factory

I

floor usage. Magnetometers lose accuracy , database management, physics-based engi- around radio frequency noise and ferrous 1 neering analysis, real-time interference metal. Ultrasonic trackers are affected by ~ (such as collisions) analysis, and distributed ultrasonic noise-from drills, rivet guns, high-performance computing among a large and so on. Consequently, over the next few number of vendors. months, Boeing will be experimenting with HEALTHY m. Virtual reality applications in optical trackers. I medicine, still years from general clinical

Once the tracking has been addressed, use, benefit from two trends in health care. the human factors issues will be tested, de- The first is the extensive use of imaging t e d g , say, whether a worker can use tools, such as ultrasound and magnetic res- the gear for two hours at a stretch without 1 onance imaging (MRI), to peer into the body. getting a headache. “We’re not even at a The second is “ a U y invasive surgery, point where we can address that, because 1 such as endoscopic procedures, in which the we’ve got crude laboratory equipment,” doctor looks not at the patient but at a video Mizell said. I screen to guide an optical-fiber light probe MILITARY DESIGN. Defense work and space , and tiny camera. Instruments are inserted exploration, both of which often take place 1 through puncture holes and manipulated in hostile environments, have had much to from outside the body. do with the creation of VR. The Pentagon’s I Researchers at the University of North Advanced Research Projects Agency (AR- Carolina (UNC) at Chapel Hill are working PA) sponsored Ivan E. Sutherland’s pio- on 3-D ultrasonic imaging of a fetus super- neering head-mounted display research in ~ imposed on a pregnant woman’s abdomen the 1960s. Shrinking government budgets [see picture, p. 25, bottom right]. Better are now forcing both defense and space ~ imagery would be possible with computer agencies to think more about virtual capa- tomography and MRI scans, said UNC bilities for design. ’ computer scientist Gary Bishop. But the ul-

“Aircraft carriers, submarines, and space trasonic scans are inexpensive and easily ac- stations are becoming unaffordable,” said cessible for tackling basic problems of Gary W. Jones, simulation program timely rendering and correlation of the su- manager in ARPA’s maritime systems tech- 1 perimposed image and the actual fetus. nology office. Their complexity is growing. ComDanies like Medical Media Svstems. The new Seawolf submarine has three times as many drawings as its predecessors. An aircraft carrier now has 30 million com- ponents, he told Spectrum, not counting nuts and bolts.

“How can humans keep up with all of that?” he asked. ARPA is betting on a com- bination of virtual environments, physics- based modeling, and distributed simulation. Last March it awarded 18-month contracts to two teams led by Lockheed Missiles & Space Co., Palo Alto, CA, and General Dynamics Corp.’s Electric Boat Division, Groton, CT, to assess virtual ship and sub marine design. The initial contracts are to experiment with pieces of the system, such as a torpedo room [see picture opposite, top]. The goal, several years off, will be “live fire” tests of virtual prototypes on a sim- ulated sea. “This will permit a much more realistic assessment of a candidate design” early in the process, Jones said, when changes can be made much more cheaply than in engineering production.

While many users are now content with 10 OOO polygons per frame, Jones’s project requires jumps to 1 000 OOO polygons per frame at 20-30 Hz. Users must also be able to interact on various scales with a large complex geometry on the order of 30 million objects. More than lo00 designs with hundreds of parameters are to be generated and evaluated and modifications (which may affect ”over 50 percent of detailed design drawings”) must be accommodated. Dynamic rapid access to more than 200 gi- gabytes of memory must be used.

Jones expects the project to drive such critical technologies as interface standards,

Hanove‘r, NH, hope to use VR in training surgeons in new techniques, much as pilots are trained for new planes on simulators. Medical Media, in cooperation with Dartmouth University’s medical and engi- neering schools, recently demonstrated a simulation of endoscopic knee surgery. High Techplanations Inc., Rockville, MD, expects to complete a surgical simulator for laproscopy. Georgia Tech in Atlanta is trying to simulate surgery on the human eye, complete with force feedback.

V i u a l renderings might also be used to help plan surgery and in remote as- sessments of patients by teams of spe- cialists. One possibility, being examined by Georgia Tech and others, is to couple tactile gloves with visual and audio links for te le diagnosis in rural areas where there are few doctors. ARPA is also preparing a medical initiative that will include VR, one possible military application of which would be its use by doctors to quickly diagnose front-line casualties remotely.

Patients are expected to begin therapy for acrophobia this fall using a 50-story virtual glass elevator, designed at Georgia Tech. In physical therapy, playing with virtual objects can be more appealing than playing with real ones. Moreover, forces, such as the resistance a patient encounters in squeezing a ball, can be scaled and his or her recuperation can be better measured and tracked. RESEARCH AND EDUCATION. There are nu- merous scientific VR visualizations, from atoms to galaxies. Users working in three di- mensions can try to dock a drug in a complex molecular structure, scaled up, to

27 Adan-Virtual reality is for real

.

feel the forces from different energy levels. Manipulators, at sites like the University of North Carolina, can vary from tweezers to a robot arm.

Some telepresence at the atomic scale was recently achieved when researchers from UNC and the University of California at Los Angeles (UCLA) announced a new way to view and control a scanning tun- neling microscope with a VR nanomanip- ulator [see picture, p. 26, left]. Examining graphite, UCLA chemistry professor R. Stanley Williams said the technique enabled him to discover that what had previously been thought of as noise was actually pieces of dislodged carbon. VR permits the viewer to rotate the sample and change the light source for different perspectives. Something intriguing can be investigated further, such as the space between atoms in a crystal lattice.

Research will reach new dimensions in November, when a Cave VR system is to be hooked to a Connection Machine 5 super- computer in Portland, OR. The result will be simulations in real time or close to it. The arrangement will enable engineers and scientists to “modify the situation as it happens,” said Carolina Cm-Neira, a Ph.D. candidate at the University of Illinois, Chicago. For instance, designs might be altered to assess aerodynamics. Initial applications for the supercomputer

VR include hurricane studies (complete with

3-D sound), galaxy configurations and black hole collisions, and representations of math- ematical worlds such as hyperbolic space.

At NASA’s Johnson Space Center, Houston, TX, R. Bowen Loftin, who ex- plores VR training for astronauts, has de- veloped a virtual physics lab for high school students. Once inside the virtual world, the students can change its properties on a control panel by adjusting gravity, friction, or time. For instance, they can stop time or slow it down in order to perceive fast- moving experiments. Among the tools are two balls with alterable (and traceable) bounce and a variable-length pendulum. AMUSEMENTS. Entertainment uses for VR naturally have received the most media at- tention, and experts agree that this large market will be a driving force in VR tech- nology development. Already, consumers with PCs can create 3-D racing-car games or air combat simulations with software such as Sense8’s WorldToolkit. Combined with Windows-based networking, the game can be played by several people together. Sega expects to introduce a low-end head- mounted system this fall. Silicon Graphics and Nintendo recently announced plans for a 64-bit 3-D game machine intended to go on sale by late 1995 for about $250.

VR arcade machines are springing up, many of them from W Industries Ltd., Leicester, UK, which sells some that can be networked so that operators can work in

teams. VR parks began in Chicago a few years ago with Battletech, which now has been expanded to Japan, too. Instead of going bowling, some teams regularly pay to wage war on each other in a futuristic setting.

Voyage to Mars is a park that is to open this year in Walnut Creek, CA, using simu- lators from Hughes Aircraft Co., Los Angeles. The company’s training division sells the Mirage, a multiseat networkable unit with a wraparound display for views of 6 meters to infinity. Another park, Magic Edge, Mountain View, CA, is building space capsule simulators with Silicon Graphics’ Reality Engines and hydraulics to simulate an F-14 fighter scrambling from an aircraft carrier to intercept hostile flyers.

Perhaps most eagerly awaited is the outcome of a Star Trek collaboration between Paramount Pictures Corp., Los Angeles, and Spectrum Holobyte, Alameda, CA (a computer-game maker). Reportedly, users are to be able to wander about the Enterprise and interact with the characters. Plans call for 50 Star Base centers, the first to open late next year.

Over the last year, there has been “lots of activity” in VR by Hollywood studios and casinos, according to Silicon Graphics’ mar- keting head, Daniel Vivoli. Most deals ap- parently remain confidential. DESIRE FOR IMPROVEMENT. If VR has piqued commercial interest, the movement beyond

28 IEEE SPECTRUM OCTOBER 1993

the research lab has also highlighted defi- ciencies-and spurred an urgent desire for improvement. “The current performance is marginal. It will stay marginal for quite a while,” Silicon Graphics’ Helman told several hundred designers at Siggraph 93 in August.

For a truly entrancing experience, a iumble of things must happen simulta- neously and seamlessly. Perhaps most im- portant is keeping up an acceptable frame rate for the user, usually about 20-30 Hz for each eye, while juggling tasks like math- ematical simulation, visual rendering, audio, force feedback, and collision detection. Input data from trackers, motion platforms, and so on must be taken into account, too.

Currently, all these tasks are too taxing for most workstation arrangements so d e signers must make tradeoffs. To orches- trate so many pressing demands, two basic approaches are employed synchronous and asynchronous.

With the synchronous method, the user always gets the most recent result, just in time. But it is limited by the speed of the slowest task. In the unpredictable VR world, noted Helman, it is hard to have all tasks run predictably in their allotted time. Consequently, frame rates may slow down, but the frame will have complete data.

With the asynchronous method, the frame rate is kept up by sacrificing completeness. For instance, if collision detection takes a tenth of a second and visual rendering takes a sixtieth of a second, the scene can be rendered six times while the collision is cal- culated. This scheduling is obviously more complex and latencies may occur. “The car may have already gone off the racetrack,” Helman observed.

A compromise between the two ap- proaches, with the tasks being divided among different machines with different pri- orities, is often necessary to keep frame rates consistently high and latencies low.

Another common compromise is just to drop features like as collision detection. Many applications also neglect sound and force feedback. Most attention is currently focused on the tug of war between image quality and frame rate. Frame rates around 10 Hz are very choppy. “The trouble is, people like complex images, so we tend to bring it to m i n i ” frame rates,” said Daniel Sandin of the University of Illinois, Chicago.

If a graphics computer can render a million polygons per second, then getting 30 frames per second would allow (simplis- tically) about 33 000 polygons to depict a scene. Nonetheless, a molecule can look much better when depicted with 100 000 polygons, so a user may put up with a frame rate of 10 Hz.

Compensatory techniques such as the blurring from anti-aliasing can make W b y - 512-pixel displays seem almost as good as 1280-by-1024 screens (which have L3 million pixels that need refreshing), said Silicon Graphics’ Helman. Texturing can save on

the order of millions of polygons in lieu of the best shading. But all this visual trickery is less impressive in stereo, he admitted.

Choosing rural scenes (canyons are a fa- vorite) rather than more difficult cityscapes is also a way of dealing with current short- comings. Precomputing scenarios and limiting paths to certain branches are two other means.

As imagery from computers improves, so must the display capability. Current head- mounted displays, usually the best for im- mersion, are often 320 by 240 pixels, giving the user the vision of the legally blind, except with a much narrower field of view. The best displays, just starting to appear commercially, have 1280 by 960 pixels. These rely on cathode-ray tubes, which are heavier than other devices. Consequently, there is a desire to create lightweight head- mounted displays that are not much more obtrusive than glasses.

Dick Urban, a project manager in MA’S electronic systems technology office, is sponsoring development by 1994 of a 1280- by-1024-pixel monochrome electrolumi- nescent display to fit in a standard ll-ounce (0.3-kg) Army goggle. A color liquid-crystal display of about 1 square inch (6.5 cm2) with that acuity is the aim of another project, which recently demonstrated pixels of 24 km. Urban told Spectrum that these are probably the world’s smallest pixels (they also have great potential for highdefinition television, offering 1OOO lines per inch, or 40 per millimeter). Even if these pixels can be manufactured, they will still provide the equivalent of only 20140 vision and a 40- degree field of view, Urban said.

But 12-km liquid-crystal display pixels are conceivable-just enough light would still pass through the pixel, he believes- and the result could be displays of 2048 by 2048 pixels, which approaches the quality of 20/20 vision.

With a commercial DataGlove, a user can “crush” a virtual soda can [left1 programmed with compliance and tensile-linearity para- meters. Small pneu- matic microcylinders, attached to the glove from the palm to the Fngertips with Velcro, produce force feedback, so the user feels a “near natural” tactile sensat- ion, said researchers at the Human-Machine Interface hboratory, Rutgers University, Piscataway, NJ

Tracking also needs improvement to better synchronize real-world motions with virtual-world depictions. When a user stops moving a data glove, for instance, and his hand keeps moving in the virtual world, the illusion suffers. “The noise and latencies are particularly bad” now, so systems cannot be predictive, said Daniel Sandin of t he University of Illinois, Chicago, a factor that could aid in the computing of complex scenes. Latency-the change in time from input to result-can also make a user unaware that an action has been received, so he repeats it, compounding the problem. Latencies may also be nauseating.

Other problems, noted recently by VR pioneer Frederick P. Brooks Jr. a t the University of North Carolina in Chapel Hill, are MITOW field of view (usually much less than peripheral vision affords), poor regis- tration with the real world (so a user might bump into real walls while walking through a virtual room), tethered ranging (which limits how far a user can wander), and dif- ficult wayfinding in virtual worlds (where it is much easier to get lost than it is in the real world).

So far, much emphasis has been placed on the visual. But sound and force “radically enhance the experience,” Brooks observed. The more senses that are bombarded with redundant information, the more real the experience appears to be, noted Scott S. Fisher, managing director of Telepresence Research, a Portolla Valley, CA, company that develops VR applications .

In addition, building synthetic envi- ronments is usually lots of work. At present, 20 OOO to 40 0o0 polygons might be used to render a house. A single room in a better model takes thousands of polygons and is equivalent to writing about 20 OOO lines of code. As for how to make authoring more efficient, Brooks said, ‘We’ve not begun to even address this problem.” +

Adam-Virtual reality is for real 29