It's virtually a reality
So far, promises of virtual reality have been little more than hype. But now, at an Ontario lab called the Cave, Star Trek fantasies are finally coming to life

Sarah Scott
Saturday Post
March 16, 2002

There is a place in southern Ontario, a small windowless room called the Cave, where National Research Council scientists are trying to bring the Holodeck to life. If you're a fan of Star Trek, you know what the Holodeck is -- a computer-created virtual reality, so real you'd think you were there. On Star Trek: The Next Generation, the Holodeck is a virtual playground for Captain Picard and his staff. The Cave is a real-life approximation, a multi-million-dollar illusion machine run by powerful computers.

Step into it, scientists say, and you will step inside a three-dimensional image -- a city street, perhaps, or a space station flying over Earth, or even a throbbing human heart. Other machines have promised similar experiences. What makes this one so convincing is that you always see the image from your perspective. Turn your head to the right, and the image is instantly redrawn -- just as in real life. You can move through the virtual world, walking or flying, if you wish. You might even meet someone there.

So, this is virtual reality, circa 2002. Manufacturing reality has been the dream of computer science since 1965, when Ivan Sutherland wrote a paper called The Ultimate Display, in which he predicted that computers would one day digitize sight, sound and the other senses so effectively that they could control "the existence of matter." This was five years before IBM invented its first personal computer. "With appropriate programming," Sutherland wrote, "such a display could literally be the Wonderland into which Alice walked." Sutherland helped launch a branch of computer science dedicated to making virtual reality a real thing -- with the assistance of helmets, wands, gloves and plenty of refrigerator-sized computers. But the gadgets have always seemed too intrusive to suspend disbelief and make you feel you were there.

Then, a decade ago, a trio of scientists from the University of Illinois at Chicago >invented the Cave, named after Plato's cave. The Cave's official manufacturer is a company based in Kitchener, Ont., called Fakespace Systems Inc., and it has made over 30 Caves so far for more than $1-million a piece, selling them to companies and research institutes around the world. (Another 70 copies, made by other manufacturers, are scattered around the world.) The Cave is not the only way to immerse yourself in a manufactured reality, of course. Virtual reality is an ever expanding world that includes everything from simple desktop screens to three-dimensional social clubs on the Web that you (or rather, a cartoon character acting as your proxy) can visit. But the Cave is probably the most convincing illusion machine, the closest thing so far to the Holodeck.

I had to see it, so I drove to the nondescript NRC building, on the wooded edge of London, Ont., to experience the ultimate in virtual reality. The Cave, located deep inside the building, is a 10x10x10 room with screens for walls. I step inside and put on special glasses to enhance the 3D image. A student working at the NRC wears the headgear; it links him to the giant computer and will track the movements of his eyes so that the 3D image that will surround us is always projected from his perspective. I move in behind him, and the lights go off. Suddenly, we're floating up an art gallery lined with paintings. I can almost forget the images are actually generated by computers and projected on to the screens and floor of the Cave. Then we're in the front seat of a car, and we're slowly crashing. I'm feeling a little weird at this point. After a side trip inside the brain of the victim of the car crash, we see what the victim sees: a blotted, obscured field of vision, the result of the brain injury. I have that odd feeling of being disjointed from reality, as if I'm inside a David Cronenberg movie.

There's nothing fantastical about the research director overseeing this project. A stocky, greying 57-year-old scientist born in Trieste, Italy, Dr. Gian Vascotto has a doctorate in biology; he has studied shellfish and shrimp, industrial nuclear waste and acid rain. He didn't get into the computer-visualization business until four years ago, when he came to the NRC, where he's now director of system simulation and control research. He then began thinking about how computers and simulation software could make manufacturing more competitive. He decided to find out whether Canadian manufacturers were interested in using these new technologies. "We thought it would be a really hard sell," he says. To his surprise, companies jumped at the idea. So the NRC created the 4,500-square-foot centre, which includes the Cave and other visualization equipment. It opened for research, testing, and demonstrations in September, 1999.

The NRC tests different uses of the Cave with several >"collaborators" who help pay for the research. One of them is General Motors, which makes light armoured vehicles for the U.S. army at a nearby plant. Until recently, designers would draw a detailed picture, which would then be turned into a physical prototype -- or several, if the first one didn't work -- for $2-million or more apiece. Now they create a 3D computer image of the vehicle. The team visits the NRC's giant screening room, where they inspect the 3D full-scale image. They can walk around the virtual vehicle, inspect it from every angle, even look at the interior. According to Joe Attard, a manager in GM's defence division, this reduces by about a third the time it usually takes to design these vehicles. "I see this as a major change in the way we develop design," Attard says. GM also uses the Cave to design cars at its vast facility outside Detroit. Legend has it that one car executive put his coffee cup on the virtual coffee holder in the Cave. It looked that real.

Now, inside the NRC'S Cave, the computer has churned out a new 3D image. This time, we're in downtown London, Ont. Vascotto explains that recently, city politicians wanted to see whether a proposed arena would overwhelm the city's historic market. As a research project, the NRC decided to show them. The city officials met at the Cave. The young university student at the controls charges up the program to show me what they saw: "There's the arena on your left," she says. She's waving her hands to move us through the virtual downtown. We swoop down to ground level. "We can go over here and stand on the street corner," she says, "and see, we have no problem seeing the market building." It's like a three-dimensional IMAX universe, except I feel like a floating giant. The buildings look like very good architectural models, built to scale, but not quite real. Maybe it's because we're missing the other part of the picture -- the hot wind on my face, the dull sound of a plane overhead. (The student has been trying to fix that, by giving the computer instructions to duplicate the subtle sound of jets that you hear if you're inside a building.) Later, when I drive back downtown to see what the scene really looks like, I look up at the roof of the building against the blue sky and I'm surprised by what I see. Reality is not how I had pictured it. It actually looks more like a computer graphic than I remembered.

Not every aspect of reality has been mimicked as successfully as sight and sound. Take touch, or haptic feedback, as the computer nerds call it.

Researchers are working on how to duplicate that sense in a virtual environment. Surgeons at some hospitals in North America can now practise on the image of a patient in a virtual environment, with an actual scalpel. The instrument is attached to a machine that feeds back resistance or vibrations to simulate cutting through bone or puncturing skin. But it's still a pretty primitive touch machine; other sensations are harder to duplicate.

Virtual-reality scientists are still working on it. Still, they're getting close enough to manufacturing reality to ask a crucial question: Just how real do you want to be?

"In some sense, virtual reality is a wish to recreate an experience close to our experience of the natural world," says Daniel Sandin, one of the Cave's creators and the director of the Electronic Visualization Laboratory at the University of Illinois. The "Killer App" is obviously design, he says, but the Cave is useful for lots of other things -- the study of the Big Bang, or the flow of water at the bottom of the ocean, the analysis of complicated things in constant flux, or the reconstruction of ancient sites.

A group of historians, scientists and artists, for instance, is working on the world's first virtual exhibit of a historic site -- the ancient Mogao Grottoes in the Gobi Desert, 492 caves with murals, wall fresco paintings and more than 3,000 painted sculptures. Imagine: >Instead of going all the way there, you could see virtually the same thing in a North American museum.

But doesn't it matter that the Grottoes are virtual, and not real? I put that question to Canadian architects Hani Rashid and Lise Anne Couture, who are designing the Virtual Guggenheim Museum, what they call "the first important virtual building in the 21st century." Their answer: Maybe not. "They removed the David, the real David, from the Palazzo Vecchio, in Florence," says Couture. "People know, but they go, 'I don't really need to see the real one because this one is good enough.' It's in the setting, whereas the real one is in a museum. Most people would rather see the fake one in the real space than see the real one in the wrong space."

"Looking at the real is a nostalgic experience," notes Rashid. He spent a year in grad school looking at pictures of Bosch's famous painting The Garden of Earthly Delights. When he finally saw the real thing, he recalls, he wasn't bowled over: "I thought it was going to be bigger." In any case, the line between real and virtual is blurred these days, he says. "Marcel Duchamp told us a long time ago that anything can be authentic. You see a real urinal or a bottle rack in a museum. You see that stuff coveted as real, and then you walk by the store and there's exactly the same bottle rack in the window. I went, 'Wow, let's buy it. Now do I have a real Duchamp?' "

As partners in the ultra-cool New York architectural firm Asymptote, Rashid and Couture have built an internationally renowned practice that includes spatial experimentation and installation, buildings and urban design, and computer-generated environments and architecture. They work out of a fifth-floor Soho office, three blocks from their home, that's lined with books -- Plato, Rousseau, the Marquis de Sade, The Decline and Fall of the Roman Empire, Gertrude Stein. They speak at cutting-edge international conferences, such as the Doors of Perception.

To Rashid and Couture, virtual reality offers incredible possibilities for altering our perception of space. "We are right now in a state -- with mathematics, understanding of biological systems, and virtual-reality technologies --that is probably unparalleled since the Renaissance," says Rashid. Consider how people saw things before the invention of perspective in the mid-15th century. Looking at a drawing by Giotto in the early 14th century, "you would get this strange, flat world, and that was real to the medieval mind." A century later, with the discovery of linear perspective, "you get an entirely different view of what is considered to be a rendition of the real," he says.

"In the Renaissance, there was a new mathematical understanding of the cosmos. We understood science; we had the printing press at our disposal." Suddenly, the medieval world looked remarkably primitive. With the invention of virtual reality, it's as if we're in the mid-15th century, when we suddenly start perceiving reality in a new way, Rashid continues.

"In the next decades, we're going to have worlds and renditions of reality that will make our modern way of thinking of the world -- perspectival and cognitive and mathematical and certain -- seem almost medieval."

But Rashid and Couture are not interested in duplicating reality. They see in virtual reality the chance to create entirely new possibilities.

Take the New York Stock Exchange, one of the last exchanges on the planet to use real people to trade stock, instead of doing it by computer. The NYSE recently hired Rashid and Couture to design a virtual stock exchange to enable NYSE officials to track the estimated US$20-billion of trades per day. The challenge was to track the movement of vast sums of money at incredible speeds. "We took it on as a spatial, architectural problem," Rashid says.

They constructed a computer world that makes you feel you are going into a three-dimensional space to retrieve information -- with the advantage of being able to zoom from an overview to a close-up in an instant, and to retrieve information and manipulate it in novel ways. The NYSE is a "machine for information exchange," said Rashid. "We had to mimic that machine in virtual space."

Down at the New York Stock Exchange, the virtual stock exchange is just off the Main Room, on the Ramp. Anne Allen, senior vice-president of the NYSE's floor operations, explains how it works. Her job is to make sure things run smoothly at the stock exchange, that there are no backups or glitches in the trading. Allen used to try to track the action on 21 computer screens filled with names and numbers. But it was hard to detect a problem quickly; she usually heard it first from the guys on the floor.

Now, she can watch the trading on nine screens. Although they are flat, the screens depict the trading floors as futuristic 3D images in blue, purple and yellow. The images match the geography of the real floor. Allen can see the floor from any vantage point, zoom down from a bird's-eye view to ground level, fly past the trading posts, and stop at a trading post to inspect the trades in detail. Whenever anything unusual happens -- hot news or a stop in trading, for instance -- flags pop up beside the virtual trading posts. Allen can fly in to inspect, and then pull away to slice and dice the information as it's flooding in; the charts are constantly bobbing as the numbers change.

Although the virtual NYSE includes live footage from the floor, its main architecture doesn't look like the real thing. Rashid and Couture rebuffed suggestions to make the walls look like marble. "We looked at them and said, what does that have to do with information? Forget that," says Rashid. "Why not put data in the walls. That's why the walls are liquid, living walls of information." In other words, click on the wall and you will find numbers, information. "The only thing they haven't been able to put in this is the energy," says Allen. The real Main Room, a financial bazaar dating back to 1791 that buzzes under a web of high-tech equipment, is a rush, she says: "I don't know how you can manufacture that synthetically."

I ask Rashid and Couture about >the Cave, the attempt to create an artificial reality that mimics the natural world. It's a "myopic, closed view," says Rashid. "We think reality is interesting enough." Last year, Rashid saw a virtual Toronto display in downtown Toronto, where he grew up. "I was trying to figure out what was missing. It wasn't just the wind and the sun. How did it turn into a hollow, emptied-out model? What's missing is memory," he says. So, the way you see something is influenced by your memory of it. "The majority of the stuff [out there now] is kind of illustrative. It's the Norman Rockwell syndrome."

Back at the Cave in London, Vascotto is pressing ahead with an ambitious goal: "To put people into virtual reality directly so they can interact with virtual people." Vascotto wants to create a convincing image of a person who walks and talks inside the Cave. Then he wants to send that 3D computer image via the Internet to another Cave thousands of miles away, where the "virtual person" could talk with a real person. In other words, you could be in two places at one time, although one of you would be virtual.

Vascotto is not joking. He hopes to demonstrate that this can work sometime next year. It would be like a video conference, only more realistic because, instead of seeing your colleague on a flat screen -- which some companies have tried and then abandoned -- he or she would be a moving 3D image. "Basically five people across Canada could send their virtual selves to a meeting place," he says. Now, this might not cut it for a client paying the big bucks to see a management consultant or an investment banker in person, but it might replace other types of business travel. "If I go to Ottawa for a meeting, it's 10 hours of my time for a two-hour meeting," says Vascotto. "An awful lot of travel takes place today because of the value of face-to-face communication."

No one has yet managed to create a virtual person and zap him somewhere else. That's largely because it takes a phenomenal amount of computer power to capture the movements and expressions of humans. To make movement look fluid, you need to create 30 frames a second, and the exercise is complicated by the fact that every time the viewer shifts her eyes a little, the computer has to instantly redraw the scene to accommodate her new perspective. "The facial features are very difficult to track because of the very subtle, very small motions," said Vascotto. At the NRC, scientists are borrowing from the world of animation to create a virtual person.

Say that a London golf pro wants to demonstrate his swing to someone in Calgary. The people at the NRC start by creating a 3D representation of the golfer's body. They can use a standard skeletal form, or the body shape of the actual golf pro. Then they photograph the golfer's swing. To do this, they stick little quarter-inch reflective balls all over the real golfer's body and then use special cameras to track each little ball as he swings. When it's time to show the swing to the person in Calgary, the real golfer demonstrates the swing in a Cave in London, and his virtual image in the Calgary Cave would duplicate the motion, like a puppet.

Vascotto has some serious competition in the United States from >Jaron Lanier, the computer genius in dreadlocks who invented the term "virtual reality". In a 2001 article in Scientific American, Lanier described how he brought together three people in different locations for a virtual conference. Rather than using an animation technique, Lanier and his team use a "sea of cameras," between seven and 60, to capture the participants, or the "moving sculptures," as he calls them. There are problems: pixelated images, a time delay that can be disorienting, even nauseating, the vast expense. Nevertheless, Lanier predicts that >tele-immersion will be commonplace in 10 years. The main obstacle, Lanier says, is not cost or computer power: "It's just that the whole thing is so damn exotic." (Others say the roadblock is human nature. Some people don't want their real selves to be projected into cyberspace. They might just want to amend the image a little, or put on a mask.)

It is perhaps inevitable that, at the end of this story, science reunites with fiction. The U.S. military has used simulators for a long time to teach trainees how to fly a plane or use other types of machinery. Now, the military is taking simulation to a whole new level. It's set up the Institute for Creative Technologies at the University of Southern California -- an assembly of people from film, games, computer science and the army -- >to create a virtual training ground filled with virtual people. "The goal is to create a prototype of something like the Holodeck," says Dr. Bill Swartout, the institute's technical director. >Some of the people at ICT even worked on Star Trek, he says. "The Holodeck is a source of inspiration. It suggests research paths we're going down."

ICT's aim is to teach soldiers headed for peacekeeping missions to deal with people in crisis. For these kinds of simulations, the army used to hire actors, but they're not always available when you need them, says Swartout. So the ICT is developing virtual characters -- like the distraught mother with an injured child in a Balkan village who's crying out for help. She's projected on to a giant curved screen that gives the illusion of immersing you in another reality. The human trainee, cast in the role of army lieutenant, has to figure out what to do. His responses are fed back into the computer. If he turns away, the mother gets really upset. If he tries to help, she calms down. The image is still a little cartoonish, but that's the price you have to pay if >you want the virtual character to be able to respond to the recruit's actions, says Dr. Norman Badler, director of the Center for Human Modeling and Simulation at the University of Pennsylvania, who's working on the virtual characters. "The level of realism is satisfactory enough for people to overlook the slightly synthetic look and get engaged."

Badler wants to turn the avatars, as these virtual representations of humans are called, into something more than mere puppets. He wants the computer characters to start thinking for themselves. To start with, they need to learn natural language, says Badler, co-author of a paper on the subject, Dynamically Altering Agent Behaviors Using Natural Language Instructions. "The long-term goal is to educate our avatars to make decisions, respond, and act in future situations on their own," Badler says. Getting the computer characters to express emotion is not easy, though. "We don't have an effective computational model for what goes on inside our head," he said. "But we're now pushing our characters to think about what they see, and react." It's digitizing emotion, essentially, and that could be the final frontier.

Emphasis mine - JJ

Note: An avatar was used in the recently released remake of H G Well's TIME MACHINE. - JJ

Reality is manufactured in lab (General Motors collaborating in Illusion Machine). National Post, Mar 16, 2002. Go to 42.The Party Tells 'Why'