Windows on the World

2D Windows for 3D Augmented Reality

Steven Feiner, Blair MacIntyre, Marcus Haupt, Eliot Solomon

The Computer Graphics and User Interfaces Laboratory
When we think of the use of head-mounted displays and 3D interaction devices to present virtual worlds, it is often in terms of environments populated solely by 3D objects. There are many situations, however, in which 2D text and graphics of the sort supported by current window systems can be useful components of these environments. This is especially true in the case of applications that run under an industry standard window system such as X. While we might imagine porting or enhancing a significant X application to take advantage of the 3D capabilities of a virtual world, the effort and cost may not be worth the return, especially if the application is inherently 2D. Therefore, we have been exploring how we can incorporate an existing 2D window system within a 3D virtual world [Feiner, MacIntyre, Haupt, and Solomon 93].

We have developed support for a full X11 window system server within our augmented reality testbed. The user wears a see-through head-mounted display, which is tracked so that the display indexes into a large X bitmap. This effectively places the user inside a display space that is mapped onto part of a surrounding virtual sphere. By tracking the user's body, and interpreting head motion relative to it, we create a portable information surround that envelopes the user as they move about. We support three kinds of windows implemented on top of the X server: windows fixed to the head-mounted display, windows fixed to the information surround, and windows fixed to locations and objects in the 3D world. Objects can also be tracked, allowing windows to move with them. To demonstrate the utility of this model, we have developed a small hypermedia system that allows links to be made between windows and windows to be attached to objects. Thus, our hypermedia system can forge links between any combination of physical objects and virtual windows.

Windows on the World testbed Windows on the World

The figure shows a virtual world populated by three X windows, photographed ``live'' through our see-through head-mounted display. The window at the right is an xload load-average meter that is fixed to the corner of a portable workstation. The xpostit window at the upper left is attached to the 3D tracker ``bolo tie'' being worn by Blair MacIntyre, so it moves when he moves. The window at the bottom is the control panel for our hypermedia system. It is fixed to the head-mounted display, so it always occupies the same location relative to the user's head.

Columbia University Computer Graphics and User Interfaces Lab

This research was supported in part by the Office of Naval Research under Contracts N00014-91-J-1872 and N00014-94-1-0564, a gift from NYNEX Science & Technology, the New York State Center for Advanced Technology in Computers and Information Systems under Contract NYSSTF-CAT-92-053, the Columbia Center for Telecommunications Research under NSF Grant ECD-88-11111, NSF Grant CDA-92-23009, and equipment grants from the Hewlett-Packard Company and Ascension Technology.

Please send comments to Blair MacIntyre at <bm@cs.columbia.edu>