KARMA

Knowledge-based Augmented Reality for Maintenance Assistance

Steven Feiner, Blair MacIntyre, Dorée Seligmann

We believe that one of the most powerful uses of virtual worlds will not be to replace the real world, but rather to augment the user's view of the real world with additional information. This idea, introduced by Ivan Sutherland's pioneering work on head-mounted displays, is often referred to as augmented reality. For example, graphics and text overlaid on the surrounding world could explain how to operate, maintain, or repair equipment, without requiring that the user refer to a separate paper or electronic manual. Similarly, participants in a business meeting could interact with a dynamic shared financial or organizational model represented in 3D and selectively supplemented with each user's personal (and private) annotations. Generating such material by hand, however, will require a tremendous amount of expertise and effort, far greater than that currently needed to design ``hand-crafted'' hypermedia and multimedia presentations. We have begun to explore how overlaid explanatory graphics can instead be designed on the fly automatically using AI techniques. This work builds on our current research on the knowledge-based generation of 3D graphics [Seligmann & Feiner 91] and multimedia presentations [Feiner & McKeown 91] that explain physical tasks.

KARMA

We are developing KARMA (Knowledge-based Augmented Reality for Maintenance Assistance) [Feiner, MacIntyre, and Seligmann 93], a prototype system that that uses our see-through head-mounted display to explain simple end-user maintenance for a laser printer. We attached several Logitech 3D trackers (the small triangles in the figure shown above) to key components of the printer, allowing the system to monitor their position and orientation. A modified version of the IBIS rule-based illustration generation system [Seligmann & Feiner 91] interactively designs graphics and simple textual callouts that fulfill a set of goals that are input to KARMA. This virtual world is intended to complement the real world on which it is overlaid. For example, one rule states that if a goal is to show the user where an object is located, the system must determine if the object is blocked by other objects. If it is blocked, it will be displayed so that it appears to be seen through the blocking objects; if it is already visible in the real world, it need not be drawn at all.

Overlaid graphics

This figure shows a virtual world designed by KARMA, viewed ``live'' through our see-through head-mounted display. It was generated to show the user how to remove the paper tray from the laser printer. The virtual world contains a highlighted representation of the paper tray that tracks the tray as it moves, a dotted line ``ghosted'' view of the tray's desired destination state, and an arrow that pulses outward to represent the action of pulling out the tray.

Columbia University Computer Graphics and User Interfaces Lab