Augmented Reality for Construction

Steven Feiner, Anthony Webster, Blair MacIntyre, and Tobias Höllerer A collaborative project at Columbia University between the Graphics and User Interfaces Lab in the Computer Science Department and the Building Technologies Group in the Graduate School of Architecture.

Introduction

Recent advances in computer interface design, and the ever increasing power and miniaturization of computer hardware, have combined to make the use of augmented reality possible in demonstration testbeds for building construction, maintenance and renovation. In the spirit of the first see-through head-mounted display developed by Sutherland (Sutherland, 1968), we and other researchers (e.g., Robinett, 1992; Caudell & Mizell, 1992; Bajura & Neumann, 1995) use the term augmented reality to refer to enrichment of the real world with a complementary virtual world. The augmented reality systems we are developing employ a see-through head-worn display that overlays graphics and sound on a person's naturally occurring sight and hearing. By tracking users and objects in space, these systems provide users with visual information that is tied to the physical environment. Unlike most virtual realities, whose virtual worlds replace the real world, augmented reality systems enhance the real world by superposing information onto it. The spatial tracking capabilities of our augmented reality systems distinguish them from the heads-up displays featured in some wearable computer systems (Quinn 1993, Patents 1994, Smailagic and Siewiorek 1994).

As part of a program aimed at developing a variety of high-performance user interfaces, we have developed a testbed augmented reality system that addresses spaceframe construction (Webster et al. 1996). Spaceframes are typically made from a large number of components of similar size and shape (typically cylindrical struts and spherical nodes). Although the exterior dimensions of all the members may be identical, the forces they carry, and therefore their inner diameters, vary with their position in the structure. Consequently it is relatively easy to assemble pieces in the wrong position-which if undetected could lead to structural failure. Our augmented reality construction system is designed to guide workers through the assembly of a spaceframe structure, to ensure that each member is properly placed and fastened.

System Description

Figure 1. Spaceframe used in our demonstration testbed.

Our prototype spaceframe structure, shown in Figure 1, is a diamond shaped, full-scale aluminum system manufactured by Starnet International (Starnet 1995). We have created a 3D computer model of the spaceframe, an ordered list of assembly steps, and a digitized set of audio files containing instructions for each step. Undergraduate Computer Science, Engineering and Architecture students helped develop the testbed as part of an NSF-sponsored educational grant in conjunction with the Columbia University's Teachers College.

Figure 2. Headworn display with optical tracker.

The system's head-worn display is a Virtual I/O see-through stereoscopic color display with integral headphones and orientation tracker (Figure 2). Position tracking is provided by an Origin Instruments DynaSight optical radar tracker, which tracks small LED targets on the head-worn display. The user interface also includes a hand-held barcode reader, which has an optical target mounted on it, and is also tracked by the DynaSight.

The spaceframe is assembled one component (strut or node) at a time. For each step of construction, the augmented reality system:

• Directs the worker to a pile of parts and tells her which part to pick up. This is currently done by displaying textual instructions and playing a sound file containing verbal instructions.
• Confirms that she has the correct piece. This is done by having her scan a barcode on the component.
• Directs her to install the component. A 3D virtual image of the component indicates where to install the component (Figures 3-5), and verbal instructions played from a sound file explain how to install it (Figure 6).
• Verifies that the component is installed by asking her to scan the component with the tracked barcode scanner (Figure 7). This checks both the identity and position of the part.

Figure 3.

Figure 4 Virtual world. User's view of the virtual world intended to overlay the view of the real world shown in Figure 3. This image was captured with the real world view blocked. It shows a virtual strut and textual instructions that explain how to install it.

Figure 5. Real + virtual worlds. User's view through the see-through head-worn display showing the real world of Figure 3 combined with the virtual world of Figure 4. In this image the user sees two real struts and a real node overlaid with a virtual strut (top) and textual instructions that instruct the user to install the strut in the location shown.

Figure 6. The user installs the strut requested in Figure 5.

Figure 7. The user scans a newly installed part (in this case, a node) with the tracked barcode scanner.

The software infrastructure in which our prototype is built is COTERIE, a multi-platform, distributed system that has been designed to allow a potentially large number of users to interact in a shared environment (MacIntyre and Feiner 1996). It runs on an assortment of hardware under UNIX, Windows NT, and Windows 95. The majority of the infrastructure was written in Modula-3, a compiled language that is well-suited for building large distributed systems. The remainder of the infrastructure, and the majority of its applications (such as the spaceframe prototype), are written in Obliq, an interpreted language that is tightly integrated with Modula-3. Application programmers are free to use either language or both. For this application we use shaded, hidden-surface-removed 3D graphics running in software under Criterion RenderWare because of the relative simplicity of the models we are rendering. We provide support for 2D applications using the native window system. This allows us to display X11 application windows on all platforms and native Windows NT/95 windows on Microsoft platforms.

The original version of our demonstration included support for only a single user (the construction worker). We are currently modifying the system to allow two additional users wearing see-through head-worn displays to each see a site supervisor's view of the construction process. Each of these additional users will see the construction worker and additional parts of the building, along with information about construction progress, current tasks, and parts availability.

Images from a Live Demonstration

The following images were taken at a live demonstration of our system at the American Society of Civil Engineers Third Congress for Computing in Civil Engineering, Anaheim, CA, June 17-19, 1996.

Figure 8. The spaceframe demonstration testbed set up at the Disneyland Hotel, Anaheim, CA.

Figure 9. A user scans a newly installed node to verify its identity and placement.

Figure 10. A user finds the next strut to be installed and verifies its identity with the scanner.

Figure 11. A user installs a strut.

Acknowledgments

This research is supported in part by the Office of Naval Research under Contract N00014-94-1-0564; NSF Gateway Engineering Coalition under NSF Grant EEC-9444246; the Columbia University CAT in High Performance Computing and Communications in Healthcare, a NY State Center for Advanced Technology supported by the NY State Science and Technology Foundation; the Columbia Center for Telecommunications Research under NSF Grant ECD-88-1111; and NSF Grant CDA-92-23009. Starnet International Inc. provided the spaceframe. The undergraduates involved in the development of the spacefame prototype were Rod Freeman (Mechanical Engineering), Jenny Wu (Architecture) and Melvin Lew (Computer Science).

Explore the work being done by the Building Technologies Group in the School of Architecture.

Learn more about the Augmented Reality for Construction project.

Return to the Columbia University Computer Graphics and User Interfaces Lab.

References

Bajura, M. and Neumann, U. "Dynamic Registration and Correction in Augmented Reality Systems." Proc. VRAIS '95 (Virtual Reality Annual International Symp.), IEEE Computer Society Press. Los Alamitos, CA, 189-196.

Caudell, T. and Mizell, D. (1992). "Augmented Reality: An Application of Heads-Up Display Technology to Manual Manufacturing Processes." Proc. Hawaii International Conf. on Systems Science, Vol. 2, 659-669.

MacIntyre, B and S. Feiner. "Language-level support for exploratory programming of distributed virtual environments." In Proc. UIST '96 (ACM Symp. on User Interface Software and Technology), Seattle, WA, November 6-8, 1996, 83-95.
(870KB gzipped postscript) (538KB Acrobat) (143KB low-resolution Acrobat)

"Patents: Portable No-hands Computer for Consulting Manuals," The New York Times, May 9, 1994, D2.

Quinn, R. "1990s Jobsite Fashion Forecast: the Computerized Hardhat," Engineering News Record, March 29, 1993.

Robinett, W. "Synthetic Experience: A Taxonomy." Presence: Teleoperators and Virtual Environments, 1(2), Summer 1992.

Smailagic, A. and D. Siewiorek. The CMU Mobile Computers: A New Generation of Computer Systems. Engineering Design Research Center Report 18-48-94. Carnegie Mellon University.

Starnet International, Inc. 200 Hope Street, Longwood, FL 32750.

Sutherland, I. "A Head-Mounted Three Dimensional Display." Proc. Fall Joint Computer Conference 1968, Thompson Books, Washington, DC, 1968, 757-764.

Webster, A., Feiner, S., MacIntyre, B., Massie, W., and T. Krueger. "Augmented reality in architectural construction, inspection and renovation." Proc. ASCE Third Congress on Computing in Civil Engineering, Anaheim, CA, June 17-19, 1996, 913-919.
(1.05MB gzipped postscript) (1.01MB Acrobat) ( 216KB low-resolution Acrobat)