View Management for Virtual and Augmented Reality

View management refers to the layout decisions that determine spatial relationships among the 2D projections of 3D objects needed to obey some desired set of constraints. For example, some objects may be sufficiently important to the tasks being performed that their visibility should be guaranteed, while others may need to be rendered at some minimum size or preferred aspect ratio to improve their legibility. Furthermore, there are many situations in which the viewpoint or the geometry of certain objects may be dynamic, yet externally determined, and relatively unpredictable. For example, the viewpoint may be constrained to that of a head-tracked display, and some of the objects may be controlled by a simulation or may be part of a changing physical world viewed through the see-through display of an augmented reality system.

Figure 1 demonstrates how automated view management is applied to a collaborative augmented reality environment. It is photographed through a see-through head-worn display from the perspective of one user sitting across from another, discussing a virtual model of the Columbia campus located between them. Labels are placed relative to buildings’ projections for readability without overlapping or ambiguity. Additional annotations in the scene include a meeting agenda (shown at the left), a virtual copy of a building that one of the users has created, and popup information related to that building (both shown at the right). The projections of these annotations avoid other objects, including the visible user’s head, to maintain a clear line of sight between users.

Figure 1. View management in an augmented reality meeting scenario (imaged through a tracked, optical see-through, head-worn display).

Figure 2 shows a simple interactive 3D information visualization system that allows users to inquire about buildings on our campus. Visible buildings are initially labeled to show their identity, and to convey what is selectable. When a user selects a building, more information about it is displayed, and placed close to the building’s visual projection with a leader line that points to the building’s projection to convey the information’s relationship. A virtual avatar is shown in the model with a comic-strip speech balloon annotation, constrained at a fixed position relative to the projection of the avatar’s mouth. All annotations are laid out so they do not overlap each other or any of the buildings’ projections. Automated placement of these annotations by our view management module ensures that important objects in the 3D scene remain unobscured when providing the user with useful information.

In the real world, many different factors influence what we see, whether done routinely, subconsciously, or without control. For example, we push aside a centerpiece to enable eye contact with others at dinner, or we hold a city guidebook so that we can read a relevant section, while simultaneously viewing an historic building that it describes as we pass by on a bus. External elements can affect what we see as well, such as someone’s head blocking the view in a movie theater or hockey game. In any of these situations, virtual and augmented reality provide the opportunity to present important information that can assist users when needed, as we have described here. Depending on the application and situation, continuously changing priorities influence what users want to view, both virtually and in the real world.

Figure 2. Screen shot of a 3D campus information visualization system.

By overlaying information onto a real or virtual world, a user interface can assist a user in understanding their environment. This is an especially important task if the user is mobile and the environment is unfamiliar: the virtual overlays need to enrich and explain, rather than clutter and confuse, the user’s physical surroundings.

We have developed a situation-awareness aid for AR and VR that is intended to provide the user with an overview of the surrounding environment and the ability to discover, select, and inquire about objects that may not be directly visible to the user within that environment. Our aid is based on a world in miniature (WIM) or exocentric “god’s eye view”: a miniature overview model of the surrounding environment is embedded within the user’s view and allows user-driven view management. The work described here addresses two key issues. First, hands-free control of our aid is designed to make it easy for the user in a mobile application to determine how much attention they wish to devote to the aid. Second, changes in user focus produce complementary modifications of overlaid annotations that are presented in the full-scale physical environment and in the miniature virtual aid. Figure 3 shows screen shots of our annotation aid in action.


Figure 3 Transfer of annotations between the physical world and virtual aid. (a) Popup annotation provides information about the computer monitor to which its arrow points at the center. (b) As user looks down, annotations are enabled in the aid, and the popup’s arrow now points at the monitor’s representation in the aid.

Figure 4. Browsing an augmented reality environment

We have developed some strategies to satisfy specific constraints for application tasks such as labeling, annotating, and placing other information. For example, Figure 4 shows an AR environmental browsing mode in which the system tries to label a number of objects that have been determined to be of interest to the user, making sure that the labels are not ambiguous. In this case, we use the two-tiered approach that was also applied Figures 1–2, creating both internal and external labels, and ensuring that no label occludes any part of an object except the object it labels. The algorithms that we use build on the space representation that we developed for our dynamic space management approach which we use to categorize the available space on the view plane.

For each object to be labeled, we determine the largest visible rectangle internal to the object’s projection that can contain the largest copy of the object’s label, given a user-settable font and size range, and taking into account an additional buffer around the label to keep adjacent labels from appearing to merge. This buffer can also help control the amount of the related object’s projection that is blocked by the label if the object has a high priority. For efficiency, the dynamic size of a label is approximated by using its aspect ratio at a certain font size, texture mapped to a polygon, and then scaled based on the result of the query.

If no visible space for an object is large enough, the object will not be labeled internally, but could instead be labeled externally. External labels can be processed in any desired order. We currently use the front-to-back order; however, since the allocation algorithm is greedy, if an importance metric were available for labels, it could be sorted instead. To lay out an external label, the system queries the empty-space representation for a rectangle that can contain the label within a user-settable range of sizes, and within a user-settable distance from the object. If the label is allocated within the empty space, it will neither be occluded by nor occlude any other object. If no such rectangle can be found, then the label is not allocated.

Since external labels are potentially ambiguous, we also generate a leader-line arrow from the label to the interior of its object. Providing the leader line helps mitigate possible misidentifications. Because internal labels occlude parts of the object that they label, we also support tagging an object to indicate whether or not certain parts should be internally labeled (e.g., internal labels may be suppressed for faces). Each external label is added to the spatial representations as it is created, so that objects added later do not occlude it.

Figure 5. Image taken directly through an optical see-through head-worn display of restaurant and related popup information.

Applications may need to show additional information about objects in the scene (Figures 1–4). This information can be placed external to related objects, much like external labels, possibly with different size and aspect ratio constraints.

Some information, such as the agenda located on the left side of Figure 1, does not refer to any objects that are visible in the scene. To place the agenda on the screen so that it doesn’t overlap any important objects, the empty-space region is queried after all other annotations are placed. There are also situations, as the AR neighborhood restaurant guide of Figure 5, where there is not enough room to place the information. In this case, we detect the limited space by the negative results of external placement. Another query is used to place the popup information so that it avoids overlapping the restaurant, but could overlap other objects, such as the building above it.

An online video describes some of the concepts behind View Management and some implementation details.

If you are interested or have more questions, please contact me at blaine [at] cs.columbia.edu or Steve Feiner (feiner [at] cs.columbia.edu), or if you wish to get more information on licensing, please contact:

Any opinions, findings, and conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF or any other organization supporting this work.