Virtual reality, ubiquitous computing, tangible interfaces, reality-based interfaces, next-generation interfaces, non-WIMP interfaces.
H.1.2 [Models and Principles]: User/Machine Systems--human factors; H.5.2 [Information Interfaces and Presentation]: User Interfaces
Is there an emerging next generation of human-computer interaction or rather simply “a thousand points of light” of disparate and unrelated innovative new developments? This workshop will bring together researchers in a range of emerging new areas of HCI to look for common ground and a common understanding of a next generation of user interfaces. If we consider command-line interfaces as the first generation, then direct manipulation and the graphical user interface defined a distinct new generation of user interfaces[5] that is still the state of practice today. Unlike the early days of graphical user interfaces, research in HCI today is developing on many fronts, making the next generation more difficult to connect and define. Yet, much current research appears to be moving away from the screen based GUI, in a related general direction.
The key components of this next generation are found in a variety of loosely-related current research areas in HCI—or, more broadly, interaction design or human-information design:
This workshop seeks ways to tie some of these and others together intellectually and to try to define a next generation from them. We will look for unifying ideas, frameworks, and theories that provide common ground for discussing, analyzing, connecting, inventing, comparing, and making predictions about emerging new interaction styles and interface designs. We also hope to use such a framework provide some explanatory power for understanding what makes particular new interfaces better or worse. In addition, it could help identify gaps or opportunities to develop a future research agenda suggested by holes or “sweet spots” in a new taxonomy.
As a starting point for discussion, we will introduce the notion of natural or realistic or reality-based interfaces as a thread to connect new developments in HCI. This notion focuses on the ways in which the new interfaces leverage users' built-in abilities. These interfaces increasingly draw their strength from exploiting the user's pre-existing skills and expectations from the real world rather than trained computer skills. For example, navigating through a conventional computer graphics system requires a set of learned commands, such as keywords to be typed in or function keys to be pressed. By contrast, navigating through virtual reality exploits the user's existing real-world “navigational commands:” positioning the head and eyes, turning the body, and walking toward something of interest. Perhaps basing the interaction on the real world reduces the mental effort required to operate the system because the user is already skilled in those aspects of the system. For casual use, this reduction can speed learning; for use in situations involving information overload, time pressure, or stress (e.g., surgery, disaster recovery), this reduction of overhead effort could improve performance.
A unifying characteristic for much next generation HCI may thus be that it increasingly taps into the users' abilities and pre-existing knowledge. Direct manipulation moved user interfaces toward more realistic interaction with the computer; next generation, reality-based interfaces push further in this direction, increasing the realism of the interface objects and allowing the user to interact even more directly with them.
We could also take this approximate notion of “realistic” or “natural” and make it more precise—perhaps by focusing on the pieces of knowledge or skills that a system requires its user to know. This could lead to a notional checklist of the knowledge the user needs, which may help in discussing and connecting different new user interfaces. However, there are many kinds of things that the user already knows. Moving the head to change point of view is one. The user may already know more arcane facts such as pressing the Alt-F4 keys will close a window. It seems intuitively better to exploit the more “basic,” more built-in knowledge that the user learned in infancy (or perhaps was born with) than to exploit more recently learned, less innate knowledge, like the Alt-F4 keys. We could explore how to measure reality-based vs. non-reality-based knowledge on a more continuous scale. This requires a way to rate the degree of reality or innate-ness for a piece of knowledge. One way is by when the user learned it; and we conjecture that younger is better. Information that is deeply ingrained in the user seems somehow more robust, perhaps more highly practiced, and should take less effort to use than information learned recently. These questions may be explored in the workshop.
Another side of this issue is that reality-based is typically not sufficient. A useful interface will rarely entirely mimic the real world, but will necessarily include some “unrealistic” or artificial features and commands. In fact much of the power of using computers comes from this “multiplier” effect, the ability to abstract from or go beyond a precise imitation of the real world.
Our goal is to find common elements for understanding and discussing a next generation of HCI and to build a community of researchers to consider this topic explicitly, in contrast to many recent developments in new interaction styles, which have thus far tended to proceed independently on unconnected and unrelated fronts.
We will use the notion of reality-based interaction to provide a concrete starting point for the workshop. We will begin by considering whether we can use that to tie together developments in next generation interaction styles into the beginning of a useful conceptual framework. From there, depending on the participants' views and contributions, we will extend or expand this approach as well as introducing alternative opposing or complementary approaches to the problem. Participants will be invited to extend, expand, discredit, or replace this initial approach, but it will provide a concrete starting point with which to agree or disagree. Current research at Tufts is fleshing out the reality-based approach with analyses of reality-based knowledge and skills needed for different interaction styles, and we will provide our latest work as input to start the discussion.
We will thus invite participants to present:
We hope a key contribution will be that ideas emerging from the workshop will serve as a lens or common language for viewing, discussing, comparing, and advancing proposed innovative new interface developments and technologies—to provide some coordinate axes on which to put them into perspective and organize them.
A second goal is to produce a research agenda for new work both in gaps suggested by our frameworks and in ways to evaluate or validate our frameworks. The initial conclusions we draw from these for simple examples may turn out to be true, but not surprising. We will focus on ways to make and test theory-based predictions that are less obvious, as a better way to test our theories.
Our final goal is to create a community of HCI researchers who are thinking specifically about connecting their research to other developments in next generation interaction. We hope the results will generate ideas that give the HCI community a new and more explicit way of thinking about and connecting next generation interaction and will suggest a research agenda for future work in this area.
The workshop will welcome researchers working in areas such as those listed above (virtual and augmented reality, ubiquitous, pervasive, and handheld interaction, tangible user interfaces, etc.) and in particular: participants researching and developing things they view as part of next generation interaction; participants with ideas or approaches for describing or defining next generation interaction; and participants with ideas for how to elaborate, formalize, test, evaluate, or expand on the reality-based starting point and to replace it with new approaches.
To date, few researchers have addressed this issue explicitly, but several have discussed sub-areas and made contributions toward it. People who have attempted to explain or organize these new styles of user interfaces have tended to concentrate more on individual classes or groups of new interfaces than on concepts that unify the classes. The time is ripe to start a discussion that connects such work. For example, Ullmer and Ishii provide a framework for tangible interfaces[6]; Fishkin, Moran, and Harrison propose the concept of embodied interfaces[3]; Bellotti, Back, Edwards, Grinter, Henderson, and Lopes define sensing interfaces, and raise a set of key problems[2]; and Beaudouin-Lafon's Instrumental Interaction model sheds new light on post-WIMP interfaces[1].
We are also pursuing this area at Tufts, under an NSF grant, and teaching a course in Fall 2005 on “Reality-based Interaction: Understanding the Next Generation of User Interfaces,” which will provide new work as input to the workshop. The NSF project will also serve as a nexus for continuing and collecting work in this topic after the workshop.
Finally, work that helped define the GUI generation is a model for us. Shneiderman took a set of disparate new user interfaces and unified them through their common characteristics[5]. Hutchins, Hollan, and Norman then explained their power and success of these interfaces with a theoretical framework[4]. Our hope is to take a first step in that direction for the emerging generation.
We thank our collaborators Andrew Afram, Eric Bahna, Georgios Christou, Michael Horn, Michael Poor, Orit Shaer, Larissa Winey, and Jamie Zigelbaum of the Computer Science Department at Tufts as well as Caroline Cao and Holly Taylor of Tufts, Leonidas Deligiannidis of University of Georgia, and Hiroshi Ishii and Sile O'Modhrain of the MIT Media Lab.
And we thank the National Science Foundation for support for our research project on this topic (NSF grant IIS-0414389).
1. M. Beaudouin-Lafon, “Instrumental Interaction: An Interaction Model for Designing Post-WIMP User Interfaces,” Proc. ACM CHI 2000 Human Factors in Computing Systems Conference, pp. 446-453, Addison-Wesley/ACM Press (2000).
2. V. Bellotti, M. Back, W.K. Edwards, R.E. Grinter, A. Henderson, and C. Lopes, “Making Sense of Sensing Systems: Five Questions for Designers and Researchers,” Proc. ACM CHI 2002 Human Factors in Computing Systems Conference, pp. 415-422, ACM Press (2002).
3. K.P. Fishkin, T.P. Moran, and B.L. Harrison, “Embodied User Interfaces: Toward Invisible User Interfaces,” Proc. of EHCI'98 European Human Computer Interaction Conference, Heraklion, Crete (1998).
4. E.L. Hutchins, J.D. Hollan, and D.A. Norman, “Direct Manipulation Interfaces,” in User Centered System Design: New Perspectives on Human-computer Interaction, ed. by D.A. Norman and S.W. Draper, pp. 87-124, Lawrence Erlbaum, Hillsdale, N.J. (1986).
5. B. Shneiderman, “Direct Manipulation: A Step Beyond Programming Languages,” IEEE Computer, vol. 16, no. 8, pp. 57-69 (1983).
6. B. Ullmer and H. Ishii, “Emerging Frameworks for Tangible User Interfaces,” in Human-Computer Interaction in the New Millenium, ed. by J.M. Carroll, Addison-Wesley/ACM Press, Reading, Mass. (2001).