There are four fundamental research questions we have been exploring. The most important is how the users will understand and correct the inferences made by the system. Any system that generalizes from examples is occasionally going to guess wrong, and it is important that users know what the system is doing so they will feel comfortable and in control. The PURSUIT visual shell (an iconic interface to a file system, like the Macintosh finder) incorporates our most successful and novel mechanism for this feedback. PURSUIT is the PhD thesis of my student, Francesmary Modugno, and was just completed in May, 1995. As the user demonstrates a program by executing example commands, PURSUIT builds a visual language representation of operations and inferences. The visual language is based on the "comic strip metaphor": the panels show the relevant data, and changes from one panel to the next represent the operation. The language provided a single medium for verifying and correcting inferences, for reviewing completed programs later, and for editing programs. Despite skepticism from other researchers that the interface would be usable, formal user studies showed that nonprogrammers could create fairly complex programs, and that the visual language was more effective than an equivalent textual language.
The second fundamental research question for demonstrational interfaces is the appropriate representations and inferencing algorithms. For a system to be acceptable, it must guess right most of the time. Our early systems used straightforward rule-based techniques and pattern matching that were empirically tuned to give acceptable performance. These resulted in highly predictable user interfaces, and this technique has been adopted by most other demonstrational systems. We are now beginning to explore a more elaborate algorithm to provide powerful inferences that can take into account the user's hints.
The third research question is what architectural support can be supplied in a toolkit to make demonstrational interfaces easier to build. Many forms of demonstrational interfaces record a sequence of operations and allow them to be repeated. We have developed an architecture where programmers structure applications around "hierarchical command objects," which encapsulate the information needed to execute, undo and re-execute each operation. They are hierarchical to support structural decomposition of user interface behaviors.
The fourth research question is to which domains can demonstrational interfaces be successfully applied. We have identified useful aspects of user interface construction, text formatting (funded by a previous NSF grant), file manipulation in a visual shell (in Pursuit discussed above), data visualization, and educational game construction, that are appropriate for being demonstrational. They share the properties that the natural way a person would describe the problem to another person is by drawing examples, and that domain knowledge can be used to narrow the range of possibilities for generalizing from the examples. Each domain also illuminates new issues for feedback and representations.
Francesmary Modugno, Albert T. Corbett and Brad A. Myers. "Evaluating Program Representation in a Demonstrational Visual Shell." Experimental Studies of Programmers Sixth Workshop. Jan. 5-7, 1996. Alexandria, VA. To appear. 1994 IEEE Workshop on Visual Languages. St. Louis, MO. pp. 304-311. Abstract and Postscript
Francesmary Modugno and Brad A. Myers. "Pursuit: A Demonstrational Visual Shell," Technical Video Program of the CHI'94 conference. SIGGRAPH Video Review, Issue 97, no. 12.
Francesmary Modugno, T.R.G. Green and Brad A. Myers. "Visual Programming in a Visual Domain: A Case Study of Cognitive Dimension," Human-Computer Interaction '94, People and Computers. Glasgow, Scotland, August, 1994. pp. 91-108. Abstract and Postscript
Francesmary Modugno and Brad A. Myers. "Exploring Graphical Feedback in a Demonstrational Visual Shell," The 1994 East-West International Conference on Human-Computer Interaction (EWHCI'94). St. Petersburg, Russia, August, 1994. pp. 262-272. An updated version appears in Lecture Notes in Computer Science 876. Brad Blumenthal, Juri Gornostaev and Claus Unger, Editors. Springer-Verlag, 1994. Abstract and Postscript
David S. Kosbie and Brad A. Myers, "Extending Programming By Demonstration With Hierarchical Event Histories," The 1994 East-West International Conference on Human-Computer Interaction. St. Petersburg, Russia, August, 1994. pp. 147-157. Abstract and Postscript
David S. Kosbie and Brad A. Myers. "A System-Wide Macro Facility Based on Aggregate Events: A Proposal," Watch What I Do: Programming by Demonstration, Allen Cypher, et. al., eds. Cambridge, MA: The MIT Press, 1993. pp. 433-444.
Andrew J. Werth and Brad A. Myers. "Tourmaline: Macrostyles by Example," Proceedings INTERCHI'93: Human Factors in Computing Systems. Amsterdam, The Netherlands, April 24-29, 1993. p. 532.
Andrew J. Werth and Brad A. Myers. "Tourmaline: Macrostyles by Example," Technical Video Program, INTERCHI'93 conference. Amsterdam, The Netherlands, April, 1993. SIGGRAPH Video Review, Issue 89, no. 17.
Brad A. Myers. "Text Formatting by Demonstration," Proceedings SIGCHI'91: Human Factors in Computing Systems. New Orleans, LA. April 28-May 2, 1991. pp. 251-256.
The motivation behind demonstrational interfaces is simple and compelling: if a user knows how to perform a task on the computer, that should be sufficient to create a program to perform a the task or a generalization of that task. The goal is to give the power of programming to all computer users without requiring them to learn how to program.
Brad A. Myers. "Demonstrational Interfaces: A Step Beyond Direct Manipulation," IEEE Computer. August, 1992. vol. 25, no. 8. pp. 61-73.
Brad A. Myers. "Invisible Programming," 1990 IEEE Workshop on Visual Languages. Skokie, Ill, October 4-6, 1990. pp. 203-208.
Visualization and Interactive Computing.
Intelligent Interfaces.
Amulet in particular is being designed with the specific goal of supporting user interface researchers, so we hope that it will prove useful to NSF ISP.