Dan R. Olsen Jr.

Computer Science Department
Brigham Young University
Provo, UT 84602



Phone: 801-378-2225

Surface: Computer Science Department, Brigham Young University, Provo, UT 84602



Adaptive Human Interfaces


Programming by demonstration, User Interfaces, Interaction, Pictures as Data, Information presentation


Original goals

Direct manipulation user interfaces are difficult to build. User Interface Management Systems (UIMS) have attempted to alleviate this problem by providing models of the interactive dialog which can drive the interface implementation. Such models, however, have failed to directly express the changes of information which comprise most operations in direct manipulation interfaces. A data pattern /transformation language based on unification matching will be developed as a model of such additional processes. Techniques will be developed for using this language as a user interface design tool. These techniques will include approaches for visually expressing the patterns / transformations interactively. This data transformation language can then be used for creating visualizations of data. Such a language can express attribute modifications, style sheets, critics (which monitor and advise users) and searches. Unifying the interaction in such a transformation language provides a foundation for more intelligent tools to reason about the behavior of the interface.


In order to apply pattern-based transformations to user interfaces and the objects that they manipulate, it is important that there be a uniform model for all information in the system. It is critical that any object in the system can be accessed through that model regardless of the object's implementation. Without such generalized access to information it is impossible to reason in an automatic way about such objects. We call our implementation of this concept NIC (Nucleus for Interactive Computing). Having defined and built such a model and then having implemented a user interface tool-kit within that model we then apply unification-style patterns to such objects, including user interface objects. Based on this model we have been able to compute search patterns from examples, infer widgets by example from drawings, infer inductive relationships in pictures, and infer class structures for data defined as pictures rather than in traditional programming languages. In addition we have developed a pattern-based agent architecture for networked information access.


Because of the general data model for representing user interfaces and the information they manipulate, we were able easily implement distributed user interfaces. In our model anything can be converted to an intermediate form and transmitted over the network. Based on this we have developed interactive services that are distributed over the World-Wide-Web. We have also used this distribution of interfaces to implement a tool kit for sharing collaborative interfaces across the network. This allows collaborative applications to be built in NIC with little more effort than single-user interfaces.



"Interactive Net Services on the WWW" with K. Rodham. INTERACT '95. (June1995).

"Distributable Interactive Objects" with K. Rodham. INTERACT '95. (June 1995).

"Pictures as Input Data" with D. Kohlert. Human Factors in Computing Systems (May 1995)

"Building Geometry-based Widgets by Example" with B. Ahlstrom and D. Kohlert. Human Factors in Computing Systems (May 1995)

"Smart Telepointers: Maintaining Telepointer Consistancy in the Presences of User Customization" with K. Rodham. ACM Transactions on Graphics (July 1994)

"Automatic Generation of Interactively Consistent Search Dialogs", Human Factors in Computing Systems (April 1994).

Theses and dissertations (available through BYU library

"GUI Tool Templates from Examples" Jun Lu, MS Thesis, 1995.

"Techniques and Tools for Network-based Interacton" Kenneth J. Rodham, PhD Dissertation, 1995.

"ADAPT: Application Data as Pictures on Terminals", Douglas C. Kohlert, PhD Dissertation, 1995.

"AGENTMAN: An Agent Manager Which Allows New Agents to Benefit Preexisting Application", Jeffery Kim Jensen, MS Thesis, 1995.

"LEMMING: Learning Easy Maps to Make Interesting New Gadgets", Bret D. Ahlstrom, 1995.

"An Intelligent Drawing System Based on Induction by Example Technique", Xinyu Deng, MS Thesis, 1994.

"Magic: Malleable Graphical Applications", Douglas C. Kohlert, MS Thesis, 1993.

"Searches by Example for GUI", Ming Cai, MS Thesis, 1993.

"A Nested Parametric Coordinate-based Interactive Interface Systems", Jinyun Lin, MS Thesis, 1993.


This project has its background in User Interface Management Systems. The goal of such systems is to abstract the definition of a user interface out of the code into an external specification. The purpose of extracting the user interface is so that enhanced tools can be created to aid interface designers. Originally this intermediate form was called the dialog specification when it was believed that most user interface software problems are related to the handling of input events. This external specification has gradually moved to a higher level with much emphasis being placed on connecting visual objects to the information that they manipulate.

The project also has roots in the "by demonstration" community where software is developed from examples of the desired behavior rather than by explicit programming. The problem is that visual information such as screen layouts and widget manipulations do not lend themselves well to textual specification in programming language. The hope is that such information can be inferred rather than encoded.


"Building Real Time Groupware with GroupKit, A Groupware Tool kit", Roseman, Mark and Greenberg, Saul, ACM Transactions on Computer-Human Interaction, March 1996.

"User Interface Management Systems: Models and Algorithms", Dan R. Olsen Jr., Morgan-Kaufmann, San Mateo, CA, (1992).

"Watch What I Do: Programming by Demonstration", ed. Allen Cypher, MIT Press, Cambridge, Mass. (1993).


The network-based user interface techniques used in NIC would be highly amenable to virtual environments. Using distributed rendering the NIC information update model would make developing such shared environments quite easy.