Susan Haller

Assistant Professor
Computer Science and Engineering Department


University of Wisconsin - Parkside
Kenosha, Wisconsin 53141-2000

E-mail: haller@cs.uwp.edu
Phone: 414-595-2343
Fax: 414-595-2114




Speech and Natural Language Understanding


Natural language interaction, turn-taking, negotiation, discourse representation, mixed initiative interaction


The usefulness of spoken natural language for human-computer interaction is not in question. A spoken natural language interface provides a flexible and efficient means of communication while a human user's eyes and hands are occupied. Providing for dialogue, whether typed or spoken, allows for humans to collaborate effectively with computer systems that are becoming increasingly complex to use because of their capabilities. Interactions for problem-solving and cooperative work are a mixture of information-seeking and information-providing behavior that a human participant and a computational system must both be able to engage in.

The research issue that will be addressed in the final proposal is how to represent and reason about interactive discourse to determine when an artificial agent should take control of a given interaction and when that agent should release control to another participant to simulate the turn-taking that takes place in task-oriented interactions. In addition to reasoning about when to take control, an artificial agent (a computational system) must know how to signal what it is doing through its choice of content and linguistic cues. The system must also be able to recognize when the human participant is taking control and reason about whether or not to allow it.

Processing interactive discourse has been considered from two different research perspectives. From the perspective of interactive generation, the system's role is active and the human user's role is passive. The system is a text planner that plans text to achieve an active discourse goal. This type of discourse processing is appropriate for systems that provide expert advice, explanations, and tutoring. In contrast, from the perspective of plan recognition, interactive discourse is a process in which the system has a passive role and the human user's role is active. The system is an inferencing system that tries to recognize a human user's unstated plan from her questions to provide the most helpful answer. This type of discourse processing is intended for question-answering systems that might be used as an interface to an information system.

An open question is what computational model is needed for a system to plan text (discourse), and also recognize the discourse plans that another participant uses. That is, what are the control strategies and discourse representations that allow a system to determine its conversational role, exchange its conversational role with a user, and interpret the user's feedback and generate appropriate responses from either role. This type of behavior is called mixed initiative interaction. Mixed initiative interactions occur in problem-solving situations where each participant needs to take control of an interaction to contribute information to a problem solution.

Three problems will be addressed in this investigation: reasoning about control, representing discourse to accommodate conversational role change, and producing/interpreting linguistic cues that indicate who has control. First, during a mixed initiative interaction, a computational system must be able to reason about control. On each conversational turn, the computational system must be able to determine whether to keep conversational control or to relinquish it. If the system is not in control, it must reason about whether or not to try to take it. Secondly, when the system does change roles, there is an associated change in how the system processes discourse that must be reflected in the discourse representation that the system uses. A discourse model must be developed that accommodates role change, and that makes it possible for the system to interpret and produce utterances from either role. Thirdly, once a decision about control has been made, several aspects of dialogue are used to signal what that decision is. Researchers have noted that cue words and ellipsis are used by speakers to indicate that conversational control is being taken or retained. Similarly, linguistic devices are used to signal that control is being offered or refused.

The proposed planning period will support activities to write a research proposal to develop and implement a computational model for processing mixed initiative natural language interactions. As part of the proposed planning period, a prototype model will be developed, implemented, and evaluated using two different application domains: an interactive spreadsheet and a medical tutoring system.


Haller, S. M. and Shapiro, S. C. IDP -- an interactive discourse planner. In M. Zock and G. Adorni editors, Trends in Natural Language Generation Springer-Verlag Publishers, 1995.

Haller, S. M. A model for cooperative interactive plan explanations. In Proceedings of the Tenth IEEE Conference on Artificial Intelligence for Applications San Antonio, Texas, March 1994.


Traditional plan recognition research deals with bottom-up plan discussions. Bottom-up plan discussions are question-answering dialogues in which the user knows, or thinks she knows most of her domain plan. The system's task is to recognize the user's domain plan from her queries and thereby provide a helpful response. In contrast, interactive text planning is concerned with top-down plan discussions. In top-down plan discussions, the user communicates her intended domain plan to the system because she knows little about how to pursue it. In this context, the system plans discourse (text) to describe and/or recommend a domain plan.

While computational models for mixed initiative interactions have been developed at the semantic and pragmatic level there is little work on integrating models at these levels into systems that interpret and generate actual natural language. Researchers have noted that linguistic devices signal an offer of control to the other participant, or retain control for the current speaker. Researchers have also investigated the role of syntax, intonation, and conversational pragmatics in the construction of turns. Other work has shown how dialogue initiative relates to discourse segmentation boundaries. The model that I propose to develop will incorporate a theory of mixed initiative interaction into a system that accepts and produces natural language text that negotiates turn-taking.


S. Carberry, Plan Recognition in Natural Language Dialogue. MIT Press, 1990.

J. Moore and W. Swartout. A reactive approach to explanation: Taking the user's feedback into account. In C. Paris, W. Swartout, and W. Mann, editors, Natural Language Generation in Artificial Intelligence and Computational Linguistics. Kluwer Academic Publishers, 1991.

R.W. Smith and D.R. Hipp. Spoken Natural Language Dialog Systems: A Practical Approach. Oxford University Press, 1994.


Intelligent Interactive Systems for Persons with Disabilities
Other Communication Modalities