Modeling Common Ground for Multimodal Communication
Abstract
The demand for more sophisticated human-computer interactions is
rapidly increasing, as users become more accustomed to conversation-
like interactions with their devices. In this talk, we examine this
changing landscape in the context of human-machine interaction in a
shared workspace to achieve a common goal. In our prototype system,
people and avatars cooperate to build blocks world structures through
the interaction of language, gesture, vision, and action. This provides
a platform to study computational issues involved in multimodal
communication. In order to establish elements of the common ground in
discourse between speakers, we have created an embodied 3D simulation,
enabling both the generation and interpretation of multiple modalities,
including: language, gesture, and the visualization of objects moving
and agents acting in their environment. The simulation is built on the
modeling language VoxML, that encodes objects with rich semantic typing
and action affordances, and actions themselves as multimodal programs,
enabling contextually salient inferences and decisions in the
environment. We illustrate this with a walk-through of multimodal
communication in a shared task.
Bio
James Pustejovsky is the TJX Feldberg professor of computer science at Brandeis University in Waltham, Massachusetts, United States. His expertise includes theoretical and computational modeling of language, specifically: Computational linguistics, Lexical semantics, Knowledge representation, temporal and spatial reasoning and Extraction. His main topics of research are Natural language processing generally, and in particular, the computational analysis of linguistic meaning.