PhD Defense: A Visual Language for Specifying and Programming Tangible User Interfaces

Abstract

Augmenting the physical world by integrating digital information with everyday physical objects, tangible user interfaces offer a more natural, intuitive and accessible form of interaction than traditional user interfaces. They draw upon users’ knowledge and skills of the everyday non-digital world, such as pre-existing spatial, motor and social skills. With a broad range of application- domains, including education, planning and design, problem solving, and entertainment, these interfaces show a potential to enhance the way people interact and leverage digital information.

However, tangible user interfaces are currently considered challenging to design and build. Designers and developers of these interfaces encounter several challenges including the lack of appropriate interaction abstractions, the shortcomings of current user interface software tools to address continuous and parallel interactions, as well as the excessive effort required to integrate various input and output technologies. This dissertation addresses these challenges by proposing a new paradigm for developing tangible user interfaces. Rather than using a specific toolkit, we propose to specify the structure and behavior of a tangible user interface using high-level constructs, which abstract away implementation details. Thus, developers would be able to use a comprehensive set of abstractions to specify, discuss and program tangible interactions. An important benefit of this approach is that these specifications can be automatically or semi automatically converted into different concrete TUI implementations by a Tangible User Interface Management System (TUIMS). In addition, such specifications could serve as a common ground for investigating both design and implementation concerns by TUI developers from different backgrounds.

Thus, the primary contribution of this dissertation is a high-level user interface description language that provides developers, from different disciplinary backgrounds, means for effectively specifying, programming and refining a broad range of tangible user interfaces. This user interface description language (UIDL) was evaluated using three evaluation methods: cognitive dimensions for notations, specification of benchmark user interfaces and application in a classroom environment. In addition we introduced a top-level architecture and a series of proof-of-concept prototypes of a TUIMS that semi-automatically converts TUIML specifications to concrete TUI implementations. Our evaluation demonstrates that the proposed language is capable of describing a broad range of TUIs and can be easily used by developers from a variety of disciplinary backgrounds to specify, refine and discuss tangible interaction.