Reasoning-Based Approaches To Situated Reference Resolution

Abstract

Quals talk:

Reference resolution is a critical facet of natural language. While the form of a referring expression serves as a cue to the intended referent, we often rely on contextual knowledge and reasoning as well. Indeed, humans use natural language, vision, and context to map referring expressions (such as pronouns) to objects/concepts in the real world. But this reasoning piece will also be required for embodied agents who will need to routinely perform reference resolution in situated interaction. This talk specifically examines social norms as modulators of reference resolution. First, I present a human subjects experiment that shows how social norms applicable in a given context influence the interpretation of referring expressions. Then, to address the lack of text-trained language models that leverage normative information and the lack of norm inference models, I offer a novel normative-based reasoning approach to reference resolution and provide a proof-of-concept implementation on an integrated system. Additionally, I highlight how current coreference tools and generative models (GPT-3) in natural language processing fail to handle ambiguous references that require normative reasoning.