Multimodal Learning for Interaction in Deeply Sensed Environments

Abstract

Instrumented environments with embedded sensing generate fine- grained spatiotemporal data that can inform computational models for interaction. This talk focuses on our work applying multimodal deep learning for context-aware interaction within such spaces. We leverage multimodal sensor fusion to generate robust contextual embeddings for optimizing timing in vehicular and human-robot interaction systems.

We also introduce GeXSe (Generative Explanatory Sensor System), a framework for multimodal representation learning that addresses cross- modal inference challenges via parallel multi-branch MLP and generative modeling. GeXSe enables extracting visual explanations from sensor data, offering intuitive insights into model decisions. Our neural architecture outperforms baselines in classifying activities from multisensor data. We demonstrate its effectiveness in producing conditioned video features aligned with sensor inputs. Finally, we discuss emerging applications in densely sensed urban settings, where we focus on how to integrate hyperlocal information and multimodal inference to enable a new class of interactive applications, spotlighting our recently funded NSF Engineering Research Center: The Center for Smart Streetscapes (CS3).

Bio:

Jorge Ortiz is an assistant professor of Electrical and Computer Engineering at Rutgers University, specializing in multimodal learning in deeply sensed computational spaces for inference and interaction, as well as in applications for multimodal human-robot interaction. Dr. Ortiz is a Principal Investigator on a recently awarded $26 million NSF Engineering Research Center, the Center for Smart Streetscapes. Before his academic appointment at Rutgers, he served as a research staff member at IBM Research, where he secured over a dozen patents. He earned his M.S. and Ph.D. in Computer Science from the University of California, Berkeley, in 2013. He received his B.S. in Computer Science and Engineering from the Massachusetts Institute of Technology in 2003.