Doctoral Thesis Defense: Algorithms and Theory for Variational Inference in Two-level Non-conjugate Models

Abstract

Work over the last two decades has developed powerful models in probabilistic machine leaning with a variety of applications: recommender systems, statistical analyses of text corpora, automated astronomy, and more. Exact inference is rarely possible in complex models and much research has been devoted to developing successful algorithms for approximate inference. Variational approximations are one of the main paradigms for approximate inference but optimization of the variational approximation is challenging, and the theoretical properties of such approximations are not fully understood.

In this talk, we focus on variational inference in a rich, two-level latent variable family that includes as members sparse Gaussian processes, correlated topic models, probabilistic matrix factorization, and mixed effects models. The first part of the talk describes recently developed algorithmic tools for approximate inference in the presence of non-conjugacy within these types of models. The ideas here extend the popular stochastic variational inference paradigm to a far larger class of models. The second part of the talk focuses on the theoretical properties of variational inference. Specifically, we discuss several variants of the variational approximation and show that some of these have strong guarantees against the best non-Bayesian procedures.