Graduate Research Talk: Amortized Variational Inference with Graph Convolutional Networks for Gaussian Processes

Abstract

Gaussian Process (GP) Inference on large datasets is computationally expensive, especially when the observation noise is non-Gaussian. Recent variational inference methods use a small number of inducing points to construct the variational distribution. However, these inference methods require strong correlations between normal data points and a small number of inducing points. These correlations may not be true in many applications. Methods in this category also lose the non-parametric flavor of GP, as the flexibility of the variational distribution is constrained by the posterior distribution on inducing points. We propose a non-parametric variational distribution for GP inference. Our method emphasizes the approximation of local correlations in GP posteriors and uses a reusable template to approximate the GP posterior at local levels while maintains a global approximation. First, we construct a variational distribution, such that the GP inference for a data point considers only its neighborhood. With this construction, the variational lower bound is highly decomposible, hence we can run stochastic optimization with small batches. Second, we train Graph Convolutional Networks as a reusable model to identify variational parameters for each data point. This amortization greatly reduces the number of parameters and the number of iterations needed in optimization. Our method retains the non-parametric flavor of GP. In empirical evaluations, the proposed method significantly speeds up the GP inference and often gets more accurate results than previous methods.