Schedule

out:

- Welcome Survey

Readings:

- CS 152 Course Syllabus

Course Overview

Slides:

- Course Overview

Notes:

- day01-A: Bayesian Linear Regression

- day01-B: From Features to Kernels

out:

- HW1

Readings:

- Ch. 2 of Rasmussen & Williams GP Textbook

--- Focus esp. on 2.1-2.5.

Gaussian Processes

Notes:

- day02: Gaussian Processes

- GP Review Material from Stanford CS229;

- David Mackay's 'Humble Gaussian'

Readings:

- Harrison's Brief Intro to Automatic Differentiation, 2021

- JAX as Accelerated NumPy

- Skim Common Gotchas in JAX

Coding Workshop

Demo Notebook:

- day03-A : Auto Differentiation with JAX [colab]

- day03-B : Gradient Descent with JAX [colab]

- day03-C : Neural Nets with JAX [colab]

- Paper on JAX: Frostig et al MLSys '18

- How to Think in JAX

due on Tue:

- HW1

Readings:

- Sec. 1.1 & 1.2 from Neal PhD Thesis

--- Focus esp. on Fig. 1.2 and its methods

Bayesian Neural Networks

Demo Notebook:

- day04 : Bayesian Neural Nets [colab]

- Mackay 1995

--- Sections 1, 2, 3, & 5

--- Focus esp. on Sec. 2

- Buntine & Weigend 1991

Readings:

- Sec. 1 - 4.2 of Betancourt 2017

--- Focus esp. on intuition behind Fig. 8 and 9

MCMC for BNNs: Part 1

Demo Notebook:

- day05 : MCMC for Neural Nets [colab]

- For another primer on MCMC, see Sec. 1.3 of Neal PhD Thesis

out:

- HW2

Readings:

- Sec. 1 - 4 of Neal Handbook of HMC

--- Focus on the algorithm on page 14

--- Can skip 4.1 altogether

MCMC for BNNs: Part 2

Slides:

- day06: HMC for BNNs

Demo Website: Interactive Viz of RW vs HMC

Readings:

- Sec. 1-2 and Sec. 5 of Blei et al. JASA '17

- --- Focus on Sec. 2.1 - 2.2, esp. the ELBO function

- --- Focus on Sec. 2.5 on practical implementation

- --- Focus on Sec. 5.3, on what to do if model is not conjugate

- --- Skim Sec. 3 if you have extra time

- --- Can skip Sec. 4 entirely

Variational Inference for BNNs

Slides:

- day07 : Intro to VI

Notes:

- day07-A: Variational Methods for Posterior Estimation

Readings:

- BBVI paper: Ranganath et al. AISTATS 2014

- --- Focus on Sec. 1 and 2 and esp. Alg. 1: could you do this for BNNs? Skim the cool experiments in Sec. 5. Skip Sec. 3&4.

- "Log Derivative Trick" Blog Post by S. Mohamed 2015

Score function trick and BBVI

Notes:

- day08: BBVI and Score Function Trick

Demo Notebook:

- day08 : ELBO and Gradient Demo [colab]

due:

- HW2

Readings:

- Bayes by Backprop: Blundell et al. 2015

- --- Focus on Sec. 1-3 and esp. the Alg. in Sec. 3.2.

- --- Focus on experiments in Sec. 5, esp. Fig. 5

- --- Skim/skip Sec. 4 (on contextual bandits)

- "Reparameterisation Tricks" Blog Post by S. Mohamed 2015

Reparam. trick and ADVI

Notes:

- day09: Bayes by Backprop and Reparameterization Trick

- ADVI paper: Kucukelbir et al JMLR '17

- An older study of VI methods for BNNs: Graves NIPS 2011

out:

- HW3

Readings:

- Stacked Denoising AEs: Vincent et al. 2010

- --- Focus esp. on Sec. 2.2 and experiments

Autoencoders (AEs)

Notes:

- day11: Autoencoders and Deep Generative Models

Demo Website: A. Karpathy's MNIST Autoencoder

- Overview chapter on AEs: Ch 14 of Goodfellow et al.'s DL Textbook 2016

- Application in medicine: Deep Patient by Miotto et al. Sci. Reports 2016

due Wed

- HW3

- Team Signup

Readings:

- Kingma & Welling ICLR 2014

Variational Autoencoders (VAEs)

Notes:

- day12: Variational Autoencoders

- Tutorial on VAE: Doersch arXiv 2016

- Concurrent paper with similar ideas: Rezende et al. ICML 2014

out:

- HW4

Readings:

- Sec. 3 of Kingma & Welling 2019