Bayesian Deep Learning

Tufts CS Special Topics Course | COMP 150 - 03 BDL | Fall 2018


Schedule might change slightly from that listed here. Please check back regularly.

For any class day with assigned readings, you should submit critical comments

Part 1: Bayesian Neural Networks

How do we model functions?

date assignments topic required optional
Tue 09/04
Course Overview + Gaussian Processes
Thu 09/06  
Gaussian Processes
Ch. 2 of Rasmusen & Williams GP Textbook - Focus esp. on 2.1-2.5.
Tue 09/11
due: HW1-GP
Bayesian Neural Nets
Sec. 1.1 & 1.2 from Neal PhD Thesis - Focus esp. on Fig. 1.2 and its methods
Sec. 1, 2, 3, and 5 from Mackay 1995 - Focus esp. on Sec. 2
Thu 09/13  
MCMC Training for BNNs
• Notes: [MCMC (PDF)]
Sec. 3 & 4 of Neal Handbook of HMC - esp. the algorithm on page 14
Sec. 1-5 of Betancourt 2017 -- esp. intuition in Fig. 22
For a refresh on MCMC, see Sec. 1.3 of Neal PhD Thesis
Tue 09/18
MCMC with HMC (day 2/2)
>>> Focus on Sec. 1 and 2 and esp. Alg. 1: could you do this for BNNs? Skim the cool experiments in Sec. 5. Feel free to skip Sec. 3&4.
To learn more about VI in general: Review by Blei et al. JASA 2017
Thu 09/20
Variational Training 1 for BNNs: Black-box VI
Tue 09/25  
Variational Training 2 for BNNs: Reparameterization Trick
- Bayes by Backprop: Blundell et al. 2015
>>> Focus on Sec. 1-3 and esp. the Alg. in Sec. 3.2. You can skim/skip Sec. 4 (on contextual bandits). Do read about experiments in Sec. 5, esp. Fig. 5
An older study of VI methods for BNNs: Graves NIPS 2011
Thu 09/27
Bayesian Nonparametric Machine Learning for Discrete Random Structures, Invited Talk by Diana Cai. Location: Halligan 102. Tufts CS Rising Star Research Seminar.
>>> Come with 1 good question to ask during the seminar

Part 2: Autoencoders, Deep Generative Models, and VAEs

How do we model structured data?

date assignments topic required optional
Tue 10/02
out: HW4-VAE
Autoencoders (AEs)
• Demo by Karpathy: [AE for MNIST (web)]
- Stacked Denoising AEs: Vincent et al. 2010
>>> Focus esp. on Sec. 2.2 and experiments
Cool application in medicine: 'Deep patient' Miotto et al. Sci. Reports 2016
Thu 10/04  
Variational Autoencoders (VAEs)
• Notes: [VAEs (PDF)]
Tutorial on VAE: Doersch arXiv 2016
Concurrent paper with similar ideas: Rezende et al. ICML 2014
Extension to Graphical Models: Johnson et al. NIPS 2016
Tue 10/09  
NO CLASS (Monday-on-Tuesday schedule per Tufts Academic Calendar )


Part 3: Research Frontiers

date assignments topic required optional
Thu 10/11
Priors for BNN with Special Guest Dr. Soumya Ghosh
Tue 10/16
PROJECT CHECKPOINT 1: In-Class Project Pitches + Feedback

Thu 10/18
VAEs with discrete random variables
-Presenters: Yirong Tang and Edwin Jain
Concurrent paper with similar idea: Jang et al. ICLR 2017
Tue 10/23
Semi-supervised VAEs
-Presenters: Ramtin Hossein and Gyan Tatiya
Thu 10/25  
Disentangled representations and VAEs
-Presenters: Yushi Liu & Linfeng Liu
Disentangled VAE: Siddharth et al. NIPS 2017
Tue 10/30  
Dropout as Bayesian Approximation
-Presenters: Jamie Heller & Minh D. Nguyen
- Dropout as Bayes Approx Paper: Gal & Ghahramani 2016
>>> Focus on understanding Fig. 2 and Fig. 4
>>> Skim and play with interactive demo.
Thu 11/01  
Modeling Different Kinds of Uncertainty
-Presenters: Eric K. Wyss & Victor Oludare
Tue 11/06
Alternatives to BNNs
-Presenters: Jeremy Shih & Duc D. Nguyen
Thu 11/08  
Improved Variational Objectives
-Presenters: Boyang Lyu & Daniel Dinjian
Importance Weighted VAEs: Burda et al. ICLR 2016
Filtering Variational: Maddison et al. NIPS 2017
Are Tighter Bounds Better?: Rainforth BDL 2017
Tue 11/13  
Flexible Approximate Posteriors #1
-Presenters: Jong Seo Yoon & Manh Duc Nguyen
Normalizing Flows: Rezende & Mohamed ICML 2015
Nonparametric Variational Inference: Gershman et al. ICML 2012
Autoregressive Flows: Kingma et al. NIPS 2016
Multiplicative Flows: Louizos & Welling ICML 2017
Thu 11/15  
PROJECT CHECKPOINT 3 in-class work time
Tue 11/20
Flexible Approximate Posteriors #2
-Presenters: Julie Jiang & Phong Hoang
Neural Autoregressive Flows: Huang et al. ICML 2018
Thu 11/22  
Tue 11/27  
Course Feedback (first 15 min.)
Thu 11/29  
CS Dept. 'Rising Stars' Research Seminar: Peter Schulam

Part 4: Final Project Deliverables

date assignments topic required optional
Tue 12/04
Thu 12/06
Tue 12/18