Schedule

Jump to: [Unit 1: Discrete] - [Unit 2: Regression] - [Unit 3: Variational Inference] - [Unit 4: Mixtures]

Deadlines

unit PS due CA due in-class exam
0 Mon Sep 17 - -
1 Thu Sep 24 Thu Oct 01 Thu Oct 08
2 Thu Oct 15 Thu Oct 22 Thu Oct 29
3 Thu Nov 05 Thu Nov 12 Thu Nov 19
4 Thu Dec 03 Thu Dec 10 -


The final project report is due Monday, December 14th.

The final exam is optional. During the final exam slot, you can retake 1-2 exams from earlier in the course for a higher score. This will take place during the scheduled final exam slot, at 3:30 on December 17.

Unit 1: Foundations for Discrete Data

Key ideas: Probability fundamentals, Point estimation strategies (ML and MAP), Posterior estimation, conjugacy

Models: Beta-Bernoulli models for binary data, Dirichlet-multinomial models for count data

Prerequisite Practice: PS0, which covers joint, conditional, and marginal distributions; Bayes rule; expectations; independence

Math Homework: PS1

Coding Assignment: CA1

Date Assignments Supplemental Resources (Before Class) Class Content Optional
Tue 09/08 day00
out:
- PS0
Readings:
- Course syllabus

Course Overview

Slides:
- Course Overview
- Annotated Slides
Math/Concept Exercises:
- Exercises 00
- Exercises 00 solutions
- Motivation for Probabilistic ML:
- Ghahramani Nature 2015
- Bishop 'Model-Based ML' 2013
Thu 09/10 day01  
Videos:
- day01-A part1: Random Vars and Probability
- day01-A part2: Joint, Conditional, Marginal
- day01-A part3: Sum Rule, Product Rule, Bayes Rule
- day01-B: Continuous r.v., PDF, CDF
Readings:
- Bishop PRML Ch. 1 Sec. 1.1, 1.2
--- Focus on 1.2.1, 1.2.2, and 1.2.3

Probability Refresher

Slides:
- Independence, Expectations, and Continuous
- Annotated Slides
Math/Concept Exercises:
- Exercises 01A: disc. r.v.
- Exercises 01B: cont. r.v.
- Exercises 01A solutions
- Exercises 01B solutions
- Intro to Probability on MIT Open Courseware
--- Lectures 1 and 2
- MathForML Ch. 6 Sec. 6.1 and 6.2
Tue 09/15 day02  
Videos:
- day02 part1: Bernoulli Distribution
- day02 part2: Models for Coin Flips
Readings:
- Bishop PRML Ch. 1 Sec. 1.2.3
--- Focus on maximum likelihood vs Bayesian approach
- Bishop PRML Ch. 2 Sec. 2.1
--- Focus on ML estimator Eq. 2.5-2.8
 Maximum Likelihood for Binary Data  
Thu 09/17 day03
due:
- PS0
out:
- PS1
Videos:
- day03 part1: Gamma functions
- day03 part2: Beta Distributions
- day03 part3: Beta-Bernoulli
Readings:
- Bishop PRML Ch. 2 Sec. 2.1
--- Focus on Beta and Bernoulli distrib.
 Bayesian Inference for Binary Data (Beta-Bernoulli)  
Tue 09/22 day04  
Videos:
- day04 part1: Categorical distributions
- day04 part2: Categorical ML Estim.
- day04 part3: Dirichlet distributions
- day04 part4: Dirichlet-Categorical model
Readings:
- Bishop PRML Ch. 2 Sec. 2.2
--- Focus on Dirichlet distribution
 Bayesian Inference for Count Data (Dirichlet-Categorical)
- For more on Dirichlet, see Frigyik, Kapila, and Gupta 2010
- For more on Lagrange Multipliers, see Bishop PRML Appendix E Lagrange Multipliers
- Additional Recorded Lectures on Lagrange Multipliers
--- Video 1/2
--- Video 2/2
Thu 09/24 day05
due:
- PS1
out:
- CA1
Readings:
- Bishop PRML Ch. 8 Sec. 8.1.1-8.1.3
--- Focus on Bayesian Networks
 Graphical Models  


Unit 2: Multivariate Gaussians and Regression

Key ideas: multivariate Gaussian distributions, model selection

Models: Bayesian linear regression, Bayesian logistic regression, generalized linear models

Algorithms: gradient descent

Math Homework: PS2

Coding Assignment: CA2

Date Assignments Supplemental Resources (Before Class) Class Content Optional
Tue 09/29 day06  
Videos:
- day06 part1: Gaussian distribution
- day06 part2: ML Estim. for Gaussians
Readings:
- Bishop PRML Ch. 1 Sec. 1.2.4
--- Focus Univariate Gaussians
--- ML estimators of mean and variance
 Univariate Gaussians
- Recorded Lecture on Biased vs Unbiased Estimator
Thu 10/01 day07
due:
- CA1
Videos:
- day07 part1: Multivariate Gaussian
- day07 part2: Visualizing Covariances
- day07 part3: Marginals of Gaussians are Gaussian
Readings:
- Bishop PRML Ch. 2 Sec. 2.3.1-2.3.5
--- Focus on multivariate Gaussian properties
- Bishop PRML Ch. 2 Sec. 2.3.5-2.3.6
--- Skim for intuition
 Multivariate Gaussians
- Wolfram: Determinants Seen Geometrically
- Immersive Linear Algebra: Determinants
- Immersive Linear Algebra: Eigenvalues and eigenvectors
Tue 10/06 day08  
Notes:
- cs135_feature_transform.pdf
- cs135_regularization.pdf
Videos:
- day08 part1: Probabilistic Linear Regression
- day08 part2: ML Estim.
 Linear Regression: ML + Bayesian Estimation
- Bishop PRML Ch. 3 Sec. 3.2
--- Bias/Variance tradeoff
Thu 10/08 day09
out:
- PS2
   Exam 1 (in-class)  
Tue 10/13 day10  
Videos:
- day10 part1: Marginal likelihood (Evidence)
- day10 part2: Model selection
Readings:
- Bishop PRML Ch. 3 Sec. 3.3
--- Focus on predictive distribution
- Bishop PRML Ch. 3 Sec. 3.4 and 3.5
--- Focus on model selection and hyperparameter estimation
 Linear Regression: Bayesian Model Selection  
Thu 10/15 day11
due:
- PS2
out:
- CA2
Videos:
- day11 part1: Generalized linear models
- day11 part2: Sigmoid function
- day11 part3: Bayes Logistic Regression
Readings:
- Bishop PRML Ch. 4 Sec. 4.3
--- Focus on linear models for binary and multi-class classification
- Bishop PRML Ch. 4 Sec. 4.5
--- Bayesian Logistic Regression
 Logistic Regression and GLMs  


Unit 3: Variational Inference

Key ideas: KL divergence, variational inference

Models: Logistic Regression, Linear regression, Bayesian networks, Variational autoencoders

Algorithms: Automatic differentiation variational inference, Coordinate ascent variational inference

Math Homework: PS3

Coding Assignment: CA3

Date Assignments Supplemental Resources (Before Class) Class Content Optional
Tue 10/20 day12  
Readings:
- Bishop PRML Ch. 1 Sec. 1.6 --- Focus on 1.6.1
 KL Divergence and the ELBO  
Thu 10/22 day13
due:
- CA2
Readings:
- Bishop PRML Ch. 10 Sec. 10.1 --- Focus on 10.1.1, Factorized Distributions
 Automatic Differentiation Variational Inference
- Automatic Differentiation Variational Inference
Tue 10/27 day14     Algorithms for Variational Inference  
Thu 10/29 day15
out:
- PS3
   Exam 2 (in-class)  
Tue 11/3 day16     Variational Autoencoders  
Thu 11/5 day17
due:
- PS3
out:
- CA3
- Project
   Hierarchical Models and Diffusion Models  
Tue 11/10     --- no class (Tufts Wednesday Schedule) ---  
Thu 11/12 day18
due:
- CA3
   Project Setup and Inverse Image Problems  


Unit 4: Clustering, Mixture Models, and E-M

Key ideas: coordinate ascent optimization, expectation-maximization (E-M) methods, local optima, entropy

Models: Mixture models with Gaussian emissions

Algorithms: k-means, EM for GMMs, gradient descent for GMMs

Math Homework: PS4

Coding Assignment: CA4

Date Assignments Supplemental Resources (Before Class) Class Content Optional
Tue 11/17 day19
due:
- Project Team Formation
   K-Means clustering  
Thu 11/19 day20     Exam 3  
Tue 11/24 day21
due:
- Project Pitch
out:
- PS4
   Gaussian mixture models  
Thu 11/26     --- no class (Thanksgiving) ---  
Tue 12/01 day22     How to train GMM  
Thu 12/03 day23
due:
- PS4
out:
- CA4
   EM for GMM  
Tue 12/08 day24
due:
- Project Initial Report
   Unit 4 review day + Course Recap  
Thu 12/10 day25
due:
- CA4
   Project Workday  
Mon 12/14
due:
- Project Final Report
      
Thu 12/17     Final Exam (Retry Exams 1-3)