Schedule

Jump to: [Unit 1: Discrete] - [Unit 2: Regression] - [Unit 3: MCMC] - [Unit 4: Mixtures]

Deadlines

unit HW due CP due in-class quiz
0 Thu Jan 25 n/a Thu Feb 01
1 Thu Feb 08 Thu Feb 15 Tue Feb 20
2 Thu Feb 29 Thu Mar 07 Thu Mar 14
3 Thu Mar 28 Thu Apr 04 Thu Apr 11
4 Thu Apr 18 Thu Apr 25 Tue May 07 (exam day)


Unit 1: Foundations for Discrete Data

Key ideas: Probability fundamentals, Point estimation strategies (ML and MAP), Posterior estimation, conjugacy

Models: Beta-Bernoulli models for binary data, Dirichlet-multinomial models for count data

Prerequisite Practice: HW0, which covers joint, conditional, and marginal distributions; Bayes rule; independence

Math Homework: HW1

Coding Practical: CP1

Date Assigned Do Before Class Class Content Optional
Thu 01/18 day00
out:
- HW0
Notes:
Readings:
- Course syllabus

Course Overview

Slides:
- Course Overview
Math/Concept Exercises:
- day00 Exercises
- day00 Solutions
- Motivation for Probabilistic ML:
- Ghahramani Nature 2015
- Bishop 'Model-Based ML' 2013
Tue 01/23 day01  
Notes:
- day01.pdf
Videos:
- day01-A part1: Random Vars and Probability
- day01-A part2: Joint, Conditional, Marginal
- day01-A part3: Sum Rule, Product Rule, Bayes Rule
- day01-B: Continuous r.v., PDF, CDF
Readings:
- Bishop PRML Ch. 1 Sec. 1.1, 1.2
--- Focus on 1.2.1, 1.2.2, and 1.2.3

Probability Refresher

Slides:
- day01 Lecture
Math/Concept Exercises:
- Exercises 01A: disc. r.v.
- Exercises 01B: cont. r.v.
- MathForML Ch. 6 Sec. 6.1 and 6.2
Thu 01/25 day02
due:
- HW0
Notes:
- day02.pdf
Videos:
- day02 part1: Bernoulli Distribution
- day02 part2: Models for Coin Flips
Readings:
- Bishop PRML Ch. 1 Sec. 1.2.3
--- Focus on maximum likelihood vs Bayesian approach
- Bishop PRML Ch. 2 Sec. 2.1
--- Focus on ML estimator Eq. 2.5-2.8

Maximum Likelihood for Binary Data

Slides:
- day02 Lecture
Math/Concept Exercises:
- Exercises 02
Lab Notebook:
- day02: Bernoulli Lab [colab]
  
Tue 01/30 day03
out:
- HW1
Notes:
- day03.pdf
- day03B.pdf: Lagrange multipliers
Videos:
- day03 part1: Gamma functions
- day03 part2: Beta Distributions
- day03 part3: Beta-Bernoulli
- day03B: Lagrange multipliers 1/2
- day03B: Lagrange multipliers 2/2
Readings:
- Bishop PRML Ch. 2 Sec. 2.1
--- Focus on Beta and Bernoulli distrib.
- Bishop PRML Appendix E Lagrange Multipliers

Beta-Bernoulli models for Binary Data

Slides:
- day03 Lecture
Lab Notebook:
- day03: Beta-Bernoulli Lab [colab]
- day03: Gamma Function Lab [colab]
  
Thu 02/01 day04 Quiz0
Notes:
- day04.pdf
Videos:
- day04 part1: Categorical distributions
- day04 part2: Categorical ML Estim.
- day04 part3: Dirichlet distributions
Readings:
- Bishop PRML Ch. 2 Sec. 2.2
--- Focus on Dirichlet distribution

Dirichlet-Categorical models for Count Data

Slides:
- day04 Lecture
Lab Notebook:
- day04: Dirichlet-Categorical Lab [colab]
- For more on Dirichlet, see Frigyik, Kapila, and Gupta 2010


Unit 2: Multivariate Gaussians and Regression

Key ideas: multivariate Gaussian distributions, model selection, Laplace approximation

Models: Bayesian linear regression, Bayesian logistic regression, generalized linear models

Algorithms: gradient descent

Math Homework: HW2

Coding Practical: CP2

Date Assigned Do Before Class Class Content Optional
Tue 02/06 day05
out:
- CP1
Notes:
- day05.pdf
Videos:
- day05 part1: Gaussian distribution
- day05 part2: ML Estim. for Gaussians
- day05 part3: Biased vs Unbiased
Readings:
- Bishop PRML Ch. 1 Sec. 1.2.4
--- Focus Univariate Gaussians
--- ML estimators of mean and variance

Univariate Gaussians

Slides:
- day05 Lecture
Lab Notebook:
- day05: Univar. Gaussian Lab [colab]
  
Thu 02/08 day06
due:
- HW1
Notes:
- day06.pdf
Videos:
- day06 part1: Multivariate Gaussian
- day06 part2: Visualizing Covariances
- day06 part3: Marginals of Gaussians are Gaussian
Readings:
- Bishop PRML Ch. 2 Sec. 2.3.1-2.3.5
--- Focus on multivariate Gaussian properties
- Bishop PRML Ch. 2 Sec. 2.3.5-2.3.6
--- Skim for intuition

Multivariate Gaussians

Slides:
- day06 Lecture
Lab Notebook:
- day06: Multivar. Gaussian Lab [colab]
- Wolfram: Determinants Seen Geometrically
- Immersive Linear Algebra: Determinants
- Immersive Linear Algebra: Eigenvalues and eigenvectors
Tue 02/13 day07  
Notes:
- day07.pdf
Videos:
- day07 part1: Probabilistic Linear Regression
- day07 part2: ML Estim.
- day07 part3: Posterior Estim.
Readings:
- Bishop PRML Ch. 3 Sec. 3.1
--- Focus on ML estimator for linear regression
- Bishop PRML Ch. 3 Sec. 3.3
--- Focus on posterior distribution

Linear Regression: ML + Bayesian Estimation

Slides:
- day07 Lecture
Lab Notebook:
- day07: Linear Regr. Lab [colab]
- Bishop PRML Ch. 3 Sec. 3.2
--- Bias/Variance tradeoff
Thu 02/15 day08
due:
- CP1
  

Unit 1 Recap

Slides:
- Unit 1 Recap
Math/Concept Exercises:
- Practice quiz
  
Tue 02/20 day09
Quiz 1
out:
- HW2
Notes:
- day09.pdf
Videos:
- day09 part1: Posterior predictive
- day09 part2: Marginal likelihood (Evidence)
- day09 part3: Model selection
Readings:
- Bishop PRML Ch. 3 Sec. 3.3
--- Focus on predictive distribution
- Bishop PRML Ch. 3 Sec. 3.4 and 3.5
--- Focus on model selection and hyperparameter estimation

Linear Regression: Bayesian Model Selection

Slides:
- day09 Lecture
Math/Concept Exercises:
- day09 Exercises
  
Thu 02/22     ----- NO CLASS ----- (Monday schedule at Tufts)  
Tue 02/27 day10  
Notes:
- day10.pdf
Videos:
- day10 part1: Discriminative vs Generative models
- day10 part2: Generalized linear models
- day10 part3: Sigmoid function
- day10 part4: Bayes Logistic Regression
Readings:
- Bishop PRML Ch. 4 Sec. 4.3
--- Focus on linear models for binary and multi-class classification
- Bishop PRML Ch. 4 Sec. 4.5
--- Bayesian Logistic Regression

Generalized Linear Models

Slides:
- day10 Lecture
- Bishop PRML Ch. 4 Sec. 4.2
--- Skim for Understanding generative classification


Unit 3: Sampling and Markov Chain Monte Carlo

Key ideas: Markov chains, revisibility, ergodicity, detailed balance, probabilistic programming

Models: Bayesian logistic regression, general directed graphical models

Algorithms: Metropolis-Hastings algorithm

Math Homework: HW3

Coding Practical: CP3

Date Assigned Do Before Class Class Content Optional
Thu 02/29 day11
due:
- HW2
Notes:
- day11.pdf
Videos:
- day11 part1: Monte Carlo estimates of expectations
- day11 part2: MCMC Intro, Stationary Distributions, Ergodicity
- day11 part3: Sampling via Inverting the CDF
- day11 part4: Transformations of Sampled Variables
Readings:
- Bishop PRML Ch. 11 Sec. 11.1
--- Focus on basic methods using transformed uniform r.v. in 11.1.1
--- Skim rejection sampling in 11.1.2
--- Skim importance sampling in 11.1.4
- Bishop PRML Ch. 11 Sec. 11.2
--- Focus on the overview section and 11.2.1 Markov chains

Sampling Methods and Markov Chain Monte Carlo

Slides:
- day11 Lecture
Math/Concept Exercises:
- day11 exercises
- Sec 11.1 - 11.3 of Art Owen's MCMC Chapter
--- More technical treatment of stationary distributions
Tue 03/05 day12  
Notes:
- day12.pdf
Videos:
- day12 part1: Markov Transitions via Accept/Reject
- day12 part2: Random Walk Proposals
- day12 part3: Detailed Balance
- day12 part4: Metropolis and Metropolis-Hastings
Readings:
- Bishop PRML Ch. 11 Sec. 11.2.2
--- Focus on Metropolis-Hastings

Random Walks and Metropolis-Hastings Algorithm

Slides:
- day12 Lecture
Lab Notebook:
- day12: MCMC [colab]
  
Thu 03/07 day13
due:
- CP2
Videos:
- Prob. Prog. with PyMC3 by Dr. C. Fonnesbeck
Readings:
- Salvatier et al. PeerJ CS 2016
--- Read thru Case Study 1

Probabilistic Programming

Slides:
- day13 Lecture
Lab Notebook:
- day13: Demo of PyMC [colab]
- Carpenter et al. 2015
--- Skim, focus on examples
Tue 03/12
out:
- HW3
- Project
  

Recap Unit 2 & Project Launch

Slides:
- Unit 2 Recap
- Project Overview
Math/Concept Exercises:
- PracticeQuiz2
  
Thu 03/14 Quiz2   Quiz 2 & Project Workday  
Tue 03/19     ----- NO CLASS -----  
Thu 03/21     ----- NO CLASS -----  


Unit 4: Clustering, Mixture Models, and E-M

Key ideas: coordinate ascent optimization, expectation-maximization (E-M) methods, local optima, entropy, KL divergence

Models: Mixture models with Gaussian emissions

Algorithms: k-means, EM for GMMs, gradient descent for GMMs

Math Practice: HW4

Coding Practice: CP4

Date Assigned Do Before Class Class Content Optional
Tue 03/26
out:
- CP3
Notes:
- day16.pdf
Videos:
- kmeans part1: Clustering overview
- kmeans part2: K-means problem & cost function
- kmeans part3: K-means algorithm
Readings:
- Bishop PRML Ch. 9 Sec. 9.1

K-Means clustering

Slides:
- K-Means Lecture
Math/Concept Exercises:
- K-means Exercises
  
Thu 03/28
due:
- HW3
Notes:
- day17.pdf
Videos:
- gmm part1: Why mixtures?
- gmm part2: GMMs with and without assignments
- gmm part3: Posterior over assignments
- gmm part4: ML for GMMs (plus log-sum-exp trick)
Readings:
- Bishop PRML Ch. 9 Sec. 9.2 - 9.2.1
--- GMMs in depth
- Bishop PRML Ch. 2 Sec. 2.3.9
--- Motivation for Gaussian mixtures

Gaussian mixture models

Slides:
- GMM Highlights
Lab Notebook:
- day17: Computing GMM PDF [colab]
  
Tue 04/02  
Notes:
- day18.pdf
Videos:
- day18 part1: Penalized ML for GMM
- day18 part2: Gradient descent for GMM
- day18 part3: Coord. descent for GMMs
- day18 part4: EM for GMM
Readings:
- Bishop PRML Ch. 9 Sec. 9.2.2, 9.3.1, and 9.3.2

How to train GMM

Slides:
- day18 Lecture
  
Thu 04/04
due:
- CP3
Notes:
- day19.pdf
Videos:
- day19 part1: GMM recap
- day19 part2: GMMs with assignments
- day19 part3: Expected complete likelihood
- day19 part4: Lower bound of incomplete likelihood
- day19 part5: EM as coord. ascent
Readings:
- Bishop PRML Ch. 9 Sec. 9.3 and 9.4

EM for GMM

Slides:
- EM for GMMs
- Background on entropy: Sec. 1.6 of Bishop PRML Ch. 1
- Background on KL divergence: Sec. 1.6.1 of Bishop PRML Ch. 1
Tue 04/09 day20
out:
- HW4
  

Unit 3 Recap

Slides:
- Unit 3 Recap
Math/Concept Exercises:
- Practice quiz 3
  
Thu 04/11 Quiz3   Quiz 3 & Project Workday  
Tue 04/16 day22
out:
- CP4
   Project Workday  
Thu 04/18 day24
due:
- Init. Report
- HW4
   Project Workday  
Tue 04/23 day24     Project Workday  
Thu 04/25 day24
due:
- CP4
  

Unit 4 review day + Course Recap

Slides:
- Unit 4 and Course Recap
Math/Concept Exercises:
- Practice quiz 4