Jump to: [Unit 1: Discrete] - [Unit 2: Regression] - [Unit 3: MCMC] - [Unit 4: Mixtures] - [Unit 5: Time Series]
For any class day with assigned readings and lecture videos, you should complete them before the start of class on that date.
Schedule might change slightly as the semester goes on. Please check here regularly and refresh the page. Look for key announcements on Piazza.
Unit 1: Foundations for Discrete Data
Key ideas: Probability fundamentals, Maximum likelihood estimation, MAP estimation, Beta-Bernoulli and Dirichlet-Multinomial distributions, conjugacy
Models: Dirichlet-multinomial models for text
Math Practice: HW1, which covers joint, conditional, and marginal distributions; ML/MAP point estimation; Bayesian posterior estimation; Beta PDF and Gamma functions; conjugacy
Coding Practice: CP1 which covers text modeling with unigram distributions
| Date |
Assigned |
Do Before Class |
Class Content |
Optional |
|---|
| Mon 02/01 day01 |
- out:
-
|
Readings:
--- Focus on 1.2.1, 1.2.2, and 1.2.3
|
- Course Overview & Probability Refresher
- Recap: Probability Fundamentals
|
- Articles motivating the probabilistic ML approach:
|
| Wed 02/03 day02 |
|
Readings:
--- Focus on maximum likelihood vs Bayesian approach
--- Focus on ML estimator Eq. 2.5-2.8
Videos:
- day02 part2 : A Spectrum of Models for Many Coin Flips - day02 part5 : Continuous Random Variables, PDFs and CDFs
|
- Maximum Likelihood for Binary Data
-
|
--- Background on probability theory
|
| Mon 02/08 day03 |
|
Readings:
--- Focus on Beta and Bernoulli distrib.
|
- Beta-Bernoulli models for Binary Data
-
|
|
| Wed 02/10 day04 |
- due:
-
|
Readings:
--- Focus on Dirichlet distribution
Videos:
- day04 part1 : From Binary to Categorical Distributions - day04 part2 : ML Estimation for Categorical Parameters - day04 part4 : Dirichlet-Categorical model and its posterior - day04 part5 : MAP Estimation for Dirichlet-Categorical - day04B (i) : Lagrange multipliers for equality constraints: Recipe and example - day04B (ii) : Lagrange multipliers for equality constraints: Why it works
|
- Dirichlet-Categorical models for Count Data
-
|
|
Unit 2: Multivariate Gaussians and Regression
Key ideas: multivariate Gaussian distributions, model selection, Laplace approximation
Models: Bayesian linear regression, Bayesian logistic regression, generalized linear models
Algorithms: gradient descent, methods for model selection
Math Practice: HW2
Coding Practice: CP2
| Date |
Assigned |
Do Before Class |
Class Content |
Optional |
|---|
| Tue 02/16 (Mon on Tues) day05 |
|
Readings:
--- Focus Univariate Gaussians
--- ML estimators of mean and variance
--- Skim multivariate Gaussian properties
|
- Univariate Gaussians
-
|
|
| Wed 02/17 day06 |
- due:
-
|
Readings:
--- Focus on multivariate Gaussian and its properties
--- Skim for intuition
Videos:
- day06 part2 : Covariance Properties; Why Contours are Elliptical - day06 part5 : Linear-Gaussian models are Joint Gaussian
|
- Multivariate Gaussians
-
|
|
| Mon 02/22 day07 |
- out:
-
|
Readings:
--- Focus on ML estimator for linear regression
--- Focus on posterior distribution
Videos:
- day07 part1 : Probabilistic view of linear regression
|
- Bayesian Linear Regression 1/2
-
|
--- Bias/Variance tradeoff
|
| Wed 02/24 day08 |
- due:
- Quiz1 (out Thu, due Fri)
|
Readings:
--- Focus on posterior predictive distribution
--- Focus on model selection and hyperparameter estimation
|
- Bayesian Linear Regression: Prediction and Model Selection
-
|
|
| Mon 03/01 day09 |
|
Readings:
--- Focus on linear models for binary and multi-class classification
Videos:
- day09 part4 : Probabilistic Logistic Regression: ML and MAP strategies - day09 part5 : 2nd-order gradient methods for Linear + Logistic Regression
|
- Bayesian Generalized Linear Models for Classification and Beyond
-
|
--- Skim for Understanding generative classification
|
| Wed 03/03 day10 |
- due:
-
|
Readings:
--- Try to understand the Laplace approximation
--- Bayesian Logistic Regression
Videos:
- day10 part2 : Laplace approximation in 1-dim and M-dims - day10 part3 : Laplace approx. posterior for Logistic Regression - day10 part4 : Predictive posteriors for Bayesian Logistic Regression
|
- Posterior Estimation and Prediction for Bayesian Generalized Linear Models
-
|
|
Unit 3: Sampling and Markov Chain Monte Carlo
Key ideas: Markov chains, revisibility, ergodicity, detailed balance, probabilistic programming
Models: Bayesian logistic regression, general directed graphical models
Algorithms: Metropolis-Hastings algorithm, Gibbs sampling
Math Practice: HW3
Coding Practice: CP3
| Date |
Assigned |
Do Before Class |
Class Content |
Optional |
|---|
| Mon 03/08 day11 |
|
Readings:
--- Focus on basic methods using transformed uniform r.v. in 11.1.1
--- Skim rejection sampling in 11.1.2
--- Skim importance sampling in 11.1.4
--- Focus on the overview section and 11.2.1 Markov chains
Videos:
- day11 part2 : Directed graphical models & ancestral sampling - day11 part3 : MCMC Intro, Stationary Distributions, Ergodicity
|
- Sampling Methods and Markov Chain Monte Carlo
-
|
|
| Wed 03/10 day12 |
- due:
- Quiz2 (out Thu, due Fri)
|
Readings:
--- Focus on Metropolis-Hastings
Videos:
- day12 part1 : Markov Transitions that Propose then Accept/Reject - day12 part3 : Detailed Balance, Proof of Random Walk's Stationary Distribution - day12 part4 : Metropolis and Metropolis-Hastings algorithms
|
- Random Walk Proposals and Metropolis-Hastings Algorithm
-
|
|
| Mon 03/15 day13 |
out: Midterm (take home) |
|
- Midterm Review
-
|
|
| Wed 03/17 day14 |
|
Videos:
- day14 part1 : Overview of Sampling Vector Random Var. - day14 part2 : Pro/con comparison of Gibbs vs. Random Walk - day14 part4 : Proof sketch of Gibbs Sampling correctness
|
- Gibbs Sampling
-
|
|
| Mon 03/22 day15 |
due: Midterm |
Readings:
--- Get intuition for MCMC from Fig 3 and Fig 7
--- Try to understand why Hamiltonian MCMC explores better (see Fig 11)
Videos:
--- Watch the first 42 min (can ignore Q&A at end)
|
- Hamiltonian Monte Carlo
-
|
|
| Wed 03/24 day16 |
- due:
-
|
Readings:
--- Read thru Case Study 1
Videos:
--- Watch the first ~40 min (can ignore Q&A at end)
|
- Probabilistic Programming
-
|
--- Skim, focus on examples
|
Unit 4: Clustering, Mixture Models, and Expectation-Maximization
Key ideas: coordinate ascent optimization, local optima, expectations
Models: Mixture models
Algorithms: k-means, expectation maximization
Math Practice: HW4
Coding Practice: CP4
| Date |
Assigned |
Do Before Class |
Class Content |
Optional |
|---|
| Mon 03/29 day17 |
|
Videos:
- day17 part4 : Guarantees, convergence, and local optima
|
- K-Means Clustering
-
|
|
| Wed 03/31 day18 |
- due:
- Quiz3 (out Thu, due Fri)
|
Readings:
--- GMMs in depth
--- Motivation for Gaussian mixtures
Videos:
- day18 part2 : Gaussian mixture model (Two views with and without assignment variables) - day18 part3 : Computing the Posterior over Assignments - day18 part4 : Estimating Parameters via Maximum Likelihood (plus logsumexp trick)
|
- Gaussian Mixture Models and ML Estimation
-
|
|
| Mon 04/05 day19 |
|
Videos:
- day19 part1 : Penalized ML Optimization Problem for GMMs - day19 part3 : Derivation of Coordinate Descent for GMMs - day19 part4 : The EM Coordinate Descent Algorithm for GMMs
|
- ML Estimation with GMMs: Expectation Maximization and Gradient Descent
-
|
|
| Wed 04/07 |
- due:
-
|
Videos:
- day20 part5 : EM as Coordinate Ascent on Lower Bound Objective
|
- Expectation Maximization for GMMs
-
|
|
Unit 5: Hidden Markov models for Time-Series
Key ideas: dynamic programming, joint MAP vs. marginal MAP
Models: Hidden markov models
Algorithms: forward-backward algorithm, Viterbi algorithm, variational inference, belief propagation
Math Practice: HW5
Coding Practice: CP5
| Date |
Assigned |
Do Before Class |
Class Content |
Optional |
|---|
| Mon 04/12 day21 |
|
Readings:
--- Only the intro before 13.2.1
Videos:
- day21 part1 : Unit 4 Motivation: Dependencies in Sequential Data
|
- Markov Models and Hidden Markov Models
-
|
|
| Wed 04/14 day22 |
- due:
-
|
Readings:
--- Think about numerical stability of calculations
Videos:
- day22 part5 : E-step for HMMs: forward algorithm and backward algorithm
|
- Expectation-Maximization for HMMs
- Breakout: Get started on HW5
|
|
| Mon 04/19 |
|
|
NO CLASS (Patriot's Day) |
|
| Wed 04/21 day23 |
- due:
-
|
|
- Viterbi for HMMs
- Breakout: Get started on CP5
|
|
| Mon 04/26 day24 |
|
Videos:
- day24 part2 : Possible optimization strategies for probabilistic models: MM, EM, ME, EE - day24 part3 : Case study for GMMs: EM and EE side-by-side - day24 part4 : Choosing the family of approximate posteriors
|
- Variational Methods
-
|
|
| Wed 04/28 day25 |
- due:
-
|
Videos:
--- Watch to gain high-level appreciation of variational methods
|
- Frontiers of probabilistic modeling
-
|
|
| Mon 05/03 day26 |
- due:
- Quiz5 (out Tue, due Wed)
|
|
- Final Exam Review
-
|
|
| FINALS WEEK |
- due:
- Exam due by Thu 5/13 end-of-day
|
|
FINAL EXAM |
|