Schedule


Jump to:

[Unit 1: Regression] - [Unit 2: Classification] - [Unit 3: Neural Nets]

[Unit 4: Trees and Ensembles] - [Unit 5: Kernels] - [Unit 6: PCA and Rec. Sys.] - [Unit 7: Frontiers]

Please complete assigned readings and videos before the start of class.

Schedule might change slightly as the semester goes on. Please check here regularly and refresh the page. Look for key announcements on Piazza.

Course Introduction

Concepts: supervised learning, unsupervised learning, difference between ML and AI

Date Assignments Do Before Class Class Content Optional Extras
Wed 09/09 day01
- HW0 out

Videos on Canvas:
Course Overview
 


Unit 1: Regression

Concepts: over-fitting, under-fitting, cross-validation

Methods: Linear regression, k-NN regression

Evaluation: mean squared error, mean absolute error

Work: HW1, Quiz1

Date Assignments Do Before Class Class Content Optional Extras
Mon 09/14 day02  
Install your Python environment

- Focus: 'Notation and Simple Matrix Algebra'

Read ISL Textbook Ch. 2 : Sec. 2.1 & Sec. 2.2
- Focus: 'Parametric Methods'
- Focus: 'Assessing Model Accuracy'
- Focus: 'K-Nearest Neighbors'

Regression Basics [slides PDF]
 
Wed 09/16 day03
- HW0 due
- HW1 out
Read ISL Textbook Ch. 3 : Sec. 3.1-3.3
- Focus: '3.1.1 Estimating Coefficients'
- '3.3.2 Extensions beyond Linear'
- '3.3.3 Potential Problems'

- Focus: Coefficient Estimation Derivation in Eq. 5.4 - 5.12

Read math notes:
- Estimating coefficients in 1-dim. and many dim. [PDF]
Linear Regression [slides PDF]
Read MML Textbook Ch. 9 : Sec. 9.1-9.2
- Derivation with probabilistic perspective
Mon 09/21 day04  

Read ISL Textbook Ch. 5 : Sec. 5.1
- Focus: 5.1.1 Validation Set Approach
- Focus: 5.1.3 k-fold Cross-Validation


- Focus: Case for Nested Cross-Validation
Polynomial Features, Hyperparameter Selection, and Cross-Validation [slides PDF]
 
Wed 09/23 day05  
Read ISL Textbook Ch. 6 : Sec. 6.2.1, 6.2.2, and 6.2.3

Video by Prof. A. Ihler (UC-Irvine): Regularization for Linear Regression
Regularization for Linear Regression [slides PDF]
 


Unit 2: Classification

Concepts: feature engineering, hyperparameters, numerical stability, gradient descent

Methods: Logistic regression, k-NN classification

Evaluation: ROC curves, confusion matrices, cross entropy

Work: HW2, Quiz2, ProjectA

Date Assignments Do Before Class Class Content Optional Extras
Mon 09/28 day06
- ProjectA out
Read DL Textbook Sec. 4.3 : Gradient Descent

Gradient Descent [slides PDF]
Some Practical Wisdom for ProjectA: "A Few Useful Things to Know about ML" by Domingos '12
Wed 09/30 day07
- HW1 due
- HW2 out
Read ISL Textbook Ch. 4 : Sec. 4.1 - 4.2

Binary Classification Overview [slides PDF]
- Nearest Neighbor Methods
- Linear Methods
 
Mon 10/05 day08  
Read EMLM Textbook Ch. 2 : Ch. 2 Evaluation Metrics

Evaluation of Binary Classification [slides PDF]
- False Positives vs. False Negatives
- ROC Curves
- PR Curves
- Log loss (cross entropy)
 
Wed 10/07 day09
- Quiz1
Read ISL Textbook Ch. 4 : Sec. 4.3 - 4.3.4

Logistic Regression [slides PDF]
 


Unit 3: Neural Nets

Concepts: backpropagation, stochastic gradient descent, hyperparameter selection

Methods: multi-layer perceptrons for regression and classification

Work: HW3, Quiz3

Date Assignments Do Before Class Class Content Optional Extras
Mon 10/12 day10  
Read: DL Textbook Ch 6 - especially:
- Sec 6.0 (Intro)
- Sec 6.1 (xor)
- Sec 6.2 (Learning)
- Sec. 6.3 Hidden Units

Neural Networks Overview [slides PDF]
- Multi-layer Perceptrons
- Universal Function Approximation [demo]
 
Wed 10/14 day11
- HW2 due

Skim for high-level understanding: DL Textbook Ch 6 :
- Sec. 6.5 Backprop
Training Neural Nets with Backprop [slides PDF]
 
Mon 10/19 day12  
Read DL Textbook Sec. 5.9 : Stochastic Gradient Descent

Read DL Textbook Ch. 8 Sec. 8.1 - 8.3
- Focus on 8.1.2 Surrogates and Early Stopping
- Focus on 8.1.3 Batch and Minibatch algorithms
- Focus on Sec. 8.3 - 8.3.1 Stochastic Gradient Descent

Stochastic Gradient Descent [slides PDF]
 
Wed 10/21 day13
- Sec. 11.4 Hyperparameter Search

Read DL Textbook Ch 6 : Sec. 6.2.2
- Focus on 6.2.2.3 : Softmax Units
Neural Nets In Practice [slides PDF]
- Multi-class Classification
- Early Stopping
- Hyperparameter Search
 


Unit 4: Trees and Ensembles

Concepts: greedy training, bagging, boosting

Methods: decision trees, random forests, XGBoost

Work: HW4, Quiz4, ProjectB

Date Assignments Do Before Class Class Content Optional Extras
Mon 10/26 day14
- ProjectA due
- ProjectB out
Read ISL Textbook Ch. 8 : Sec. 8.1
- Read 8.1 - 8.1.4
- Focus on figures for intuition

Decision Trees for Regression and Classification [slides PDF]
- An algorithmic focus on decision trees for classification
Wed 10/28 day15
- HW3 due
- HW4 out
Read ISL Textbook Ch. 5 : Sec. 5.2
- 5.2 'The Bootstrap'
Read ISL Textbook Ch. 8 : Sec. 8.2
- Read 8.2.1 Bagging
- Read 8.2.2 Random Forests

(No videos on Canvas today)
Random Forests [slides PDF]
- Ensembles of independent predictors
 
Mon 11/02 day16  
Read ISL Textbook Ch. 8 : Sec. 8.2
- Read 8.2.3 Boosting

(No videos on Canvas today)
Boosting [slides PDF]
- Boosting and XGBoost
 
Wed 11/04 day17
Read sklearn docs on Text Feature Extraction

- From 6.2.3 through 6.2.3.7


(No videos on Canvas today)
Project B Material: Text Representations
 


Unit 5: Kernel Methods

Concepts: kernel functions

Methods: kernelized linear regression, support vector machines

Work: HW5, Quiz5

Date Assignments Do Before Class Class Content Optional Extras
Mon 11/09 day18  

- Focus on Sec 9.1 - 9.2
- Skim Sec. 9.4
- Focus on Sec. 9.5 relating SVMs to Logistic Regression
Support Vector Machines [slides PDF]
 
Tue 11/10 (Tufts-Wed-on-Tues) day19  

- Focus on Sec 9.3 (which covers Kernels for SVMs)
Kernel Methods for Regression and Classification [slides PDF]
- Skim Sec. 6.1 Dual Representations
- Focus on Sec. 6.2 Constructing Kernels
Wed 11/11
- HW4 due
  NO CLASS (Veteran's Day Holiday)  
Mon 11/16 day20  
- Read Sec. 9.4 one-vs-one and one-vs-all

- Read Sec. 10.1 Unsupervised Learning

- Read Sec. 10.2.1 What are Principal Components?

(No videos on Canvas today)
Multi-class SVMs / Intro to Dimensionality Reduction [slides]
 
Wed 11/18 day21
- HW5 out
Fairness in Supervised Learning [slides]
- Breakout: Discussion Guide
- No lab materials or notebook today
Read about other high-stakes applications


Unit 6: PCA and Recommender Systems

Concepts: dimensionality reduction, matrix factorization, recommendation systems

Methods: principal components analysis, collaborative filtering models

Work: ProjectC

Date Assignments Do Before Class Class Content Optional Extras
Mon 11/23 day22
- ProjectB due
- ProjectC out
- Read Sec. 10.2.2 Another Interpretation of PCA
- Read Sec. 10.2.3 More on PCA
- Read Rec. 10.2.4 Other Uses of PCA (then go skim 6.3.1)

Principal Components Analysis [slides PDF]
 
Wed 11/25     NO CLASS (Thanksgiving Holiday)  
Mon 11/30 day23  

Matrix Factorization for Recommender Systems [slides PDF]
 
Wed 12/02 day24
- HW5 due

Recommender Systems [slides PDF]