Comp 150FML: Foundations of Machine Learning
Department of Computer Science
Tufts University
Fall 2009

Course Web Page (this page): http://www.cs.tufts.edu/comp/150FML/

Announcement(s):

(10/18) Note for programming portion in assignment 2. The mean squared error of the true (hidden) vector is 3.78, 3.78, 4.015 for datasets 1,2,3 respectively. (The numbers quoted in the assignments are after taking square root of the error.)
Class is canceled for Thursday 10/15. A make up class is held on Wednesday 10/14 12:00-1:15 (open block) in H-106.
Please check table below for updated list of reading, handouts and assignments.

Syllabus:

An advanced course focusing on statistical foundations and algorithms for machine learning. Topics include: classification and regression problems, regularization, kernel methods, model selection, boosting, support vector machines, and unsupervised learning.

Prerequisites:

Prior exposure to machine learning (as in comp150ML or in comp135 or perhaps an AI course), some calculus (as in math13), algebra (as in math46), and probability (as in math 161/2). Or permission of instructor.

Class Times:

Tuesday, Thursday 10:30-11:45 Halligan 106

Instructor:

Roni Khardon
Office hours: Tuesday 12:00-13:00 or by appointment.

Textbooks

We will be mainly following [B]. The other texts can provide a different perspective on (or exposition of) similar material and can be helpful.

[B] Pattern Recognition and Machine Learning, by C. M. Bishop, Springer, 2006.
[DHS] Pattern Classification (2nd edition), by R. Duda, P. Hart, and D. Stork, John Wiley & Sons, 2001.
[HTF] The elements of Statistical Learning, T. Hastie, R. Tibshirani, J. Friedman, Springer, 2001.
[CST]: An introduction to support vector machines : and other kernel-based learning methods. N. Cristianini and J. Shawe-Taylor. Cambridge University Press, 2000.

Course Work and Marking

Work includes regular exercises to reinforce and further develop course material (accounting for 50% of the final grade) and an empirical project applying ideas learned to an application of student's choice (50% as well).

Tentative List of Topics and Reading Assignments

Review of topics from probability theory and algebra. Linear regression and classification. Kernel methods. Gaussian processes. Graphical models, and learning and inference algorithms for them.
Lecture schedule from 2007 offering of the course is here ; we will follow a similar schedule.

Unit Topics/Comments Reading/ Dates

Background Reading This is more of an overview than an introduction; skim through w/o expecting to get all the details. Chapter 1

Lectures 1-2 Maximum likelihood and Bayesian estimates for beta, Bernoulli and univariate Gaussians Sections 2.1, 2.2, 1.2.4

Lecture 3 Linear Regression Section 3.1

Assignment 1 HW1.txt 9/24

Lecture 4 Linear Algebra Review Any introductory linear algebra text; appendix C

Lectures 5,6,7 Multivariate Normal Distributions Section 2.3
Some handy formulas

Lecture 8,9 Bayesian Linear Regression Section 3.3
Review Slides

Assignment 2 hw2.pdf
Data for assignment 2 is in this directory 10/14 and 10/20

Lecture 10 Model Selection Section 3.4-5

Lecture 11 Exponential Family Distributions Section 2.4

Assignment 3 hw3.pdf 11/3

Lecture 12 Generative Models for Classification Section 4.2

Lecture 13 Fisher's Linear discriminant Section 4.1
FLD Equations Slide

Lecture 14 Logistic Regression Section 4.3

Lecture 15 Bayesian Logistic Regression Section 4.4-5

Lecture 16-17 Introduction to Kernel Methods: perceptron, nearest neighbors, least squares, and Gaussian processes Chapter 6
Chapters 2,3 of [CST]

Assignment 4 hw4.pdf
Data for assignment 4 is in this directory 11/12 and 11/17

Lecture 18-19 Quadratic Optimization and SVM Section 7.1
Chapter 5 of [CST]

Lecture 20 Automatic Relevance Determination and Relevance Vector Machines Section 7.2

Lecture 21 PCA and Kernel PCA Sections 12.1, 12.3

Assignment 5 hw5.pdf 12/3

Upcoming topics Graphical Models, Sampling Methods, EM Chapters 8,11,9

Unit	Topics/Comments	Reading/ Dates
Background Reading	This is more of an overview than an introduction; skim through w/o expecting to get all the details.	Chapter 1
Lectures 1-2	Maximum likelihood and Bayesian estimates for beta, Bernoulli and univariate Gaussians	Sections 2.1, 2.2, 1.2.4
Lecture 3	Linear Regression	Section 3.1
Assignment 1	HW1.txt	9/24
Lecture 4	Linear Algebra Review	Any introductory linear algebra text; appendix C
Lectures 5,6,7	Multivariate Normal Distributions	Section 2.3 Some handy formulas
Lecture 8,9	Bayesian Linear Regression	Section 3.3 Review Slides
Assignment 2	hw2.pdf Data for assignment 2 is in this directory	10/14 and 10/20
Lecture 10	Model Selection	Section 3.4-5
Lecture 11	Exponential Family Distributions	Section 2.4
Assignment 3	hw3.pdf	11/3
Lecture 12	Generative Models for Classification	Section 4.2
Lecture 13	Fisher's Linear discriminant	Section 4.1 FLD Equations Slide
Lecture 14	Logistic Regression	Section 4.3
Lecture 15	Bayesian Logistic Regression	Section 4.4-5
Lecture 16-17	Introduction to Kernel Methods: perceptron, nearest neighbors, least squares, and Gaussian processes	Chapter 6 Chapters 2,3 of [CST]
Assignment 4	hw4.pdf Data for assignment 4 is in this directory	11/12 and 11/17
Lecture 18-19	Quadratic Optimization and SVM	Section 7.1 Chapter 5 of [CST]
Lecture 20	Automatic Relevance Determination and Relevance Vector Machines	Section 7.2
Lecture 21	PCA and Kernel PCA	Sections 12.1, 12.3
Assignment 5	hw5.pdf	12/3
Upcoming topics	Graphical Models, Sampling Methods, EM	Chapters 8,11,9

Comp 150FML: Foundations of Machine Learning Department of Computer Science Tufts University Fall 2009