Syllabus

Course Overview

This course provides the theoretical and computational foundations for probabilistic machine learning. The focus is on probabilistic models, which are especially useful for any application where observed data could be noisy, sometimes missing, or not available in large quantities. We emphasize representing uncertainty with formal distributions and trying to average over these distributions when making decisions (as done in the Bayesian approach).

We will study the following probabilistic models:

• models for discrete data (e.g. next word prediction)
• linear models for regression
• generalized linear models for classification
• mixture models
• hidden Markov models for sequential data

We will try to understand all models through a unifying conceptual framework of directed graphical models.

Algorithms studied include:

• sampling via Markov chain Monte Carlo methods
• optimization via gradient descent (first-order and second-order)
• optimization via coordinate descent, as in expectation maximization and variational inference

Objectives

After completing this course, students will be able to:

• Demonstrate formal mathematical understanding of probabilistic models.
• Given an applied data analysis task, select a relevant probabilistic model, fit the model on a relevant dataset using an appropriately chosen approximate inference method, and analyze the results.
• Analyze numerical accuracy and stability of common probabilistic ML algorithms (e.g. avoiding overflow/underflow)
• Analyze scalability considerations of common probabilistic ML methods (including runtime and memory complexity requirements).

Prerequisites

This course intends to provide students a solid foundation in statistical machine learning methods.

To achieve this objective, we expect students to be familiar with the following before taking the course:

• Probability theory
• • e.g. you could explain the difference between a probability density function (PDF) and a cumulative distribution function (CDF)
• Basic linear algebra (comfort with matrix/vector products; have seen inverses & determinants before)
• • e.g. you could write the closed-form solution of least squares linear regression using basic matrix operations (multiply, inverse)
• First-order gradient-based optimization
• • e.g. you could code up a simple gradient descent procedure in Python to find the minimum of functions like f(x) = x^2
• Basic supervised machine learning methods
• • e.g. you can describe the mathematical optimization problem that we solve to train linear regression and logistic regression
• Coding in Python with modern open-source data science libraries
• • Basic array operations in numpy (computing inner products, inverting matrices, etc.)
• • Making basic plots or grids of plots in matplotlib
• • Training basic classifiers (like LogisticRegression) in scikit-learn

Practically, this means having successfully completed at least one and ideally both of these courses (or their equivalent outside of Tufts):

• EE 104 (Probabilistic Systems Analysis)
• CS 135 (Introduction to Machine Learning)

With instructor permission, diligent students who are lacking in a few of these areas of coursework could be able to catch-up on core concepts via self study and thus still be able to complete the course effectively. Please see the community-sourced Resources Page for a list of potentially useful resources for self-study.

Enrolling and Wait Lists

As of the start of semester, we expect to have about 40 students enrolled in the course. We are currently at capacity, but some students may drop the course and leave openings for others (usually we see 5-10 openings in the first week of classes as schedules shift).

Our top priority is to provide each enrolled student with our full support, including the ability to get prompt answers to questions on Piazza and in office hours as well as the ability to get high-quality feedback on submitted homeworks, exams, and projects in a timely manner.

We understand some students are on the wait list (either formally on the wait list on SIS system, or just conceptually would like to be in the course). It is possible that students currently on the wait list may be added, but only if there is adequate staff support.

Prof. Mike Hughes will make the final decision about all wait list candidates by end of day Mon 1/30, before the ADD deadline of 2/1.

To be considered for enrollment, you must do these two things:

• Complete and submit HW0 by end of day Thu 1/26.
• • This action shows you have the necessary skills and would take the course seriously
• Message the instructor by end of day Thu 1/26 via email with subject "CS 136 Wait List Request"
• • Explain your current state within your degree program (e.g. sophomore undergraduate in CS, Ph.D. student in Math)
• • Explain why taking the course this semester would be important to you.

Class Format for Spring 2023

The course is organized into 5 topical units (each 2-3 weeks long), which will govern in-class and out-of-class work.

Synchronous course meetings will be in-person throughout Spring 2023.

As of end of first day of class (Wed 1/30) we will expect students to be signed up on Piazza. This is how we will communicate any changes to the course meeting locations with you. If you have not heard otherwise, expect course meetings and office hours to be in person in their usual locations.

Workload

Each week, you should expect to spend about 8-12 hours on this class.

Here's our recommended break-down of how you'll spend time each week:

• 1.75 hr / wk preparation before Mon class (reading, lecture videos)
• 1.00 hr / wk active participation in Mon class (excludes flex time)
• 1.75 hr / wk preparation before Wed class (reading, lecture videos)
• 1.00 hr / wk active participation in Wed class (excludes flex time)
• 4.00 hr / wk on HW or CP assignment (one assignment due each week, takes ~4 hr total)
• 0.50 hr / wk preparing for and taking quiz (quizzes happen every few weeks, so get 0.5 hr prep plus 0.5 time taking the quiz)

This totals to ~10.00 hr / wk.

Grading

Final grades will be computed based on a numerical score via the following weighted average:

• 25% homeworks (3% HW0, 4.4% each for HW1-HW5)
• 22% coding practicals (4.4% each for CP1-CP5)
• 36% short quizzes (one per unit, averaged after dropping the lowest quiz)
• 15% project (including final report and intermediate checkpoint reports)
• 2% participation in class and on Piazza discussion forum

When assigning grades given a final numerical score (from 0.0 to 1.0), the following scale will be used:

• 0.94-1.00 : A
• 0.90-0.94 : A-
• 0.87-0.90 : B+
• 0.83-0.87 : B
• 0.80-0.83 : B-
• 0.77-0.80 : C+
• 0.73-0.77 : C
• 0.70-0.73 : C-
• 0.67-0.70 : D
• 0.63-0.67 : D
• 0.60-0.63 : D-
• Below 0.60: F

The highest possible grade of "A+" will be awarded at the instructor's discretion.

We do not round up grades. This means you must earn at least an 0.83 (not 0.825 or 0.8295 or 0.8299) to earn a B.

Textbook

As a primary textbook, we will use "Pattern Recognition and Machine Learning" by Christopher M. Bishop.

A free PDF is available online from the author:

<https://www.microsoft.com/en-us/research/uploads/prod/2006/01/Bishop-Pattern-Recognition-and-Machine-Learning-2006.pdf>

Other suggested resources can be found on the Resources Page.

Collaboration Policy

Our ultimate goal is for each student to fully understand the course material. With this goal in mind, we have the following policy:

For quizzes, all work should be done individually, with no collaboration with others whatsoever.

For homeworks and coding practicals (HWs and CPs), we have the following policy for student work:

You must write the text of anything that will be turned in -- all code and all written solutions -- on your own without help from other human or artificial agents. You may not share any code or solutions with others, regardless of if they are enrolled in the class or not.

We do encourage high-level interaction with your classmates. After you have spent at least 10 minutes thinking about the problem on your own, you may verbally discuss assignments with others in the class. You may work out solutions together on whiteboards, laptops, or other media, but you are not allowed to take away any written or electronic information from joint work sessions with others. No notes, no diagrams, and no code. Emails, text messages, and other forms of virtual communication also constitute “notes” and should not be used preparing solutions.

When preparing your solutions, you may always consult textbooks, materials on the course website, or existing content on the web for general background knowledge. However, you cannot ask for answers through any question answering websites such as (but not limited to) Quora, StackOverflow, ChatGPT, etc. If you see any material having the same problem and providing a solution, you cannot check or copy the solution provided. If general-purpose material was helpful to you, please cite it in your solution within your "Collaboration Statement".

Academic Integrity Policy

This course will strictly follow the Academic Integrity Policy of Tufts University. Students are expected to finish course work independently when instructed, and to acknowledge all collaborators appropriately when group work is allowed. Submitted work should truthfully represent the time and effort applied.

Please refer to the Academic Integrity Policy at the following URL: https://students.tufts.edu/student-affairs/student-life-policies/academic-integrity-policy

Accessibility

Tufts and the instruction team of CS 136 strive to create a learning environment that is welcoming students of all backgrounds.

If you feel unwelcome for any reason, please talk to your instructor so we can work to make things better. If you feel uncomfortable talking to members of the teaching staff, consider reaching out to your academic advisor, the department chair, or your dean.

Please see the detailed accessibility policy at the following URL: <https://students.tufts.edu/student-accessibility-services>