Bayesian Deep Learning

Tufts CS Special Topics Course | COMP 150 - 03 BDL | Fall 2018

Syllabus


COMP 150 - 03 BDL: Bayesian Deep Learning
Department of Computer Science, Tufts University
Class Meetings for Fall 2018: Tues and Thurs 3:00-4:15pm in Halligan Hall 111A (J block)
Instructor: Mike Hughes, Assistant Professor of Computer Science.
  • Office hours in Halligan 210: Mon 4-5pm and Thurs 4:15-5:30pm
  • Contact: mhughes(AT)cs.tufts.edu for personal issues only (otherwise use Canvas forums)
Grad TA: Xinmeng Li, Ph.D. student, Tufts CS.
  • Office hours in Halligan extension (1st floor): Mon & Thurs 7-9pm
  • Contact: xinmeng.li(AT)tufts.edu for personal issues only (otherwise use Canvas forums)
Canvas Site: https://canvas.tufts.edu/courses/2088 (requires tufts.edu login)
  • Use discussion forums for any question of general interest!
  • Use for submitting reading comment assignments

Course Overview

The emerging research area of Bayesian Deep Learning seeks to combine the benefits of modern deep learning methods (scalable gradient-based training of flexible neural networks for regression and classification) with the benefits of modern Bayesian statistical methods to estimate probabilities and make decisions under uncertainty. The goal of this course is to bring students to the forefront of knowledge in this area through coding exercises, student-led discussion of recent literature, and a long-term research project. Covered topics include key modeling innovations (e.g. models for functions and deep generative models), learning paradigms (e.g. MCMC and variational inference), and probabilistic programming platforms (e.g. Tensorflow, PyTorch, PyMC3). By completing a 2-month self-designed research project, students will gain experience with designing, implementing, and evaluating new contributions in this exciting research space.

Objectives

After completing this course, students will be able to:

  • read a new published paper within the field and identify its contributions, strengths, and limitations
  • implement a presented method in Python and apply it to an appropriate dataset
  • suggest new research ideas and appropriate experiments for evaluation

Prerequisites

This course intends to bring students near the current state-of-the-art. An ambitious final project could represent a viable submission to a workshop at a major machine learning conference such as NIPS or ICML.

To achieve this objective, we expect students to be familiar with:

  • Coding in Python with modern open-source data science libraries, such as:
    • Basic array operations in numpy (computing inner products, inverting matrices, etc.)
    • Making basic plots or grids of plots in matplotlib
    • Training basic classifiers (like LogisticRegression) in scikit-learn
  • Probability theory
    • e.g. you could explain the difference between a probability density function and a cumulative density function
  • First-order gradient-based optimization
    • e.g. you could code up a simple gradient descent procedure in Python to find the minimum of f(x) = x^2
  • Basic supervised machine learning methods
    • e.g. you can describe the difference between linear regression or logistic regression
  • Linear algebra
    • e.g. you could write the closed-form solution of least squares linear regression using basic matrix operations (multiply, inverse)

Practically, at Tufts this means having successfully completed one of:

  • COMP 135 (Introduction to Machine Learning)
  • COMP 136 (Statistical Pattern Recognition)
  • OR permission of the instructor

With instructor permission, diligent students who are lacking in a few of these areas will hopefully be able to catch-up on core concepts via self study and thus still be able to complete the course effectively. Please see the community-sourced Prereq. Catchup Resources Page for a list of potentially useful resources for self-study.

Coursework and Deliverables

There are four primary tasks for students throughout the course:

  • 30% : Complete 4 homework assignments
    • Short PDF writeups will be turned into Gradescope.
    • Please turn in by the posted due date. Each student has up to 2 late days to use for homeworks.
  • 10% : Participate in discussion during class meetings
  • 10% : Lead 1 or 2 in-class discussions
    • Pairs of students will be assigned to each class after 10/04
    • Read the paper well in advance of the assigned date
    • Meet with instructor during office hours beforehand to discuss strategy
  • 50% : Self-designed research project
    • 15% deliver in-class presentation

Throughout, our evaluation will focus on your process. We wish to train you to thinking scientifically about problems, think critically about strengths and limitations of published methods, propose good hypotheses, and confirm or refute theories with well-designed experiments. It doesn't matter too much if your proposed idea works or doesn't work in the end, just that you understand why.

Collaboration Policy

For homeworks: we encourage you to work actively with other students, but you must be an active participant (asking questions, contributing ideas) and you should write your solutions document alone. At the top of your writeup, you must include the names of any people you worked with, and in what way you worked them (discussed ideas, debugged math, team coding). We may occasionally check in with groups to ascertain that everyone in the group was participating in accordance with this policy.

For final projects: we encourage you to work in teams of 2 or 3, but you may work alone if you really wish to. Each team should submit one report at each checkpoint and will give one presentation. Each member of the team is expected to actively participate in every stage of the project (ideation, math, coding, writing, etc.). Please write all names at the top of every report, with brief notes about how work was divided among team members. Larger teams will be expected to produce more interesting content.

Academic Integrity Policy

This course will strictly follow the Academic Integrity Policy of Tufts University. Students are expected to finish course work independently when instructed, and to acknowledge all collaborators appropriately when group work is allowed. Submitted work should truthfully represent the time and effort applied.

Please refer to the Academic Integrity Policy at the following URL: https://students.tufts.edu/student-affairs/student-life-policies/academic-integrity-policy

Accessibility

Tufts and the instructor of COMP 150 strive to create a learning environment that is welcoming students of all backgrounds. Please see the detailed accessibility policy at the following URL: https://students.tufts.edu/student-accessibility-services