Project

Last modified: 2026-09-07 11:42

Status: NOT YET RELEASED.

Jump to: Deliverables • Task Selection • Grading • Baseline • Upgrade

Goals

Throughout this class, you've learned about the foundational methods for probabilistic modeling. The goal of this project is to give you hands-on experience in applying such models to real data.

You'll complete 3 steps that occur when developing models:

1. Select a specific data analysis task and suitable baseline model

2. Design and implement a promising upgrade to this model

   • Should be hypothesis driven: Why is your upgrade promising?

3. Evaluate your proposed upgrade via experiments on real data

   • Analyze whether your hypothesis was correct, or not.

In the real world, you'll often iterate between steps 2 and 3 multiple times to get better performance.

Deliverables

• (1) by Tue Nov 17: Complete Team Formation Form on Gradescope (Link to be added)

  • Commits you to your team
  • Suggests a chosen task, dataset, and baseline (all these can change down the road except your team)

• (2) by Tue Nov 24: Turn in 1-page Project Pitch

  • Summarize chosen dataset and task
  • Suggest possible baseline
  • Suggest possible upgrade
  • Primarily used for instructor feedback
  • Graded mostly for completion of key elements

• (3) week of Nov 30 - Dec 4: Meet with course staff

  • Discuss your pitch at a whiteboard for 10 min
  • Based on your project pitch, you will be assigned a day/time to meet with course staff.

• (4) by Tue Dec 08: Turn in Initial Report

  • 1-2 page report (you can reuse any text from this in your final report; thinks of it as a partially-written final report)
  • Describe data and task
  • Describe baseline method clearly, with mathematical depth
  • Demonstrate baseline performance on task
  • Provide plan for proposed upgrade and experiments

• (5) by Mon Dec 14: Turn in Final Report

  • 5-6 page report
  • Describe task, baseline, and upgrade clearly
  • At least 1 figure/table evaluating your hypothesis

Teamwork

You should work in teams of 2 or 3. Teams of 4 or more are not allowed.

If you wish to work alone, please discuss with the course instructor for specific approval. We only recommend this if you are an advanced research student and would benefit from solo work to make rapid progress or because data access approvals are difficult.

If you work alone, you will be required to do the same amount of work as a team of 2. If you work in a team of 3, you will be required to do 2 upgrades instead of 1 to appropriately scale the amount of work.

Task and Dataset selection

You should select a dataset that meets these requirements:

• Publicly available
• At least 50 instances
• At least 2 features
• Suitable for your chosen analysis task (regression, classification, clustering, etc.)
• Already available in format for ML analysis (we want you to focus on model building, not data cleaning)

These are suggested but not set in stone: if you have an idea you're excited about (e.g. you want to use a private dataset you're already involved with for ongoing research), please discuss with course staff.

You should be confident that your dataset is appropriate for your desired analysis task.

Baseline method selection

Your "baseline" method is a concrete specification of the following 3 components:

• probabilistic model (likelihood, possibly also a prior)
• optimization problem (e.g. ML estimation or MAP estimation or posterior estimation)
• algorithm (e.g. gradient descent, coordinate descent, or closed-form formula)

Possibilities include:

• MAP estimation for Dir-Cat model for unigrams, as in CP1
• MAP estimation for linear regression, as in CP2
• MAP estimation for logistic regression
• ADVI for heteroscedastic linear regression
• ADVI for topic models
• MAP estimation for Gaussian mixtures, as in CP4
• MAP or posterior estimation on time series data using a hidden Markov model or linear dynamical system

If you wish to pursue a model that we do not cover in this course, please discuss with course staff. The main requirement will be that it needs to be viewable as a probabilistic model.

On the day the project releases, we will have spent a week talking about neural networks. Due to their computational demands and fickleness you may not build a deep learning model unless specifically given course staff approval. If you have specific research experience working with deep models, can clearly describe your proposed model probabilistically, and can implement the baseline easily, reach out to course staff to get special approval.

Upgrade method selection

The emphasis of this project will be on creating and testing hypotheses about how to improve model performance. Toward this end, we want you to design, implement, and evaluate one specific, feasible upgrade for your baseline model.

Your proposed upgrade should:

• have a compelling story about why it could deliver improvement on your task / dataset
• have a possible performance gain that is concretely measurable
• be completable by you and your team within about 2 weeks of effort
• fit within the concepts of this course
• pursue an idea we have not already covered in-depth in homeworks or coding practicals.

Your upgrade does not need to be a novel idea (e.g. it can be described already in a textbook or research paper). Your upgrade does not need to succeed (we care more about understanding why it works or does not work than on what your final performance metric is).

For a list of possible ideas, see our Project Brainstorming Google Doc.

There are two major kinds of upgrades, enumerated below.

Upgrade Option 1: Changing the Model (Prior or Likelihood)

For this kind of upgrade, you'd change the concrete PDF/PMF of your prior or likelihood

Examples:

• Change a Normal prior on regression weights to a Laplace prior (corresponds to LASSO regression)
• Change a Normal likelihood for regression to a likelihood for robust regression less sensitive to outliers

Upgrade Option 2: Changing the Estimation Objective and/or Algorithm

Here, we're considering changes to how parameters (or distributions over parameters) are estimated.

Examples:

• Compare a baseline first-order gradient descent for MAP estimation of logistic regression with an upgrade that uses second-order gradient descent.
• Compare MAP estimation of logistic regression with a Laplace approximation to the posterior, testing whether posterior predictive probabilities are better calibrated than plug-in point predictions.
• Compare coordinate ascent variational inference, which uses closed-form conjugate updates, with ADVI, which uses Monte Carlo gradient estimates, on the same model.
• Compare a mean-field variational family with a full-rank Gaussian family in ADVI, testing whether the factorization assumption hurts when parameters are correlated.

We especially suggest considering changes to algorithms that enable scaling to larger datasets.

Examples:

• Compare the baseline gradient descent for logistic regression to stochastic gradient descent with minibatches
• Compare full-batch MAP estimation in a mixture model with Stochastic Variational Inference (SVI) which uses minibatches

Examples that are not allowed:

• Changing from an MLE estimate of the parameters to a MAP estimate is technically a change in the estimation objective, but we will generally not allow this one since we have studied this extensively in CP1 and CP2. We will only consider if the idea is particularly interesting/novel/impactful for your target task.
• Very simple, straightforward code optimizations (for example changing a for loop to a matrix operation) do not count. Your proposed upgrade needs take about a week or two of effort, not a change to a few lines of code.
• As discussed before, any deep learning-based models, unless specifically approved.

Other Options

You may wish to pursue an upgrade that doesn't fall neatly into the options enumerated above. If so, please discuss with course staff!

Forming a Hypothesis

We want you to practice forming and testing concrete hypotheses about your modeling task.

Your hypothesis for why you are pursuing your upgrade should have the following structure:

We hypothesize that compared to [BASELINE], [UPGRADE] should improve [PERFORMANCE METRIC] on our dataset, especially when [SPECIFIC SCENARIO], because of [PROPERTY OF UPGRADE]

For example, in CP1, we could have said the following

We hypothesize that compared to [the ML estimator for unigrams], [the MAP estimator for unigrams] should improve [heldout log likelihood] on our dataset, especially when [datasets are smaller than a few thousand words], because [MAP offers a smoother way to handle unseen words, while ML is known to overfit and concentrate mass on too few words].

This clear statement of your hypothesis illuminates exactly what tests we need to run to evaluate. Remember, in science, all good hypotheses are testable and thus falsifiable.

Grading

• 80% of project grade depends on final report
• 10% of project grade is initial report
• 10% of project grade is participation in earlier parts (completing the team signup, completing the pitch, meeting with course staff)