HW2: SSL to the Moon without a GPU

Last modified: 2024-10-10 16:50

Status: RELEASED.

Due date: Thu. Oct. 17, 2024 by 11:59pm ET (Boston time), with up to 4 late days possible

What to turn in:

Your ZIP file should include

  • All starter code files (.py and .ipynb) with your edits (in the top-level directory)

Your PDF should include (in order):

  • Your full name
  • Collaboration statement
  • Problem 1: figures 1a and 1b
  • Problem 2: figures 2a, 2b, and 2c; short answer for 2d
  • Problem 3: figures 3a, 3b, and 3c (with captions)
  • Problem 4: table 4a and code listing 4b

Please use provided LaTeX Template: https://github.com/tufts-ml-courses/cs152l3d-24f-assignments/blob/main/hw2/hw2_template.tex

Questions?

Jump to: Starter Code   Problem 1   Problem 2   Problem 3   Problem 4

Updates to Instructions:

Use this fragment: Instead of:

Goals

We spent Weeks 3 and 4 learning all about self- and semi-supervised learning.

In this HW2, you'll implement a common method for each style of SSL, (self- and semi-), and then evaluate your implementation on a toy dataset.

  • Problem 1: Establish a baseline for supervised training on labeled-set-only.

  • Problem 2: Can we gain value from pseudo-labeling?

  • Problem 3: Can we gain value from SimCLR?

Background

To read up on Pseudo-labeling for Problem 2, see the Pseudo-label SSL for HW2 page.

To read up on SimCLR for Problem 3, see the SimClr for HW2 page.

Problem Setup

Starter Code and Provided Data

You can find the starter code and our provided "two half moons" dataset in the course Github repository here:

https://github.com/tufts-ml-courses/cs152l3d-24f-assignments/tree/main/hw2/

Datasets

Run the top cells of hw2.ipynb, which load and plot the data we'll classify.

Two Half Moons dataset

Two Half Moons toy dataset. What can we learn from only 5 red and 5 blue points in training (and 5 more of each in validation)? Can more unlabeled data help?

This is a variant of the common "two half moons" dataset that is widely used to illustrate various binary classification tasks, especially in semi-supervised.

Classifier architecture

Skim MLPClassifier.py, which defines the very simple core architecture we will use:

  • input for each example is a 2-dim array/tensor
  • hidden layer with 32 units followed by ReLu
  • hidden layer with 2 units followed by ReLu
  • hidden layer with 2 units followed by L2normalization
  • output layer that produces a 2-dim array/tensor of logits

We could get predicted probabilities for the two classes by applying softmax to the logits produced by the network.

Note that the last hidden layer produces an "embedding" of the input feature vector on the unit circle in 2 dimensions. This is a special case of the common practice in representation learning of embedding on a unit hyperspheres in a desired number of dimensions (often 500+).

Problem 1

Worth 15 out of 100 points

Tasks for Code Implementation

Target time for these coding steps: 20 min.

Skim train_super.py, which defines a function for performing training. This is very similar to what we used in HW1. There's nothing you need to implement here, but focus on understanding what this function does.

Tasks for Experiment Execution

Target time for all these steps: 30 min.

Step through hw2.ipynb to achieve the following.

EXPERIMENT 1(a): Explore settings of lr, n_epochs and seed to find a setting that works well (low validation set xent) on the bigger labeled set version of half-moons (see hw2.ipynb). We have plenty of data here, so keep l2pen_mag = 0.0. This step is so we know that a solid solution to this data is achievable, if we had enough data.

EXPERIMENT 1(b): Explore settings of lr, n_epochs and seed to find a setting that works well (low validation set xent) on the smaller version of half-moons (see hw2.ipynb). Please set l2pen_mag = 2.0, to avoid overly confident predicted probabilities. This smaller version is what we'll try to improve throughout Problems 2 and 3 below.

Tasks for Report

Figure 1(a): Show the two-panel visualization (trace plot of losses, decision boundary) representing the best run from Experiment 1(a). No caption necessary here.

Figure 1(b): Show the two-panel visualization (trace plot of losses, decision boundary) representing the best run from Experiment 1(b). No caption necessary here.

Problem 2: Semi-supervised learning via Curriculum Pseudo-labeling

Worth 30 out of 100 points

Tasks for Code Implementation

Target time for these steps: 30 min.

Inside data_utils_pseudolabel.py, make the following edits

CODE 2(i) Edit make_pseudolabels_for_most_confident_fraction so that, given a trained model and a unlabeled dataset stored in tensor xu_ND (shape N x D), you are computing phat_N and yhat_N, tensors indicating the maximum predicted probability and the corresponding class label for each of the N=2048 examples in the unlabeled set.

Also read through make_expanded_data_loaders so you understand how we are merging the original train set with this new pseudolabel dataset to make new data loaders. But no edits necessary.

Tasks for Experiment Execution

Target time for these steps: 2 hrs.

Step through hw2.ipynb to achieve the following.

EXPERIMENT 2(a): For the best model from 1b (of the smaller dataset), use your code to make pictures of the pseudolabels obtained by thresholding at 0.9, 0.5, and 0.1 quantiles of predicted probabilities of each class.

EXPERIMENT 2(b): Perform PHASE ONE of curriculum pseudo-labeling. Build a dataset using the model from 1b, with quantile threshold set to 0.5 (keeping the most confident 50% of all unlabeled data). Run train_super with suitable settings of seed, lr, n_epochs so you see reasonable convergence and low validation-set xent. Set l2pen_mag=200 because even though we have lots of "data", we want to avoid overconfident predicted probabilities.

EXPERIMENT 2(c): Perform PHASE TWO of curriculum pseudo-labeling. Build a dataset using the phase1 model from 2b, with quantile threshold set to 0.25 (keeping the most confident 75% of all unlabeled data). Run train_super with suitable settings of seed, lr, n_epochs so you see reasonable convergence and low validation-set xent. Again, set l2pen_mag=200 because even though we have lots of "data", we want to avoid overconfident predicted probabilities.

Tasks for Report

Figure 2(a) Show the dataset visualization figure resulting from experiment 2a. The purpose is to sanity check your pseudolabel dataset construction. No caption necessary.

Figure 2(b) Show two-panel visualization (trace plot of losses, decision boundary) from the best training run of Experiment 2(b). No caption necessary.

Figure 2(c) Show two-panel visualization (trace plot of losses, decision boundary) from the best training run of Experiment 2(c). No caption necessary.

Short answer 2(d) Reflect on the similarities and differences between the decision boundaries seen here in Problem 2 (with pseudolabels), and the earlier boundary in Fig 1b (which only used the labeled set). Try to align what you see with conceptual knowledge of how pseudolabeling works. Hint: How do your results comparing supervised and psuedo-label semi-supervised learning contrast with Fig. 1 of the Ouali et al. paper we read?

Problem 3: Self-supervised learning via SimCLR

Worth 50 out of 100 points

Tasks for Code Implementation

Target time for these steps: 3 hr.

In losses_simclr.py, complete the following tasks:

  • Task 3(i): Complete the todos in calc_self_loss_for_batch
  • Task 3(ii): Implement calc_simclr_loss__forloop
  • Task 3(iii): (optional, recommended for speed) Implement calc_simclr_loss__fast

That last task is optional, focusing on how to speed up Python/PyTorch code via using built-in functions that are vectorized, avoiding any for loops. You can skip this step, but note your later experiments might be much slower, with training for 50 epochs taking several minutes rather than a few seconds.

Next, skim train_self.py to see how we perform training with a self-supervised objective. You don't need to edit anything here, but you should understand how it works.

Tasks for Experiment Execution

Target time for these steps: 2 hr.

Step through the Problem 3 section of hw2.ipynb to achieve the following.

EXPERIMENT 3(a) : Using your SimCLR loss implementation and the provided train_self module, fit an MLP encoder to the unlabeled half-moons dataset (all U=2048 examples). You'll want to explore lr, n_epochs, and seed to find a setting that works well (low training set loss, we're looking for values less than 4.0 after convergence).

EXPERIMENT 3(b) : Visualize the learned representations for your best SimCLR model from 3(a), and compare these to the best supervised model from 1b (using the smaller data) and from 1a (bigger data). Use provided code in hw2.ipynb.

EXPERIMENT 3(c) : Freeze the learned representations from 3a, via a call to set_trainable_layers. Next, call train_super.train_model to train a linear output "classifier head" to classify on the provided tr_loader and va_loader. Keep l2pen_mag=0.0, and find reasonable values of other settings (lr, n_epochs, seed).

Tasks for Report

In your submitted report, include:

Figure 3a with caption : Plot the loss over time for your best SimCLR runs. In the caption, describe your strategy for (1) selecting a seed and (2) selecting a learning rate.

Figure 3b with caption : Show side-by-side the unit-circle embeddings for the test set from (1) your best SimCLR model from 3a, (2) the best supervised model from 1b, and (3) the best supervised model from 1a, using code provided in hw2.ipynb. In the caption, please reflect on the differences between the panels. What about how each method is trained leads to the embeddings you see?

Figure 3c with caption: Show the two-panel visualization (loss traces on left, decision boundary on right) for your fine-tuned SimCLR classifier from 3c. In the caption, reflect on the decision boundary here and how it is similar/different to the earlier one from 1b. Also describe how the classifier boundary you see aligns with the unit circle embedding visualization for SimCLR in 3b.

Problem 4: Head-to-head comparison

Worth 5 out of 100 points

Tasks for experiment execution

In hw2.ipynb, compute and report the test-set cross-entropy (base2) and accuracy for each of

  • best supervised model from 1(b)
  • best semi-supervised model from 2(c)
  • best self-supervised model plus fine-tuning from 3(c)

Tasks for Report

In your submitted report, include:

Table 4(a): Report the final cross entropy and accuracy of each model on the test set. No caption is necessary.

Short answer 4(b) : Include the code for your best implementation of calc_simclr_loss (either forloop or fast versions). Use the provided style in hw2_template.tex.

Credits

Problem 3 on SimCLR adapted in part from