This website covers a past offering of CS 135. For the current offering, go to https://www.cs.tufts.edu/cs/135/.

HW6: Automatic Differentiation and Explainability

Last modified: 2026-03-29 14:52

Status: RELEASED.

Due date: Wed Apr 1, 2026 by end of day (11:59 pm ET) in Medford, MA

Jump to: Background   Problem 1   Problem 2  

Turn-in links:

Overview

This assignment will have you explore a pre-trained neural network for image classification. You will construct:

  • adversarial examples via activation maximization,
  • and feature attribution maps via Lime.

To complete this assignment, you will be using Pytorch. Because I want you to play around with GPU acceleration, I am expecting this assignment to be completed in Google Colab. You can save Colab notebooks, but you will have to rerun cells whenever you disconnect from the server, and you will need to re-upload image files each time (as well as rerun cells, as your runtime will be killed after staying idle for a bit). These annoyances are worth the experience of seeing what techniques are possible with more computational resources.

Evaluation Rubric

The worth of each problem is

  • 90% PDF report
  • 5% python notebook submission
  • 5% reflection

See the PDF submission portal on Gradescope for the point values of each PDF subproblem.

Files to Turn In:

PDF report:

  • Prepare a short PDF report.
  • This document will be manually graded.
  • Can use your favorite report writing tool (Word or G Docs or LaTeX or ....)
  • Should be human-readable. Do not include code. Do NOT just export a jupyter notebook to PDF.
  • Should have each subproblem marked via the in-browser Gradescope annotation tool

Starter Notebook

The starter notebook can be found in the hw6 folder of the public assignments repo for this class:

https://github.com/tufts-ml-courses/cs135-26s-assignments/tree/main/hw6/homework_6.ipynb

The beginning of the notebook is a tutorial on (1) loading image data and passing it into a pre-trained classification neural network, and (2) using GPUs to accelerate code.

Background

To complete this HW, you'll need some specific knowledge from the following sessions of class:

  • Explainability (day17)
  • Pytorch and Automatic Differentiation (day18)

In particular, you'll start by attempting activation maximization. You're given a pre-trained ResNet18 neural network, which takes in images of size 3 x 324 x 224 (the first "3" is the 3 color channels: red, green, and blue) and performs multiclass classification with 1000 classes. It does so by outputting a continuous score \(s_c(x)\) for \(c=1,\dots,1000\) and then using the softmax function to map those scores to probabilities (\(\hat{p}(x)\), see day09 for details).

Your goal is to take an image of your choice and update it until the neural network is very confident it is from a particular (wrong) class of your choosing. Mathematically, we initialize \(x\) to be the image you choose, and iterate:

$$x \leftarrow x + \alpha\nabla_x s_{c^*}(x)$$

Where \(c^*\) is a class of your choice and \(\alpha\) is a step size. This looks a lot like some previous optimization problems, but instead of optimizing our model parameters (in this case, neural network weights and biases), here we are optimizing the input image itself. In addition, instead of minimizing a loss function, we're trying to maximize the model output for class \(c^*\). This is sometimes called gradient ascent, although we can easily reframe it as a gradient descent problem by saying our loss is \(-s_{c^*}(x)\).

Intuitively, if \(c^*\) is e.g. the goldfish class, we're saying: "take this image and update it incrementally until the neural network is very confident it is a goldfish".

There is no provided scaffolding code for the tasks below, but most of the code you need can be assembled from previous assignments and labs. If you run into problems with runtime or memory costs on Google Colab, some tips:

  • Use a GPU-accelerated runtime (helps with speed but can create memory issues)
  • If you run out of GPU memory, make sure that you aren't accidentally logging whole computation graphs (e.g. make sure your losses are detached when stored in history). If you're still running out of memory, consider just logged less information: e.g. you can log values every 5 or 10 optimization steps instead.

Problem 1: Activation Maximization

These implementation steps should all be performed on a class-appropriate image of your choosing. You should choose an image that the resnet18 classifier outputs a reasonable prediction for. If your image contains many things inside of it, the prediction of the classifier might not be "correct", but you should not pick an image for which the classifier is completely incorrect.

Implementation Step 1A : Activation Maximization on Images

Take your image, and pick a class that is not correct for it (the mapping of indices to classes can be found in the starter notebook). Perform activation maximization by gradient descent. Although you are optimizing the continuous score \(s_{c^*}(x)\), you should check the value of \(\hat{p}_{c^*}(x)\) at every iteration, and stop as soon as \(\hat{p}_{c^*}(x)>0.9999999\) (that's 7 nines). Recall that you can compute \(\hat{p}(x)\) from \(s(x)\) using softmax. You might have to do some work tuning the step size.

Note: why does it make a difference to optimize \(s_{c^*}(x)\) and not \(\hat{p}_{c^*}(x)\), when they're monotonically related (making one bigger makes the other bigger)? As it turns out it's easier to optimize \(z\)s because as they get very large, the \(\hat{p}\)s get very close to 1 and the gradient \(\nabla_x \hat{p}_{c^* }(x)\) gets close to 0. The gradient \(\nabla_x \hat{z}_{c^*}(x)\) doesn't saturate, so it doesn't have that problem.

Note 2: The starter notebook provides a normalization step for the input image, to get it in the form that the neural network expects. This introduces a question: should you normalize the image once at the beginning and then optimize the normalized image? Or should you optimize the unnormalized image and treat normalization as part of the loss computation (i.e. making it part of the forward pass)? Because matplotlib expects unnormalized images, I encourage you to use the 2nd approach and optimize the "raw"/unnormalized image.

Figure 1 in Report.

Please provide all of these:

  • Your original image
  • The new image you obtain by performing activation maximization.
  • Your optimization trajectory: either a loss curve, depicting \(-s_{c^*}(x)\) decreasing, or a curve showing \(s_{c^*}(x)\) increasing. The x-axis should count the number of gradient iterations.

In a short caption below the figure, please provide:

  • The top 3 classes assigned by the classifier to your original image, and those classes' associated probabilities \(\hat{p}_c(x)\)
  • The top 3 classes assigned by the classifier to your optimized image, and those classes' associated probabilities \(\hat{p}_c(x)\)

You are very very likely to get the following error message when using plt.imshow to visualize your optimized image:

WARNING:matplotlib.image:Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). Got range [.....].

If you get this error, do not worry; we will address it in the next subproblem.

Implementation Step 1B : Fixing Error Message

Why does this error message appear? It is because images are constrained: we expect every channel of every pixel to be in the range \([0,1]\). But when we perform a gradient update

$$x \leftarrow x + \alpha\nabla_x s_{c^*}(x)$$

there is no guarantee that after this update, every component of \(x\) will be in \([0,1]\). Perhaps taking a step in the direction of the gradient causes us to walk outside the set of images, in which case our conclusions are nonsense. The neural network allows you to hand it images with values below 0 or above 1, but it never saw values like that at training time so its behavior might be odd. If you got lucky and did not encounter this error in the previous problem, make sure you understand why it is possible to encounter.

We want to find an input that is a valid image and maximizes the model's output for class \(c^*\). Thus, we'll need to correct our previous approach. Luckily, there's a tool in the ML toolbox that can handle this easily, sometimes called reparameterizing. The new optimization problem is solved as follows:

Initialize \(x' = \sigma^{-1}(x)\), where \(\sigma\) is the sigmoid function from Units 2-3. In other words, let \(x'\) be an array of size 3 x 224 x 224, where if you take every value and sigmoid it, you get \(x\) back. Then iterate:

$$x' \leftarrow x' + \alpha\nabla_{x'} s_{c^*}(\sigma(x'))$$

What's happening here? If we compare this equation to our previous update equation, we'll notice that we've added a sigmoid. Thus every gradient step, we:

  • sigmoid \(x'\) to get an array where every entry is in \([0,1]\), aka a valid image, (then normalize it as before,) then
  • pass that valid image through our classifier to get \(s_{c^*}(\sigma(x'))\).

We then ask Pytorch to compute derivatives of the combined function, and update \(x'\). What we really care about is the image \(\sigma(x')\), but by optimizing \(x'\) we ensure that the image is always a valid image. No matter what values \(x'\) takes, \(\sigma(x')\) will always have values between 0 and 1. This is sometimes called reparameterizing because we can think of \(x'\) as an alternative way of representing our image in numbers (aka an unconstrained representation or unconstrained parameterization). It is not the "direct" representation of RGB values in each pixel, but an indirect representation that defines an image nonetheless.

Implement this new optimization problem in code. The beautiful thing about automatic differentiation is that this tasks requires minimal editing of your code for problem 1A. Note that the inverse of the sigmoid function (which you'll need to initialize) is called the "logit" function, and can be called via torch.logit. Again, stop your optimization as soon as \(\hat{p}_{c^*}(\sigma(x'))>0.9999999\) (that's 7 nines).

NOTE: When starting this problem, check whether your original image contains any pixel values that are exactly 0 or exactly 1. If it does, then the logit function will return \(\pm \infty\) because no value, when passed through sigmoid, gives exactly 0 or 1. To resolve this issue, you can use torch.clamp(input, min=0.001, max=0.999) to increase any value below 0.001 to 0.001 and decrease any value above 0.999 to 0.999. Clamp your image before computing its logit.

Figure 2 in Report.

Please provide all of these:

  • Your original image (again)
  • The new image you obtain by performing activation maximization on \(x'\). Remember that you want to visualize \(\sigma(x')\), not \(x'\) directly.
  • Your optimization trajectory: either a loss curve, depicting \(-s_{c^*}(\sigma(x'))\) decreasing, or a curve showing \(s_{c^*}(\sigma(x'))\) increasing. The x-axis should count how many gradient iterations.

In a short caption below the figure, please provide:

  • The top 3 classes assigned by the classifier to your original image, and those classes' associated probabilities \(\hat{p}_c(x)\) (again)
  • The top 3 classes assigned by the classifier to your optimized image, and those classes' associated probabilities \(\hat{p}_c(\sigma(x'))\)

Implementation Step 1C : Again but with smaller threshold

Repeat step 1B, but:

  • Select a different class. If before you chose a class that was "close to right" (e.g. the 2nd or 3rd top result of the original classifier) then this time choose a class that is very wrong. If before you chose something very wrong, this time choose something close to right.
  • In addition, you should now stop optimization as soon as \(\hat{p}_{c^*}(\sigma(x'))>0.9\). Rather than going to extreme lengths to make the image look like class \(c^*\), we want to see how small of a change we can make to \(x'\) in order to make the model pretty confident. If your learning rate is too high, it might be the case that the same iteration you step over 0.9 you also go to an extremely high value (e.g. 0.999). If that happens, reduce your learning rate and rerun until you get a \(\hat{p}_{c^*}(\sigma(x'))\) value between 0.9 and 0.99.

Figure 3 in Report

Follow the instructions for Figure 2, but this time for the result of Implementation Step 1C.

Short answer 1a: What does this tell you about the neural network?

Discuss the visualizations you made in the previous few problems. In particular, please answer:

  • How are the images changing in order to switch classification?
  • Can you reconcile the fact that this classifier is very accurate with the results you're seeing above?
  • If this model was going to be deployed somewhere for object recognition, would you be worried about adversarial examples (examples specifically designed to fool the detector)? Why or why not?

Problem 2: Lime

The starter notebook provides code for you to play with Lime, a method for feature attribution. Lime separates images into regions, and then tries to identify which regions are most important to a particular classification. We did not go into the math of how Lime works, but you now have the opportunity to see it in action.

For this problem, you may use the same image as above, a new image, or multiple images.

Implementation Step 2A : Feature Attribution

Use Lime to produce 2 feature attribution maps. Each map is an image with some of the pixels in the image painted over. Note that you should fiddle with the argument num_features, which selects how many regions of the image to show to you. You can make both with the same image but multiple classes (if the image contains multiple recognizable objects inside it) or you can make them with 2 different images.

You should choose the 2 features maps so that: * One of them is intuitive and clearly "correct". * One of them is surprising, but has some logic to it. Think about the wolf vs husky classifier that was focused on the snow: the feature map you choose should not just be garbage, it should be possible for you to justify it even if it appears at first glance to be wrong. For an additional example, running Lime on Bubblegum by Dan Jackson for the class "Vase" revealed it was focusing on the greenery, not the vase itself. This is sensible, because many pictures of vases will contain stems and leaves, but it is wrong. If you want to look at an example image from the model's training dataset for each class, see here.

Please get creative, play around, and find something that surprises you!

Figure 4 in Report.

Visualize the 2 feature maps above by showing the masked image (i.e. all pixels not in the selected region painted over with black). If the original image is different from the previous problems, please show the full original image as well. Just below the figure, for each map please write:

  • The class being explained,
  • and the probability of that class under the classifier.

Short Answer 2a in Report

Please discuss what you noticed about the feature maps in step 2A: why did the surprising one surprise you?