Project B: Working with Images

Last modified: 2026-04-23 09:49

Updates

• 2026-04-10: Leaderboards and submission open.
• 2026-04-09: Updated test data. Be sure to re-download the test data if you downloaded it before 4/9.
• 2026-04-09: Updated part 2 grading.

Status: RELEASED

Due date: Monday, Apr 27, 2026 by 11:59pm Boston time. See late policy for projects. Remember, HW late days don't work on projects.

Jump to:

  Background   Code   Dataset   Problem 1: Random Forest   Problem 2: Open Challenge   Rubric

Turn-in links

• PDF report turned in to: https://www.gradescope.com/courses/1220989/assignments/7970598
• TXT file of test-set predictions for Problem 1's: https://www.gradescope.com/courses/1220989/assignments/7954580
• TXT file of test-set predictions for Problem 2's Open-Ended Leaderboard: https://www.gradescope.com/courses/1220989/assignments/7954589
• Reflection on Project B: https://www.gradescope.com/courses/1220989/assignments/7970628 (Both group members need to complete this) See What to Turn In Instructions for details about each item.

Overview

This is a multi-week project with lots of open-ended programming. Get started right away!

Deadlines

We will be enforcing the following deadlines:

• Release on Tue 3/31
• Form partners and complete ProjectB Team Formation Form by Fri 4/3 11:59pm
• Submit your model's test set predictions for Problem 1 by Tue 4/14 11:59pm
• Problem 2's Leaderboard closes Mon 4/27 at 11:59pm
• Final Report due Mon 4/27 at 11:59pm

We recommend the following workload:

• By 4/8: Have the data loaded, cleaned, and ready for classification. Make all major data cleaning and processing decisions.
• By 4/14: Complete Problem 1 and submit your Problem 1 predictions.
• By 4/22: Build a functional pipeline for Problem 2, train it, and evaluate/investigate improvements.
• By 4/27: Complete last tweaks to Problem 2 and write the report.

Note that while you overall have more time for project B than project A, you also have more concurrent homeworks.

Late tokens cannot be used on this project. The final report can be turned in late with a grade penalty as detailed in the course syllabus. Late tokens also cannot be used to extend the intermediate project deadlines. These intermediate deadlines can be extended in exceptional circumstances.

Team Formation

By Fri 4/2 you should have identified your partner and signed up here:

• ProjectB Team Formation Form

Even if you decide to work alone, you should fill this form out acknowledging that. You may form a team with someone who is enrolled in the other section of CS135.

In this project, you are encouraged to work as a team of 2 people. If you prefer, you can work individually. Individual teams still need to complete all the parts below and will be evaluated no differently than other teams. We strongly recommend working in pairs to keep your workload manageable.

If you need help finding a teammate, please post to our "Finding a Partner for Project B" post on Piazza.

Work to Complete

As a team, you will work on one restricted problem, and then a completely open problem.

The 2 problems look at different models for using multiple data types for medical prediction.

• Problem 1 looks at using a random forest model and creating engineered image features.
• Problem 2 is an open-ended problem, where any model and any feature representation is allowed.

Throughout problems 1 and 2, you will practice the development cycle of an ML practitioner:

• Propose a reasonable ML pipeline (feature extraction + classifier)
• Train that pipeline on available data
• Evaluate results carefully on available data
• Revise the pipeline and repeat

You will not be submitting the code of your models. Instead, you will submit your model's predictions on a test dataset, whose true \(y\) values we do not share with you. The majority of your grade will be determined by your report, but some of your grade will be determined by the quality of your predictions:

• For full credit on Problem 1, we expect you to hit a particular AUROC on test data.
• For Problem 2 we will have a leaderboard where you can compare your model's performance to your peers. This is designed to reflect the reality of building ML models, where your goal is often not to hit a particular performance level but to either produce the highest-quality model possible or to outperform your competition. In the real world, there's no performance threshold at which your work is "done".

For project B no portion of your grade will be based on how your model compares to other models in the class. Instead, part 2 will be graded the same as part 1 where you have to reach a reasonable threshold. There will still be a leaderboard competition and the top 3 teams may recieve prizes and/or some bonus points. To ensure fairness on the leaderboard we require students to NOT pay for any additional computational resources that might improve your models' performance above your peers'. If computational resources are a big concern for you in this project, reach out to your instructor.

What to Turn In

PDF report: Each team will prepare one PDF report covering all problems.

• Suggested length 6 pages (upper limit is 8 pages) including figures and tables
• This document will be manually graded according to our rubric
• Can use your favorite report writing tool (Word or G Docs or LaTeX or ....)
• Should be human-readable. Do not include code. Do NOT just export a jupyter notebook to PDF.
• Should have each subproblem marked via the in-browser Gradescope annotation tool.
• To make grading easier, if you are working in pairs please make sure to add your partner to your Gradescope submission.

GET STARTED EARLY. Writing a report takes a lot of time! If some of the figures/results in your report are reflective of a previous implementation and not your final implementation submitted to the leaderboard, we will not penalize you for that.

Submitting Model Predictions: Each team will prepare one plain-text file of test-set predicted probabilities to submit to each of Problem 1 Predictions and Problem 2 Leaderboard.

• File name must be: yproba1_test.txt
• One line for each example in the test set
• Each line contains float probability that the corresponding example should be classified as positive ** given its features
• Should be loadable into NumPy as a 1D array via: np.loadtxt('yproba1_test.txt')
• Just like with the PDF, you can add your partner to your submission once you've uploaded it.

SUBMISSION LIMITS. For both problems, you should periodically submit the predictions of your best model on the test set. In order to prevent attempts to optimize the test loss directly, we limit you to 10 submissions in a 24 hour period. We will not release the true labels of the test set to you during or after the project.

Reflection: Each individual will turn in a reflection form (after completing the report). Use the link above under Turn-In links.

Datasets, Starter Code and Code Restrictions

For all required code and .csv files, see the projectB folder of the public assignments repo for this class:

https://github.com/tufts-ml-courses/cs135-26s-assignments/tree/main/projectB

The starter code provides important functionality for loading and cleaning the required data. Be sure to look through and use the starter code for Problem 1.

Image data can be found in Google Drive:

• x_train_img.npz
• x_test_img.npz

You can download this data from the link, or load it directly into a Google Colab Notebook:

import gdown
!gdown 1JTvMkEaR3AxqAagYzfLebnS_RavA7agR #x_train_img.npz
!gdown 1bQxyUXwvUn2cPjb5gIp-jB3CUpqD4Ca1 #x_test_img.npz
x_train_imgs, x_train_img_ids = load_img_data("x_train_img.npz") # function provided in starter notebook

For this assignment, you can use any Python package you like (sklearn, nltk, torch, transformers, etc). You are welcome to consult the sklearn documentation website or other external web resources for snippets of code to guide your usage of different classifiers. However, you should understand every line of the code you use and not just copy-paste without thinking carefully. In your report, you should cite and acknowledge third-party code that made a significant impact on your work.

Remember to keep the course collaboration policy in mind: do your own work!

Background

Warning: This dataset includes real medical images. They are isolated images of small skin lesions, but some students might find them disturbing to look at. If you don't want to look at the lesion images you are not required to for this project. Please reach out if you have any questions.

For those with limited access to healthcare due to financial, mobility, or other constraints, deciding to go to a medical facility can be a difficult decision. In project B we will explore constructing a model that could be run on patient cellphones to help them in their decision of whether or not a skin lesion requires medical attention.

Specifically, this task focuses on skin cancer. Given limited patient information and a cellphone image of a skin lesion, your model will determine whether that lesion requires medical attention (is cancerous or precancerous), or is a harmless lesion such as a mole.

Skin cancer, like many cancers, is much more common among older patient than among younger patients. Thus, age can be an important consideration in whether or not a lesion requires attention. However, when a disease has a strong age bias it can also throw machine learning models off, as they may assume all older patients have the disease while all younger patients do not. We want to create a model that is useful for all age groups in identifying whether or not they need treatment.

This task is multimodal. You are given both image data and clinical features in tabular format. Additionally, this dataset contains common challenges in medical data: there is significant missing data and high class imbalance.

Classes

Below are representative pictures of each of the six possible diagnoses in the dataset.

The six original classes. In project B you will predict the binary task of BCC, MEL, SCC, or ACK (positive class) versus NEV or SEK (negative class)

These diagnoses are three types of skin cancer:

• Basal Cell Carcinoma of skin (BCC). Requires treatment.
• Malignant Melanoma (MEL). Requires treatment.
• Squamous Cell Carcinoma (SCC). Requires treatment.

and three other common skin conditions:

• Actinic Keratosis (ACK): considered precancerous. Requires treatment.
• Melanocytic Nevus of Skin (NEV): the medical term for a normal mole. No treatment required.
• Seborrheic Keratosis (SEK): sometimes called "age spots". No treatment required.

For project B we will focus on the binary classification task of predicting whether or not a skin lesion needs treatment Thus, you will be given a binary classification task where BCC, MEL, SCC, and ACK are the positive class, and then the two harmless conditions NEV and SEK are the negative class.

Dataset citation

This dataset comes from work from a research group working with the Dermatological and Surgical Assistance Program (PAD) of the Federal University of Espírito Santo, a nonprofit program that provides free skin lesion treatment, in particular, to low-income people who cannot afford private treatment. The data is released under a CC BY 4.0 license. See also the publication describing the dataset:

PAD-UFES-20: A skin lesion dataset composed of patient data and clinical images collected from smartphones
Andre GC Pacheco et. al.
August 2020
Data Brief . 2020 Aug 25;32:106221. doi: 10.1016/j.dib.2020.106221

We are grateful to these authors for making the dataset available.

Provided data

You are given data in two files per split:

• x_*.csv: This file contains 21 clinical features: 12 features pertaining to the patient as a whole and 9 features about the particular lesion. Some patients in the dataset have multiple lesions. Similar to author in project A, there is no overlap in patients between the training and testing sets.
• x_*.npz: This is a compressed numpy array of images corresponding to each patient.

There are 1178 input,output pairs in the training set, and 277 possible input-only instances in the test set. By design, we do not release the true labels ('outputs') of the test set.

By design, our provided train/test data have no overlap in patients. We want to make sure the model we build can accurately classify new patients.

You are prohibited in all problems from directly training any model on data outside the released CSV files in this project's starter code repository. You can use pretrained models, but in the spirit of the model being runnable on a cellphone any pre-trained model must have fewer than 10 million parameters.

CSV contents: Common keys

All CSV files have 3 common key columns to ensure we can understand data provenance.

• Column 1: 'patient_id' : Unique ID for each patient.
• Column 2: 'img_id' : Unique ID for each lesion image, corresponding to the image file.

CSV contents: Input features

The clinical features for our problem are in files named x_*.csv

CSV format for x_*.csv files:

We provide two such files: x_train.csv and x_test.csv

NPZ contents: Input images

The image files contain two components, the image data and image ids. They can be loaded with np.load, and code to load the images is included in the starter notebook.

The images are stored as a numpy array, where each value represents a particular color channel and value. The images have all been resized to be 256 by 256 pixels and contain 3 color channels (RGB), thus the loaded images numpy array for training images has a shape of (1197, 256, 256, 3). The training images are 171MB and the testing images are 40MB.

The image IDs are in the same order as the images; for the training data the array has shape (1197,).

We provide two such files: x_train_img.npz and x_test_img.npz

Labels to predict

Ground-truth labels for our problem are in files named y_*.csv

CSV format for y_*.csv files:

• Columns 1-2: common keys that identify each passage
• Column 3: 'coarse_label': either 0 (NEV or SEK) or 1 (BEL, MEL, SCC, or ACK). What you should try to predict.
• Column 4: 'fine_label': one of six diagnoses in [BCC, MEL, SCC, ACK, NEV or SEK ']

We provide only y_train.csv.

We don't release y_test.csv. Please submit to the Gradescope assignments to see your performance on the test set.

Performance metrics

Your Problem 1 predictions will be assessed via Area under the ROC curve (AUROC), judging your classifier's ability to distinguishing between diagnoses that require medical attention or those that do not, which are stored as 'Coarse Label' in y_*.csv data.

For Problem 2, we want to assess that your model can perform across all patient age groups. Thus, your leaderboard score for Problem 2 will be assessed as the minimum AUROC over 3 age groups: Young (<30), Adult (30-60), and Senior (60+). We will split the test patients into these three groups, calculate the AUROC of your predictions for each group separately, then calculate your final score as the lowest of the three. For patient age \(a\) your leaderboard score \(= \min(AUROC_{a<30}, AUROC_{30 \le a < 60}, AUROC_{a \ge 60})\).

Code is provided in the starter notebook to create these age groups for your own model validation.

Preprocessing

This dataset is real medical dataset, and processing it requires multiple preprocessing decisions. We provide one possible data processing pipeline for you in the starter code, but invite you to try changing these data preprocessing steps in Problem 2.

Missing Data

For many of the instances, some of the lesion-specific clinical features are missing (often marked 'UNK' for unknown in the data). This means that most likely the patient filling out questions about their lesion skipped these items. In the starter code, we simply set these missing values for binary features to be false, as a patient not answering it more likely means it does not exhibit that feature (i.e. a patient whose lesion is not itching might skip the itching question when filling in data about their lesion). However, this is not necessarily the case. One possible direction to explore in Problem 2 would be if a more sophisticated imputation of missing data improves performance.

Categorical features

The most common way to handle categorical features is to encode them in a one-hot encoding. For instance, the region feature denotes the location of the lesion on the body. We can encode the region into a set of mutually exclusive binary features: is_face, is_back, is_arm, etc. We have provided code to perform a one-hot encoding for all categorical features using the sklearn.preprocessing function OneHotEncoder. In Problem 2 you could explore other feature encodings.

Class Balance

This dataset is imbalanced. In Problems 1 and 2 you should explore how to deal with this imbalance. Scikit-learn has built-in ways for models to handle imbalanced data, but other approaches such as data augmentation could be appropriate. For Problem 2 you might also want to explore balancing your data across age groups to improve for minimum age-group AUROC performance.

Problem 1: Random Forest with engineered image features

Goals and Tasks for Problem 1

In Problem 1, you will explore using a random forest classifier that merges the tabular and image data by creating engineered image features.

One way to approach integrating multiple data modalities is to simply turn one modality into another. In Problem 1, you will manually transform images into at least 2 distinct features that you can treat directly as additional features.

One example of this would be that malignant lesions often exhibit increased color variation as compared to more benign lesions. Thus, the color variation in an image may be predictive of whether or not it requires treatment. An easy way to calculate this would be to calculate the standard deviation of each color channel in each image (you are allowed to use this as one of your two features):

color_N_3 = np.std(images_N_256_256_3, axis=(1, 2))

You could also explore features such as an image's mean intensity, it's level of symmetry, the difference between the inner and outer image regions, pixel intensity histograms for different regions, etc. You may also use computer vision packages such as OpenCV to create more complex image features, but that is not required to get a full score on Problem 1.

1A : Feature Construction Description

Well-written paragraph describing your chosen pipeline, with sufficient detail that another student in this class could reproduce it. You are encouraged to use just plain English prose. You might include a brief, well-written pseudocode block if you think it is helpful. Remember that you should create at least 2 image features, and then combine them with the clinical features in x_*.csv to create your final model inputs for Problem 1.

You should describe and justify all major decisions, such as:

• What normalization or standardization did you perform for the input features?
• What image features did you create and try?
• How many total images features did you create?
• Did you use the entire image to create your features, or only regions of the image?

1B : Cross Validation Design Description

Well-written paragraph describing how you use cross-validation to perform both classifier training and any hyperparameter selection needed for the classifier pipeline. Remember that patient_id plays a similar role in this project to author in Project A.

For Problem 1, you must use cross validation with at least 3 folds, searching over at least 5 possible hyperparameter configurations to avoid overfitting.

You should describe and justify all major decisions, such as:

• What performance metric will your search try to optimize on heldout data?
• How many folds? How big is each fold? Even when counts can be inferred from the number of folds, reporting how many data points there are in each fold is very helpful for a reader to understand the scope of your experiments.
• After using CV to identify a selected hyperparameter configuration, how will you then build one "final" model to apply on the test set?
• Did you use off-the-shelf libraries? Or implement from scratch?

1C : Hyperparameter Selection for Random Forest Classifier

For this step 1C, we want you to design and execute a hyperparameter search for a RandomForest classifier. Please use your features from 1A and your CV design from 1B. That procedure should allow you to estimate the heldout performance of a well-fit model with each candidate hyperparameter. Remember, your ultimate goal is to build a classifier pipeline that will achieve the best performance on the provided test set, as evaluated via Test Set Prediction submission later in 1D. But in this step, you only use estimates from CV.

In one paragraph about experimental design, you should describe and justify all major decisions, such as

• Which hyperparameters are you searching?
• What concrete grid of values will you try?
• What RandomForest hyperparameters are you setting to a fixed non-default value (such as class_weight)?

Next, you should include a figure that visualizes performance as a function of one hyperparameter (details in the hyperparameter selection rubric below). Finally, you should include a caption paragraph summarizing the results of your hyperparameter search.

Note that while you are only required to visualize one hyperparameter, you likely will need to search over multiple hyperparameters in order to achieve a full score for Problem 1.

For all 3 parts above, please follow the detailed hyperparameter selection rubric which is common across Problem 1 and Problem 2.

1D : Reflecting on Your Test Set Performance

Report your AUROC on test data as provided on Gradescope, and in a short paragraph reflect on your model's performance.

Create your "final" classifier using the selected hyperparameters from 1C. Apply your classifier to each test lesion in x_test.csv/npz. Store your probabilistic predictions into a single-column plain-text file yproba1_test.txt, as described above under What to Turn In. Upload this file to our Problem 1 Test Set Prediction Assignment and report your AUROC on test data.

Problem 2: Open-ended challenge

Goals and Tasks for Problem 2

For this problem, your goal is to obtain the best performance on heldout data, using any feature representation you want, any classifier you want, and any hyperparameter selection procedure you want.

Here are some concrete examples of different approaches you can try:

• The OpenCV library has a number of useful vision algorithms for detecting connected regions of pixels or basic shape sizes, which could be useful alternate features.
• Image augmentation is a common approach when using image datasets in machine learning, where images can be rotated, flipped, or otherwise slightly altered to increase the amount of training data available. Again OpenCV has a number of useful methods for performing such transformations.
• A natural approach would be to create a convolutional neural network. You are welcome to reuse code from homework 5. You could either use a CNN to create features to feed into a traditional classifier or build more complex neural network architecture that integrates both data types.
• You could explore more sophisticated methods for imputing missing clinical features, such as those provided by Pandas.
• A challenge in Problem 2 is that you have to create a model that performs well for all age groups. You could explore creating an ensemble, performing data weighting, or other approaches for increasing performance across different age groups.

Among classifiers, you can use any classifier in sklearn (nearest neighbor, random forest, MLP, etc.). You can also go beyond sklearn if you want. Be sure you understand enough about this classifier to discuss it clearly and define a reasonable hyperparameter search strategy.

As instructed below, your detailed steps for this Problem 2 will mimic Problem 1: you'll write sections like 1A (overall design of feature representation), 1B (overall CV experimental strategy), 1C (chosen classifier and its hyperparameter search strategy), and 1D (performance analysis).

For full credit, we expect that at least 2 parts out of 2A, 2B, and 2C explore substantially different methods than those used in Problems 1A, 1B, 1C. Each choice must be plausibly motivated by improving your classifier's performance.

Some examples of what is and is not considered substantially different:

• Changing a single function is not considered different enough. For instance, using a randomizedCV search instead of a grid search. However, if you used a Bayesian optimization library to perform hyperparameter optimization that would be considered substantially different.
• Using raw pixel values as part of a CNN is considered a substantially different feature representation.
• Changing which hyperparamters you search over with a Random Forest model would not be considered substantially different.
• Creating an ensemble of Random Forest classifiers to account for different age groups would be considered substantially different.
• Only changing the objective of your hyperparameter search to use the same metric as the Problem 2 leaderboard is not considered sufficiently different.
• Performing data augmentation is considered a sufficiently different feature representation.

As stated above, you are allowed to use pre-trained models. However, those models must be noted in your report and cannot be more than 10 million parameters in size. A table of pretrained models available in Pytorch, with their corresponding parameter counts, can be found here.

2A : Feature Representation description

Include a paragraph describing and justifying how you created features for your final model from the images and clinical features.

Include enough detail that another student could roughly reproduce your work. If this process is substantially similar to 1A, it is OK to say so and keep this paragraph brief (you don't need to repeat yourself).

2B : Cross Validation (or Equivalent) description

Include a paragraph describing and justifying how you set up your training and hyperparameter selection process. Remember you can only use provided data in x_train.csv/npz,y_train.csv. This will be assessed similarly to 1B.

Include enough detail that another student could roughly reproduce your work. If this is substantially similar to 1B, it is OK to say so and keep this paragraph brief.

2C : Classifier description with Hyperparameter search

Include a paragraph describing and justifying which classifier you selected, how you trained it, and how you performed hyperparameter search (including concrete values you explored). Include enough detail that another student could roughly reproduce your work.

Include a figure that shows how classifier performance changes over at least one hyperparameter. Provide a paragraph caption interpreting the results of your search. In all these steps, please follow the common rubric for hyperparameter selection.

2D : Reflecting on Your Model Performance

Apply your best pipeline from 2A - 2D above to the test sentences in x_test.csv/npz. Store your probabilistic predictions into a single-column plain-text file yproba1_test.txt, as described above under What to Turn In. Upload this file to our Problem 2 leaderboard.

For both your Problem 1 model and your Problem 2 model, report:

• Your test performance (overall AUROC for part 1 and minimum group AUROC for part 2), as recorded on Gradescope.
• The models' 3 most confidently-correct and 3 most confidently-incorrect predictions on heldout data. Please display the actual images as demonstrated in the starter notebook, as well as any other features/parts of the passage which are relevant to your concluding paragraph (see below). However, if you are uncomfortable looking at the images you may instead discuss class balance and the clinical features.

The heldout part is important: if you use a fixed validation set for Problem 2, you can simply make predictions on the validation set to find the 3 most correct and incorrect.

In a summary paragraph, discuss the relative performance of your two models. By investigating what each model gets right and wrong (and you're welcome to investigate beyond just the top-3's we request) make some hypotheses about either (a) what each model does better than the other, or (b) where both models are still struggling, and what key weakness you would try to fix if you were to continue working on this problem in the future.

Rubric for Overall Performance

We'll get a final number for this project by averaging:

• 80% : your PDF report, using the rubric below
• 5% : your Problem 1 submission, using the rubric below
• 10% : your Problem 2 leaderboard submissions, using the rubric below
• 5% : completion of your reflection on the project

Rubric for Evaluating Model Prediction Submissions

For both Problems 1 and 2, we've defined a particular threshold for a "reasonable" score (e.g. how we'd expect a standard pipeline trained using good practices to perform).

For Problem 1, 100% of your score is determined by whether you hit that score. For Problem 2, your score is broken down:

• 85% of points are awarded for hitting the "reasonable score" threshold.
• 15% of points awarded to the top 3 submissions in the class. Partial credit is awarded to other submissions, linearly interpolating between the "reasonable" score and the "top" score (e.g. a submission which is 2/3 of the way from reasonable minimum age group AUROC to top 3 minimum age group AUROC will get 10 of the 15 points).

Rubric for Evaluating PDF Report

Earning full credit on this assignment requires a well-thought-out report that demonstrates you made reasonable design decisions for feature preprocessing and classifiers and followed machine learning best practices throughout, especially for hyperparameter selection. Achieving top-scores on the leaderboard is far less important than understanding why some methods and choices outperform others.

Points will be allocated across the various parts as follows:

• Problem 1: 40%
• 10%: Paragraph 1A on Feature representation design decisions
• 10%: Paragraph 1B on cross validation design decisions
• 15%: Paragraph 1C on training and selection for your classifier
• 5%: Paragraph 1D reflection on heldout performance
• Problem 2: 60%
• 15%: Paragraph 2A on Feature representation design decisions
• 15%: Paragraph B on cross validation design decisions
• 20%: Paragraph C on training and selection for your classifier
• 10%: Paragraph D reflection both models' performance

Hyperparameter Selection Rubric

Paragraph on Search Experiment Design:

When you describe your hyperparameter search, we'll look for a paragraph that describes the design of your Hyperparameter Search Experiment:

• 1-3 sentences: describe the potential advantages of the chosen classifier for the task at hand.
• 1-3 sentences: describe which model complexity hyperparameter(s) were explored and how these values control model complexity. Describe why the chosen candidate value grids (or random distributions) are reasonable to explore the transition between under and over fitting and find the "sweet spot" in-between.
• 1-3 sentences: describe any necessary details about the training process (e.g. are there convergence issues? step-size selection issues? should you stop early to avoid overfitting?)

Figure Requirements:

Your figure should show model performance on the metric you choose for model selection (y-axis) as a function of one hyperparameter value (x-axis). This hyperparameter should control model complexity. If you searched over multiple hyperparameters in constructing your model, you can fix all of them but 1 and make a plot by varying the last hyperparameter.

The figure should show performance across at least 5 distinct hyperparameter values, covering both underfitting and overfitting cases. That is, if at all possible, at least one candidate value should show clear underfitting and at least one should show clear overfitting.

Your figure should:

• Show both training set and validation set performance trends in the same plot.
• Show the typical performance at each hyperparameter via the average over multiple CV folds
• Label each axis.
• Include a legend.
• Include a brief caption.

Your figure should also communicate uncertainty around this typical value, by exposing the variation across the multiple CV folds

• A simple way to show uncertainty is show the empirical range across all folds, or the empirical standard deviation
• A potentially better way to do this is show a separate dot for the direct performance of each fold (so 5 dots for 5 folds). Don't summarize when you can show the raw data.

The big idea here is that your figure should help the reader understand if one hyperparameter is definitely better than another (e.g. performance is better on most or all folds) or if there isn't much difference. Many plotting libraries have functions or arguments for producing neat error bars: e.g. matplotlib's errorbar or seaborn's errorbar and err_style arguments. If you add many overlapping lines and/or error bars, take some time to make sure the plot is still organized and easy to understand.

Paragraph on Interpreting Search Experiment Results

Provide a brief paragraph interpretating the results in your figure.

• Which hyperparameter is preferred and why?
• Is the evidence decisive, or uncertain?

General tips for Figures

Please do your best to keep figures close to the related paragraph, ideally on the same page.

If a figure contains multiple elements such as multiple lines or multiple sets of bars, please make sure that they are on the same scale. Alternatively, if different scales are necessary, adjust them appropriately to maintain reasonable and easily interpretable trends. For instance, it is inappropriate to represent a line fluctuating between 0 and 1 on the same scale as a line oscillating between 100 and 1000, as this would distort the representation and interpretation of the data points.

-->