This website covers a past offering of CS 135. For the current offering, go to https://www.cs.tufts.edu/cs/135/.

Project A: Classifying Reading-Level Difficulty from Text

Last modified: 2026-02-23 15:06

Status: RELEASED

Due date: Friday, Mar 6, 2026 by 11:59pm Boston time. See late policy for projects. Remember, HW late days don't work on projects.

Jump to:

  Background   Code   Datasets   Problem 1: Bag-of-Words   Problem 2: Open Challenge   Rubric

Turn-in links

See What to Turn In Instructions for details about each item.

Overview

This is a multi-week project with lots of open-ended programming. Get started right away!

Deadlines

We will be enforcing the following deadlines:

  • Release on Thu 2/12
  • Form partners and complete ProjectA Team Formation Form by Tue 2/17 11:59pm
  • Submit your model's test set predictions for Problem 1 by Tue 2/24 11:59pm
  • Problem 2's Leaderboard closes Thu 3/5 at 11:59pm
  • Final Report due Fri 3/6 at 11:59pm

We highly recommend the following workload:

  • Week of 2/16: Complete Problem 1 (~4 hours, each partner) and start brainstorming and reading to prepare for Problem 2 (~2 hours, each partner)
  • Week of 2/23: Build a functional pipeline for Problem 2, train it, and evaluate/investigate improvements (~6 hours, each partner)
  • Week of 3/2: Complete last tweaks to Problem 2 and write the report (~4 hours, each partner). There is also a homework due this week, so try as much as you can to plan ahead.

Late tokens cannot be used on this project. The final report can be turned in late with a grade penalty as detailed in the course syllabus. Late tokens also cannot be used to extend the intermediate project deadlines (Problem 1 test set predictions by 2/24 and Problem 2's test set predictions leaderboard by 3/5). These intermediate deadlines can be extended in exceptional circumstances.

Team Formation

By Tue 2/17, you should have identified your partner and signed up here:

Even if you decide to work alone, you should fill this form out acknowledging that. You may form a team with someone who is enrolled in the other section of CS135.

In this project, you are encouraged to work as a team of 2 people. If you prefer, you can work individually. Individual teams still need to complete all the parts below and will be evaluated no differently than other teams. We strongly recommend working in pairs to keep your workload manageable.

If you need help finding a teammate, please post to our "Finding a Partner for Project A" post on Piazza.

Work to Complete

As a team, you will work on one restricted problem, and then a completely open problem.

The 2 problems look at different models and different representations of text for a common task.

  • Problem 1 looks at using a logistic regression model on bag-of-word feature representations
  • Problem 2 is an open-ended problem, where any model and any feature representation is allowed

Throughout Problems 1 and 2, you will practice the development cycle of an ML practitioner:

  • Propose a reasonable ML pipeline (feature extraction + classifier)
  • Train that pipeline on available data
  • Evaluate results carefully on available data
  • Revise the pipeline and repeat

You will not be submitting the code of your models. Instead, you will submit your model's predictions on a test dataset, whose true \(y\) values we do not share with you. The majority of your grade will be determined by your report, but some of your grade will be determined by the quality of your predictions:

  • For full credit on Problem 1, we expect you to hit a particular AUROC on test data.
  • For Problem 2 we will have a leaderboard where you can compare your model's performance to your peers. This is designed to reflect the reality of building ML models, where your goal is often not to hit a particular performance level but to either produce the highest-quality model possible or to outperform your competition. In the real world, there's no performance threshold at which your work is "done". A small number of points are reserved for submissions which hit (or at close to) the top 3 in the leaderboard.

Leaderboard as Competition: While we explained the motivation of the leaderboard above, we want to highlight that a very small portion of your grade on this assignment is affected by the performance of other students. However, to ensure fairness on the leaderboard we require students to NOT pay for any additional computational resources that might improve your models' performance above your peers'. If computational resources are a big concern for you in this project, reach out to your instructor.

What to Turn In

PDF report: Each team will prepare one PDF report covering all problems.

GET STARTED EARLY. Writing a report takes a lot of time! If some of the figures/results in your report are reflective of a previous implementation and not your final implementation submitted to the leaderboard, we will not penalize you for that.

Submitting Model Predictions: Each team will prepare one plain-text file of test-set predicted probabilities to submit to each of Problem 1 Predictions and Problem 2 Leaderboard.

  • File name must be: yproba1_test.txt
  • One line for each example in the test set
  • Each line contains float probability that the corresponding example should be classified as positive (Class 1: UK Key Stage 4-5) given its features
  • Should be loadable into NumPy as a 1D array via: np.loadtxt('yproba1_test.txt')
  • Just like with the PDF, you can add your partner to your submission once you've uploaded it.

SUBMISSION LIMITS. For both problems, you should periodically submit the predictions of your best model on the test set. In order to prevent attempts to optimize the test loss directly, we limit you to 10 submissions in a 24 hour period. We will not release the true labels of the test set to you during or after the project.

Reflection: Each individual will turn in a reflection form (after completing the report). Use the link above under Turn-In links.

Datasets, Starter Code and Code Restrictions

For all required data and code, see the projectA folder of the public assignments repo for this class:

https://github.com/tufts-ml-courses/cs135-26s-assignments/tree/main/projectA

Our starter code repo provides a few scripts helping you load the data for each problem, but otherwise offers no other code. This is meant to simulate the lack of code you'd have in the "real world", trying to build a text classifier from scratch using your machine learning skills.

For this assignment, you can use any Python package you like (sklearn, nltk, torch, transformers, etc). You are welcome to consult the sklearn documentation website or other external web resources for snippets of code to guide your usage of different classifiers. However, you should understand every line of the code you use and not just copy-paste without thinking carefully. In your report, you should cite and acknowledge third-party code that made a significant impact on your work.

Remember to keep the course collaboration policy in mind: do your own work!

Background

In today's school systems, modern youth spend years learning to read and critically interpret written text. To best support this learning, it is often critical to select text passages that have the appropriate level of difficulty. We'll refer to this level of difficulty as the reading level.

Manual assessment of a text passage's reading level is a time-intensive process. It can take a long time for new works to be labeled appropriately. Due to short supply of well-labeled texts, students in school often read the "same old" stories in a one-size-fits-all approach.

The goal of this project is to explore the possibility of automatic assessment of reading level. Accurate automatic assessment of reading level could empower students to select more possible materials (recent novels, current event articles) while being confident the difficulty of the text remains in reach.

We have remixed a public dataset, UK Key Stage Readability for English Texts prepared by J. Bird in 2024 cited below.

Our dataset (available in the starter repository) contains thousands of text passages from many authors. Each passage comes from a larger literary work included within Project Gutenberg. While the primary language is English, as works of literature these texts may includes words or phrases that come from any other language an author may wish to invoke.

Each passage is marked with a fine-grained label indicating one of four levels in the United Kingdom's national curriculum. These labels are determined via a Lexile score, a proprietary measure of reading difficulty level that is not open to the public.

Fine-grained labels: The 4 possible fine-grained labels are:

  • Age 7-11: UK Key Stage 2, corresponding to Lexile score 400-800
  • Age 11-14: UK Key Stage 3, corresponding to Lexile score 801-1000
  • Age 14-16: UK Key Stage 4, corresponding to Lexile score 1001-1200
  • Age 16-18: UK Key Stage 5, corresponding to Lexile score 1201 or higher

Coarse labels: Your goal is to develop a binary classifier that can correctly distinguish between a basic coarse notion of difficulty: lower-level vs. upper-level. We define the 2 Coarse Labels as:

  • Class 0: Lower-level (Age 7-14, UK Key Stage 2-3)
  • Class 1: Upper-level (Age 14-18, UK Key Stage 4-5)

Example lower-level passages

From The Marvelous Land of Oz by L. Frank Baum:

"I will carry them hidden in the straw which stuffs my body, and when I give them the signal by unbuttoning my jacket, they have only to rush out and scamper home again as fast as they can. By doing this they will assist me to regain my throne, which the Army of Revolt has taken from me." "In that case," said the Queen, "I will not refuse your request. Whenever you are ready, I will call twelve of my most intelligent subjects." "I am ready now" returned the Scarecrow.

Row 99 from The mystery of the Blue Train by Agatha Christie:

Knighton spoke French like a Frenchman, he had been in America, France, and England at much the same time as the Marquis was operating. The Marquis was last heard of as engineering various jewel robberies in Switzerland, and it was in Switzerland that you had come across Major Knighton; and it was at precisely that time that the first rumours were going round of your being in treaty for the famous rubies." "But why murder?" murmured Van Aldin brokenly.

Row 714 from The Adventures of Pinocchio by Carlo Collodi:

But the Cricket, who was a wise old philosopher, instead of being offended at Pinocchio's impudence, continued in the same tone: "If you do not like going to school, why don't you at least learn a trade, so that you can earn an honest living?" "Shall I tell you something?" asked Pinocchio, who was beginning to lose patience. "Of all the trades in the world, there is only one that really suits me." "And what can that be?"

Example upper-level passages

Row 423 from The Ethics of Aristotle by Aristotle:

For this reason it is thought to be a greater proof of Courage to be fearless and undisturbed under the pressure of sudden fear than under that which may be anticipated, because Courage then comes rather from a fixed habit, or less from preparation: since as to foreseen dangers a man might take his line even from calculation and reasoning, but in those which are sudden he will do so according to his fixed habit of mind.

Row 802 from The Pickwick Papers by Charles Dickens:

But still a cloud seemed to hang over his companions which that great man could not but be sensible of, and was wholly at a loss to account for. There was a mysterious air about them both, as unusual as it was alarming. 'And how,' said Mr. Pickwick, when he had grasped his followers by the hand, and exchanged warm salutations of welcome--'how is Tupman?' Mr. Winkle, to whom the question was more peculiarly addressed, made no reply. He turned away his head, and appeared absorbed in melancholy reflection.

Row 4931 from Leviathan by Thomas Hobbes:

But it is not declared in what consisted the praeeminence of the manner of Gods speaking to Moses, above that of his speaking to other Prophets, as to Samuel, and to Abraham, to whom he also spake by a Voice, (that is, by Vision) Unlesse the difference consist in the cleernesse of the Vision. For Face to Face, and Mouth to Mouth, cannot be literally understood of the Infinitenesse, and Incomprehensibility of the Divine Nature.

Dataset citation

This dataset comes from research work by Jordan J. Bird, released in December 2024. His UK Key Stage Readability for English Texts dataset is available on Kaggle under an MIT license. See also his preprint manuscript:

WHAT DIFFERENTIATES EDUCATIONAL LITERATURE? A MULTIMODAL FUSION APPROACH OF TRANSFORMERS AND COMPUTATIONAL LINGUISTICS
Jordan J. Bird
December 2024
arXiv preprint 2411.17593, version 3

We are grateful to these authors for making the dataset available.

Provided data

You are given data in CSV file format. There are 5557 input,output pairs in the training set, and 1197 possible input-only instances in the test set. By design, we do not release the true labels ('outputs') of the test set.

By design, our provided train/test data have no overlap in authors or titles. We want to make sure the model we build can accurately classify the reading level of never-before-seen authors.

You are prohibited in all problems from directly training any model on data outside the released CSV files in this project's starter code repository. You can use pretrained models, but they must be approved by course staff first and cannot be models where a primary use case is to assess reading difficulty levels.

CSV contents: Common keys

All CSV files have 3 common key columns to ensure we can understand data provenance.

  • Column 1: 'author' : string, name of author
  • Column 2: 'title' : string, title of the published work
  • Column 3: 'passage_id' : int, maps this passage back to original dataset release

CSV contents: Input features

The inputs to our problem are in files named x_*.csv

CSV format for x_*.csv files:

  • Columns 1-3: common keys that identify each passage. DO NOT use these as features for classification.
  • Column 4: 'text' : string, passage of text to be classified
  • Column 5: 'char_count' : int, count of characters in text
  • Column 6: 'word_count' : int, count of words in text
  • Column 7: 'sentence_count': int, count of sentences in text
  • Column 8-13: other numerical counts
  • Column 14-15: sentiment metrics
  • Column 16-23: readability metrics
  • Column 24-: other numerical features

We provide two such files: x_train.csv and x_test.csv

To understand the numerical features in Columns 5 and onward, see Sec. 3.1.1 of the paper by Bird (2024).

CSV contents: Labels to predict

Ground-truth labels for our problem are in files named y_*.csv

CSV format for y_*.csv files:

  • Columns 1-3: common keys that identify each passage
  • Column 4: 'Coarse Label': either 'Key Stage 2-3' or 'Key Stage 4-5'
  • Column 5: 'Fine Label': one of four reading levels in ['Key Stage 2', 'Key Stage 3', 'Key Stage 4', 'Key Stage 5']

We provide only y_train.csv.

We don't release y_test.csv. Please submit to the Gradescope assignments to see your performance on the test set.

Suggested Way to Load Data into Python

We suggest loading the data using the read_csv method in Pandas:

x_train_df = pd.read_csv('x_train.csv')
tr_list_of_text = x_train_df['text'].values.tolist()

You can see a short example working Python script here: https://github.com/tufts-ml-courses/cs135-26s-assignments/blob/main/projectA/load_train_data.py

Performance metric

Your Problem 1 and Problem 2 predictions will be assessed via Area under the ROC curve (AUROC), judging your classifier's ability to distinguishing between the lower-level (Key Stages 2-3) and upper-level (Key Stages 4-5) reading levels which are stored as 'Coarse Label' in y_*.csv data.

Preprocessing

As discussed in class, when classifying text there are many possible approaches to feature representation. By feature representation, we mean the process of transforming any possible natural language document (often represented as an ordered list of words which can be of variable length) into a feature vector \(x_n\) of a standard length.

In this project, we will explore several approaches, including bag-of-words vectors (explored in Problem 1). Later, you'll be allowed to try any feature representation approach you want (Problem 2).

Problem 1: Bag-of-Words Feature Representation

Background on Bag-of-Words Representations

As discussed in class on day10, the "Bag-of-Words" (BoW) representation assumes a fixed, finite-size vocabulary of V possible words is known in advance, with a defined index order (e.g. the first word is "stegosaurus", the second word is "dinosaur", etc.).

Each document is represented as a count vector of length V, where entry at index v gives the number of times that the vocabulary word with index v appears in the document.

The key constraint with BoW representations is that each input feature must directly correspond to one human-readable unigram in a finite vocabulary.

That said, you have many design decision to make when applying a BoW representation:

  • How big is your vocabulary?
  • Do you exclude rare words (e.g. appearing in less than 10 documents)?
  • Do you exclude common words (like 'the' or 'a', or appearing in more than 50% of documents)?
  • Do you keep the count values, or only store present/absent binary values?

You are strongly encouraged to take advantage of the many tools that sklearn provides related to BoW representations:

Goals and Tasks for Problem 1

For Problem 1, you will develop an effective BoW representation plus binary Logistic Regression classifier pipeline, aiming to produce the best possible performance on heldout data.

You should experiment with several possible ways of performing BoW preprocessing.

You must use a LogisticRegression classifier for this problem.

You should use best practices in hyperparameter selection techniques to avoid overfitting and generalize well to new data. Within your hyperparameter selection, you should use cross-validation over multiple folds to assess the range of possible performance numbers that might be observed on new data.

Your report should contain the following sections:

1A : Bag-of-Words Design Decision Description

Well-written paragraph describing your chosen BoW feature representation pipeline, with sufficient detail that another student in this class could reproduce it. You are encouraged to use just plain English prose. You might include a brief, well-written pseudocode block if you think it is helpful.

You should describe and justify all major decisions, such as:

  • Did you perform any "cleaning" of the data? (e.g. handle upper vs. lower case, strip punctuation or unusual characters, etc.). Hint: don't spend too much time on cleaning. Simpler is better.
  • How did you determine the final vocabulary set? Did you exclude any words? If so, why?
  • What was your final vocabulary size? If size varies across folds because it depends on the training set, please provide the typical size ranges. Reporting exact numbers is helpful for a reader to understand the scope and complexity of your experiments.
  • Did you use counts or binary values or something else?
  • How does your approach handle out-of-vocabulary words in the test set?
  • Did you use off-the-shelf libraries? Or implement from scratch?

1B : Cross Validation Design Description

Well-written paragraph describing how you use cross-validation to perform both classifier training and any hyperparameter selection needed for the classifier pipeline.

For Problem 1, you must use cross validation with at least 3 folds, searching over at least 5 possible hyperparameter configurations to avoid overfitting.

You should describe and justify all major decisions, such as:

  • What performance metric will your search try to optimize on heldout data?
  • How many folds? How big is each fold? Even when counts can be inferred from the number of folds, reporting how many data points there are in each fold is very helpful for a reader to understand the scope of your experiments.
  • After using CV to identify a selected hyperparameter configuration, how will you then build one "final" model to apply on the test set?
  • Did you use off-the-shelf libraries? Or implement from scratch?

1C : Hyperparameter Selection for Logistic Regression Classifier

For this step 1C, we want you to design and execute a hyperparameter search for a LogisticRegression classifier. Please use your BoW preprocessing from 1A and your CV design from 1B. Your CV procedure may only use the provided data in x_train.csv, y_train.csv. That procedure should allow you to estimate the heldout performance of a well-fit model with each candidate hyperparameter. Remember, your ultimate goal is to build a classifier pipeline that will achieve the best performance on the provided test set, as evaluated via Test Set Prediction submission later in 1E. But in this step, you only use estimates from CV.

In one paragraph about experimental design, you should describe and justify all major decisions, such as

  • Which hyperparameters are you searching?
  • What concrete grid of values will you try?

Next, you should include a figure that visualizes performance as a function of one hyperparameter (details in the hyperparameter selection rubric below). Finally, you should include a caption paragraph summarizing the results of your hyperparameter search.

For all 3 parts above, please follow the detailed hyperparameter selection rubric which is common across Problem 1 and Problem 2.

1D : Reflecting on Your Test Set Performance

Report your AUROC on test data as provided on Gradescope, and in a short paragraph reflect on your model's performance.

Create your "final" classifier using the selected hyperparameters from 1C. Apply your classifier to each test sentence in x_test.csv. Store your probabilistic predictions into a single-column plain-text file yproba1_test.txt, as described above under What to Turn In. Upload this file to our Problem 1 Test Set Prediction Assignment and report your AUROC on test data.

If your test set performance is very different from your validation performance (estimated via cross validation), then you should suspect something is amiss. Your validation performance should be an estimate of how well your model generalizes. Small variations are expected because datasets are unique and contain idiosyncracies, but large differences in performance indicate that one dataset is easier to generalize to than the other. If this is happening, it's bad! It means your validation set will favor hyperparameters that are either too flexible or not flexible enough for the true generalization task.

Investigate the dataset. Why might one generalization task be easier than the other? In this case of this dataset, there is a specific reason for the gap you observe. Include a short paragraph describing your hypothesis for why this gap exists, and your main takeaways for how your performance in Problem 1 can be improved.

To maximize your performance on the leaderboard, you will likely want to address this issue in your Problem 2 training pipeline.

Problem 2: Open-ended challenge

Goals and Tasks for Problem 2

For this problem, your goal is to obtain the best performance on heldout data, using any feature representation you want, any classifier you want, and any hyperparameter selection procedure you want.

Here are some concrete examples of different features you can try:

  • Can you use off-the-shelf embedding vector representations of text produced by neural nets trained on lots of data? Think of methods like word2vec or GloVe or BERT, or maybe even more recent large language models. To get you started, we provide BERT embeddings of each document in the starter code repository.
  • Can you use the numerical (non-text) features provided in x_*.csv files? These include counts of words, counts of sentences, sentiment metrics, readability measures, etc. To understand the provided numerical features, see Sec. 3.1.1 of the paper by Bird (2024).
  • Can you extract your own separate numerical features from the text?
  • Can you use smart reweighting techniques like term-frequency/inverse-document-frequency? See sklearn.feature_extraction.text.TfIdfVectorizer

Among classifiers, you can use any classifier in sklearn (nearest neighbor, random forest, MLP, etc.). You can also go beyond sklearn if you want. Be sure you understand enough about this classifier to discuss it clearly and define a reasonable hyperparameter search strategy.

As instructed below, your detailed steps for this Problem 2 will mimic Problem 1: you'll write sections like 1A (overall design of feature representation), 1B (overall CV experimental strategy), 1C (chosen classifier and its hyperparameter search strategy), and 1D (performance analysis).

For full credit, we expect that at least 2 parts out of 2A, 2B, and 2C explore substantially different methods than those used in Problems 1A, 1B, 1C. Each choice must be plausibly motivated by improving your classifier's performance.

2A : Feature Representation description

Include a paragraph describing and justifying how you used the provided features in x_*.csv (text and numeric fields) into fixed-length feature vectors suitable for classification.

Include enough detail that another student could roughly reproduce your work. If this process is substantially similar to 1A, it is OK to say so and keep this paragraph brief (you don't need to repeat yourself).

2B : Cross Validation (or Equivalent) description

Include a paragraph describing and justifying how you set up your training and hyperparameter selection process. Remember you can only use provided data in x_train.csv,y_train.csv. This will be assessed similarly to 1B.

For fair assessment, you can only use the provided x_train.csv,y_train.csv data for any models that you train. Use of other data sources for direct training is prohibited. Use of pretrained models is allowed, as long as they were not developed specifically for the purpose of reading-level assessment.

Include enough detail that another student could roughly reproduce your work. If this is substantially similar to 1B, it is OK to say so and keep this paragraph brief.

2C : Classifier description with Hyperparameter search

Include a paragraph describing and justifying which classifier you selected, how you trained it, and how you performed hyperparameter search (including concrete values you explored). Include enough detail that another student could roughly reproduce your work.

Include a figure that shows how classifier performance changes over at least one hyperparameter. Provide a paragraph caption interpreting the results of your search. In all these steps, please follow the common rubric for hyperparameter selection.

2D : Reflecting on Your Model Performance

Apply your best pipeline from 2A - 2D above to the test sentences in x_test.csv. Store your probabilistic predictions into a single-column plain-text file yproba1_test.txt, as described above under What to Turn In. Upload this file to our Problem 2 leaderboard.

For both your Problem 1 model and your Problem 2 model, report:

  • Your Test AUROC, as recorded on Gradescope.
  • The models' 3 most confidently-correct and 3 most confidently-incorrect predictions on heldout data. You do not need to include the entire passages, only the author and booktitle, as well as any other features/parts of the passage which are relevant to your concluding paragraph (see below).

The heldout part is important: if you use a fixed validation set for Problem 2, you can simply make predictions on the validation set to find the 3 most correct and incorrect. For Problem 1 (and if you use cross validation on Problem 2), you can make use of sklearn.cross_val_predict which tells you, for each data instance, what prediction it is given when it is in the heldout fold.

In a summary paragraph, discuss the relative performance of your two models. By investigating what each model gets right and wrong (and you're welcome to investigate beyond just the top-3's we request) make some hypotheses about either (a) what each model does better than the other, or (b) where both models are still struggling, and what key weakness you would try to fix if you were to continue working on this problem in the future.

Rubric for Overall Performance

We'll get a final number for this project by averaging:

  • 80% : your PDF report, using the rubric below
  • 5% : your Problem 1 submission, using the rubric below
  • 10% : your Problem 2 leaderboard submissions, using the rubric below
  • 5% : completion of your reflection on the project

Rubric for Evaluating Model Prediction Submissions

For both Problems 1 and 2, we've defined a particular threshold for a "reasonable" score (e.g. how we'd expect a standard pipeline trained using good practices to perform).

For Problem 1, 100% of your score is determined by whether you hit that score. For Problem 2, your score is broken down:

  • 85% of points are awarded for hitting the "reasonable score" threshold.
  • 15% of points awarded to the top 3 submissions in the class. Partial credit is awarded to other submissions, linearly interpolating between the "reasonable" score and the "top" score (e.g. a submission which is 2/3 of the way from reasonable AUROC to top 3 AUROC will get 10 of the 15 points).

Rubric for Evaluating PDF Report

Earning full credit on this assignment requires a well-thought-out report that demonstrates you made reasonable design decisions for feature preprocessing and classifiers and followed machine learning best practices throughout, especially for hyperparameter selection. Achieving top-scores on the leaderboard is far less important than understanding why some methods and choices outperform others.

Points will be allocated across the various parts as follows:

  • Problem 1: 40%
  • 10%: Paragraph 1A on Feature representation design decisions
  • 10%: Paragraph 1B on cross validation design decisions
  • 15%: Paragraph 1C on training and selection for your classifier
  • 5%: Paragraph 1D reflection on heldout performance
  • Problem 2: 60%
  • 15%: Paragraph 2A on Feature representation design decisions
  • 15%: Paragraph B on cross validation design decisions
  • 20%: Paragraph C on training and selection for your classifier
  • 10%: Paragraph D reflection both models' performance

Hyperparameter Selection Rubric

Paragraph on Search Experiment Design:

When you describe your hyperparameter search, we'll look for a paragraph that describes the design of your Hyperparameter Search Experiment:

  • 1-3 sentences: describe the potential advantages of the chosen classifier for the task at hand.
  • 1-3 sentences: describe which model complexity hyperparameter(s) were explored and how these values control model complexity. Describe why the chosen candidate value grids (or random distributions) are reasonable to explore the transition between under and over fitting and find the "sweet spot" in-between.
  • 1-3 sentences: describe any necessary details about the training process (e.g. are there convergence issues? step-size selection issues? should you stop early to avoid overfitting?)

Figure Requirements:

Your figure should show model performance on the metric you choose for model selection (y-axis) as a function of one hyperparameter value (x-axis). This hyperparameter should control model complexity. If you searched over multiple hyperparameters in constructing your model, you can fix all of them but 1 and make a plot by varying the last hyperparameter.

The figure should show performance across at least 5 distinct hyperparameter values, covering both underfitting and overfitting cases. That is, if at all possible, at least one candidate value should show clear underfitting and at least one should show clear overfitting.

Your figure should:

  • Show both training set and validation set performance trends in the same plot.
  • Show the typical performance at each hyperparameter via the average over multiple CV folds
  • Label each axis.
  • Include a legend.
  • Include a brief caption.

Your figure should also communicate uncertainty around this typical value, by exposing the variation across the multiple CV folds

  • A simple way to show uncertainty is show the empirical range across all folds, or the empirical standard deviation
  • A potentially better way to do this is show a separate dot for the direct performance of each fold (so 5 dots for 5 folds). Don't summarize when you can show the raw data.

The big idea here is that your figure should help the reader understand if one hyperparameter is definitely better than another (e.g. performance is better on most or all folds) or if there isn't much difference. Many plotting libraries have functions or arguments for producing neat error bars: e.g. matplotlib's errorbar or seaborn's errorbar and err_style arguments. If you add many overlapping lines and/or error bars, take some time to make sure the plot is still organized and easy to understand.

Paragraph on Interpreting Search Experiment Results

Provide a brief paragraph interpretating the results in your figure.

  • Which hyperparameter is preferred and why?
  • Is the evidence decisive, or uncertain?

General tips for Figures

Please do your best to keep figures close to the related paragraph, ideally on the same page.

If a figure contains multiple elements such as multiple lines or multiple sets of bars, please make sure that they are on the same scale. Alternatively, if different scales are necessary, adjust them appropriately to maintain reasonable and easily interpretable trends. For instance, it is inappropriate to represent a line fluctuating between 0 and 1 on the same scale as a line oscillating between 100 and 1000, as this would distort the representation and interpretation of the data points.