HW3: Classification

Last modified: 2026-02-08 19:35

Updates

• 2026-02-06 11:00: Starter code was updated to include the line X_dev = X_dev.drop(columns=cols_to_drop) when dropping columns. If you downloaded hw3.ipynb before this time please add this line to your code, or re-download.
• 2026-02-06 19:00: 2A ROC curve should be on the validation set, not the training set.
• 2026-02-06 19:00: Clarified tie-breaking.

Status: RELEASED

Due date: Wed Feb 11, 2026 by end of day (11:59 pm ET) in Medford, MA

Jump to: Background   Starter Code   Dataset   Problems

Turn-in links:

• PDF report turned in to: https://www.gradescope.com/courses/1220989/assignments/7605011
• CODE files turned in to: https://www.gradescope.com/courses/1220989/assignments/7604968
• Finally, complete your reflection here: https://forms.gle/C9yJDbPtgGt8wVLi6

Files to Turn In: Your code submission should contain this file, without any folder structure:

• hw3.ipynb

Your report PDF should be typeset using an application of your choosing (e.g. Word, Google Docs, Latex, etc.). It should not include code, but can include pasted code outputs.

PDF report is a short human-readable report that will be manually graded.

• The report should be no more than 5 pages.
• Can use your favorite report writing tool (Word or G Docs or LaTeX or ....)
• Should be human-readable. Do not include code. Do NOT just export a jupyter notebook to PDF.
• Always provide explanations for your answers. Responses that do not include justification or reasoning will result in a deduction of points.
• For any questions that ask you to report values, all results should keep 3 digits.
• When you submit to Gradescope, be sure to mark each subproblem via in-browser Gradescope annotation tool

Evaluation Rubric:

• 90% PDF submission
• 5% reflection
• 5% python notebook submission

Background

In this HW, you'll perform an analyses using logistic regression and k-nearest neighbors.

You'll examine how different factors effect evaluation and model implementation in cervical cancer risk prediction and algorithmic decision support in a medical setting.

To complete this HW, you'll need some specific knowledge from the following sessions of class:

• Classification Basics (day06)
• Evaluating Classifiers (day07)

Starter Code

See the hw3 folder of the public assignments repo for this class: https://github.com/tufts-ml-courses/cs135-26s-assignments/tree/main/hw3

This starter code includes code for loading, cleaning, and imputing missing data (where needed) for each dataset. Do not alter this code, as it may change your results as compared to what we expect when grading. You will need to run the code and we encourage you to read through the starter code to understand how we are handling missing data in this situation.

Also included are some functions you may find helpful during your analysis. Similar to homework 2, we give you a function that creates a pipeline which you should use for your analysis.

Dataset Cervical Cancer Screening

We are examining a dataset of cervical cancer risk factors collected at 'Hospital Universitario de Caracas' in Caracas, Venezuela, comprising demographic information, habits, and historic medical records of 858 patients. The original dataset contained 4 target binary variables consisting of 4 different Cervical cancer diagnostic tests.

Your task is to create classification model that predicts whether or not a patient should get a follow-up biopsy, assuming that if that biopsy is positive we were correct in recommending that a biopsy should take place. We will thus only focus on the biopsy target variable. We will include the other diagnostic test results as a features: they consists of less invasive screen tests that are often performed as a part of routine medical care such as cervical cytology, also referred to as a Pap test. This screening test is typically recommended for all patients who can get cervical cancer from the ages of 21-65, so it is conceivable that we have a cytology result for every patient in the dataset as it was collected from patients receiving health care.

One confounding factor in this dataset is that several patients decided not to answer some of the questions because of privacy concerns, thus we have missing data for some features.

We have already split the data into training, test, and validation sets for you. Note that the merged training and validation datasets are referred to as the development set.

Problems

Problem 1: Model Creation and Training

1A: Summary Statistics

Calculate and report the class balance in the development set; the proportion of the positive and negative classes. What would the accuracy of a baseline model at always predicts the negative class be in this scenario? Why isn't this model good enough to use for our cancer screening task?

1B: Logistic Regression

Train a logistic regression model with \(L_2\) regularization using the pipeline creation function included in the starter code. Perform a hyperparameter search using the training and validation sets, selecting the hyperparameter combination that achieves the highest AUCROC on the validation data. Perform a grid search (check all combinations of hyperparameter values) of the regularization strength C and the n_neighbors (k) hyperparameter used for imputing missing data. Note that because our missing data imputation method uses a hyperparamter, we need to search over that in addition to hyperparamters for our main model.

Check the following ranges of C and n_neighbors values:

C_grid = np.logspace(-4, 4, 17)
k_grid = range(1, 18)

You can use sklearn's roc_auc_score function to score your model on the validation data.

Report the best found C, n_neighbors for imputation, and validation AUCROC. In the case of a tie select the model with a smaller imputation hyperparamter, with a secondary tiebreaker of the smaller regularization parameter. Hint: This and 1C may take a few minutes to run depending on your computer. You might want to do a dry-run where you just print out parameter combinations without actually training models to make sure your code is setup correctly before performing the true run.

1C: K-nearest Neighbors

Train a k-nearest neighbors model using the pipeline creation function included in the starter code. Perform a hyperparameter search using the training and validation sets, selecting the hyperparameter combination that achieves the highest AUCROC on the validation data. Perform a grid search (check all combinations of hyperparameter values) of the regularization the n_neighbors hyperparameter used for the actually classifier and the n_neighbors hyperparameter used for imputing missing data. Both should search the same k_grid used above, but be sure that you are checking all combinations of different n_neighbors values. In the case of a tie select the model with a smaller imputation hyperparamter, with a secondary tiebreaker of the smaller classifier n_neighbors value. You may use the same sklearn method mentioned above.

Report the best found n_neighbors for the classifier, n_neighbors for imputation, and validation AUCROC.

Problem 2: Choosing a decision threshold from binary classification metrics

2A: ROC Curves

Plot ROC curves for the best performing logistic regression and k-nearest neighbors models from 1B and 1C on the validation set in the same plot, with the logistic regression model using a red line ("r-") and the k-nearest neighbors model using a blue line ("b-"). Be sure to label your axes (plt.xlabel() and plt.ylabel()) and include a legend (plt.legend()).

You should use the sklearn roc_curve function to perform this step, passing the first two return values into plt.plot. To understand how to use this function, consult the function's User Guide and documentation.

Under the figure, in a sentence or two discuss the ROC curves. Does one model generally outperform the other, or does a different model dominate at different points along the curve?

2B: Selecting a decision threshold

To implement our model for medical decision making we need to choose a threshold at which a biopsy is recommended. One way we can do this is to maximize a binary classification metric under some other performance threshold. For instance, here we may want to maximize the number of true cancer patients found while limiting the number of unneeded biopsies performed.

Note that roc_curve also returns a list of decision thresholds at each point in the ROC curve. For each of the two trained models, choose a threshold that maximizes TPR while satisfying TNR >= 0.9 on the validation set. Report the two chosen decision thresholds, their corresponding TNR and TPR rates at those thresholds, and which model performs better under this decision thershold. Hint: Recall that \(TNR = 1 - FPR\)

2C: Test-Set Performance

Using the better-performing model and threshold from 2B, train your model on the full development set and make binary predictions on the test set. Compare these predictions to the corresponding true labels by printing out the confusion matrix (either in a table or as the TP, FP, TN, and FN counts). You can calculate the confusion matrix using sklearn's confusion_matrix function. Be sure to label which of the 4 values is which.

Problem 3: Choosing a decision threshold based on cost

Different correct and incorrect classification metrics are often associated with a cost. For instance, here, we can consider cost of a false negative as the change in mortality based on how much more likely a person is to survive if they receive early treatment for cervical cancer, and the cost of a false positive as the small chance for complications during a biopsy.

We often consider these costs in terms of proportional risk (i.e. a false negative is X times worse than a false positive), but for this problem we will extrapolate our model performance as if it were used on the female population of Venezuela where this dataset was collected. Note that normally there are more accurate but complex ways we would want to extrapolate model performance to an entire population, but for the purposes of this analysis we'll assume that the data we have is representative of the Venezuelan population.

Venezuela's female population is about 14 million (which we can write in python as 1.4e7). We will approximate that beginning cervical cancer treatment early results in a 70% reduction in a patient's mortality (\(c_{fn}\)), and that complications from a biopsy result in a 0.5% increase in mortality for a patient that does not have cancer (\(c_{fp}\)).

We also need the base rate, i.e. the proportion of positive instances in the validation set. We can thus calculate the extrapolated total increase in mortality \(c_t\) from our model compared to a theoretical perfect model as:

c_fn=0.7
c_fp=0.005
population=1.4e7
fn = fnr*base_rate
fp = fpr*(1-base_rate)
c_t = ((fn * c_fn) + (fp * c_fp))
excess_deaths = c_t*population

3A: Baseline cost

Calculate the extrapolated mortality cost for two baseline models, a model that predicts all positive and a model that predicts all negatives on the validation set. Report these two costs.

3B: Calculating Mortality

Find the decision thresholds for each model that minimizes the extrapolated mortality cost on the validation set (using the same hyperparameters you found in question 2). Report the decision threshold, tpr, fpr, and extrapolated mortality cost for the best performing logistic regression threshold and k-nearest neighbors threshold.

Which model does better? How does each model compare to the baselines? Considering the better model, do you think this threshold is realistic or would we need to consider other factors? Hint: what would the consequences be if we used this model to decide treatment on the entire population?

3C: Test Set Performance

Train the model you consider better from 3B on the entire development dataset. Using the chosen threshold for this model from 3B, now make binary predictions on the test set and compare them to corresponding true labels by printing out the confusion matrix (either in a table or as the TP, FP, TN, FN). Be sure to label which of the 4 values is which.

Calculate the mortality cost of this model and compare it to the total mortality cost of a baseline all positive baseline on the test set. Does it do better or worse? Does it do relatively better or worse than the model performed on the validation set?

3D: Changing Base Rates

We refer to the proportion of a population that has a particular condition as the prevalence of that condition. Almost all instances of cervical cancer are caused by a long-term infection of Human papillomavirus (HPV). While some form of HPV vaccine has been available since 2006, Venezuela has still not incorporated the HPV vaccine into its national vaccination program. Estimates are that vaccination at a young age can reduce the prevalence of cervical cancer by almost 90%.

Assuming other factors are equal, explore how your optimal decision threshold would change for both models if the HPV vaccine was rolled out and the prevalence of cervical cancer was reduced by 90%. We can simulate this by re-calculating cost with a base positive instance rate by divided by 10, as if the class balance was changed by a factor of 10.

Using the validation set, report the new decision threshold for your kNN and logistic regression models with this new base rate, and the TPR and FPR at these new thresholds. How did the change in the base positive instance rate affect the decision thresholds (you might need to look past 3 digits to see a change)? Why do you think that is the case?