HW2: Cross-Validation, Regularization, and Hyperparameter

Last modified: 2026-02-01 12:04

Status: RELEASED

Due date: Wed Feb 4, 2026 by end of day (11:59 pm ET) in Medford, MA

Jump to: Background   Starter Code   Problem 1   Problem 2   Problem 3

Turn-in links:

• PDF report turned in to: https://www.gradescope.com/courses/1220989/assignments/7560308
• CODE files turned in to: https://www.gradescope.com/courses/1220989/assignments/7561899
• Complete your reflection here: https://forms.gle/hRv99u7pwb8xpJbs7

Files to Turn In: Your code submission should contain this file, without any folder structure:

• hw2.ipynb

Your report PDF should be typeset using an application of your choosing (e.g. Word, Google Docs, Latex, etc.). It should not include code, but can include pasted code outputs.

Evaluation Rubric:

• 90% PDF submission (results + conceptual question)
• 5% reflection
• 5% python notebook submission

Background

To complete this HW, you'll need some specific knowledge from the following sessions of class:

• Hyperparameter selection & cross validation (day04)
• Regularization (day05)

Starter Code

See the hw2 folder of the public assignments repo for this class: https://github.com/tufts-ml-courses/cs135-26s-assignments/tree/main/hw2

This starter code includes the file hw2.ipynb, which will help you organize your analysis for the report and provides starter code for loading in data and getting started.

You can work on this notebook locally on your computer (recommended). You can also use Colab or other cloud environments, though you'll have to upload your completed .ipynb file.

All questions should be answered in your PDF report, not your notebook. We ask you to submit your code only to double-check your work.

Coding notes

Try to take advantage of sklearn and numpy functionality where possible to save yourself time. If you're not sure where to start, take a look at the labs. They show examples of a lot of the functionality you need to use in this homework.

All errors should be expressed using root mean squared error (sklearn.metrics.root_mean_squared_error) and show at least 2 decimal places.

Normalizing Inputs and Sanitizing Outputs

As discussed in class, when running a penalized regression model such as ridge regression we want to make sure our features are normalized so that their overall scale does not influence the model. It turns out this is also a good practice for unpenalized regression as well; you'll explore why in question 2C. Use scikit-learns MinMaxScaler method for this purpose.

When making predictions on a physical property like MPG, there are values we know are physically impossible or implausible. A common practice is to sanitize model predictions to force them to be in a plausible range. You have been given a function, sanititze in the starter code that performs this opperation on a vector. Be sure to sanitize your model predictions whenever you are calculating error.

Prediction Task: Predicting Miles-per-Gallon efficiency from Vehicle Engine Properties

We'll return to the MPG efficiency dataset we looked at towards the end of HW1. To refresh, this is a data set containing ratings of power, size, and other information for 395 makes and models of vehicles. For each vehicle, we have the following information:

You have been asked to build a predictor for fuel efficiency (measured in miles per gallon or "mpg") as a function of other vehicle characteristics.

In the starter code, we have provided an existing train/validation/test split of this dataset, stored on-disk in comma-separated-value (CSV) files: x_train.csv, y_train.csv, x_valid.csv, y_valid.csv, x_test.csv, and y_test.csv.

• x_train.csv and y_train.csv contain features and outcomes for 192 examples
• x_valid.csv and y_valid.csv contain features and outcomes for 100 examples
• x_test.csv and y_test.csv contain features and outcomes for 100 examples

Get the data files here

Models

You will be implementing the following models to answer questions 1-3. Read through all the questions before you implement the models to make sure you understand how you're going to need to use them.

Model 0: Baseline Guess-Train-Set-Mean

It is a good engineering practice to try simple approaches before complex models. Here's a very simple regression model: given a training set of size \(N\), compute the mean of all response values \(y_i\), and store that as \(\bar{y}\). The prediction function of this model is then

That is, no matter what feature vector \(x_*\) is provided, always guess the train-set-mean \(\bar{y}\). We considered this model in class as a linear regressor with no input features.

In the starter notebook, to implement this model you simply need to compute the mean of training set \(y\) values.

Model 1: Polynomial Regression

Model 1 is an unpenalized polynomial regression model. You will need to implement models with different max degree polynomial transformations.

It is recommended that you use the pipeline creation helper function provided for you in the starter notebook. Your code should chain together the PolynomialFeatures and LinearRegression classes provided by sklearn.

You can find an example of using a pipeline in the day04 lab.

Model 2: Penalized Polynomial Regression

Model 2 is an L2-penalized polynomial regression, also known as a ridge regression. You should use the provided pipeline code as a basis to chain together the PolynomialFeatures and Ridge implementations provided by sklearn.

Report Questions

Question 1: Investigating Polynomial Regression

In Question 1, consider only the unpenalized polynomial regression (model 1). Use the included fixed training, validation, and testing sets.

Question 1A: Interpreting learned parameters

Fit a linear regression model to a polynomial feature transform when degree = 1.

Provide the weight coefficient values (to 2 decimal places) for each of the \(F\) features of your degree = 1 model. With these values, which feature has the highest positive impact on MPG? Which has the highest negative impact? Do these make sense?

Hint: Engine displacement refers to the overall volume of air that can move through the engine. Larger engine means larger displacement.

Question 1B: Comparing learned parameters

Fit a linear regression model to a polynomial feature transform when degree = 4.

Provide the weight coefficient values (to 2 decimal places) for each of the \(F\) features of your degree = 4 model.

Compare the 1-degree vs. 4-degree learned model parameters. Which model tends to have larger parameters? Why do you think that is?

Question 2: Fixed-Set Hyperparameter Searches

We'll now perform a "complete" experiment for our polynomial regression model and polynomial ridge regression models.

• For the polynomial regression model, try all possible degrees from 1 to 7 (inclusive).

• For the penalized model, fix degree=4. Consider possible alpha values:

alpha_list = np.asarray([1.e-10, 1.e-08, 1.e-06, 1.e-04, 1.e-02, 1.e+00, 1.e+02, 1.e+04, 1.e+06])

For each model you'll want to build a fresh pipeline, fit the model of specified degree or alpha value, and record the train error and the validation error, finally, find the best perfoming model on the validation set and save it's performance on the test set.

Question 2A: Figure 1 in Report: Error vs degree

Using the included plotting code, make a line plot of RMSE on y-axis vs. polynomial degree on x-axis. Show two lines, one for error on training set (in blue) and one for error on validation (in red) for the polynomial regression model. Please do not change any styling from the provided code.

Provide a brief caption that answers the two questions:

• (i) Based on this plot, what degree value do you recommend we use to "deploy" the model on new instances?

• (ii) At what degree values (if any) do you see signs of overfitting? Explain what you see that suggests overfitting.

Question 2B: Figure 2 in Report: Error vs. alpha

Make a line plot of RMSE on y-axis vs. alpha on x-axis. Show two lines, one for error on training set (in blue) and one for error on validation (in red) for the ridge polynomial model.

Since alpha is on a log scale, we should transform the plot to also have a log scale with plt.gca().set_xscale('log') after the plotting function is called.

Provide a brief caption that answers:

Does this plot look as you expect based on course concepts? What specific alpha value do you recommend if we want to "deploy" on new instances and get the lowest possible error?

Question 2C: Removing MinMaxScalar

The provided pipeline uses a MinMaxScalar preprocessor to rescale each of the raw input features to values between 0.0 and 1.0 before the polynomial transformation.

What happens to training set error as a function of degree when MinMaxScalar preprocessing is omitted for the unpenalized regression model? Can we make sense of this trend using the concept of overfitting, or is there some other explanation? For a LR model with polynomial features and degree above 2, why might it be useful to rescale each raw input feature (like weight or displacement) to be in the interval 0.0 to 1.0? Hint: The behavior might not match what we theoretically expect to happen for an unpenalized regression model.

Question 3: Hyperparameter Searches with Cross-Validation and final analysis

Instead of fixed training and validation sets, we will combine the training and validation sets to use as a large "development" set that should contain 292 examples total for cross validation. You can merge the data with np.hstack and np.vstack:

x_trva_LF = np.vstack([x_tr_MF, x_va_NF])
y_trva_L = np.hstack([y_tr_M, y_va_N])

For each possible alpha value as well as each possible polynomial degree listed below (i.e. every possible combination of alpha and degree), train and evaluate a Ridge regression model across the entire train+validation set using 10-fold cross validation.

degree_cv_list = [1, 2, 3, 4, 5, 6, 7]
alpha_cv_list = np.logspace(-10, 6, 17)

To perform cross-validation, you must use the scikit-learn function KFold and include the arguments shuffle=True and random_state=SEED. If you do not do this your values will not match what we expect in the results. Your cross validation loop should look something like:

    kf = sklearn.model_selection.KFold(n_splits=K, shuffle=True, random_state=SEED)
    for train_ind, val_ind in kf.split(x_trva_LF, y_trva_L):
        ...fit and calculate errors on your model

For each possible hyperparameter configuration (alpha value and degree value), your 10-fold CV procedure will give you an estimate of the training error and heldout validation error across all K folds. Compute the mean validation error across all K folds to get your estimated cross-validation error.

Question 3A: Hyperparameter grid-search

Select the model hyperparameters that minimize your estimated cross-validation error. That is, the average validation error across the K different train-val splits. Report the best-performing hyperparameter values and the corresponding training and validation set errors.

Using these best hyperparameters, retrain the model using the full development set (as shown above). Then compute that (retrained) model's error on the test set.

Question 3B: Table 1 in Report

In one neat table, please compare the test set root-mean-squared-error (RMSE) performance for the following regressors:

• Baseline: Always guess the mean \(y\) value of the training set, regardless of the new instance features
• The best Poly+Linear pipeline, picking degree to minimize val set error
• The best Poly+Ridge pipeline (from Question 2), fixing degree=4 and picking alpha to minimize val set error
• The best Poly+Ridge pipeline (from Question 3), picking degree and alpha to minimize 10-fold cross validation error

Your table should include the model name, chosen hyperparameter values, test error, validation error, and training error.

Below the table answer the questions:

• (i) What are two possible reasons the best ridge polynomial regression model from Question 3 would outperform the best ridge polynomial regression model from Question 2?
• (ii) Why can't we just select degree/alpha values to directly minimize test set error?