Module 08: Regularization & Model Selection - Participation Exercises

Open In Colab

Module 08: Regularization & Model Selection - Participation Exercises#

Exercise 8.1: Discussion - The Bias-Variance Tradeoff#

Type: 💬 Discussion (5 min)

Explain the bias-variance tradeoff to a partner as if they’ve never heard of it:

  1. What is bias? What is variance?

  2. Why can’t we minimize both at the same time?

  3. Give a real-world analogy (not from machine learning)

Challenge: Can you explain it in under 30 seconds?

Your explanation:

Exercise 8.2: Mini-Exercise - Ridge vs Lasso#

Type: 🔧 Mini-Exercise (7 min)

Observe how Ridge and Lasso affect coefficients differently.

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.preprocessing import StandardScaler

# Create data with some useless features
np.random.seed(42)
n = 100
X = np.random.randn(n, 5)
# Only first 2 features matter, rest are noise
y = 3*X[:, 0] + 2*X[:, 1] + np.random.randn(n)*0.5

feature_names = ['important_1', 'important_2', 'noise_1', 'noise_2', 'noise_3']

# Scale features
X_scaled = StandardScaler().fit_transform(X)

# Fit models with different regularization
models = {
    'OLS': LinearRegression(),
    'Ridge (alpha=1)': Ridge(alpha=1),
    'Lasso (alpha=0.1)': Lasso(alpha=0.1)
}

results = {}
for name, model in models.items():
    model.fit(X_scaled, y)
    results[name] = model.coef_

coef_df = pd.DataFrame(results, index=feature_names)
print("Coefficients:")
print(coef_df.round(3))

# TASK: 
# 1. Which model correctly identifies the noise features?
# 2. When would you prefer Ridge over Lasso?

Coefficients:
               OLS  Ridge (alpha=1)  Lasso (alpha=0.1)
important_1  2.712            2.680              2.584
important_2  2.025            1.999              1.905
noise_1     -0.010           -0.006              0.000
noise_2      0.077            0.074              0.000
noise_3     -0.004           -0.007             -0.000

Your observations:

Exercise 8.3: Critique - Cross-Validation Mistakes#

Type: 🔍 Critique (5 min)

Find the data leakage in this cross-validation workflow:

from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import Ridge

# BUGGY cross-validation workflow
# X, y = load_data()

# Step 1: Scale all data
# scaler = StandardScaler()
# X_scaled = scaler.fit_transform(X)  # <-- PROBLEM HERE!

# Step 2: Cross-validate
# scores = cross_val_score(Ridge(), X_scaled, y, cv=5)

# TASK: What's wrong with this workflow?
# How should it be done correctly?

The problem:

The fix:

Contents