Homework 13: Model Interpretability

Open In Colab

Homework 13: Model Interpretability#

Interpret machine learning models to understand what they learned.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
from sklearn.model_selection import train_test_split
from sklearn.inspection import PartialDependenceDisplay, permutation_importance
import shap

Dataset: Reactor Performance#

Predict reactor conversion from operating conditions.

# Load reactor data from URL
url = "https://raw.githubusercontent.com/jkitchin/s26-06642/main/dsmles/data/hw13_reactor_data.csv"
data = pd.read_csv(url)

feature_names = ['temperature', 'pressure', 'catalyst_loading', 'residence_time', 'feed_purity']
X = data[feature_names]
y = data['conversion']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train model
model = GradientBoostingRegressor(n_estimators=100, max_depth=4, random_state=42)
model.fit(X_train, y_train)
print(f"Test R²: {model.score(X_test, y_test):.3f}")

Test R²: 0.887

Problem 1: Feature Importance#

1a. Plot the built-in feature importances from the Gradient Boosting model.

# Your code here

1b. Calculate permutation importance on the test set. How does it compare to the built-in importance?

# Your code here

1c. Why might permutation importance give different results than built-in importance?

Your answer here:

Problem 2: Partial Dependence Plots#

2a. Create partial dependence plots for temperature and pressure.

# Your code here

2b. Describe the relationship the model learned between temperature and conversion. Is it linear?

Your answer here:

2c. Create a 2D partial dependence plot for temperature and pressure together. Is there an interaction?

# Your code here

Problem 3: SHAP Values#

3a. Calculate SHAP values for the test set using TreeExplainer.

# Your code here

3b. Create a SHAP summary plot. Which features have the largest impact?

# Your code here

3c. Pick a specific test sample and create a SHAP waterfall plot. Explain what each bar means.

# Your code here

3d. Based on the SHAP analysis, what recommendations would you give a plant operator to maximize conversion?

Your answer here:

3e. A colleague says “The model found that high temperature increases conversion, so we should run at 600 K.” What concerns would you raise?

Your answer here:

Contents