What today is about

1. Last time, slower: model capacity, validation curves, learning curves
2. Classification: predicting a category, on the plant
3. Measuring a classifier: the confusion matrix, precision and recall, the threshold
4. Tracking and search: Optuna runs the search, MLflow records every trial

Last time, four questions to ask of any model

Four questions to ask of any model, last week's or your own:

What does it know?

Only its data.

The polynomial scored R² = 0.9999975 on the water data, and gave 223 MPa at 300 °C, where NIST gives 517.7.

What did training solve?

An optimization problem, and the family picks it.

A line has one minimum and a network many; a GP tunes a few bounded hyperparameters.

What did the score measure?

The question your split asked.

The tree beat the line on random folds and lost to it by 2 MPa on grouped folds.

What is holding it back?

A gap is variance, a high level is bias.

The tree kept a gap of 8.5 MPa. The line closed its gap but ended 1.4 MPa above a model with more capacity.

Start simple, and put what you know into the features: a line with two physics features came within 0.3 MPa of a Gaussian process on grouped folds.

Model capacity

Capacity is the range of functions a model can represent. More capacity fits more complicated relationships, and more of the noise.

Every family from last time turns a different knob:

• Polynomial degree, for the line
• Tree depth
• The number of hidden units, for the network
• The ridge or lasso penalty
• A kernel's length scale, for the Gaussian process

Model capacity, overfitting and underfitting

A model overfits when its training error is much lower than its validation error. It has learned the noise and the particular rows it was given. It underfits when both errors are high and close together, because it is too simple for the relationship.

The bias-variance trade-off: more capacity lowers bias and raises variance, so the validation error is lowest somewhere in between.

Hastie, Tibshirani and Friedman, section 7.3, eq. 7.9

Model capacity, validation curves

A validation curve plots training and validation error against one capacity knob, with the data held fixed.

• Training error keeps falling as a tree gets deeper, and validation error stops improving past depth 9

Model capacity, learning curves

A learning curve plots training and validation error against training-set size, model fixed.

Classification, what it is

• Regression (Lecture 9) predicts a number: a strength, a pressure
• Classification predicts a category: normal or faulty, pass or fail
• Most classifiers work in two steps: a probability, then a threshold

Classification, the plant

The Tennessee Eastman process (TEP), the simulated plant from Lecture 5 on. The question: is the plant faulty right now?

Fault 4: the cooling water entering the reactor gets warmer, so the controller opens the cooling water valve, xmv_10, further. P&ID: Lyu et al. (2026)

Classification, logistic regression

Classification, one cut on the valve

Below 43.4% open: normal. Above: fault 4. On 51,000 test samples, this one cut is wrong once.

Classification, fault 14

Fault 14: the same valve sticks. The controller pushes, the valve jumps too far, the controller pushes back: big swings around the same average.

P&ID: Lyu et al. (2026)

Classification, fault 14, one cut or two

Classification, four model families

Replay all

Step 3000

Logistic regression

Minimizes the log loss, also called cross-entropy: small when the true class gets a high probability.

Depth 3

Decision tree

Splits to lower the Gini impurity of each box: zero when a box holds one class.

Trained (172 iterations)

Neural network

A softmax turns its scores into probabilities; training minimizes the cross-entropy.

150 points

Gaussian process

A smooth random function goes through a link . It also gives an uncertainty.

Measuring a classifier, accuracy

• Imagine a "detector" that answers normal for every sample, whatever the data. On the 59,000 test samples:

• Accuracy 85.4%, yet it never catches a single fault
• So we need metrics that tell the two mistakes apart: missed faults and false alarms

Measuring a classifier, the confusion matrix

The numbers: a neural network on all 52 channels, trained on normal runs and nine faults, tested on the 59,000 test samples (any of the nine faults counts as a fault), threshold 0.5.

Measuring a classifier, precision and recall

• Why two numbers: an alarm you cannot trust (low precision) and a fault you miss (low recall) cost different things
• The network: of its 8,313 alarms, 8,300 were real faults, so precision is 0.998
• Of the 8,640 real faults, it caught 8,300, so recall is 0.961

Measuring a classifier, four classifiers

• Always normal never predicts a fault, so TP = 0 and recall is 0
• It raises no alarm at all, so precision is 0/0, which scikit-learn reports as 0
• Accuracy barely separates the last three rows. Recall does.

Measuring a classifier, the threshold

The slider moves the threshold on the network's probability of fault, to show what each choice costs. In scikit-learn, model.predict(X) gives the class (threshold 0.5); model.predict_proba(X) gives the probability, so you pick the threshold.

Threshold ↓: more alarms, recall ↑, precision ↓. Threshold ↑: fewer alarms, precision ↑, recall ↓.

Measuring a classifier, faults it never saw

A classifier only catches new faults that look like the ones it was shown.

Tracking and search, back to Lecture 9

Lecture 9 picked the tree's depth by eye. Its RMSE (root mean squared error) over grouped folds: 9.42 MPa with no depth limit, 9.10 MPa at depth 9.

• : the hyperparameters, such as the tree depth and the minimum leaf size
• : the model's parameters, the tree's splits, found by training
• Every evaluation of the outer problem trains a model: a search needs many, and a record of each

Tracking and search, Optuna and MLflow

Tracking and search, grid versus random

Two hyperparameters, nine trials for each search. Only the x-axis hyperparameter matters in this example.

The bump (made up for the picture): the validation score against the x-axis hyperparameter. Its peak is the best setting.

Takeaway: same 9 trials, but random search tries 9 values of what matters and grid only 3, so random gets closer to the best.

Tracking and search, how Optuna picks the next trial

Bergstra et al. (2011), where TPE comes from

Tracking and search, Optuna on the concrete strength dataset

Each dot: one trial's validation RMSE. Each line: the best RMSE so far, so it only goes down.

Green dotted lines: Lecture 9's two trees, 9.42 and 9.10 MPa. TPE's best, 8.90 MPa, beats both.

Tracking and search, the run hierarchy

A search trains a model once per trial. MLflow keeps every one of them, in one place:

Each child run is one trial: a new max_depth and min_samples_leaf, and the validation RMSE they give. The data, the folds and the model stay the same. The registry holds the model to use, with a name and a version anyone can load.

Tracking and search, one trial in code

def objective(trial):
    params = dict(
        max_depth=trial.suggest_int('max_depth', 2, 20),
        min_samples_leaf=trial.suggest_int('min_samples_leaf', 1, 50),
    )
    model = DecisionTreeRegressor(
        random_state=SEED,
        **params,
    )
    rmse = -cross_val_score(
        model,
        X_tr,
        y_tr,
        groups=groups_tr,
        cv=GroupKFold(5),
        scoring='neg_root_mean_squared_error',
    ).mean()
    with mlflow.start_run(nested=True):
        mlflow.log_params(params)
        mlflow.log_metric('val_rmse', rmse)
    return rmse

Tracking and search, the search in code

parent = mlflow.start_run(run_name='concrete-tree-search')
study = optuna.create_study(
    direction='minimize',
    sampler=optuna.samplers.TPESampler(seed=SEED),
)
study.optimize(
    objective,
    n_trials=20,
)

winner = DecisionTreeRegressor(
    random_state=SEED,
    **study.best_params,
).fit(X_tr, y_tr)
mlflow.sklearn.log_model(
    winner,
    name='model',
    registered_model_name='concrete-tree',
    skops_trusted_types=['sklearn.tree._tree.Tree'],
)
mlflow.end_run()

Tracking and search, test once

Report the test number, 7.41 MPa. The test mixes never took part in the search.

It is lower than 8.90 here: 195 rows from 86 mixes is a small test set, and its score depends on which mixes landed in it.

Recap

Four questions this session added to Lecture 9's four:

What turns a number into a category?

A threshold on a probability.

Fault 4's boundary sits at 43.4% open; fault 14 needs more than one straight cut.

How do you judge a detector?

Its confusion matrix, not accuracy alone.

The baseline scores 85.4% accuracy and 0 recall; precision and recall expose it.

What can a classifier recognize?

New faults only if they look like the ones it was shown.

Two faults it never saw: fault 18 was caught 92% of the time, fault 19 only 0.1%.

How do you trust a search?

Track every trial, test the winner once.

TPE reached 8.90 MPa at trial 11, random search 9.00 at trial 38. The test set gave 7.41.

Report the confusion matrix with its threshold, track every trial, and test once.

This week

Practice module for this session, for participation credit
Demos l10-classification.ipynb and l10-tracking-search.ipynb, to rerun after class