What today is about

Lecture 7 built a table and fitted a model. It never asked whether the model was any good.

Today:

1. Time series as supervised learning, with a horizon
2. What makes a series forecastable at all
3. Baselines: better than what?
4. Direct and recursive forecasts
5. Evaluating on time
6. Residuals as a detector, and the limits
7. The notebook, then questions

Why this matters

Fit a one-step model on reactor pressure: predictions hug the data, R-squared near 0.99.

• "nothing changes in 3 minutes" is already close to right (Lecture 7)
• most of that score belongs to the free guess
• an operator needs 30 to 60 minutes of warning, not 3

A model is worth keeping only if it beats the free guesses, at the horizon somebody needs, on data it never saw.

Why this matters, beyond the plant

Same three questions everywhere: how far ahead, better than what, scored how.

Supervised learning

• Lecture 7 already did this: lstsq on a design matrix
• scikit-learn: every model has fit(X, y) and predict(X)
• linear model, random forest, boosting: swap in one line

Supervised learning, two kinds of number

• a dataset you used to choose hyperparameters is no longer a fair test of them
• so: train fits parameters, validation picks hyperparameters, test is touched once
• held-out score: the error on rows the fit never saw. Lecture 7 never computed one.
• honest only if the held-out rows look like the future the model will face, the hard part for a time series

Supervised learning, the horizon

cols = {f"y[t-{k}]": pl.col("xmeas_7").shift(k).over(RUN) for k in range(10)}
cols["target"] = pl.col("xmeas_7").shift(-h).over(RUN)   # the only new line

.over(RUN): Lecture 7's run-boundary rule, still required.

Supervised learning, what is known at time t

Anything in the right column is leakage, stretched across rows.

Forecastability, autocorrelation

• plot it against : the ACF
• tells you before fitting whether the past can help
• statsmodels.graphics.tsaplots.plot_acf, or three lines of numpy

Forecastability, three kinds of series

Pressure and separator level: run 1, Rieth et al. (2017). Random walk: simulated. figures/make_figures.py

Forecastability, what each one calls for

7 of 22 continuous TEP channels look like separator level: white noise.

Forecastability, stationarity

• every model here assumes the future looks like the past
• fix for a drifting level: differencing, model y[t] - y[t-1]
• finance: prices to returns; the "I" in ARIMA

Forecastability, trend and seasonality

• Trend: a sustained change in level (demand growing year on year)
• Seasonality: a repeating calendar pattern (load peaks every weekday evening)
• Standard fixes: seasonal differencing, calendar features

A plant at an operating point has mostly neither.
A daily cycle there usually has a measured cause (cooling water temperature): put that column in the table, not the hour.

Hyndman and Athanasopoulos, FPP3, section 9.1

Forecastability, a question

Baselines

FPP3, section 5.2

Baselines, which one wins

For a stationary series with spread and autocorrelation :

• persistence wins exactly when
• white noise (): persistence is worse; separator level measures 1.415
• negative : persistence bets on the wrong side of the swing

Baselines, reactor pressure

Train runs 1 to 300, test runs 401 to 500. Test standard deviation 7.51 kPa. figures/make_figures.py

Baselines, reactor pressure, the numbers

RMSE in kPa. The model earns most where neither free guess is good.

Baselines, skill scores

• reference = the better baseline at that horizon
• MASE: mean absolute error divided by the in-sample one-step naive error; below 1 beats naive, unitless, averages across series
• also report kPa: an operator can judge 5.0 kPa, not 0.14

Hyndman and Koehler (2006), author's copy

Baselines, a question

Baselines, case study: Meese and Rogoff

Exchange-rate models of the 1970s (money supply, interest rates, trade balances):

• forecast 1 to 12 months ahead, re-fit with rolling regressions (only past data)
• reference: a random walk
• "a random walk model would have outperformed all the other models"
• also lost: univariate models, a VAR, optimal combinations of the forecasts

They fitted history well. Only the baseline comparison showed they could not forecast.

Meese and Rogoff, Fed IFDP 184 (1981), the working paper of the 1983 J. Int. Econ. article

Baselines, case study: the M4 competition

100,000 real series, every team scored the same way. Benchmarks included Naïve2 and Comb (three simple exponential smoothers).

• 12 of the 17 most accurate methods: combinations of mostly statistical methods
• the six pure ML methods: none beat Comb, one beat Naïve2
• best surprise: a statistical/ML hybrid, about 10 % better than Comb

Makridakis, Spiliotis and Assimakopoulos (2018), IJF 34, abstract

Model families

• they meet at the linear model: ridge on 10 lags is an AR(10)
• statsmodels for the first route, scikit-learn and skforecast for the second

statsmodels tsa · scikit-learn lagged features

Model families, ridge and scaling

• lag columns are near-copies (0.996 on pressure, Lecture 7)
• ridge penalizes coefficient size, which stabilizes the fit
• the penalty is unit-dependent: scaling now changes the answer
• fit the scaler on training rows only, so put it in the pipeline

model = make_pipeline(StandardScaler(), Ridge(alpha=1.0))
model.fit(X_train, y_train)     # scaler sees training rows only

Multi-step strategies

• recursive: one model, the whole path, but it runs on its own guesses
• errors compound: this is "running free" from Lecture 7
• recursive needs the whole future input path u[t+1] ... u[t+h-1]; direct needs only what is known at t

Multi-step strategies, measured

Test RMSE in kPa, runs 401 to 500. Same 10 lags, same ridge.
At two hours, recursive is worse than the mean.
Adding the 11 current valve positions to direct: 5.02 to 4.71 kPa at 30 min.

Multi-step strategies, not a law

On the 111 series of the NN5 competition, recursive beat direct.

• best there: multi-output strategies that predict the whole horizon at once
• seasonal adjustment helped in 38 of 39 models
• so: measure both on your data

Taieb, Bontempi, Atiya and Sorjamaa (2012), arXiv preprint

Multi-step strategies, window features

Lecture 7 showed rolling_mean. The open question was the window length.

The horizon is the criterion: try two or three lengths, keep what lowers the held-out error at your .

Evaluating on time, why shuffling leaks

• K-fold: shuffle rows, hold out a fifth, repeat
• neighbouring rows are near-copies (the ACF)
• shuffled, each test row has neighbours before and after it in training
• a flexible model recalls them

The score measures how densely you sampled, not how well you forecast. Lecture 7 named this train-test contamination.

Evaluating on time, measured

• bars are error: lower is better, and the dashed line is persistence
• below the line the model beats the free forecast; above it it loses
• same model and features; only the split changed

One run at a time, , pressure lags and valves, mean of runs 1 to 10. figures/make_figures.py

Evaluating on time, what the bars say

• shuffled: forest 4.32, ridge 4.82, both far below persistence (~6.1)
• time-ordered: forest 6.92, ridge 6.67, both worse than persistence
• one says "ship it", the other says "don't"

The boundary: pooled over 200 independent runs, ridge scores 4.74 shuffled and 4.71 by run. K-fold is valid for a purely autoregressive model with uncorrelated errors.
Shuffling bites hardest on one short series, a flexible model, correlated errors.

Bergmeir, Hyndman and Koo (2018), author's copy

Evaluating on time, names from finance

Both inflate the score. Neither raises an error.

Residuals, as a detector

1. fit on normal data
2. set the threshold on other normal data (99th percentile of the absolute residual)
3. alarm when a new residual exceeds it

Pressure, one step: residual sd 1.70 kPa, threshold 4.38 kPa. No fault examples needed.

Residuals, false alarms

• 99th percentile means 1 % of normal samples exceed it
• 480 samples a day, so 4.8 false alarms a day
• require 3 in a row: 0.01 a day, at the cost of delay

Residuals, one fault

Fault 1 (A/C feed ratio step), run 1, faulty training file of Rieth et al. (2017). 3-in-a-row alarm 45 min after onset.

Residuals, what the fault shows

• pressure swings 100 kPa; the residual only 5 to 10: it reacts to surprise, not distance
• the residual returns to the band while the fault is still there: the controller compensated
• across 20 faults on this channel: alarm within an hour for 1, 7, 12; never for 2, 3, 4, 9, 10, 11, 15, 16

One channel sees only what that channel sees.

Limitations, when a forecaster fails

Each one returns a confident number, not an error.

Recap

• A forecast is supervised learning with a horizon: features at , target at
• The ACF says whether the past can help; white noise and random walks have simple best forecasts
• Beat the better of persistence and the mean, at every ; persistence wins while
• Direct beat recursive on pressure; measure both
• TimeSeriesSplit(gap=h), or split by series; a shuffled score flattered two models past persistence
• Residuals are a detector; the threshold sets the false alarms

Standings

Nicknames only. Everyone who skipped one still counted in every bar you saw.

Next

Assignment 4 is released today, due 2026-09-28: an h-step forecaster for stripper temperature
Practice module for this session, for participation credit
Reading FPP3 chapter 5, https://otexts.com/fpp3/toolbox.html

Full notes, with all sources: lectures/l08/notes.md