Roadmap

1. Match the architecture to the input
2. MLPs, briefly
3. CNNs for fields and images
4. Sequence models for sensor data
5. Training a deep net well
6. Does the architecture earn its keep?
7. Live demo: a 1D-CNN for turbofan RUL

Match the architecture to the input

An MLP treats its input as a flat vector, no relationships among the numbers.

Right for tabular data. Wrong for data with a shape:

• a temperature field is a grid; a pixel's neighbors matter
• a vibration trace is a sequence; the recent past matters

Flatten either and the model relearns, from scratch, structure you already knew.

MLPs, briefly

Lecture 11 built one. The choices that shape it:

• width and depth: capacity, and overfitting past what the data supports
• activation: ReLU or GELU, not saturating sigmoids
• regularizers: dropout, weight decay, batch/layer norm

Scaling still matters: fit the scaler on training data only (Lecture 7). A neural net does not repeal the leakage rules.

CNNs for fields and images

For data on a grid: an image, or a stress / temperature / velocity field from simulation.

• kernel: a small grid of weights slid across the input
• stride, padding: how it moves, and the border
• channels: stacked feature maps
• pooling: downsample, so deeper layers see a coarser, wider view

CNNs for fields and images, kernel

CNNs for fields and images, stride and padding

CNNs for fields and images, channels

CNNs for fields and images, pooling

CNNs for fields and images, the receptive field

Goodfellow: convolution leverages sparse interactions, parameter sharing, and equivariant representations.

CNNs for fields and images, why fewer parameters

An MLP would learn each position's edge detector separately, from far more data.

• the kernel slide: 9 weights for all 36 outputs; a dense layer from 8×8 to 6×6 needs 64 × 36 = 2,304
• the vertical-edge kernel is a finite-difference stencil for ∂T/∂x, summed over three rows (Prewitt)

Goodfellow, ch. 9, CS231n

CNNs for fields and images, a question

Sequence models, 1D-CNN and RNN

A sensor time series is a 1D grid, time, with a channel per sensor.

• Recurrent network (LSTM, GRU): walk the sequence step by step, carrying a gated hidden state that remembers the past

Sequence models, a 1D convolution

Sequence models, the receptive field

Sequence models, recurrence

Sequence models, gradients through time

Sequence models, a question

Sequence models, four wirings

Sequence models, which one

The reflex was an RNN for anything sequential. That has shifted.

Bai et al. (2018): "a simple convolutional architecture outperforms canonical recurrent networks such as LSTMs across a diverse range of tasks."

• temporal CNN: fixed windows, faster (timesteps compute in parallel), the sensible default
• LSTM / GRU: genuinely long-range, variable dependence
• attention / transformers: next, and they assume the least

Normalize on train stats only; never window across a unit boundary (Lecture 8).

Attention and transformers

• every cycle reads every other in one step; weights computed from the data
• "based solely on attention mechanisms, dispensing with recurrence and convolutions entirely" (Vaswani et al. 2017)
• message passing on a complete graph: (Gilmer et al. 2017)
• a convolution is the same update on a grid, with fixed weights per neighbour

Attention and transformers, one step

• token = one cycle: 14 sensors → linear → 32 numbers
• query , key , value , all learned
• keeps large dot products from saturating the softmax
• 4 heads of 8 dimensions in parallel; then residual, layer norm, per-cycle MLP

Attention and transformers, by hand

Attention and transformers, order

• without it, attention + averaging is blind to order: a window is a bag of readings
• "we must inject some information about the relative or absolute position" (Vaswani)
• sinusoidal or learned: "nearly identical results"; ours is a learned 30 × 32 table

Attention and transformers, a question

Attention and transformers, does order matter?

• no positions: identical on every fold; order-blindness is exact
• positions help (1.6 cycles), but the GRU relies on order far more

Attention and transformers, in PyTorch

self.inp = nn.Linear(14, 32)
self.pos = nn.Parameter(torch.zeros(1, 30, 32))
layer = nn.TransformerEncoderLayer(32, 4, dim_feedforward=64, dropout=0.1,
                                   batch_first=True, norm_first=True)
self.enc = nn.TransformerEncoder(layer, 2)
self.head = nn.Linear(32, 1)

h = self.enc(self.inp(x.transpose(1, 2)) + self.pos)
return self.head(h.mean(1))

18,561 parameters (GRU: 15,425); 960 of them are the position table.

Attention and transformers, what it learned

Attention and transformers, the weakest prior

• assumes almost nothing, so it must learn order and locality from about 70 engines
• the same weak prior is why it wins at LLM scale
• Zeng et al. 2023: one linear layer beat five forecasting transformers "in all cases" on nine datasets

Training a deep net well, the schedule

Minibatch gradient = true gradient + noise, and the step scales both. Near the minimum only the noise is left, so a constant rate never settles.

• step: drop by a factor every few epochs
• cosine: anneal smoothly to near zero (SGDR, CosineAnnealingLR)
• one-cycle: up to a max and back down in one run (Smith)

Training a deep net well, the schedule on a noisy bowl

Training a deep net well, the schedule on our GRU

Five seeds, 60 epochs, stopping off; validation RMSE in cycles:

The kept epoch is 9 to 17; with T_max = 60, cosine is still at 83% of its rate at epoch 17. Check the rate at the epoch you keep.

Training a deep net well, early stopping

• training error keeps falling; validation turns up once the net fits the training engines
• seed 1: train 12.1 → 9.7, validation 13.4 → 16.1 from epoch 17 to 60
• for a linear model under gradient descent it is L2 regularization (Goodfellow 7.8)
• uses the validation split; the test set is still touched once
• checkpointing: keep the best weights on disk, so a crash costs nothing

Training a deep net well, early stopping replayed

Training a deep net well, gradient clipping

• one factor for the whole gradient: direction kept, length capped
• clip_grad_norm_(model.parameters(), c) between backward() and step()
• recurrent nets have steep walls in the loss; a step at a wall "would bring us very far" (Pascanu et al. 2013)
• set c above ordinary norms; clip_grad_norm_ returns the norm, so log it

Training a deep net well, a question

Training a deep net well, gradient clipping at a wall

Training a deep net well, weight decay

• the ridge penalty from Lecture 9: shrink every weight by a fixed fraction each step
• shrinks most where the data is weakest: factor per Hessian direction (Goodfellow 7.1.1)
• Adam(weight_decay=...) is coupled L2; AdamW is true decay (Loshchilov and Hutter, Lecture 11)

Training a deep net well, weight decay shrinks the weak direction

Training a deep net well, reading the curves

Lecture 11's broken loops were all the third kind, the one mistaken for a modeling problem.

Does the architecture earn its keep?, the setup

Remaining useful life on C-MAPSS FD001 turbofans (from Lecture 8):

• sliding 30-cycle windows over 17 sensor/setting channels
• piecewise-linear RUL target (constant at 125, then linear)
• GroupKFold by engine: no engine in both train and validation (Lecture 8)

Two models: a 1D-CNN on raw windows vs gradient boosting on window features (mean, std, last).

Does the architecture earn its keep?, the result

1D-CNN 19.9 ± 2.0 vs baseline 18.3 ± 1.0 cycles RMSE, four engine-grouped folds. The quick sequence model does not beat the baseline, and the gap is inside its own spread.

Does the architecture earn its keep?, the lesson

A deep net is not a default.

Grinsztajn et al. (45 datasets): "tree-based models remain state-of-the-art on medium-sized data" (~10K samples).

The CNN earns its advantage from scale, from tuning (schedule, regularization), and from raw structure a hand-crafted feature cannot capture. On a small, well-summarized benchmark, the baseline is the model to beat.

Limitations and trade-offs

• An architecture is an assumption to check. A CNN assumes locality and translation invariance; a global constraint or symmetry can make that a liability.
• Depth needs data. A hundred engines is small; parameter sharing reduces but does not remove the appetite. Small data favors a strong classical model.
• Accelerators and larger models have costs (Lecture 11: GPU 2.5x slower on a small model). Past the capacity the data supports, more layers buy overfitting. Debug on CPU.

What to watch

The comparison, on a grouped split.

The sequence model does not automatically win.

The grouping by engine is what keeps the comparison honest instead of flattering.

Recap

• Match architecture to input structure: vector to MLP, field to 2D-CNN, sequence to 1D-CNN or RNN
• CNN economy = sparse interactions + parameter sharing; the receptive field grows with depth
• Temporal CNN is the sensible default over an RNN for fixed windows
• Train well: LR schedule, early stopping, clipping, weight decay, read the loss curves
• A deep net is not a default; run a strong baseline, grouped by unit

Standings

Nicknames only. Everyone who skipped one still counted in every bar you saw.

Next

Reading PyTorch CNN/LSTM tutorials; Goodfellow ch. 9-10; Grinsztajn et al.

Full notes, with all sources: lectures/l12/notes.md