Validation, measured

This page is generated. cargo xtask validate wrote it from commit 03ec3d6-dirty. Every number on it was captured from that run's own output. To change what it says, change what the suite measures and run the command again.

The narrative version of this material, with the reasoning behind each tier and the history of what it caught, is in Validation. This section is the other half: the numbers, written by the command that ran the suite, from the suite's own output. Nothing here was transcribed.

A tier with no chapter has not been generated. That is stated rather than left to be inferred from an absence, because a missing page and a page nobody updated look the same from the table of contents.

Toolchain on the machine that ran this: rustc 1.97.1 (8bab26f4f 2026-07-14), gfortran 15.2.0.

Fidelity preflight

cargo xtask fidelity runs the port forward from the nominal state and diffs states, derivatives, measurements and the generator word against a golden oracle trace committed to the repository. It needs no Fortran toolchain, so it runs everywhere in about a second.

steps diffed	worst	where	gate	trace recorded with
100 of 100	3.521323734565789e-14	`YP(12) at step 77`	1e-12	gfortran 15.2.0

What the long tiers look like

Tiers 4 and 5 run but write no chapter yet, for the reason GENERATED_TIERS gives in xtask. Their figures are here, each drawn by the run named under it.

How far apart the trajectories get, in units of instrument noise. Every disturbance scenario, run on both implementations, with the worst disagreement anywhere in the run divided by the noise standard deviation XNS(i) of the channel it happened on. The reference line is therefore not zero but the point at which the plant's own instruments could resolve the difference. The claim is false if any marker reaches the shaded band, and the explanation is false if the platform libm markers ever leave the = 0 lane: that would mean the divergence is something other than transcendental rounding. Drawn by cargo xtask validate --tiers 4 at commit b6ef4ee-dirty; the measurement it repeats was first recorded in B-0034 (LOG.org).

The two sources against the reference's own run-to-run spread. There is no absolute scale on which a Frobenius distance or a Kolmogorov-Smirnov statistic is small, so the battery measures one: it splits the Fortran's own runs in half and computes the same statistic Fortran against Fortran. The horizontal axis is the statistic across the two sources, the vertical axis is that null, and both axes share a scale so the diagonal is equality. The claim is false if a cross-source point crosses to the low side of the diagonal; the two that already sit there are the battery's own positive control, where one variable of the reference was shifted by ten standard deviations before comparing. Drawn by cargo xtask validate --tiers 5 --smoke at commit 03ec3d6-dirty; the measurement it repeats was first recorded in B-0047b (LOG.org).

Each variable's mean difference against its own equivalence margin. The calibration figure above says the two sources are closer to each other than the reference is to itself. It does not say how much room is left, and a TOST verdict does not either: a p-value reports that a test did not reject, not by what distance. This does. Each marker is one of the 53 variables in one scenario, its paired mean difference divided by that variable's own equivalence margin, which is a tenth of the reference's standard deviation for that variable. One is the gate and the region past it is shaded, so the claim is false if any solid marker reaches the shaded band. Hollow markers are the variables the moment gate does not apply to and are drawn rather than dropped: a reference that never moved has no mean to compare, and a valve stuck by the scenario's own fault is judged on its distribution instead, which is the decision recorded as B-0047d. Drawn by cargo xtask validate --tiers 5 --smoke at commit 03ec3d6-dirty; the measurement it repeats was first recorded in B-0047b (LOG.org).

The ladder

tier	what it proves	chapter
1	`TESUB1` to `TESUB8` match the oracle	tier 1, from `b6ef4ee-dirty` by `cargo xtask validate --tiers 1,2,3 --smoke`
2	single-step derivatives match, over all three pools	tier 2, from `b6ef4ee-dirty` by `cargo xtask validate --tiers 1,2,3 --smoke`
3	the generator call order matches, draw for draw	tier 3, from `b6ef4ee-dirty` by `cargo xtask validate --tiers 1,2,3 --smoke`
4	trajectories stay inside the measurement noise (diagnostic)	runs, but writes no chapter yet
5	statistical equivalence: TOST, KS, ACF, spectra, correlations	runs, but writes no chapter yet
6	downstream detectors cannot tell the two sources apart	runs, but writes no chapter yet
7	the published `d00` to `d21` files are reproduced	runs, but writes no chapter yet
8	differential fuzzing finds no counterexample	no harness yet
9	identical digests across platforms, wasm included	runs, but writes no chapter yet
10	every quirk fix ships with a measured delta	runs, but writes no chapter yet

This run selected tier(s) [5] and wrote chapter(s) for []. Tiers 4 to 7, 9 and 10 run but do not write a chapter yet; tier 8 has no harness. The delta index is generated separately, by cargo xtask deltas.

Keyboard shortcuts

TEP-Rust

Validation, measured

Fidelity preflight

What the long tiers look like

The ladder