Boiling Frog Cross-Session Drift Simulation

If accuracy falls a little every session, does a rolling baseline notice? Does a running-maximum floor?

PurposeTest whether slow, sub-threshold decline is visible to a rolling 10-session baseline, and whether a floor anchored to the best result seen would catch it.

StatusPartly circular — the first half is arithmetic. The second half is not tested with a rule. A check added when this page was built finds the floor rule would false-alarm on noise.

Sample100 sessions, seed 42. Accuracy = 0.85 − 0.005 × session + Normal(0, 0.02), clamped to 0.40–0.98.

Touches NCFCA codeNo. A self-contained numpy simulation.

RelatedThe floor rule is specified in the ACDE hardening addendum, which is not published on this site.

AttributionCode AI-generated (tool not recorded). Run in the lab; results unchanged.

ReviewedBy no outside party. The reading below was done when this page was built, directly from the code and CSV.

What the data show

The injected drift takes accuracy from 0.86 down to the 0.40 clamp, where 9 of the 100 sessions end up. The rolling mean follows it down with a lag of about 0.02, ending at 0.4016.
The script checks only the final observation against the rolling mean. The gap is 0.0016, below its 0.05 threshold, and it prints “NOT DETECTED: Silent drift below threshold”. The largest single-session gap in the run is 0.081, and 6 of 100 sessions exceed 0.05.
The running maximum (“floor”) reaches 0.8655 by session 3. That is a noise draw above the true 0.85 baseline, and it never moves again. The gap between the floor and observed accuracy grows to 0.465.
The script computes that floor gap but never applies a detection rule to it. “The floor would catch it” is true of any decline and was not tested.

Check added when this page was built

This part is not from the lab’s run. I simulated 2,000 runs of 100 sessions with no drift, accuracy fixed at 0.85 plus noise, and counted how often an observation after session 20 fell below 0.92 × the running-maximum floor, the line the hardening addendum uses as its anchor.

Noise (standard deviation)	Share of sessions below 0.92 × floor, no drift
0.02 (this script)	about 12%
0.04 (the other scripts)	about 64%

A running maximum of noisy data ratchets upward even when nothing is wrong, so a rule that flags “below the decayed floor” would raise a false alarm on stable performance.

What it shows

A rolling mean cannot see a slow decline that is smaller than its window. A running-maximum anchor does diverge from a declining series, but it also diverges from a flat one.

What it does not show

A detector. The test that would show one, flagging drift while staying quiet on stable data, is not run.
Anything about real learners or real systems.

Download (unchanged from the lab’s run): boiling_frog_cross_session_drift_test.py · boiling_frog_results.csv
The scripts write to a path from the lab’s own machine (/home/workdir/artifacts/); change it to run them elsewhere. Each uses seed 42, so a rerun reproduces the CSV.