dynamicsystemsarchitecture.org

Backtest-Only Audit

Full list of what needs forward-walk testing. Every item below has only ever been run against the specific case it was designed around — none checked against genuinely independent, unplanned material. Ordered by real priority, not file order.

PurposeSystematic self-audit distinguishing what's been forward-walk tested from what's only ever been backtested against its own construction — the discipline of catching your own scope gaps.
StatusVerified — the document's own claims (what's untested) are accurate; most listed items are intentionally still open
Built fromDirect review of every major component against its own test history
Depends onLaundering Vulnerability finding (Tier 1, item 2)
Superseded by—
EvidenceThis is itself the evidence document — see AI Systems Evaluation and QA/Verification field pages for how it's used.

Tier 1 — highest priority, real consequences if wrong

  1. The assignment app's calibration engine (pickCalibratedDifficulty, the frontier probability targets 75/70/65/60/55%). The single most important item on this list. Never run through more than a handful of manually-generated example problems. The entire design claim — that difficulty converges toward the stated target hit-rates over repeated trials — has never actually been tested with repeated trials. This is about to be used with real students. Needs a real simulation: thousands of synthetic student-response trials with varied ability levels, before a real student ever sees it.
  2. TaintedValue / the laundering fix. Verified only against the one specific attack construction it was built to catch (weighted-average blend of C into D). Never tested against: laundering through multiple hops (C → I → D), laundering via a nested/aggregated TaintedValue structure, or laundering through S or I specifically.
  3. Claims 2, 3, 6, 7 (prerequisite_dag, circuit_breaker, constitutional_behavioral_unit, rbac). Already formally flagged as stale by the project's own verification_registry.py — never re-checked against the value-provenance invariant discovered during the Claim 1 fix.
  4. The proof's core invariants (Theorem 1, Theorem 2, Corollary 3.1, Proposition 1.3). Verified primarily via a test suite built to match the proof's own claims — closer to backtesting the proof against itself than independent adversarial checking. The long-timescale C→D pathway question remains flagged as open, not closed, by design.

Tier 2 — real gaps, lower immediate stakes

  1. The isolation comparison scripts. Each tested only against the specific adversarial taxonomy invented for it — never against a genuinely creative, unplanned attack.
  2. story_logic_checker.py. Tested exactly once, on the example built to demonstrate it. Never run against a real, independently-authored story with a real plot hole not constructed to fit the tool.
  3. red_team_panel.py. Tested only on the same scenario, with only two experts. Never tested with a third, differently-shaped expert.
  4. multi_stage_pipeline.py. Tested only on one invented toy example. Never applied to a real, existing pipeline in the project.
  5. epistemic_protocols.py, Rules 2, 4, 5, 6, 8. Only Rule 3 has had one real forward-walk test so far. The other five are still backtest-only.

Tier 3 — real but lower stakes, or already partially addressed

  1. verification_registry.py. Tested only on the exact Claim 1 timeline it was built to formalize.
  2. governance_v2.py, genesis.py, lattice.py, audit_gate_and_coordinator.py. Verified individually and in one integration test — that test itself is a single constructed scenario, not varied or repeated.
  3. reference_ensemble_estimator.py's bootstrap fix. Verified with one specific dataset. Never tested against very small N or highly skewed data.

New rule this audit itself suggests

Cross-document citation consistency has no code yet — a real, named gap with no implementation, itself an example of something backtested (found once) but not yet built into anything that could catch a second instance automatically.

What "forward-walked" would actually mean here, stated plainly

For code: run against inputs, attacks, or scenarios that were not used to write the code. For the proof: independent verification of the still-open empirical questions, not internal consistency checking. For the calibration engine specifically: real trial volume, not example generation — the one item where "we'll get to it" isn't good enough given real students are the next step.