formal-mas / docs /results /results.md
NicolasT08's picture
Formal-MAS source code
dfe1826 verified
|
Raw History Blame Contribute Delete
37.1 kB

Tables for the test split (n = 30)

Generated by docs/results/make_tables.py from runs/ (read-only), following docs/analysis_plan.md. Rates in %, intervals 95 %.

Runs used

Run Config Profile Seed Split Commit Tree Degraded Run status vs first run
20261103T174646Z_C5_experiment C5 experiment 2026 test ad70ec2 clean False running same
20261103T212359Z_C5_experiment_resume C5 experiment 2026 test ad70ec2 dirty (plan file only) False budget_stop same
20261103T225140Z_C5_experiment_resume2 C5 experiment 2026 test 93bc701 dirty (plan file only) False finished same
20261103T235556Z_C4_experiment C4 experiment 2026 test ea4db4b dirty (plan file only) False finished same
20261104T194056Z_C3_experiment C3 experiment 2026 test 6c61d69 clean False finished same
20261104T232056Z_C2_experiment_check C2 experiment 2026 test 6c61d69 clean False finished same
20261105T131443Z_C1_experiment C1 experiment 2026 test 5ed7507 clean False running same
20261107T200233Z_C1_experiment_resume C1 experiment 2026 test 2c8eae5 dirty (plan file only) False finished same

Row checks

Config Rows Status counts lean_down split/config mismatch area/difficulty mismatch
C1 30 budget_exhausted: 7, failed: 3, solved: 20 0 0 0
C2 30 claimed_solved: 30 0 0 0
C3 30 budget_exhausted: 3, failed: 7, solved: 20 0 0 0
C4 30 failed: 16, solved: 14 0 0 0
C5 30 failed: 6, solved: 24 0 0 0

Table 1. Verified success rate (SR) and Baseline B

Config Solved SR (%) Wilson 95 % CI (%) C2 claimed rate (%) [Wilson] FPR_B (%) [Wilson]
C1 20/30 66.7 [48.8, 80.8] — —
C3 20/30 66.7 [48.8, 80.8] — —
C4 14/30 46.7 [30.2, 63.9] — —
C5 24/30 80.0 [62.7, 90.5] — —
C2 (Baseline B) — — — 30/30 = 100.0 [88.6, 100.0] 11/30 = 36.7 [21.9, 54.5]

Note. C2 is not verified, so it has no SR; it enters only through its claimed rate B⁺/n and FPR_B (plan §7.1). Of the 30 claimed proofs the formal check verified 19 and did not verify 11; 0 left out of FPR_B.

Table 1b. Baseline B claim check (FPR_B, Eq. 16)

Quantity Value
Problems 30
Claimed (B⁺) 30
Not claimed 0
Check status among B⁺ not_verified: 11, verified: 19
Verified (Acc on the assembled theorem) 19
Not verified (budget_exhausted included) 11
Left out of FPR_B (provider_error, budget_stop, error, lean_down) 0
FPR_B = not verified / (verified + not verified) 11/30 = 36.7 % [21.9, 54.5]
B⁺ problems for the manual review (§4.2) 10 drawn with random.Random(2026).sample(sorted ids, 10) (listed below)

Note. Manual review set (B⁺ not verified): amc12b_2002_p3, imo_1964_p1_2, imo_1965_p2, imo_1978_p5, imo_1979_p1, imo_1984_p6, imo_1987_p6, imo_1993_p5, putnam_1965_b2, putnam_1995_b1.

Table 2. Planned comparisons H1–H3 (exact McNemar, Bonferroni ×3)

ID Comparison b c ΔSR (pp) Paired bootstrap 95 % CI (pp) Exact p Bonferroni p Reject at 0.05/3
H1 C5 vs C1 4 0 +13.3 [+3.3, +26.7] 0.1250 0.3750 no
H2 C5 vs C3 5 1 +13.3 [+0.0, +30.0] 0.2188 0.6562 no
H3 C5 vs C4 10 0 +33.3 [+16.7, +50.0] 0.0020 0.0059 yes

Note. n = 30, so power is low: a non-significant result is no evidence of a difference, not evidence of no difference. H2 is Δ_RAG = SR_C5 − SR_C3; H3 is the Critic contribution SR_C5 − SR_C4.

Table 2b. Exploratory comparisons (uncorrected, no claims)

ID Comparison b c ΔSR (pp) Exact p (uncorrected)
E1 C3 vs C4 9 3 +20.0 0.1460
E2 C1 vs C3 3 3 +0.0 1.0000
E3 C1 vs C4 8 2 +20.0 0.1094

Table 3. Solved / n per cell of the 3 × 3 grid (descriptive)

Difficulty Area n C1 C3 C4 C5 C2 (claimed)
easy Algebra 5 5/5 5/5 4/5 5/5 5/5
easy Number theory 5 5/5 4/5 5/5 5/5 5/5
easy Combinatorics 0 — — — — —
medium Algebra 5 4/5 5/5 1/5 5/5 5/5
medium Number theory 4 2/4 2/4 2/4 4/4 4/4
medium Combinatorics 1 0/1 0/1 1/1 1/1 1/1
hard Algebra 3 1/3 2/3 0/3 1/3 3/3
hard Number theory 4 1/4 1/4 0/4 1/4 4/4
hard Combinatorics 3 2/3 1/3 1/3 2/3 3/3

Table 3b. SR by difficulty and by area (margins, Wilson 95 % CI)

Config easy (n = 10) medium (n = 10) hard (n = 10) Algebra (n = 13) Number theory (n = 13) Combinatorics (n = 4)
C1 10/10 6/10 4/10 10/13 8/13 2/4
100.0 [72.2, 100.0] 60.0 [31.3, 83.2] 40.0 [16.8, 68.7] 76.9 [49.7, 91.8] 61.5 [35.5, 82.3] 50.0 [15.0, 85.0]
C3 9/10 7/10 4/10 12/13 7/13 1/4
90.0 [59.6, 98.2] 70.0 [39.7, 89.2] 40.0 [16.8, 68.7] 92.3 [66.7, 98.6] 53.8 [29.1, 76.8] 25.0 [4.6, 69.9]
C4 9/10 4/10 1/10 5/13 7/13 2/4
90.0 [59.6, 98.2] 40.0 [16.8, 68.7] 10.0 [1.8, 40.4] 38.5 [17.7, 64.5] 53.8 [29.1, 76.8] 50.0 [15.0, 85.0]
C5 10/10 10/10 4/10 11/13 10/13 3/4
100.0 [72.2, 100.0] 100.0 [72.2, 100.0] 40.0 [16.8, 68.7] 84.6 [57.8, 95.7] 76.9 [49.7, 91.8] 75.0 [30.1, 95.4]
C2 (claimed) 10/10 10/10 10/10 13/13 13/13 4/4
100.0 [72.2, 100.0] 100.0 [72.2, 100.0] 100.0 [72.2, 100.0] 100.0 [77.2, 100.0] 100.0 [77.2, 100.0] 100.0 [51.0, 100.0]

Table 4. Efficiency per solved problem (RNF-3)

Config Solved Calls / solved k tokens / solved Nominal USD / solved Real USD / solved Wall min / solved
C1 20 12.85 [7.19, 22.60] 219.7 [114.6, 394.7] 0.1285 [0.0668, 0.2313] 0.1235 [0.0643, 0.2218] 17.7 [9.2, 32.0]
C2 (per claimed) 30 2.17 [2.00, 2.40] 5.3 [2.8, 8.6] 0.0027 [0.0013, 0.0047] 0.0027 [0.0013, 0.0047] 0.4 [0.2, 0.7]
C3 20 26.50 [16.15, 43.67] 150.4 [79.3, 274.6] 0.0675 [0.0335, 0.1279] 0.0675 [0.0335, 0.1278] 9.2 [4.6, 17.3]
C4 14 9.43 [5.72, 16.67] 70.7 [38.2, 137.8] 0.0278 [0.0135, 0.0571] 0.0000 [0.0000, 0.0000] 0.2 [0.1, 0.3]
C5 24 16.92 [10.63, 26.21] 148.0 [74.4, 266.4] 0.0566 [0.0245, 0.1097] 0.0490 [0.0236, 0.0897] 7.6 [3.3, 14.6]

Note. C2 has no verified solutions: its ratio uses claimed solutions and excludes the claim check (calls, tokens, cost and wall time up to claim_check_start).

Table 4b. Efficiency per problem (means)

Config Calls / problem k tokens / problem Nominal USD / problem Real USD / problem Wall min / problem
C1 8.57 [5.83, 11.33] 146.4 [91.1, 203.5] 0.0857 [0.0532, 0.1192] 0.0823 [0.0505, 0.1152] 11.8 [7.2, 16.5]
C2 2.17 [2.00, 2.40] 5.3 [2.8, 8.6] 0.0027 [0.0013, 0.0047] 0.0027 [0.0013, 0.0047] 0.4 [0.2, 0.7]
C3 17.67 [12.97, 22.37] 100.3 [63.8, 140.8] 0.0450 [0.0269, 0.0658] 0.0450 [0.0269, 0.0658] 6.1 [3.7, 9.0]
C4 4.40 [3.37, 5.43] 33.0 [22.8, 44.4] 0.0130 [0.0081, 0.0186] 0.0000 [0.0000, 0.0000] 0.1 [0.1, 0.1]
C5 13.53 [9.57, 17.60] 118.4 [67.9, 178.7] 0.0453 [0.0225, 0.0737] 0.0392 [0.0218, 0.0588] 6.1 [3.0, 9.7]

Table 4c. Totals, cache use and real cost

Config LLM calls Cached calls Cached (%) Tokens (M) Nominal USD (rows) Real USD (rows) Real USD (ledger, all runs)
C1 257 47 18.3 4.393 2.5700 2.4695 2.5679
C2 65 0 0.0 0.158 0.0814 0.0814 0.7083
C3 530 18 3.4 3.009 1.3508 1.3500 1.3500
C4 132 132 100.0 0.990 0.3887 0.0000 0.0000
C5 406 34 8.4 3.551 1.3590 1.1770 1.3585

Note. C2 claim check (not in the C2 figures of the other columns): 77 calls (11 cached), 0.6274 nominal, 0.6269 real.

RNF-3 (C̄_solved ≤ 20 for C5): C̄_solved = 406/24 = 16.92 calls per solved problem: met.

Table 4d. Median [IQR] of LLM calls and wall time per problem (supplementary)

Config Calls, median [Q1, Q3] Wall min, median [Q1, Q3]
C1 4.5 [1.0, 18.0] 2.7 [0.6, 24.2]
C2 2.0 [2.0, 2.0] 0.1 [0.1, 0.3]
C3 16.5 [3.2, 28.0] 3.7 [0.6, 6.7]
C4 5.0 [1.0, 5.8] 0.1 [0.0, 0.1]
C5 11.5 [1.0, 23.8] 2.4 [0.2, 5.2]

Table 5. Wall time per problem by difficulty, minutes (descriptive)

Config easy (n = 10): mean / median [Q1, Q3] medium (n = 10): mean / median [Q1, Q3] hard (n = 10): mean / median [Q1, Q3]
C1 0.7 / 0.4 [0.1, 1.1] 14.3 / 16.2 [1.8, 23.2] 20.3 / 25.3 [7.0, 32.4]
C2 0.1 / 0.1 [0.1, 0.2] 0.1 / 0.1 [0.1, 0.2] 1.0 / 0.5 [0.2, 1.1]
C3 0.8 / 0.1 [0.0, 1.0] 4.0 / 2.6 [1.2, 6.0] 13.6 / 12.3 [6.6, 19.8]
C4 0.1 / 0.0 [0.0, 0.1] 0.1 / 0.1 [0.0, 0.1] 0.1 / 0.1 [0.1, 0.1]
C5 0.4 / 0.0 [0.0, 0.5] 2.7 / 2.6 [0.7, 4.5] 15.1 / 14.1 [4.0, 21.0]

Table 6. Refinement robustness: SR(k) of C5 vs the independence curve (Eq. 14)

k SR_C5(k) [Wilson] (%) Δ̂(k) = SR(k) − SR(k−1) (pp) Indep. Δ(k) = p̂(1−p̂)^(k−1) (%) Subgoals solved at attempt k Solved at k / reached k
1 15/30 = 50.0 [33.2, 66.8] +20.0 40.9 18/44 = 40.9 18/44 = 40.9
2 19/30 = 63.3 [45.5, 78.1] +13.3 24.2 8/44 = 18.2 8/26 = 30.8
3 24/30 = 80.0 [62.7, 90.5] +16.7 14.3 5/44 = 11.4 5/18 = 27.8

Note. SR_C5(0) = 9/30 (accepted plan with no LLM subgoal attempt). p̂ = 18/44 = 40.9 %. k = 1 from C4 (no retries, no Critic advice): 14/30 = 46.7 % [30.2, 63.9]. The last two columns are subgoal-level diagnostics (supplementary).

Table 7. κ error taxonomy over all failed attempts (Eq. 11)

Config Failed attempts syntax missing lemma tactic failure misformalization timeout planning
C1 186 60 (32.3) 50 (26.9) 68 (36.6) 0 (0.0) 8 (4.3) 0 (0.0)
C2 (claim check) 45 14 (31.1) 8 (17.8) 22 (48.9) 0 (0.0) 1 (2.2) 0 (0.0)
C3 82 20 (24.4) 17 (20.7) 42 (51.2) 0 (0.0) 2 (2.4) 1 (1.2)
C4 16 5 (31.2) 4 (25.0) 7 (43.8) 0 (0.0) 0 (0.0) 0 (0.0)
C5 57 19 (33.3) 18 (31.6) 18 (31.6) 0 (0.0) 2 (3.5) 0 (0.0)

Note. C3–C5: classes recorded by the Critic. C1: attempts classified offline with policy.classify (first 3 Lean messages, 300 characters each, as stored; no k_max escalation). C2 has no verified attempts of its own: its row is the claim-check loop.

Table 7b. κ taxonomy per area

Config Area Failed attempts syntax missing lemma tactic failure misformalization timeout planning
C1 Algebra 77 14 (18.2) 40 (51.9) 22 (28.6) 0 (0.0) 1 (1.3) 0 (0.0)
C1 Number theory 83 27 (32.5) 3 (3.6) 46 (55.4) 0 (0.0) 7 (8.4) 0 (0.0)
C1 Combinatorics 26 19 (73.1) 7 (26.9) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0)
C2 (claim check) Algebra 14 5 (35.7) 4 (28.6) 5 (35.7) 0 (0.0) 0 (0.0) 0 (0.0)
C2 (claim check) Number theory 23 5 (21.7) 4 (17.4) 13 (56.5) 0 (0.0) 1 (4.3) 0 (0.0)
C2 (claim check) Combinatorics 8 4 (50.0) 0 (0.0) 4 (50.0) 0 (0.0) 0 (0.0) 0 (0.0)
C3 Algebra 18 3 (16.7) 9 (50.0) 5 (27.8) 0 (0.0) 0 (0.0) 1 (5.6)
C3 Number theory 43 13 (30.2) 3 (7.0) 26 (60.5) 0 (0.0) 1 (2.3) 0 (0.0)
C3 Combinatorics 21 4 (19.0) 5 (23.8) 11 (52.4) 0 (0.0) 1 (4.8) 0 (0.0)
C4 Algebra 8 1 (12.5) 3 (37.5) 4 (50.0) 0 (0.0) 0 (0.0) 0 (0.0)
C4 Number theory 6 2 (33.3) 1 (16.7) 3 (50.0) 0 (0.0) 0 (0.0) 0 (0.0)
C4 Combinatorics 2 2 (100.0) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0)
C5 Algebra 25 7 (28.0) 8 (32.0) 10 (40.0) 0 (0.0) 0 (0.0) 0 (0.0)
C5 Number theory 24 8 (33.3) 9 (37.5) 6 (25.0) 0 (0.0) 1 (4.2) 0 (0.0)
C5 Combinatorics 8 4 (50.0) 1 (12.5) 2 (25.0) 0 (0.0) 1 (12.5) 0 (0.0)

Table 7c. Final failure reason per problem (status @ stage)

Config Problems status @ stage: count
C1 10 budget_exhausted: 7; failed: 3
C2 30 claimed, check verified: 19; claimed, check not_verified: 11
C3 10 failed @ verify: 5; budget_exhausted @ formalize: 3; failed @ formalize: 2
C4 16 failed @ verify: 11; failed @ formalize: 5
C5 6 failed @ verify: 4; failed @ formalize: 2

Table 8. Subgoals closed by the tactic pre-pass vs the LLM (supplementary)

Config Pre-pass (accepted plans) LLM (accepted plans) Pre-pass (all subgoal_solved) Solved with no LLM subgoal Solved
C3 7/32 = 21.9 25/32 = 78.1 8/39 = 20.5 8 20
C4 9/18 = 50.0 9/18 = 50.0 11/21 = 52.4 9 14
C5 11/39 = 28.2 28/39 = 71.8 11/42 = 26.2 9 24

Table 9. Infrastructure and error statuses in the kept rows (plan §5, §7.7)

Config provider_error budget_stop error node_error invalid_statement lean_down
C1 0 0 0 0 0 0
C2 0 0 0 0 0 0
C3 0 0 0 0 0 0
C4 0 0 0 0 0 0
C5 0 0 0 0 0 0

Table 10. Solved test problems with a fidelity note (plan §6)

Problem Solved by Manual review
amc12b_2003_p6 C1, C3, C4, C5 2026-11-07: counts as solved; report note: The formalization proves that u 0 = +-2/sqrt 3 (a necessary condition), not that -2*sqrt(3)/3 is attained; this is enough to rule out the other options.
mathd_algebra_234 C1, C3, C4, C5 2026-11-07: counts as solved; no note
mathd_algebra_455 C1, C3, C4, C5 2026-11-07: counts as solved; no note
putnam_1965_b2 C1, C3, C5 2026-11-07: counts as solved; no note

Table 11. Per-problem outcomes

Problem Difficulty Area C1 C3 C4 C5 C2 (claim, check)
algebra_absapbon1pabsapbleqsumabsaon1pabsa medium Algebra 0 1 0 1 claimed, verified
amc12_2000_p20 medium Algebra 1 1 0 1 claimed, verified
amc12_2001_p9 medium Algebra 1 1 0 1 claimed, verified
amc12a_2008_p15 medium Number theory 0 0 0 1 claimed, not_verified
amc12a_2008_p25 medium Algebra 1 1 0 1 claimed, verified
amc12b_2002_p3 medium Number theory 0 1 1 1 claimed, not_verified
amc12b_2003_p6 medium Algebra 1 1 1 1 claimed, verified
imo_1964_p1_2 hard Number theory 1 1 0 1 claimed, not_verified
imo_1965_p2 hard Algebra 1 1 0 1 claimed, not_verified
imo_1978_p5 hard Algebra 0 0 0 0 claimed, not_verified
imo_1979_p1 hard Number theory 0 0 0 0 claimed, not_verified
imo_1984_p6 hard Number theory 0 0 0 0 claimed, not_verified
imo_1987_p6 hard Number theory 0 0 0 0 claimed, not_verified
imo_1993_p5 hard Algebra 0 1 0 0 claimed, not_verified
mathd_algebra_160 easy Algebra 1 1 1 1 claimed, verified
mathd_algebra_206 easy Algebra 1 1 0 1 claimed, verified
mathd_algebra_234 easy Algebra 1 1 1 1 claimed, verified
mathd_algebra_288 easy Algebra 1 1 1 1 claimed, verified
mathd_algebra_455 easy Algebra 1 1 1 1 claimed, verified
mathd_numbertheory_321 easy Number theory 1 0 1 1 claimed, verified
mathd_numbertheory_328 easy Number theory 1 1 1 1 claimed, verified
mathd_numbertheory_345 easy Number theory 1 1 1 1 claimed, verified
mathd_numbertheory_430 easy Number theory 1 1 1 1 claimed, verified
mathd_numbertheory_64 easy Number theory 1 1 1 1 claimed, verified
numbertheory_3pow2pownm1mod2pownp3eq2pownp2 medium Number theory 1 0 0 1 claimed, verified
numbertheory_notequiv2i2jasqbsqdiv8 medium Number theory 1 1 1 1 claimed, verified
numbertheory_sumkmulnckeqnmul2pownm1 medium Combinatorics 0 0 1 1 claimed, verified
putnam_1964_b2 hard Combinatorics 1 0 1 1 claimed, verified
putnam_1965_b2 hard Combinatorics 1 1 0 1 claimed, not_verified
putnam_1995_b1 hard Combinatorics 0 0 0 0 claimed, not_verified

Table 11b. Hard problems per configuration (supplementary; post hoc, descriptive, not in the analysis plan)

Problem Area C1 C3 C4 C5 Solved by
imo_1964_p1_2 Number theory 1 1 0 1 3
imo_1965_p2 Algebra 1 1 0 1 3
imo_1978_p5 Algebra 0 0 0 0 0
imo_1979_p1 Number theory 0 0 0 0 0
imo_1984_p6 Number theory 0 0 0 0 0
imo_1987_p6 Number theory 0 0 0 0 0
imo_1993_p5 Algebra 0 1 0 0 1
putnam_1964_b2 Combinatorics 1 0 1 1 3
putnam_1965_b2 Combinatorics 1 1 0 1 3
putnam_1995_b1 Combinatorics 0 0 0 0 0
Solved 4/10 4/10 1/10 4/10

Note. Post hoc, descriptive (not in the analysis plan): no tests, no intervals. 1 = verified solved. C1 and C5 solve the same hard problems (4 and 4); 5 hard problems are solved by no configuration (imo_1978_p5, imo_1979_p1, imo_1984_p6, imo_1987_p6, putnam_1995_b1); C3 and C5 solve 4 and 4 hard problems, but only C3 solves imo_1993_p5 and only C5 solves putnam_1964_b2.

Table 12. Exploratory: C5-Flash vs C5-Pro on the hard problems (post hoc, descriptive, not in the analysis plan)

Problem Flash solved Flash calls Flash USD Flash min Pro outcome Pro solved Pro calls Pro USD Pro min
imo_1964_p1_2 1 15 0.0261 3.4 not attempted — — — —
imo_1965_p2 1 14 0.0312 3.7 failed 0 30 0.3608 33.6
imo_1978_p5 0 31 0.1347 39.6 provider_error (connection drop, not a system failure) — 30 0.2938 40.7
imo_1979_p1 0 28 0.1463 19.7 failed 0 24 0.2722 29.3
imo_1984_p6 0 32 0.3366 27.6 interrupted — — — —
imo_1987_p6 0 36 0.1793 21.4 budget_exhausted 0 40 0.4607 57.4
imo_1993_p5 0 27 0.0790 10.6 failed 0 24 0.3807 53.6
putnam_1964_b2 1 9 0.0160 2.3 not attempted — — — —
putnam_1965_b2 1 17 0.0409 4.9 not attempted — — — —
putnam_1995_b1 0 25 0.1268 17.5 not attempted — — — —
Both finished (4) 1/4 105 0.4357 55.5 0/4 118 1.4744 173.9
All hard (10) 4/10 234 1.1169 150.9 — — — —

Note. Post hoc, exploratory (decided 2026-11-08, after all confirmatory runs): descriptive only, no tests, no intervals; it changes no main result. C5-Flash = the kept C5 rows of the main test result; C5-Pro = run 20261108T130803Z_C5_experiment_pro_pro_hard (profile experiment_pro: deepseek-v4-pro in every role, everything else as frozen), stopped by the team and not resumed. Cost is nominal USD (cache hits priced as fresh calls). provider_error, interrupted and not attempted do not count as failures of Pro and are left out of its solved count; only problems both runs finished are compared. imo_1984_p6 was interrupted after 4 paid calls (0.1725 USD real, 18.1 min) and left no row. Real cost of the Pro run: 1.9407 USD (151 paid calls, all in the ledger); project ledger total: 9.3003 USD.

Pro vs Flash on the 4 problems both finished: 3.38 × the nominal cost and 3.13 × the wall time.

Project spend (ledger, all runs): main experiments (every run except the exploratory Pro run, dev included) 7.3596 USD; exploratory Pro run 1.9407 USD; total 9.3003 USD.

Discordant pairs (candidates for illustrative traces)

  • C5 solved, C4 not (Critic loop): algebra_absapbon1pabsapbleqsumabsaon1pabsa (C5: solved, 24 calls, replans 1, max attempts 1); amc12_2000_p20 (C5: solved, 23 calls, replans 0, max attempts 3); amc12_2001_p9 (C5: solved, 7 calls, replans 0, max attempts 2); amc12a_2008_p15 (C5: solved, 16 calls, replans 0, max attempts 3); amc12a_2008_p25 (C5: solved, 23 calls, replans 0, max attempts 2); imo_1964_p1_2 (C5: solved, 15 calls, replans 0, max attempts 3); imo_1965_p2 (C5: solved, 14 calls, replans 0, max attempts 2); mathd_algebra_206 (C5: solved, 28 calls, replans 0, max attempts 2); numbertheory_3pow2pownm1mod2pownp3eq2pownp2 (C5: solved, 15 calls, replans 0, max attempts 3); putnam_1965_b2 (C5: solved, 17 calls, replans 0, max attempts 3)
  • C4 solved, C5 not: —
  • C5 solved, C3 not (RAG): amc12a_2008_p15 (C5: solved, 16 calls, replans 0, max attempts 3); mathd_numbertheory_321 (C5: solved, 1 calls, replans 0, max attempts 0); numbertheory_3pow2pownm1mod2pownp3eq2pownp2 (C5: solved, 15 calls, replans 0, max attempts 3); numbertheory_sumkmulnckeqnmul2pownm1 (C5: solved, 5 calls, replans 0, max attempts 1); putnam_1964_b2 (C5: solved, 9 calls, replans 0, max attempts 1)
  • C3 solved, C5 not: imo_1993_p5 (C5: failed, 27 calls, replans 1, max attempts -)
  • C5 solved, C1 not: algebra_absapbon1pabsapbleqsumabsaon1pabsa (C5: solved, 24 calls, replans 1, max attempts 1); amc12a_2008_p15 (C5: solved, 16 calls, replans 0, max attempts 3); amc12b_2002_p3 (C5: solved, 9 calls, replans 0, max attempts 1); numbertheory_sumkmulnckeqnmul2pownm1 (C5: solved, 5 calls, replans 0, max attempts 1)
  • C1 solved, C5 not: —
  • C2 claimed, check did not verify: amc12a_2008_p15 (C5: solved, 16 calls, replans 0, max attempts 3); amc12b_2002_p3 (C5: solved, 9 calls, replans 0, max attempts 1); imo_1964_p1_2 (C5: solved, 15 calls, replans 0, max attempts 3); imo_1965_p2 (C5: solved, 14 calls, replans 0, max attempts 2); imo_1978_p5 (C5: failed, 31 calls, replans 1, max attempts -); imo_1979_p1 (C5: failed, 28 calls, replans 1, max attempts -); imo_1984_p6 (C5: failed, 32 calls, replans 1, max attempts -); imo_1987_p6 (C5: failed, 36 calls, replans 1, max attempts -); imo_1993_p5 (C5: failed, 27 calls, replans 1, max attempts -); putnam_1965_b2 (C5: solved, 17 calls, replans 0, max attempts 3); putnam_1995_b1 (C5: failed, 25 calls, replans 1, max attempts -)

Generated data notes

  • C1 merged file checked row by row: each row equals (as parsed JSON) exactly one source row; 23 from 20261105T131443Z_C1_experiment, 7 from 20261107T200233Z_C1_experiment_resume. No source row is left out.
  • Kept 30 rows from 20261104T232056Z_C2_experiment_check.
  • Kept 30 rows from 20261104T194056Z_C3_experiment.
  • Kept 30 rows from 20261103T235556Z_C4_experiment.
  • Kept 23 rows from 20261103T174646Z_C5_experiment.
  • Discarded row imo_1984_p6 (status budget_stop) of 20261103T212359Z_C5_experiment_resume, as the C5 Deviations entry says.
  • Kept 2 rows from 20261103T212359Z_C5_experiment_resume.
  • Kept 5 rows from 20261103T225140Z_C5_experiment_resume2.
  • C1: 52 LLM responses ended at max_tokens (finish_reason length) in 9 rows.
  • C1: ledger spend in no kept row = 0.09845 USD (real cost only): 20261105T131443Z_C1_experiment / imo_1987_p6: 0.09845 USD, 5 paid calls.
  • C2: 0 LLM responses ended at max_tokens (finish_reason length) in 0 rows; 12 more in the claim check.
  • C3: 18 LLM responses ended at max_tokens (finish_reason length) in 6 rows.
  • C4: 2 LLM responses ended at max_tokens (finish_reason length) in 2 rows.
  • C5: 14 LLM responses ended at max_tokens (finish_reason length) in 5 rows.
  • C5: ledger spend in no kept row = 0.18153 USD (real cost only): 20261103T174646Z_C5_experiment / imo_1987_p6: 0.02309 USD, 8 paid calls; 20261103T212359Z_C5_experiment_resume / imo_1984_p6: 0.15844 USD, 15 paid calls.

Method notes (interpretations of docs/analysis_plan.md and data anomalies)

Hand-written; appended verbatim to results.md by make_tables.py. Each item is a point where the plan had to be read literally or was silent, or a fact found in the data.

Analysis code

  1. Analysis code. Plan §7 places the statistics in src/formal_mas/analysis/stats.py. The first version of this analysis implemented them inside docs/results/make_tables.py (the file did not exist yet); on 2026-11-07, by team decision, they were moved to src/formal_mas/analysis/stats.py (Wilson, exact McNemar, Bonferroni, bootstrap, quartiles), with offline tests against closed-form values in tests/test_stats.py. make_tables.py imports them, and the regenerated results.md and tables.tex were checked to be byte for byte identical to the earlier output (before Table 10 was switched to the post-test fidelity review). Exact binomial tests and Wilson intervals come from scipy.stats.binomtest (Wilson without continuity correction). The bootstrap uses numpy.random.default_rng. The κ classifier is imported read-only from src/formal_mas/orchestrator/policy.py. scripts/compare.py was not written: make_tables.py produces every table of §7.
  2. Dev before test (plan §7). The script was first run in --dev mode on dev runs, using stand-ins on the same 15 dev problems, and checked there: Wilson 11/15 = [48.0, 89.1], exact McNemar p values computed by hand, and the C1 κ classification on 20261103T161957Z_C1_experiment_c1_cost. Unit tests of the statistics were added later (item 1). Disclosure: before the script existed, the analyst had seen the first test row of C5 (mathd_algebra_206, solved) and the summaries stored in the run_config.json of the C5 resume runs and of C4, while checking the configurations. After the first test run, the script changed in three ways: (a) C2 wall time now stops at claim_check_start (the row's wall_s includes the claim check, and the 2026-11-03 Clarification excludes the check from C2's figures); (b) a note on responses cut at max_tokens was added; (c) LaTeX layout. No test, comparison, correction or metric of the plan was changed.

Merges and integrity

  1. C5 merge follows the C5 Deviations entry exactly. It keeps 23 rows of 20261103T174646Z_C5_experiment, imo_1987_p6 and imo_1979_p1 of ..._resume, and 5 rows of ..._resume2. The budget_stop row of imo_1984_p6 in resume 1 is discarded; the script asserts its status. The script fails if a row is kept twice or if a source row is neither kept nor discarded.
  2. C1 uses runs/20261107T200233Z_C1_experiment_resume/C1_test_merged_results.jsonl. Every merged row equals (as parsed JSON) exactly one source row: 23 from 20261105T131443Z_C1_experiment and 7 from the resume.
  3. All five configurations have exactly 30 rows with the same 30 test ids. There is no provider_error, budget_stop, error, node_error, invalid_statement or lean_down row, and no degraded run. The plan's re-run rule and sensitivity analysis (§5) therefore do not apply. Area and difficulty in each row match data/eval.jsonl.
  4. All run_config.json files agree on profile, seed, split, Lean toolchain, Mathlib revision and model settings. git diff ad70ec2 2c8eae5 -- src config shows that only config/pipeline.yaml: spend.run_usd changed (the documented spend-cap deviations). The input sha256 of the resume runs match the Deviations entry. Anomaly: run_config.json records dirty: true for C5 resume 1, C5 resume 2, C4 and the C1 resume, with changed_files = [docs/analysis_plan.md] only. The Deviations entry calls these trees "clean". The only uncommitted file was the plan itself.
  5. The first C5 run and the first C1 run have run status running: they were killed, as documented. Their results files hold only finished problems.

Primary outcome and comparisons

  1. C2 in the SR table. Following §7.1, C2 has no verified SR. It enters through its claimed rate |B⁺|/30 = 30/30 and FPR_B. The 19 claims that the check verified are reported as a count only, because the plan defines no SR for them. C2 is not in any McNemar test (§3). In the per-cell and margin tables, C2 shows claimed counts, labelled as such, because §4.1 says "for each configuration".
  2. McNemar: b = first configuration solved and second not; c = the reverse; two-sided exact binomial on b | b + c with p = 1/2 (p = 1 if b + c = 0). Bonferroni-adjusted p = min(1, 3p); a test is significant if p < 0.05/3.
  3. Paired bootstrap (plan: 10 000 resamples of problems, seed 2026, percentile). The generator is a fresh numpy.random.default_rng(2026) for each statistic, so all comparisons use the same 10 000 index vectors. The plan does not name a generator. For H2 the lower bound is exactly 0.0 pp.
  4. Exploratory comparisons (C3 vs C4, C1 vs C3, C1 vs C4) get b, c, ΔSR and an uncorrected exact p only. The plan specifies no bootstrap CI for them.

FPR_B (Eq. 16, §4.2, Clarification 2026-11-03)

  1. B⁺ holds all 30 problems (every C2 row is claimed_solved). Check status: 19 verified, 11 not_verified. No check ended in budget_exhausted, provider_error, budget_stop, error or lean_down, so none is left out. FPR_B = 11/30.
  2. Manual-review sample. 11 claims are not verified, which is more than 10, so 10 are drawn "with seed 2026". The plan does not give the drawing procedure; random.Random(2026).sample(sorted(ids), 10) was used, which leaves out amc12a_2008_p15. The manual review itself (informal proof wrong vs. formalization failure) is a task for the team and has not been done.
  3. Factual cross-reference, not a conclusion: for 5 of the 11 not-verified claims (amc12a_2008_p15, amc12b_2002_p3, imo_1964_p1_2, imo_1965_p2, putnam_1965_b2), C5 has a kernel-accepted proof of the same formal statement.

Efficiency (§4.3)

  1. C(x) = llm.llm_calls of the row. It includes cache hits and JSON-repair calls, as the budget counts them. Tokens = prompt + completion (reasoning tokens are part of completion in the ledger). Nominal cost (cost_usd_nominal) is the compared figure (§1). Real cost is shown as well.
  2. Real cost. The real cost of every kept row equals the ledger sum for (run_id, problem_id); no mismatch was found. The ledger totals include spend that is in no kept row: C1 +0.09845 USD (imo_1987_p6 in the first C1 run: 10 calls, 5 paid and 5 cached JSON repairs), so C1 = 2.4695 + 0.0985 = 2.5679 USD. C5 +0.18153 USD (imo_1987_p6 in the first run: 0.02309 USD, 8 paid calls; imo_1984_p6 in resume 1: 0.15844 USD, 15 paid calls), so C5 = 1.1770 + 0.1815 = 1.3585 USD. C2 = 0.0814 (C2 itself) + 0.6269 (claim check) = 0.7083 USD. C3 = 1.3500 USD. C4 = 0.0000 USD. Per-problem and per-solved figures come from kept rows only.
  3. C2 efficiency leaves out the claim check: its calls, tokens and cost come from the row's llm, and its wall time runs from problem_start to claim_check_start. C̄_solved needs a "solved" count, and C2 has no verified one, so its ratio uses claimed solutions (30).
  4. C4: all 132 LLM calls were cache hits (requests identical to C5's), so its real cost is 0 and its LLM latency is near zero. Lean verification was real (5 of 259 lean_verify events were Lean-cache hits). C4's wall time (about 0.1 min per problem) therefore does not measure LLM computation. Nominal cost is the comparable figure (§1).
  5. Bootstrap CIs for the efficiency ratios: plan §4.3 gives no seed, so the same procedure as note 10 is used (seed 2026, percentile). For Σx/#solved, resamples with no solved problem would be dropped; none occurred on test.
  6. The median/IQR table (4d) and wall time by difficulty (Table 5) were requested by the team. They are descriptive only and not in the plan.

Refinement robustness (§4.4, Eq. 14)

  1. Accepted plan. The trace of a solved C5 problem is split at each plan_proposed / plan_fallback that follows subgoal work (attempt counters restart after a replan). The accepted plan is the last segment. "Attempts on a subgoal" is the attempts field of subgoal_solved; pre-pass subgoals count as 0. Read literally ("its accepted proof used at most k attempts on every subgoal"), a replanned problem counts by its accepted plan only. This affects algebra_absapbon1pabsapbleqsumabsaon1pabsa: C5 replanned once, then solved every subgoal on its first attempt, so it counts at k = 1. It is the only problem that differs between SR_C5(1) = 15/30 and SR_C4 = 14/30. SR_C5(3) = SR_C5 = 24/30, as expected.
  2. SR(0). Δ̂(1) = SR(1) − SR(0) needs SR(0), which the plan does not define. The same rule with k = 0 is used: the accepted plan has no LLM-attempted subgoal (pre-pass or automatic closing only). This gives 9/30.
  3. Independence curve. The plan says p_i is "estimated from first-attempt subgoal success in C5". Each subgoal has one first attempt, so a per-subgoal p_i would be 0 or 1. A pooled p̂ = 18/44 is used instead, counting every LLM-attempted subgoal of every plan (including abandoned plans and failed problems). The curve reported is Eq. 14's per-subgoal marginal gain Δ(k) = p̂(1 − p̂)^(k−1). It sits at a different level (subgoal) from the problem-level Δ̂(k). The last two columns of Table 6 give the subgoal-level empirical counterpart and are supplementary. The figure the plan asks for (§7.5) is docs/results/figures/srk_curve.pdf (and .png), drawn from the same numbers: (a) SR_C5(k), k = 0..3, with Wilson CIs and C4 at k = 1; (b) Δ̂(k) against p̂(1 − p̂)^(k−1).

Error taxonomy (§4.5)

  1. For C3–C5 the classes are the rows' error_classes, one per Critic event (checked equal to the critique events), including the k_max escalation to planning. Only kept rows count; the discarded attempts of the merges are excluded.
  2. C1 attempts are classified offline with policy.classify(Trace(reason, messages, None)), using the stored lean_errors (at most 3 messages of 300 characters, versus 5 × 1500 in the graph). The statement end is unknown, so misformalization cannot fire, which fits since C1 never writes statements. escalate is not applied because C1 has no k_max or strategies. The 44 sanitize:empty attempts (classed as syntax) happen alongside 52 C1 responses that ended at max_tokens.
  3. C2 has no verified attempts of its own. Its taxonomy row is the claim-check loop (labelled). The plan does not say how C2 enters §4.5.
  4. Final failure reasons group stages by prefix (verify:sg2 becomes verify).

Other data facts

  1. Responses that ended at max_tokens (finish_reason = length): C1 52 (9 rows), C3 18 (6 rows), C5 14 (5 rows), C4 2 (2 rows), C2 0 (12 in the claim check).
  2. C1 budget_exhausted rows (7) reached b_max = 18 calls after 9–17 recorded attempts, because JSON-repair calls count against the budget. The 3 failed C1 rows used all 18 attempts.
  3. Fidelity (§6). Four solved test problems carry a fidelity note: amc12b_2003_p6, mathd_algebra_234, mathd_algebra_455 and putnam_1965_b2. The team reviewed them on 2026-11-07 (post_test_review in data/curation/fidelity_review.yaml; Table 10): all four count as solved, amc12b_2003_p6 with a note in the report. The verified SR is reported unchanged, as §6 requires.
  4. Supplementary, not in the plan: Table 8 (pre-pass vs LLM solved_by) and Table 4d.

Illustrative traces (for the appendix)

  • amc12a_2008_p15 (Critic loop). C5 solved it on the third attempt at its only subgoal (two tactic_failure retries with Critic feedback, 16 calls, first C5 run). C4 abandoned after the first failure. C3 failed after a replan (30 calls). C1 failed. The C2 claim was not verified.
  • numbertheory_sumkmulnckeqnmul2pownm1 (RAG). C5 solved it with 5 calls on the first attempt. C3 failed after 4 missing_lemma and 2 tactic_failure attempts over two plans (30 calls).
  • imo_1965_p2 (FPR_B as an upper bound). C2 claimed it, and its check failed (missing_lemma, tactic_failure, missing_lemma). C5 found a kernel-accepted proof: the pre-pass closed one subgoal and the LLM closed the other two (one retry after missing_lemma).