# Tables for the `test` split (n = 30) Generated by `docs/results/make_tables.py` from `runs/` (read-only), following `docs/analysis_plan.md`. Rates in %, intervals 95 %. ### Runs used | Run | Config | Profile | Seed | Split | Commit | Tree | Degraded | Run status | vs first run | |---|---|---|---|---|---|---|---|---|---| | 20261103T174646Z_C5_experiment | C5 | experiment | 2026 | test | ad70ec2 | clean | False | running | same | | 20261103T212359Z_C5_experiment_resume | C5 | experiment | 2026 | test | ad70ec2 | dirty (plan file only) | False | budget_stop | same | | 20261103T225140Z_C5_experiment_resume2 | C5 | experiment | 2026 | test | 93bc701 | dirty (plan file only) | False | finished | same | | 20261103T235556Z_C4_experiment | C4 | experiment | 2026 | test | ea4db4b | dirty (plan file only) | False | finished | same | | 20261104T194056Z_C3_experiment | C3 | experiment | 2026 | test | 6c61d69 | clean | False | finished | same | | 20261104T232056Z_C2_experiment_check | C2 | experiment | 2026 | test | 6c61d69 | clean | False | finished | same | | 20261105T131443Z_C1_experiment | C1 | experiment | 2026 | test | 5ed7507 | clean | False | running | same | | 20261107T200233Z_C1_experiment_resume | C1 | experiment | 2026 | test | 2c8eae5 | dirty (plan file only) | False | finished | same | ### Row checks | Config | Rows | Status counts | lean_down | split/config mismatch | area/difficulty mismatch | |---|---|---|---|---|---| | C1 | 30 | budget_exhausted: 7, failed: 3, solved: 20 | 0 | 0 | 0 | | C2 | 30 | claimed_solved: 30 | 0 | 0 | 0 | | C3 | 30 | budget_exhausted: 3, failed: 7, solved: 20 | 0 | 0 | 0 | | C4 | 30 | failed: 16, solved: 14 | 0 | 0 | 0 | | C5 | 30 | failed: 6, solved: 24 | 0 | 0 | 0 | ### Table 1. Verified success rate (SR) and Baseline B | Config | Solved | SR (%) | Wilson 95 % CI (%) | C2 claimed rate (%) [Wilson] | FPR_B (%) [Wilson] | |---|---|---|---|---|---| | C1 | 20/30 | 66.7 | [48.8, 80.8] | — | — | | C3 | 20/30 | 66.7 | [48.8, 80.8] | — | — | | C4 | 14/30 | 46.7 | [30.2, 63.9] | — | — | | C5 | 24/30 | 80.0 | [62.7, 90.5] | — | — | | C2 (Baseline B) | — | — | — | 30/30 = 100.0 [88.6, 100.0] | 11/30 = 36.7 [21.9, 54.5] | *Note.* C2 is not verified, so it has no SR; it enters only through its claimed rate B⁺/n and FPR_B (plan §7.1). Of the 30 claimed proofs the formal check verified 19 and did not verify 11; 0 left out of FPR_B. ### Table 1b. Baseline B claim check (FPR_B, Eq. 16) | Quantity | Value | |---|---| | Problems | 30 | | Claimed (B⁺) | 30 | | Not claimed | 0 | | Check status among B⁺ | not_verified: 11, verified: 19 | | Verified (Acc on the assembled theorem) | 19 | | Not verified (budget_exhausted included) | 11 | | Left out of FPR_B (provider_error, budget_stop, error, lean_down) | 0 | | FPR_B = not verified / (verified + not verified) | 11/30 = 36.7 % [21.9, 54.5] | | B⁺ problems for the manual review (§4.2) | 10 drawn with random.Random(2026).sample(sorted ids, 10) (listed below) | *Note.* Manual review set (B⁺ not verified): amc12b_2002_p3, imo_1964_p1_2, imo_1965_p2, imo_1978_p5, imo_1979_p1, imo_1984_p6, imo_1987_p6, imo_1993_p5, putnam_1965_b2, putnam_1995_b1. ### Table 2. Planned comparisons H1–H3 (exact McNemar, Bonferroni ×3) | ID | Comparison | b | c | ΔSR (pp) | Paired bootstrap 95 % CI (pp) | Exact p | Bonferroni p | Reject at 0.05/3 | |---|---|---|---|---|---|---|---|---| | H1 | C5 vs C1 | 4 | 0 | +13.3 | [+3.3, +26.7] | 0.1250 | 0.3750 | no | | H2 | C5 vs C3 | 5 | 1 | +13.3 | [+0.0, +30.0] | 0.2188 | 0.6562 | no | | H3 | C5 vs C4 | 10 | 0 | +33.3 | [+16.7, +50.0] | 0.0020 | 0.0059 | yes | *Note.* n = 30, so power is low: a non-significant result is no evidence of a difference, not evidence of no difference. H2 is Δ_RAG = SR_C5 − SR_C3; H3 is the Critic contribution SR_C5 − SR_C4. ### Table 2b. Exploratory comparisons (uncorrected, no claims) | ID | Comparison | b | c | ΔSR (pp) | Exact p (uncorrected) | |---|---|---|---|---|---| | E1 | C3 vs C4 | 9 | 3 | +20.0 | 0.1460 | | E2 | C1 vs C3 | 3 | 3 | +0.0 | 1.0000 | | E3 | C1 vs C4 | 8 | 2 | +20.0 | 0.1094 | ### Table 3. Solved / n per cell of the 3 × 3 grid (descriptive) | Difficulty | Area | n | C1 | C3 | C4 | C5 | C2 (claimed) | |---|---|---|---|---|---|---|---| | easy | Algebra | 5 | 5/5 | 5/5 | 4/5 | 5/5 | 5/5 | | easy | Number theory | 5 | 5/5 | 4/5 | 5/5 | 5/5 | 5/5 | | easy | Combinatorics | 0 | — | — | — | — | — | | medium | Algebra | 5 | 4/5 | 5/5 | 1/5 | 5/5 | 5/5 | | medium | Number theory | 4 | 2/4 | 2/4 | 2/4 | 4/4 | 4/4 | | medium | Combinatorics | 1 | 0/1 | 0/1 | 1/1 | 1/1 | 1/1 | | hard | Algebra | 3 | 1/3 | 2/3 | 0/3 | 1/3 | 3/3 | | hard | Number theory | 4 | 1/4 | 1/4 | 0/4 | 1/4 | 4/4 | | hard | Combinatorics | 3 | 2/3 | 1/3 | 1/3 | 2/3 | 3/3 | ### Table 3b. SR by difficulty and by area (margins, Wilson 95 % CI) | Config | easy (n = 10) | medium (n = 10) | hard (n = 10) | Algebra (n = 13) | Number theory (n = 13) | Combinatorics (n = 4) | |---|---|---|---|---|---|---| | C1 | 10/10 | 6/10 | 4/10 | 10/13 | 8/13 | 2/4 | | | 100.0 [72.2, 100.0] | 60.0 [31.3, 83.2] | 40.0 [16.8, 68.7] | 76.9 [49.7, 91.8] | 61.5 [35.5, 82.3] | 50.0 [15.0, 85.0] | | C3 | 9/10 | 7/10 | 4/10 | 12/13 | 7/13 | 1/4 | | | 90.0 [59.6, 98.2] | 70.0 [39.7, 89.2] | 40.0 [16.8, 68.7] | 92.3 [66.7, 98.6] | 53.8 [29.1, 76.8] | 25.0 [4.6, 69.9] | | C4 | 9/10 | 4/10 | 1/10 | 5/13 | 7/13 | 2/4 | | | 90.0 [59.6, 98.2] | 40.0 [16.8, 68.7] | 10.0 [1.8, 40.4] | 38.5 [17.7, 64.5] | 53.8 [29.1, 76.8] | 50.0 [15.0, 85.0] | | C5 | 10/10 | 10/10 | 4/10 | 11/13 | 10/13 | 3/4 | | | 100.0 [72.2, 100.0] | 100.0 [72.2, 100.0] | 40.0 [16.8, 68.7] | 84.6 [57.8, 95.7] | 76.9 [49.7, 91.8] | 75.0 [30.1, 95.4] | | C2 (claimed) | 10/10 | 10/10 | 10/10 | 13/13 | 13/13 | 4/4 | | | 100.0 [72.2, 100.0] | 100.0 [72.2, 100.0] | 100.0 [72.2, 100.0] | 100.0 [77.2, 100.0] | 100.0 [77.2, 100.0] | 100.0 [51.0, 100.0] | ### Table 4. Efficiency per solved problem (RNF-3) | Config | Solved | Calls / solved | k tokens / solved | Nominal USD / solved | Real USD / solved | Wall min / solved | |---|---|---|---|---|---|---| | C1 | 20 | 12.85 [7.19, 22.60] | 219.7 [114.6, 394.7] | 0.1285 [0.0668, 0.2313] | 0.1235 [0.0643, 0.2218] | 17.7 [9.2, 32.0] | | C2 (per claimed) | 30 | 2.17 [2.00, 2.40] | 5.3 [2.8, 8.6] | 0.0027 [0.0013, 0.0047] | 0.0027 [0.0013, 0.0047] | 0.4 [0.2, 0.7] | | C3 | 20 | 26.50 [16.15, 43.67] | 150.4 [79.3, 274.6] | 0.0675 [0.0335, 0.1279] | 0.0675 [0.0335, 0.1278] | 9.2 [4.6, 17.3] | | C4 | 14 | 9.43 [5.72, 16.67] | 70.7 [38.2, 137.8] | 0.0278 [0.0135, 0.0571] | 0.0000 [0.0000, 0.0000] | 0.2 [0.1, 0.3] | | C5 | 24 | 16.92 [10.63, 26.21] | 148.0 [74.4, 266.4] | 0.0566 [0.0245, 0.1097] | 0.0490 [0.0236, 0.0897] | 7.6 [3.3, 14.6] | *Note.* C2 has no verified solutions: its ratio uses claimed solutions and excludes the claim check (calls, tokens, cost and wall time up to claim_check_start). ### Table 4b. Efficiency per problem (means) | Config | Calls / problem | k tokens / problem | Nominal USD / problem | Real USD / problem | Wall min / problem | |---|---|---|---|---|---| | C1 | 8.57 [5.83, 11.33] | 146.4 [91.1, 203.5] | 0.0857 [0.0532, 0.1192] | 0.0823 [0.0505, 0.1152] | 11.8 [7.2, 16.5] | | C2 | 2.17 [2.00, 2.40] | 5.3 [2.8, 8.6] | 0.0027 [0.0013, 0.0047] | 0.0027 [0.0013, 0.0047] | 0.4 [0.2, 0.7] | | C3 | 17.67 [12.97, 22.37] | 100.3 [63.8, 140.8] | 0.0450 [0.0269, 0.0658] | 0.0450 [0.0269, 0.0658] | 6.1 [3.7, 9.0] | | C4 | 4.40 [3.37, 5.43] | 33.0 [22.8, 44.4] | 0.0130 [0.0081, 0.0186] | 0.0000 [0.0000, 0.0000] | 0.1 [0.1, 0.1] | | C5 | 13.53 [9.57, 17.60] | 118.4 [67.9, 178.7] | 0.0453 [0.0225, 0.0737] | 0.0392 [0.0218, 0.0588] | 6.1 [3.0, 9.7] | ### Table 4c. Totals, cache use and real cost | Config | LLM calls | Cached calls | Cached (%) | Tokens (M) | Nominal USD (rows) | Real USD (rows) | Real USD (ledger, all runs) | |---|---|---|---|---|---|---|---| | C1 | 257 | 47 | 18.3 | 4.393 | 2.5700 | 2.4695 | 2.5679 | | C2 | 65 | 0 | 0.0 | 0.158 | 0.0814 | 0.0814 | 0.7083 | | C3 | 530 | 18 | 3.4 | 3.009 | 1.3508 | 1.3500 | 1.3500 | | C4 | 132 | 132 | 100.0 | 0.990 | 0.3887 | 0.0000 | 0.0000 | | C5 | 406 | 34 | 8.4 | 3.551 | 1.3590 | 1.1770 | 1.3585 | *Note.* C2 claim check (not in the C2 figures of the other columns): 77 calls (11 cached), 0.6274 nominal, 0.6269 real. **RNF-3 (C̄_solved ≤ 20 for C5):** C̄_solved = 406/24 = 16.92 calls per solved problem: **met**. ### Table 4d. Median [IQR] of LLM calls and wall time per problem (supplementary) | Config | Calls, median [Q1, Q3] | Wall min, median [Q1, Q3] | |---|---|---| | C1 | 4.5 [1.0, 18.0] | 2.7 [0.6, 24.2] | | C2 | 2.0 [2.0, 2.0] | 0.1 [0.1, 0.3] | | C3 | 16.5 [3.2, 28.0] | 3.7 [0.6, 6.7] | | C4 | 5.0 [1.0, 5.8] | 0.1 [0.0, 0.1] | | C5 | 11.5 [1.0, 23.8] | 2.4 [0.2, 5.2] | ### Table 5. Wall time per problem by difficulty, minutes (descriptive) | Config | easy (n = 10): mean / median [Q1, Q3] | medium (n = 10): mean / median [Q1, Q3] | hard (n = 10): mean / median [Q1, Q3] | |---|---|---|---| | C1 | 0.7 / 0.4 [0.1, 1.1] | 14.3 / 16.2 [1.8, 23.2] | 20.3 / 25.3 [7.0, 32.4] | | C2 | 0.1 / 0.1 [0.1, 0.2] | 0.1 / 0.1 [0.1, 0.2] | 1.0 / 0.5 [0.2, 1.1] | | C3 | 0.8 / 0.1 [0.0, 1.0] | 4.0 / 2.6 [1.2, 6.0] | 13.6 / 12.3 [6.6, 19.8] | | C4 | 0.1 / 0.0 [0.0, 0.1] | 0.1 / 0.1 [0.0, 0.1] | 0.1 / 0.1 [0.1, 0.1] | | C5 | 0.4 / 0.0 [0.0, 0.5] | 2.7 / 2.6 [0.7, 4.5] | 15.1 / 14.1 [4.0, 21.0] | ### Table 6. Refinement robustness: SR(k) of C5 vs the independence curve (Eq. 14) | k | SR_C5(k) [Wilson] (%) | Δ̂(k) = SR(k) − SR(k−1) (pp) | Indep. Δ(k) = p̂(1−p̂)^(k−1) (%) | Subgoals solved at attempt k | Solved at k / reached k | |---|---|---|---|---|---| | 1 | 15/30 = 50.0 [33.2, 66.8] | +20.0 | 40.9 | 18/44 = 40.9 | 18/44 = 40.9 | | 2 | 19/30 = 63.3 [45.5, 78.1] | +13.3 | 24.2 | 8/44 = 18.2 | 8/26 = 30.8 | | 3 | 24/30 = 80.0 [62.7, 90.5] | +16.7 | 14.3 | 5/44 = 11.4 | 5/18 = 27.8 | *Note.* SR_C5(0) = 9/30 (accepted plan with no LLM subgoal attempt). p̂ = 18/44 = 40.9 %. k = 1 from C4 (no retries, no Critic advice): 14/30 = 46.7 % [30.2, 63.9]. The last two columns are subgoal-level diagnostics (supplementary). ### Table 7. κ error taxonomy over all failed attempts (Eq. 11) | Config | Failed attempts | syntax | missing lemma | tactic failure | misformalization | timeout | planning | |---|---|---|---|---|---|---|---| | C1 | 186 | 60 (32.3) | 50 (26.9) | 68 (36.6) | 0 (0.0) | 8 (4.3) | 0 (0.0) | | C2 (claim check) | 45 | 14 (31.1) | 8 (17.8) | 22 (48.9) | 0 (0.0) | 1 (2.2) | 0 (0.0) | | C3 | 82 | 20 (24.4) | 17 (20.7) | 42 (51.2) | 0 (0.0) | 2 (2.4) | 1 (1.2) | | C4 | 16 | 5 (31.2) | 4 (25.0) | 7 (43.8) | 0 (0.0) | 0 (0.0) | 0 (0.0) | | C5 | 57 | 19 (33.3) | 18 (31.6) | 18 (31.6) | 0 (0.0) | 2 (3.5) | 0 (0.0) | *Note.* C3–C5: classes recorded by the Critic. C1: attempts classified offline with policy.classify (first 3 Lean messages, 300 characters each, as stored; no k_max escalation). C2 has no verified attempts of its own: its row is the claim-check loop. ### Table 7b. κ taxonomy per area | Config | Area | Failed attempts | syntax | missing lemma | tactic failure | misformalization | timeout | planning | |---|---|---|---|---|---|---|---|---| | C1 | Algebra | 77 | 14 (18.2) | 40 (51.9) | 22 (28.6) | 0 (0.0) | 1 (1.3) | 0 (0.0) | | C1 | Number theory | 83 | 27 (32.5) | 3 (3.6) | 46 (55.4) | 0 (0.0) | 7 (8.4) | 0 (0.0) | | C1 | Combinatorics | 26 | 19 (73.1) | 7 (26.9) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | | C2 (claim check) | Algebra | 14 | 5 (35.7) | 4 (28.6) | 5 (35.7) | 0 (0.0) | 0 (0.0) | 0 (0.0) | | C2 (claim check) | Number theory | 23 | 5 (21.7) | 4 (17.4) | 13 (56.5) | 0 (0.0) | 1 (4.3) | 0 (0.0) | | C2 (claim check) | Combinatorics | 8 | 4 (50.0) | 0 (0.0) | 4 (50.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | | C3 | Algebra | 18 | 3 (16.7) | 9 (50.0) | 5 (27.8) | 0 (0.0) | 0 (0.0) | 1 (5.6) | | C3 | Number theory | 43 | 13 (30.2) | 3 (7.0) | 26 (60.5) | 0 (0.0) | 1 (2.3) | 0 (0.0) | | C3 | Combinatorics | 21 | 4 (19.0) | 5 (23.8) | 11 (52.4) | 0 (0.0) | 1 (4.8) | 0 (0.0) | | C4 | Algebra | 8 | 1 (12.5) | 3 (37.5) | 4 (50.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | | C4 | Number theory | 6 | 2 (33.3) | 1 (16.7) | 3 (50.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | | C4 | Combinatorics | 2 | 2 (100.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | | C5 | Algebra | 25 | 7 (28.0) | 8 (32.0) | 10 (40.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | | C5 | Number theory | 24 | 8 (33.3) | 9 (37.5) | 6 (25.0) | 0 (0.0) | 1 (4.2) | 0 (0.0) | | C5 | Combinatorics | 8 | 4 (50.0) | 1 (12.5) | 2 (25.0) | 0 (0.0) | 1 (12.5) | 0 (0.0) | ### Table 7c. Final failure reason per problem (status @ stage) | Config | Problems | status @ stage: count | |---|---|---| | C1 | 10 | budget_exhausted: 7; failed: 3 | | C2 | 30 | claimed, check verified: 19; claimed, check not_verified: 11 | | C3 | 10 | failed @ verify: 5; budget_exhausted @ formalize: 3; failed @ formalize: 2 | | C4 | 16 | failed @ verify: 11; failed @ formalize: 5 | | C5 | 6 | failed @ verify: 4; failed @ formalize: 2 | ### Table 8. Subgoals closed by the tactic pre-pass vs the LLM (supplementary) | Config | Pre-pass (accepted plans) | LLM (accepted plans) | Pre-pass (all subgoal_solved) | Solved with no LLM subgoal | Solved | |---|---|---|---|---|---| | C3 | 7/32 = 21.9 | 25/32 = 78.1 | 8/39 = 20.5 | 8 | 20 | | C4 | 9/18 = 50.0 | 9/18 = 50.0 | 11/21 = 52.4 | 9 | 14 | | C5 | 11/39 = 28.2 | 28/39 = 71.8 | 11/42 = 26.2 | 9 | 24 | ### Table 9. Infrastructure and error statuses in the kept rows (plan §5, §7.7) | Config | provider_error | budget_stop | error | node_error | invalid_statement | lean_down | |---|---|---|---|---|---|---| | C1 | 0 | 0 | 0 | 0 | 0 | 0 | | C2 | 0 | 0 | 0 | 0 | 0 | 0 | | C3 | 0 | 0 | 0 | 0 | 0 | 0 | | C4 | 0 | 0 | 0 | 0 | 0 | 0 | | C5 | 0 | 0 | 0 | 0 | 0 | 0 | ### Table 10. Solved test problems with a fidelity note (plan §6) | Problem | Solved by | Manual review | |---|---|---| | amc12b_2003_p6 | C1, C3, C4, C5 | 2026-11-07: counts as solved; report note: The formalization proves that u 0 = +-2/sqrt 3 (a necessary condition), not that -2*sqrt(3)/3 is attained; this is enough to rule out the other options. | | mathd_algebra_234 | C1, C3, C4, C5 | 2026-11-07: counts as solved; no note | | mathd_algebra_455 | C1, C3, C4, C5 | 2026-11-07: counts as solved; no note | | putnam_1965_b2 | C1, C3, C5 | 2026-11-07: counts as solved; no note | ### Table 11. Per-problem outcomes | Problem | Difficulty | Area | C1 | C3 | C4 | C5 | C2 (claim, check) | |---|---|---|---|---|---|---|---| | algebra_absapbon1pabsapbleqsumabsaon1pabsa | medium | Algebra | 0 | 1 | 0 | 1 | claimed, verified | | amc12_2000_p20 | medium | Algebra | 1 | 1 | 0 | 1 | claimed, verified | | amc12_2001_p9 | medium | Algebra | 1 | 1 | 0 | 1 | claimed, verified | | amc12a_2008_p15 | medium | Number theory | 0 | 0 | 0 | 1 | claimed, not_verified | | amc12a_2008_p25 | medium | Algebra | 1 | 1 | 0 | 1 | claimed, verified | | amc12b_2002_p3 | medium | Number theory | 0 | 1 | 1 | 1 | claimed, not_verified | | amc12b_2003_p6 | medium | Algebra | 1 | 1 | 1 | 1 | claimed, verified | | imo_1964_p1_2 | hard | Number theory | 1 | 1 | 0 | 1 | claimed, not_verified | | imo_1965_p2 | hard | Algebra | 1 | 1 | 0 | 1 | claimed, not_verified | | imo_1978_p5 | hard | Algebra | 0 | 0 | 0 | 0 | claimed, not_verified | | imo_1979_p1 | hard | Number theory | 0 | 0 | 0 | 0 | claimed, not_verified | | imo_1984_p6 | hard | Number theory | 0 | 0 | 0 | 0 | claimed, not_verified | | imo_1987_p6 | hard | Number theory | 0 | 0 | 0 | 0 | claimed, not_verified | | imo_1993_p5 | hard | Algebra | 0 | 1 | 0 | 0 | claimed, not_verified | | mathd_algebra_160 | easy | Algebra | 1 | 1 | 1 | 1 | claimed, verified | | mathd_algebra_206 | easy | Algebra | 1 | 1 | 0 | 1 | claimed, verified | | mathd_algebra_234 | easy | Algebra | 1 | 1 | 1 | 1 | claimed, verified | | mathd_algebra_288 | easy | Algebra | 1 | 1 | 1 | 1 | claimed, verified | | mathd_algebra_455 | easy | Algebra | 1 | 1 | 1 | 1 | claimed, verified | | mathd_numbertheory_321 | easy | Number theory | 1 | 0 | 1 | 1 | claimed, verified | | mathd_numbertheory_328 | easy | Number theory | 1 | 1 | 1 | 1 | claimed, verified | | mathd_numbertheory_345 | easy | Number theory | 1 | 1 | 1 | 1 | claimed, verified | | mathd_numbertheory_430 | easy | Number theory | 1 | 1 | 1 | 1 | claimed, verified | | mathd_numbertheory_64 | easy | Number theory | 1 | 1 | 1 | 1 | claimed, verified | | numbertheory_3pow2pownm1mod2pownp3eq2pownp2 | medium | Number theory | 1 | 0 | 0 | 1 | claimed, verified | | numbertheory_notequiv2i2jasqbsqdiv8 | medium | Number theory | 1 | 1 | 1 | 1 | claimed, verified | | numbertheory_sumkmulnckeqnmul2pownm1 | medium | Combinatorics | 0 | 0 | 1 | 1 | claimed, verified | | putnam_1964_b2 | hard | Combinatorics | 1 | 0 | 1 | 1 | claimed, verified | | putnam_1965_b2 | hard | Combinatorics | 1 | 1 | 0 | 1 | claimed, not_verified | | putnam_1995_b1 | hard | Combinatorics | 0 | 0 | 0 | 0 | claimed, not_verified | ### Table 11b. Hard problems per configuration (supplementary; post hoc, descriptive, not in the analysis plan) | Problem | Area | C1 | C3 | C4 | C5 | Solved by | |---|---|---|---|---|---|---| | imo_1964_p1_2 | Number theory | 1 | 1 | 0 | 1 | 3 | | imo_1965_p2 | Algebra | 1 | 1 | 0 | 1 | 3 | | imo_1978_p5 | Algebra | 0 | 0 | 0 | 0 | 0 | | imo_1979_p1 | Number theory | 0 | 0 | 0 | 0 | 0 | | imo_1984_p6 | Number theory | 0 | 0 | 0 | 0 | 0 | | imo_1987_p6 | Number theory | 0 | 0 | 0 | 0 | 0 | | imo_1993_p5 | Algebra | 0 | 1 | 0 | 0 | 1 | | putnam_1964_b2 | Combinatorics | 1 | 0 | 1 | 1 | 3 | | putnam_1965_b2 | Combinatorics | 1 | 1 | 0 | 1 | 3 | | putnam_1995_b1 | Combinatorics | 0 | 0 | 0 | 0 | 0 | | Solved | | 4/10 | 4/10 | 1/10 | 4/10 | | *Note.* Post hoc, descriptive (not in the analysis plan): no tests, no intervals. 1 = verified solved. C1 and C5 solve the same hard problems (4 and 4); 5 hard problems are solved by no configuration (imo_1978_p5, imo_1979_p1, imo_1984_p6, imo_1987_p6, putnam_1995_b1); C3 and C5 solve 4 and 4 hard problems, but only C3 solves imo_1993_p5 and only C5 solves putnam_1964_b2. ### Table 12. Exploratory: C5-Flash vs C5-Pro on the hard problems (post hoc, descriptive, not in the analysis plan) | Problem | Flash solved | Flash calls | Flash USD | Flash min | Pro outcome | Pro solved | Pro calls | Pro USD | Pro min | |---|---|---|---|---|---|---|---|---|---| | imo_1964_p1_2 | 1 | 15 | 0.0261 | 3.4 | not attempted | — | — | — | — | | imo_1965_p2 | 1 | 14 | 0.0312 | 3.7 | failed | 0 | 30 | 0.3608 | 33.6 | | imo_1978_p5 | 0 | 31 | 0.1347 | 39.6 | provider_error (connection drop, not a system failure) | — | 30 | 0.2938 | 40.7 | | imo_1979_p1 | 0 | 28 | 0.1463 | 19.7 | failed | 0 | 24 | 0.2722 | 29.3 | | imo_1984_p6 | 0 | 32 | 0.3366 | 27.6 | interrupted | — | — | — | — | | imo_1987_p6 | 0 | 36 | 0.1793 | 21.4 | budget_exhausted | 0 | 40 | 0.4607 | 57.4 | | imo_1993_p5 | 0 | 27 | 0.0790 | 10.6 | failed | 0 | 24 | 0.3807 | 53.6 | | putnam_1964_b2 | 1 | 9 | 0.0160 | 2.3 | not attempted | — | — | — | — | | putnam_1965_b2 | 1 | 17 | 0.0409 | 4.9 | not attempted | — | — | — | — | | putnam_1995_b1 | 0 | 25 | 0.1268 | 17.5 | not attempted | — | — | — | — | | Both finished (4) | 1/4 | 105 | 0.4357 | 55.5 | | 0/4 | 118 | 1.4744 | 173.9 | | All hard (10) | 4/10 | 234 | 1.1169 | 150.9 | | — | — | — | — | *Note.* Post hoc, exploratory (decided 2026-11-08, after all confirmatory runs): descriptive only, no tests, no intervals; it changes no main result. C5-Flash = the kept C5 rows of the main test result; C5-Pro = run 20261108T130803Z_C5_experiment_pro_pro_hard (profile experiment_pro: deepseek-v4-pro in every role, everything else as frozen), stopped by the team and not resumed. Cost is nominal USD (cache hits priced as fresh calls). provider_error, interrupted and not attempted do not count as failures of Pro and are left out of its solved count; only problems both runs finished are compared. imo_1984_p6 was interrupted after 4 paid calls (0.1725 USD real, 18.1 min) and left no row. Real cost of the Pro run: 1.9407 USD (151 paid calls, all in the ledger); project ledger total: 9.3003 USD. **Pro vs Flash on the 4 problems both finished:** 3.38 × the nominal cost and 3.13 × the wall time. **Project spend (ledger, all runs):** main experiments (every run except the exploratory Pro run, `dev` included) 7.3596 USD; exploratory Pro run 1.9407 USD; total 9.3003 USD. ### Discordant pairs (candidates for illustrative traces) - C5 solved, C4 not (Critic loop): algebra_absapbon1pabsapbleqsumabsaon1pabsa (C5: solved, 24 calls, replans 1, max attempts 1); amc12_2000_p20 (C5: solved, 23 calls, replans 0, max attempts 3); amc12_2001_p9 (C5: solved, 7 calls, replans 0, max attempts 2); amc12a_2008_p15 (C5: solved, 16 calls, replans 0, max attempts 3); amc12a_2008_p25 (C5: solved, 23 calls, replans 0, max attempts 2); imo_1964_p1_2 (C5: solved, 15 calls, replans 0, max attempts 3); imo_1965_p2 (C5: solved, 14 calls, replans 0, max attempts 2); mathd_algebra_206 (C5: solved, 28 calls, replans 0, max attempts 2); numbertheory_3pow2pownm1mod2pownp3eq2pownp2 (C5: solved, 15 calls, replans 0, max attempts 3); putnam_1965_b2 (C5: solved, 17 calls, replans 0, max attempts 3) - C4 solved, C5 not: — - C5 solved, C3 not (RAG): amc12a_2008_p15 (C5: solved, 16 calls, replans 0, max attempts 3); mathd_numbertheory_321 (C5: solved, 1 calls, replans 0, max attempts 0); numbertheory_3pow2pownm1mod2pownp3eq2pownp2 (C5: solved, 15 calls, replans 0, max attempts 3); numbertheory_sumkmulnckeqnmul2pownm1 (C5: solved, 5 calls, replans 0, max attempts 1); putnam_1964_b2 (C5: solved, 9 calls, replans 0, max attempts 1) - C3 solved, C5 not: imo_1993_p5 (C5: failed, 27 calls, replans 1, max attempts -) - C5 solved, C1 not: algebra_absapbon1pabsapbleqsumabsaon1pabsa (C5: solved, 24 calls, replans 1, max attempts 1); amc12a_2008_p15 (C5: solved, 16 calls, replans 0, max attempts 3); amc12b_2002_p3 (C5: solved, 9 calls, replans 0, max attempts 1); numbertheory_sumkmulnckeqnmul2pownm1 (C5: solved, 5 calls, replans 0, max attempts 1) - C1 solved, C5 not: — - C2 claimed, check did not verify: amc12a_2008_p15 (C5: solved, 16 calls, replans 0, max attempts 3); amc12b_2002_p3 (C5: solved, 9 calls, replans 0, max attempts 1); imo_1964_p1_2 (C5: solved, 15 calls, replans 0, max attempts 3); imo_1965_p2 (C5: solved, 14 calls, replans 0, max attempts 2); imo_1978_p5 (C5: failed, 31 calls, replans 1, max attempts -); imo_1979_p1 (C5: failed, 28 calls, replans 1, max attempts -); imo_1984_p6 (C5: failed, 32 calls, replans 1, max attempts -); imo_1987_p6 (C5: failed, 36 calls, replans 1, max attempts -); imo_1993_p5 (C5: failed, 27 calls, replans 1, max attempts -); putnam_1965_b2 (C5: solved, 17 calls, replans 0, max attempts 3); putnam_1995_b1 (C5: failed, 25 calls, replans 1, max attempts -) ## Generated data notes - C1 merged file checked row by row: each row equals (as parsed JSON) exactly one source row; 23 from `20261105T131443Z_C1_experiment`, 7 from `20261107T200233Z_C1_experiment_resume`. No source row is left out. - Kept 30 rows from `20261104T232056Z_C2_experiment_check`. - Kept 30 rows from `20261104T194056Z_C3_experiment`. - Kept 30 rows from `20261103T235556Z_C4_experiment`. - Kept 23 rows from `20261103T174646Z_C5_experiment`. - Discarded row `imo_1984_p6` (status `budget_stop`) of `20261103T212359Z_C5_experiment_resume`, as the C5 Deviations entry says. - Kept 2 rows from `20261103T212359Z_C5_experiment_resume`. - Kept 5 rows from `20261103T225140Z_C5_experiment_resume2`. - C1: 52 LLM responses ended at max_tokens (finish_reason length) in 9 rows. - C1: ledger spend in no kept row = 0.09845 USD (real cost only): `20261105T131443Z_C1_experiment` / imo_1987_p6: 0.09845 USD, 5 paid calls. - C2: 0 LLM responses ended at max_tokens (finish_reason length) in 0 rows; 12 more in the claim check. - C3: 18 LLM responses ended at max_tokens (finish_reason length) in 6 rows. - C4: 2 LLM responses ended at max_tokens (finish_reason length) in 2 rows. - C5: 14 LLM responses ended at max_tokens (finish_reason length) in 5 rows. - C5: ledger spend in no kept row = 0.18153 USD (real cost only): `20261103T174646Z_C5_experiment` / imo_1987_p6: 0.02309 USD, 8 paid calls; `20261103T212359Z_C5_experiment_resume` / imo_1984_p6: 0.15844 USD, 15 paid calls. ## Method notes (interpretations of `docs/analysis_plan.md` and data anomalies) Hand-written; appended verbatim to `results.md` by `make_tables.py`. Each item is a point where the plan had to be read literally or was silent, or a fact found in the data. ### Analysis code 1. **Analysis code.** Plan §7 places the statistics in `src/formal_mas/analysis/stats.py`. The first version of this analysis implemented them inside `docs/results/make_tables.py` (the file did not exist yet); on 2026-11-07, by team decision, they were moved to `src/formal_mas/analysis/stats.py` (Wilson, exact McNemar, Bonferroni, bootstrap, quartiles), with offline tests against closed-form values in `tests/test_stats.py`. `make_tables.py` imports them, and the regenerated `results.md` and `tables.tex` were checked to be byte for byte identical to the earlier output (before Table 10 was switched to the post-test fidelity review). Exact binomial tests and Wilson intervals come from `scipy.stats.binomtest` (Wilson without continuity correction). The bootstrap uses `numpy.random.default_rng`. The κ classifier is imported read-only from `src/formal_mas/orchestrator/policy.py`. `scripts/compare.py` was not written: `make_tables.py` produces every table of §7. 2. **Dev before test (plan §7).** The script was first run in `--dev` mode on dev runs, using stand-ins on the same 15 dev problems, and checked there: Wilson 11/15 = [48.0, 89.1], exact McNemar p values computed by hand, and the C1 κ classification on `20261103T161957Z_C1_experiment_c1_cost`. Unit tests of the statistics were added later (item 1). Disclosure: before the script existed, the analyst had seen the first `test` row of C5 (`mathd_algebra_206`, solved) and the summaries stored in the `run_config.json` of the C5 resume runs and of C4, while checking the configurations. After the first `test` run, the script changed in three ways: (a) C2 wall time now stops at `claim_check_start` (the row's `wall_s` includes the claim check, and the 2026-11-03 Clarification excludes the check from C2's figures); (b) a note on responses cut at `max_tokens` was added; (c) LaTeX layout. No test, comparison, correction or metric of the plan was changed. ### Merges and integrity 3. **C5 merge** follows the C5 Deviations entry exactly. It keeps 23 rows of `20261103T174646Z_C5_experiment`, `imo_1987_p6` and `imo_1979_p1` of `..._resume`, and 5 rows of `..._resume2`. The `budget_stop` row of `imo_1984_p6` in resume 1 is discarded; the script asserts its status. The script fails if a row is kept twice or if a source row is neither kept nor discarded. 4. **C1** uses `runs/20261107T200233Z_C1_experiment_resume/C1_test_merged_results.jsonl`. Every merged row equals (as parsed JSON) exactly one source row: 23 from `20261105T131443Z_C1_experiment` and 7 from the resume. 5. All five configurations have exactly 30 rows with the same 30 `test` ids. There is no `provider_error`, `budget_stop`, `error`, `node_error`, `invalid_statement` or `lean_down` row, and no degraded run. The plan's re-run rule and sensitivity analysis (§5) therefore do not apply. Area and difficulty in each row match `data/eval.jsonl`. 6. All `run_config.json` files agree on profile, seed, split, Lean toolchain, Mathlib revision and model settings. `git diff ad70ec2 2c8eae5 -- src config` shows that only `config/pipeline.yaml: spend.run_usd` changed (the documented spend-cap deviations). The input sha256 of the resume runs match the Deviations entry. **Anomaly:** `run_config.json` records `dirty: true` for C5 resume 1, C5 resume 2, C4 and the C1 resume, with `changed_files = [docs/analysis_plan.md]` only. The Deviations entry calls these trees "clean". The only uncommitted file was the plan itself. 7. The first C5 run and the first C1 run have run status `running`: they were killed, as documented. Their results files hold only finished problems. ### Primary outcome and comparisons 8. **C2 in the SR table.** Following §7.1, C2 has no verified SR. It enters through its claimed rate |B⁺|/30 = 30/30 and FPR_B. The 19 claims that the check verified are reported as a count only, because the plan defines no SR for them. C2 is not in any McNemar test (§3). In the per-cell and margin tables, C2 shows **claimed** counts, labelled as such, because §4.1 says "for each configuration". 9. **McNemar:** b = first configuration solved and second not; c = the reverse; two-sided exact binomial on b | b + c with p = 1/2 (p = 1 if b + c = 0). Bonferroni-adjusted p = min(1, 3p); a test is significant if p < 0.05/3. 10. **Paired bootstrap** (plan: 10 000 resamples of problems, seed 2026, percentile). The generator is a fresh `numpy.random.default_rng(2026)` for each statistic, so all comparisons use the same 10 000 index vectors. The plan does not name a generator. For H2 the lower bound is exactly 0.0 pp. 11. Exploratory comparisons (C3 vs C4, C1 vs C3, C1 vs C4) get b, c, ΔSR and an uncorrected exact p only. The plan specifies no bootstrap CI for them. ### FPR_B (Eq. 16, §4.2, Clarification 2026-11-03) 12. B⁺ holds all 30 problems (every C2 row is `claimed_solved`). Check status: 19 `verified`, 11 `not_verified`. No check ended in `budget_exhausted`, `provider_error`, `budget_stop`, `error` or `lean_down`, so none is left out. FPR_B = 11/30. 13. **Manual-review sample.** 11 claims are not verified, which is more than 10, so 10 are drawn "with seed 2026". The plan does not give the drawing procedure; `random.Random(2026).sample(sorted(ids), 10)` was used, which leaves out `amc12a_2008_p15`. The manual review itself (informal proof wrong vs. formalization failure) is a task for the team and has not been done. 14. Factual cross-reference, not a conclusion: for 5 of the 11 not-verified claims (`amc12a_2008_p15`, `amc12b_2002_p3`, `imo_1964_p1_2`, `imo_1965_p2`, `putnam_1965_b2`), C5 has a kernel-accepted proof of the same formal statement. ### Efficiency (§4.3) 15. C(x) = `llm.llm_calls` of the row. It includes cache hits and JSON-repair calls, as the budget counts them. Tokens = prompt + completion (reasoning tokens are part of completion in the ledger). Nominal cost (`cost_usd_nominal`) is the compared figure (§1). Real cost is shown as well. 16. **Real cost.** The real cost of every kept row equals the ledger sum for (run_id, problem_id); no mismatch was found. The ledger totals include spend that is in no kept row: C1 +0.09845 USD (`imo_1987_p6` in the first C1 run: 10 calls, 5 paid and 5 cached JSON repairs), so C1 = 2.4695 + 0.0985 = 2.5679 USD. C5 +0.18153 USD (`imo_1987_p6` in the first run: 0.02309 USD, 8 paid calls; `imo_1984_p6` in resume 1: 0.15844 USD, 15 paid calls), so C5 = 1.1770 + 0.1815 = 1.3585 USD. C2 = 0.0814 (C2 itself) + 0.6269 (claim check) = 0.7083 USD. C3 = 1.3500 USD. C4 = 0.0000 USD. Per-problem and per-solved figures come from kept rows only. 17. **C2 efficiency** leaves out the claim check: its calls, tokens and cost come from the row's `llm`, and its wall time runs from `problem_start` to `claim_check_start`. C̄_solved needs a "solved" count, and C2 has no verified one, so its ratio uses claimed solutions (30). 18. **C4:** all 132 LLM calls were cache hits (requests identical to C5's), so its real cost is 0 and its LLM latency is near zero. Lean verification was real (5 of 259 `lean_verify` events were Lean-cache hits). C4's wall time (about 0.1 min per problem) therefore does not measure LLM computation. Nominal cost is the comparable figure (§1). 19. Bootstrap CIs for the efficiency ratios: plan §4.3 gives no seed, so the same procedure as note 10 is used (seed 2026, percentile). For Σx/#solved, resamples with no solved problem would be dropped; none occurred on `test`. 20. The median/IQR table (4d) and wall time by difficulty (Table 5) were requested by the team. They are descriptive only and not in the plan. ### Refinement robustness (§4.4, Eq. 14) 21. **Accepted plan.** The trace of a solved C5 problem is split at each `plan_proposed` / `plan_fallback` that follows subgoal work (attempt counters restart after a replan). The accepted plan is the last segment. "Attempts on a subgoal" is the `attempts` field of `subgoal_solved`; pre-pass subgoals count as 0. Read literally ("its accepted proof used at most k attempts on every subgoal"), a replanned problem counts by its accepted plan only. This affects `algebra_absapbon1pabsapbleqsumabsaon1pabsa`: C5 replanned once, then solved every subgoal on its first attempt, so it counts at k = 1. It is the only problem that differs between SR_C5(1) = 15/30 and SR_C4 = 14/30. SR_C5(3) = SR_C5 = 24/30, as expected. 22. **SR(0).** Δ̂(1) = SR(1) − SR(0) needs SR(0), which the plan does not define. The same rule with k = 0 is used: the accepted plan has no LLM-attempted subgoal (pre-pass or automatic closing only). This gives 9/30. 23. **Independence curve.** The plan says p_i is "estimated from first-attempt subgoal success in C5". Each subgoal has one first attempt, so a per-subgoal p_i would be 0 or 1. A pooled p̂ = 18/44 is used instead, counting every LLM-attempted subgoal of every plan (including abandoned plans and failed problems). The curve reported is Eq. 14's per-subgoal marginal gain Δ(k) = p̂(1 − p̂)^(k−1). It sits at a different level (subgoal) from the problem-level Δ̂(k). The last two columns of Table 6 give the subgoal-level empirical counterpart and are supplementary. The figure the plan asks for (§7.5) is `docs/results/figures/srk_curve.pdf` (and `.png`), drawn from the same numbers: (a) SR_C5(k), k = 0..3, with Wilson CIs and C4 at k = 1; (b) Δ̂(k) against p̂(1 − p̂)^(k−1). ### Error taxonomy (§4.5) 24. For C3–C5 the classes are the rows' `error_classes`, one per Critic event (checked equal to the `critique` events), including the k_max escalation to `planning`. Only kept rows count; the discarded attempts of the merges are excluded. 25. **C1** attempts are classified offline with `policy.classify(Trace(reason, messages, None))`, using the stored `lean_errors` (at most 3 messages of 300 characters, versus 5 × 1500 in the graph). The statement end is unknown, so `misformalization` cannot fire, which fits since C1 never writes statements. `escalate` is not applied because C1 has no k_max or strategies. The 44 `sanitize:empty` attempts (classed as `syntax`) happen alongside 52 C1 responses that ended at `max_tokens`. 26. **C2** has no verified attempts of its own. Its taxonomy row is the claim-check loop (labelled). The plan does not say how C2 enters §4.5. 27. Final failure reasons group stages by prefix (`verify:sg2` becomes `verify`). ### Other data facts 28. Responses that ended at `max_tokens` (`finish_reason = length`): C1 52 (9 rows), C3 18 (6 rows), C5 14 (5 rows), C4 2 (2 rows), C2 0 (12 in the claim check). 29. C1 `budget_exhausted` rows (7) reached b_max = 18 calls after 9–17 recorded attempts, because JSON-repair calls count against the budget. The 3 `failed` C1 rows used all 18 attempts. 30. **Fidelity (§6).** Four solved test problems carry a fidelity note: `amc12b_2003_p6`, `mathd_algebra_234`, `mathd_algebra_455` and `putnam_1965_b2`. The team reviewed them on 2026-11-07 (`post_test_review` in `data/curation/fidelity_review.yaml`; Table 10): all four count as solved, `amc12b_2003_p6` with a note in the report. The verified SR is reported unchanged, as §6 requires. 31. **Supplementary, not in the plan:** Table 8 (pre-pass vs LLM `solved_by`) and Table 4d. ### Illustrative traces (for the appendix) - `amc12a_2008_p15` (Critic loop). C5 solved it on the third attempt at its only subgoal (two `tactic_failure` retries with Critic feedback, 16 calls, first C5 run). C4 abandoned after the first failure. C3 failed after a replan (30 calls). C1 failed. The C2 claim was not verified. - `numbertheory_sumkmulnckeqnmul2pownm1` (RAG). C5 solved it with 5 calls on the first attempt. C3 failed after 4 `missing_lemma` and 2 `tactic_failure` attempts over two plans (30 calls). - `imo_1965_p2` (FPR_B as an upper bound). C2 claimed it, and its check failed (`missing_lemma`, `tactic_failure`, `missing_lemma`). C5 found a kernel-accepted proof: the pre-pass closed one subgoal and the LLM closed the other two (one retry after `missing_lemma`).