Tables for the test split (n = 30)
Generated by docs/results/make_tables.py from runs/ (read-only), following docs/analysis_plan.md. Rates in %, intervals 95 %.
Runs used
| Run |
Config |
Profile |
Seed |
Split |
Commit |
Tree |
Degraded |
Run status |
vs first run |
| 20261103T174646Z_C5_experiment |
C5 |
experiment |
2026 |
test |
ad70ec2 |
clean |
False |
running |
same |
| 20261103T212359Z_C5_experiment_resume |
C5 |
experiment |
2026 |
test |
ad70ec2 |
dirty (plan file only) |
False |
budget_stop |
same |
| 20261103T225140Z_C5_experiment_resume2 |
C5 |
experiment |
2026 |
test |
93bc701 |
dirty (plan file only) |
False |
finished |
same |
| 20261103T235556Z_C4_experiment |
C4 |
experiment |
2026 |
test |
ea4db4b |
dirty (plan file only) |
False |
finished |
same |
| 20261104T194056Z_C3_experiment |
C3 |
experiment |
2026 |
test |
6c61d69 |
clean |
False |
finished |
same |
| 20261104T232056Z_C2_experiment_check |
C2 |
experiment |
2026 |
test |
6c61d69 |
clean |
False |
finished |
same |
| 20261105T131443Z_C1_experiment |
C1 |
experiment |
2026 |
test |
5ed7507 |
clean |
False |
running |
same |
| 20261107T200233Z_C1_experiment_resume |
C1 |
experiment |
2026 |
test |
2c8eae5 |
dirty (plan file only) |
False |
finished |
same |
Row checks
| Config |
Rows |
Status counts |
lean_down |
split/config mismatch |
area/difficulty mismatch |
| C1 |
30 |
budget_exhausted: 7, failed: 3, solved: 20 |
0 |
0 |
0 |
| C2 |
30 |
claimed_solved: 30 |
0 |
0 |
0 |
| C3 |
30 |
budget_exhausted: 3, failed: 7, solved: 20 |
0 |
0 |
0 |
| C4 |
30 |
failed: 16, solved: 14 |
0 |
0 |
0 |
| C5 |
30 |
failed: 6, solved: 24 |
0 |
0 |
0 |
Table 1. Verified success rate (SR) and Baseline B
| Config |
Solved |
SR (%) |
Wilson 95 % CI (%) |
C2 claimed rate (%) [Wilson] |
FPR_B (%) [Wilson] |
| C1 |
20/30 |
66.7 |
[48.8, 80.8] |
— |
— |
| C3 |
20/30 |
66.7 |
[48.8, 80.8] |
— |
— |
| C4 |
14/30 |
46.7 |
[30.2, 63.9] |
— |
— |
| C5 |
24/30 |
80.0 |
[62.7, 90.5] |
— |
— |
| C2 (Baseline B) |
— |
— |
— |
30/30 = 100.0 [88.6, 100.0] |
11/30 = 36.7 [21.9, 54.5] |
Note. C2 is not verified, so it has no SR; it enters only through its claimed rate B⁺/n and FPR_B (plan §7.1). Of the 30 claimed proofs the formal check verified 19 and did not verify 11; 0 left out of FPR_B.
Table 1b. Baseline B claim check (FPR_B, Eq. 16)
| Quantity |
Value |
| Problems |
30 |
| Claimed (B⁺) |
30 |
| Not claimed |
0 |
| Check status among B⁺ |
not_verified: 11, verified: 19 |
| Verified (Acc on the assembled theorem) |
19 |
| Not verified (budget_exhausted included) |
11 |
| Left out of FPR_B (provider_error, budget_stop, error, lean_down) |
0 |
| FPR_B = not verified / (verified + not verified) |
11/30 = 36.7 % [21.9, 54.5] |
| B⁺ problems for the manual review (§4.2) |
10 drawn with random.Random(2026).sample(sorted ids, 10) (listed below) |
Note. Manual review set (B⁺ not verified): amc12b_2002_p3, imo_1964_p1_2, imo_1965_p2, imo_1978_p5, imo_1979_p1, imo_1984_p6, imo_1987_p6, imo_1993_p5, putnam_1965_b2, putnam_1995_b1.
Table 2. Planned comparisons H1–H3 (exact McNemar, Bonferroni ×3)
| ID |
Comparison |
b |
c |
ΔSR (pp) |
Paired bootstrap 95 % CI (pp) |
Exact p |
Bonferroni p |
Reject at 0.05/3 |
| H1 |
C5 vs C1 |
4 |
0 |
+13.3 |
[+3.3, +26.7] |
0.1250 |
0.3750 |
no |
| H2 |
C5 vs C3 |
5 |
1 |
+13.3 |
[+0.0, +30.0] |
0.2188 |
0.6562 |
no |
| H3 |
C5 vs C4 |
10 |
0 |
+33.3 |
[+16.7, +50.0] |
0.0020 |
0.0059 |
yes |
Note. n = 30, so power is low: a non-significant result is no evidence of a difference, not evidence of no difference. H2 is Δ_RAG = SR_C5 − SR_C3; H3 is the Critic contribution SR_C5 − SR_C4.
Table 2b. Exploratory comparisons (uncorrected, no claims)
| ID |
Comparison |
b |
c |
ΔSR (pp) |
Exact p (uncorrected) |
| E1 |
C3 vs C4 |
9 |
3 |
+20.0 |
0.1460 |
| E2 |
C1 vs C3 |
3 |
3 |
+0.0 |
1.0000 |
| E3 |
C1 vs C4 |
8 |
2 |
+20.0 |
0.1094 |
Table 3. Solved / n per cell of the 3 × 3 grid (descriptive)
| Difficulty |
Area |
n |
C1 |
C3 |
C4 |
C5 |
C2 (claimed) |
| easy |
Algebra |
5 |
5/5 |
5/5 |
4/5 |
5/5 |
5/5 |
| easy |
Number theory |
5 |
5/5 |
4/5 |
5/5 |
5/5 |
5/5 |
| easy |
Combinatorics |
0 |
— |
— |
— |
— |
— |
| medium |
Algebra |
5 |
4/5 |
5/5 |
1/5 |
5/5 |
5/5 |
| medium |
Number theory |
4 |
2/4 |
2/4 |
2/4 |
4/4 |
4/4 |
| medium |
Combinatorics |
1 |
0/1 |
0/1 |
1/1 |
1/1 |
1/1 |
| hard |
Algebra |
3 |
1/3 |
2/3 |
0/3 |
1/3 |
3/3 |
| hard |
Number theory |
4 |
1/4 |
1/4 |
0/4 |
1/4 |
4/4 |
| hard |
Combinatorics |
3 |
2/3 |
1/3 |
1/3 |
2/3 |
3/3 |
Table 3b. SR by difficulty and by area (margins, Wilson 95 % CI)
| Config |
easy (n = 10) |
medium (n = 10) |
hard (n = 10) |
Algebra (n = 13) |
Number theory (n = 13) |
Combinatorics (n = 4) |
| C1 |
10/10 |
6/10 |
4/10 |
10/13 |
8/13 |
2/4 |
|
100.0 [72.2, 100.0] |
60.0 [31.3, 83.2] |
40.0 [16.8, 68.7] |
76.9 [49.7, 91.8] |
61.5 [35.5, 82.3] |
50.0 [15.0, 85.0] |
| C3 |
9/10 |
7/10 |
4/10 |
12/13 |
7/13 |
1/4 |
|
90.0 [59.6, 98.2] |
70.0 [39.7, 89.2] |
40.0 [16.8, 68.7] |
92.3 [66.7, 98.6] |
53.8 [29.1, 76.8] |
25.0 [4.6, 69.9] |
| C4 |
9/10 |
4/10 |
1/10 |
5/13 |
7/13 |
2/4 |
|
90.0 [59.6, 98.2] |
40.0 [16.8, 68.7] |
10.0 [1.8, 40.4] |
38.5 [17.7, 64.5] |
53.8 [29.1, 76.8] |
50.0 [15.0, 85.0] |
| C5 |
10/10 |
10/10 |
4/10 |
11/13 |
10/13 |
3/4 |
|
100.0 [72.2, 100.0] |
100.0 [72.2, 100.0] |
40.0 [16.8, 68.7] |
84.6 [57.8, 95.7] |
76.9 [49.7, 91.8] |
75.0 [30.1, 95.4] |
| C2 (claimed) |
10/10 |
10/10 |
10/10 |
13/13 |
13/13 |
4/4 |
|
100.0 [72.2, 100.0] |
100.0 [72.2, 100.0] |
100.0 [72.2, 100.0] |
100.0 [77.2, 100.0] |
100.0 [77.2, 100.0] |
100.0 [51.0, 100.0] |
Table 4. Efficiency per solved problem (RNF-3)
| Config |
Solved |
Calls / solved |
k tokens / solved |
Nominal USD / solved |
Real USD / solved |
Wall min / solved |
| C1 |
20 |
12.85 [7.19, 22.60] |
219.7 [114.6, 394.7] |
0.1285 [0.0668, 0.2313] |
0.1235 [0.0643, 0.2218] |
17.7 [9.2, 32.0] |
| C2 (per claimed) |
30 |
2.17 [2.00, 2.40] |
5.3 [2.8, 8.6] |
0.0027 [0.0013, 0.0047] |
0.0027 [0.0013, 0.0047] |
0.4 [0.2, 0.7] |
| C3 |
20 |
26.50 [16.15, 43.67] |
150.4 [79.3, 274.6] |
0.0675 [0.0335, 0.1279] |
0.0675 [0.0335, 0.1278] |
9.2 [4.6, 17.3] |
| C4 |
14 |
9.43 [5.72, 16.67] |
70.7 [38.2, 137.8] |
0.0278 [0.0135, 0.0571] |
0.0000 [0.0000, 0.0000] |
0.2 [0.1, 0.3] |
| C5 |
24 |
16.92 [10.63, 26.21] |
148.0 [74.4, 266.4] |
0.0566 [0.0245, 0.1097] |
0.0490 [0.0236, 0.0897] |
7.6 [3.3, 14.6] |
Note. C2 has no verified solutions: its ratio uses claimed solutions and excludes the claim check (calls, tokens, cost and wall time up to claim_check_start).
Table 4b. Efficiency per problem (means)
| Config |
Calls / problem |
k tokens / problem |
Nominal USD / problem |
Real USD / problem |
Wall min / problem |
| C1 |
8.57 [5.83, 11.33] |
146.4 [91.1, 203.5] |
0.0857 [0.0532, 0.1192] |
0.0823 [0.0505, 0.1152] |
11.8 [7.2, 16.5] |
| C2 |
2.17 [2.00, 2.40] |
5.3 [2.8, 8.6] |
0.0027 [0.0013, 0.0047] |
0.0027 [0.0013, 0.0047] |
0.4 [0.2, 0.7] |
| C3 |
17.67 [12.97, 22.37] |
100.3 [63.8, 140.8] |
0.0450 [0.0269, 0.0658] |
0.0450 [0.0269, 0.0658] |
6.1 [3.7, 9.0] |
| C4 |
4.40 [3.37, 5.43] |
33.0 [22.8, 44.4] |
0.0130 [0.0081, 0.0186] |
0.0000 [0.0000, 0.0000] |
0.1 [0.1, 0.1] |
| C5 |
13.53 [9.57, 17.60] |
118.4 [67.9, 178.7] |
0.0453 [0.0225, 0.0737] |
0.0392 [0.0218, 0.0588] |
6.1 [3.0, 9.7] |
Table 4c. Totals, cache use and real cost
| Config |
LLM calls |
Cached calls |
Cached (%) |
Tokens (M) |
Nominal USD (rows) |
Real USD (rows) |
Real USD (ledger, all runs) |
| C1 |
257 |
47 |
18.3 |
4.393 |
2.5700 |
2.4695 |
2.5679 |
| C2 |
65 |
0 |
0.0 |
0.158 |
0.0814 |
0.0814 |
0.7083 |
| C3 |
530 |
18 |
3.4 |
3.009 |
1.3508 |
1.3500 |
1.3500 |
| C4 |
132 |
132 |
100.0 |
0.990 |
0.3887 |
0.0000 |
0.0000 |
| C5 |
406 |
34 |
8.4 |
3.551 |
1.3590 |
1.1770 |
1.3585 |
Note. C2 claim check (not in the C2 figures of the other columns): 77 calls (11 cached), 0.6274 nominal, 0.6269 real.
RNF-3 (C̄_solved ≤ 20 for C5): C̄_solved = 406/24 = 16.92 calls per solved problem: met.
Table 4d. Median [IQR] of LLM calls and wall time per problem (supplementary)
| Config |
Calls, median [Q1, Q3] |
Wall min, median [Q1, Q3] |
| C1 |
4.5 [1.0, 18.0] |
2.7 [0.6, 24.2] |
| C2 |
2.0 [2.0, 2.0] |
0.1 [0.1, 0.3] |
| C3 |
16.5 [3.2, 28.0] |
3.7 [0.6, 6.7] |
| C4 |
5.0 [1.0, 5.8] |
0.1 [0.0, 0.1] |
| C5 |
11.5 [1.0, 23.8] |
2.4 [0.2, 5.2] |
Table 5. Wall time per problem by difficulty, minutes (descriptive)
| Config |
easy (n = 10): mean / median [Q1, Q3] |
medium (n = 10): mean / median [Q1, Q3] |
hard (n = 10): mean / median [Q1, Q3] |
| C1 |
0.7 / 0.4 [0.1, 1.1] |
14.3 / 16.2 [1.8, 23.2] |
20.3 / 25.3 [7.0, 32.4] |
| C2 |
0.1 / 0.1 [0.1, 0.2] |
0.1 / 0.1 [0.1, 0.2] |
1.0 / 0.5 [0.2, 1.1] |
| C3 |
0.8 / 0.1 [0.0, 1.0] |
4.0 / 2.6 [1.2, 6.0] |
13.6 / 12.3 [6.6, 19.8] |
| C4 |
0.1 / 0.0 [0.0, 0.1] |
0.1 / 0.1 [0.0, 0.1] |
0.1 / 0.1 [0.1, 0.1] |
| C5 |
0.4 / 0.0 [0.0, 0.5] |
2.7 / 2.6 [0.7, 4.5] |
15.1 / 14.1 [4.0, 21.0] |
Table 6. Refinement robustness: SR(k) of C5 vs the independence curve (Eq. 14)
| k |
SR_C5(k) [Wilson] (%) |
Δ̂(k) = SR(k) − SR(k−1) (pp) |
Indep. Δ(k) = p̂(1−p̂)^(k−1) (%) |
Subgoals solved at attempt k |
Solved at k / reached k |
| 1 |
15/30 = 50.0 [33.2, 66.8] |
+20.0 |
40.9 |
18/44 = 40.9 |
18/44 = 40.9 |
| 2 |
19/30 = 63.3 [45.5, 78.1] |
+13.3 |
24.2 |
8/44 = 18.2 |
8/26 = 30.8 |
| 3 |
24/30 = 80.0 [62.7, 90.5] |
+16.7 |
14.3 |
5/44 = 11.4 |
5/18 = 27.8 |
Note. SR_C5(0) = 9/30 (accepted plan with no LLM subgoal attempt). p̂ = 18/44 = 40.9 %. k = 1 from C4 (no retries, no Critic advice): 14/30 = 46.7 % [30.2, 63.9]. The last two columns are subgoal-level diagnostics (supplementary).
Table 7. κ error taxonomy over all failed attempts (Eq. 11)
| Config |
Failed attempts |
syntax |
missing lemma |
tactic failure |
misformalization |
timeout |
planning |
| C1 |
186 |
60 (32.3) |
50 (26.9) |
68 (36.6) |
0 (0.0) |
8 (4.3) |
0 (0.0) |
| C2 (claim check) |
45 |
14 (31.1) |
8 (17.8) |
22 (48.9) |
0 (0.0) |
1 (2.2) |
0 (0.0) |
| C3 |
82 |
20 (24.4) |
17 (20.7) |
42 (51.2) |
0 (0.0) |
2 (2.4) |
1 (1.2) |
| C4 |
16 |
5 (31.2) |
4 (25.0) |
7 (43.8) |
0 (0.0) |
0 (0.0) |
0 (0.0) |
| C5 |
57 |
19 (33.3) |
18 (31.6) |
18 (31.6) |
0 (0.0) |
2 (3.5) |
0 (0.0) |
Note. C3–C5: classes recorded by the Critic. C1: attempts classified offline with policy.classify (first 3 Lean messages, 300 characters each, as stored; no k_max escalation). C2 has no verified attempts of its own: its row is the claim-check loop.
Table 7b. κ taxonomy per area
| Config |
Area |
Failed attempts |
syntax |
missing lemma |
tactic failure |
misformalization |
timeout |
planning |
| C1 |
Algebra |
77 |
14 (18.2) |
40 (51.9) |
22 (28.6) |
0 (0.0) |
1 (1.3) |
0 (0.0) |
| C1 |
Number theory |
83 |
27 (32.5) |
3 (3.6) |
46 (55.4) |
0 (0.0) |
7 (8.4) |
0 (0.0) |
| C1 |
Combinatorics |
26 |
19 (73.1) |
7 (26.9) |
0 (0.0) |
0 (0.0) |
0 (0.0) |
0 (0.0) |
| C2 (claim check) |
Algebra |
14 |
5 (35.7) |
4 (28.6) |
5 (35.7) |
0 (0.0) |
0 (0.0) |
0 (0.0) |
| C2 (claim check) |
Number theory |
23 |
5 (21.7) |
4 (17.4) |
13 (56.5) |
0 (0.0) |
1 (4.3) |
0 (0.0) |
| C2 (claim check) |
Combinatorics |
8 |
4 (50.0) |
0 (0.0) |
4 (50.0) |
0 (0.0) |
0 (0.0) |
0 (0.0) |
| C3 |
Algebra |
18 |
3 (16.7) |
9 (50.0) |
5 (27.8) |
0 (0.0) |
0 (0.0) |
1 (5.6) |
| C3 |
Number theory |
43 |
13 (30.2) |
3 (7.0) |
26 (60.5) |
0 (0.0) |
1 (2.3) |
0 (0.0) |
| C3 |
Combinatorics |
21 |
4 (19.0) |
5 (23.8) |
11 (52.4) |
0 (0.0) |
1 (4.8) |
0 (0.0) |
| C4 |
Algebra |
8 |
1 (12.5) |
3 (37.5) |
4 (50.0) |
0 (0.0) |
0 (0.0) |
0 (0.0) |
| C4 |
Number theory |
6 |
2 (33.3) |
1 (16.7) |
3 (50.0) |
0 (0.0) |
0 (0.0) |
0 (0.0) |
| C4 |
Combinatorics |
2 |
2 (100.0) |
0 (0.0) |
0 (0.0) |
0 (0.0) |
0 (0.0) |
0 (0.0) |
| C5 |
Algebra |
25 |
7 (28.0) |
8 (32.0) |
10 (40.0) |
0 (0.0) |
0 (0.0) |
0 (0.0) |
| C5 |
Number theory |
24 |
8 (33.3) |
9 (37.5) |
6 (25.0) |
0 (0.0) |
1 (4.2) |
0 (0.0) |
| C5 |
Combinatorics |
8 |
4 (50.0) |
1 (12.5) |
2 (25.0) |
0 (0.0) |
1 (12.5) |
0 (0.0) |
Table 7c. Final failure reason per problem (status @ stage)
| Config |
Problems |
status @ stage: count |
| C1 |
10 |
budget_exhausted: 7; failed: 3 |
| C2 |
30 |
claimed, check verified: 19; claimed, check not_verified: 11 |
| C3 |
10 |
failed @ verify: 5; budget_exhausted @ formalize: 3; failed @ formalize: 2 |
| C4 |
16 |
failed @ verify: 11; failed @ formalize: 5 |
| C5 |
6 |
failed @ verify: 4; failed @ formalize: 2 |
Table 8. Subgoals closed by the tactic pre-pass vs the LLM (supplementary)
| Config |
Pre-pass (accepted plans) |
LLM (accepted plans) |
Pre-pass (all subgoal_solved) |
Solved with no LLM subgoal |
Solved |
| C3 |
7/32 = 21.9 |
25/32 = 78.1 |
8/39 = 20.5 |
8 |
20 |
| C4 |
9/18 = 50.0 |
9/18 = 50.0 |
11/21 = 52.4 |
9 |
14 |
| C5 |
11/39 = 28.2 |
28/39 = 71.8 |
11/42 = 26.2 |
9 |
24 |
Table 9. Infrastructure and error statuses in the kept rows (plan §5, §7.7)
| Config |
provider_error |
budget_stop |
error |
node_error |
invalid_statement |
lean_down |
| C1 |
0 |
0 |
0 |
0 |
0 |
0 |
| C2 |
0 |
0 |
0 |
0 |
0 |
0 |
| C3 |
0 |
0 |
0 |
0 |
0 |
0 |
| C4 |
0 |
0 |
0 |
0 |
0 |
0 |
| C5 |
0 |
0 |
0 |
0 |
0 |
0 |
Table 10. Solved test problems with a fidelity note (plan §6)
| Problem |
Solved by |
Manual review |
| amc12b_2003_p6 |
C1, C3, C4, C5 |
2026-11-07: counts as solved; report note: The formalization proves that u 0 = +-2/sqrt 3 (a necessary condition), not that -2*sqrt(3)/3 is attained; this is enough to rule out the other options. |
| mathd_algebra_234 |
C1, C3, C4, C5 |
2026-11-07: counts as solved; no note |
| mathd_algebra_455 |
C1, C3, C4, C5 |
2026-11-07: counts as solved; no note |
| putnam_1965_b2 |
C1, C3, C5 |
2026-11-07: counts as solved; no note |
Table 11. Per-problem outcomes
| Problem |
Difficulty |
Area |
C1 |
C3 |
C4 |
C5 |
C2 (claim, check) |
| algebra_absapbon1pabsapbleqsumabsaon1pabsa |
medium |
Algebra |
0 |
1 |
0 |
1 |
claimed, verified |
| amc12_2000_p20 |
medium |
Algebra |
1 |
1 |
0 |
1 |
claimed, verified |
| amc12_2001_p9 |
medium |
Algebra |
1 |
1 |
0 |
1 |
claimed, verified |
| amc12a_2008_p15 |
medium |
Number theory |
0 |
0 |
0 |
1 |
claimed, not_verified |
| amc12a_2008_p25 |
medium |
Algebra |
1 |
1 |
0 |
1 |
claimed, verified |
| amc12b_2002_p3 |
medium |
Number theory |
0 |
1 |
1 |
1 |
claimed, not_verified |
| amc12b_2003_p6 |
medium |
Algebra |
1 |
1 |
1 |
1 |
claimed, verified |
| imo_1964_p1_2 |
hard |
Number theory |
1 |
1 |
0 |
1 |
claimed, not_verified |
| imo_1965_p2 |
hard |
Algebra |
1 |
1 |
0 |
1 |
claimed, not_verified |
| imo_1978_p5 |
hard |
Algebra |
0 |
0 |
0 |
0 |
claimed, not_verified |
| imo_1979_p1 |
hard |
Number theory |
0 |
0 |
0 |
0 |
claimed, not_verified |
| imo_1984_p6 |
hard |
Number theory |
0 |
0 |
0 |
0 |
claimed, not_verified |
| imo_1987_p6 |
hard |
Number theory |
0 |
0 |
0 |
0 |
claimed, not_verified |
| imo_1993_p5 |
hard |
Algebra |
0 |
1 |
0 |
0 |
claimed, not_verified |
| mathd_algebra_160 |
easy |
Algebra |
1 |
1 |
1 |
1 |
claimed, verified |
| mathd_algebra_206 |
easy |
Algebra |
1 |
1 |
0 |
1 |
claimed, verified |
| mathd_algebra_234 |
easy |
Algebra |
1 |
1 |
1 |
1 |
claimed, verified |
| mathd_algebra_288 |
easy |
Algebra |
1 |
1 |
1 |
1 |
claimed, verified |
| mathd_algebra_455 |
easy |
Algebra |
1 |
1 |
1 |
1 |
claimed, verified |
| mathd_numbertheory_321 |
easy |
Number theory |
1 |
0 |
1 |
1 |
claimed, verified |
| mathd_numbertheory_328 |
easy |
Number theory |
1 |
1 |
1 |
1 |
claimed, verified |
| mathd_numbertheory_345 |
easy |
Number theory |
1 |
1 |
1 |
1 |
claimed, verified |
| mathd_numbertheory_430 |
easy |
Number theory |
1 |
1 |
1 |
1 |
claimed, verified |
| mathd_numbertheory_64 |
easy |
Number theory |
1 |
1 |
1 |
1 |
claimed, verified |
| numbertheory_3pow2pownm1mod2pownp3eq2pownp2 |
medium |
Number theory |
1 |
0 |
0 |
1 |
claimed, verified |
| numbertheory_notequiv2i2jasqbsqdiv8 |
medium |
Number theory |
1 |
1 |
1 |
1 |
claimed, verified |
| numbertheory_sumkmulnckeqnmul2pownm1 |
medium |
Combinatorics |
0 |
0 |
1 |
1 |
claimed, verified |
| putnam_1964_b2 |
hard |
Combinatorics |
1 |
0 |
1 |
1 |
claimed, verified |
| putnam_1965_b2 |
hard |
Combinatorics |
1 |
1 |
0 |
1 |
claimed, not_verified |
| putnam_1995_b1 |
hard |
Combinatorics |
0 |
0 |
0 |
0 |
claimed, not_verified |
Table 11b. Hard problems per configuration (supplementary; post hoc, descriptive, not in the analysis plan)
| Problem |
Area |
C1 |
C3 |
C4 |
C5 |
Solved by |
| imo_1964_p1_2 |
Number theory |
1 |
1 |
0 |
1 |
3 |
| imo_1965_p2 |
Algebra |
1 |
1 |
0 |
1 |
3 |
| imo_1978_p5 |
Algebra |
0 |
0 |
0 |
0 |
0 |
| imo_1979_p1 |
Number theory |
0 |
0 |
0 |
0 |
0 |
| imo_1984_p6 |
Number theory |
0 |
0 |
0 |
0 |
0 |
| imo_1987_p6 |
Number theory |
0 |
0 |
0 |
0 |
0 |
| imo_1993_p5 |
Algebra |
0 |
1 |
0 |
0 |
1 |
| putnam_1964_b2 |
Combinatorics |
1 |
0 |
1 |
1 |
3 |
| putnam_1965_b2 |
Combinatorics |
1 |
1 |
0 |
1 |
3 |
| putnam_1995_b1 |
Combinatorics |
0 |
0 |
0 |
0 |
0 |
| Solved |
|
4/10 |
4/10 |
1/10 |
4/10 |
|
Note. Post hoc, descriptive (not in the analysis plan): no tests, no intervals. 1 = verified solved. C1 and C5 solve the same hard problems (4 and 4); 5 hard problems are solved by no configuration (imo_1978_p5, imo_1979_p1, imo_1984_p6, imo_1987_p6, putnam_1995_b1); C3 and C5 solve 4 and 4 hard problems, but only C3 solves imo_1993_p5 and only C5 solves putnam_1964_b2.
Table 12. Exploratory: C5-Flash vs C5-Pro on the hard problems (post hoc, descriptive, not in the analysis plan)
| Problem |
Flash solved |
Flash calls |
Flash USD |
Flash min |
Pro outcome |
Pro solved |
Pro calls |
Pro USD |
Pro min |
| imo_1964_p1_2 |
1 |
15 |
0.0261 |
3.4 |
not attempted |
— |
— |
— |
— |
| imo_1965_p2 |
1 |
14 |
0.0312 |
3.7 |
failed |
0 |
30 |
0.3608 |
33.6 |
| imo_1978_p5 |
0 |
31 |
0.1347 |
39.6 |
provider_error (connection drop, not a system failure) |
— |
30 |
0.2938 |
40.7 |
| imo_1979_p1 |
0 |
28 |
0.1463 |
19.7 |
failed |
0 |
24 |
0.2722 |
29.3 |
| imo_1984_p6 |
0 |
32 |
0.3366 |
27.6 |
interrupted |
— |
— |
— |
— |
| imo_1987_p6 |
0 |
36 |
0.1793 |
21.4 |
budget_exhausted |
0 |
40 |
0.4607 |
57.4 |
| imo_1993_p5 |
0 |
27 |
0.0790 |
10.6 |
failed |
0 |
24 |
0.3807 |
53.6 |
| putnam_1964_b2 |
1 |
9 |
0.0160 |
2.3 |
not attempted |
— |
— |
— |
— |
| putnam_1965_b2 |
1 |
17 |
0.0409 |
4.9 |
not attempted |
— |
— |
— |
— |
| putnam_1995_b1 |
0 |
25 |
0.1268 |
17.5 |
not attempted |
— |
— |
— |
— |
| Both finished (4) |
1/4 |
105 |
0.4357 |
55.5 |
|
0/4 |
118 |
1.4744 |
173.9 |
| All hard (10) |
4/10 |
234 |
1.1169 |
150.9 |
|
— |
— |
— |
— |
Note. Post hoc, exploratory (decided 2026-11-08, after all confirmatory runs): descriptive only, no tests, no intervals; it changes no main result. C5-Flash = the kept C5 rows of the main test result; C5-Pro = run 20261108T130803Z_C5_experiment_pro_pro_hard (profile experiment_pro: deepseek-v4-pro in every role, everything else as frozen), stopped by the team and not resumed. Cost is nominal USD (cache hits priced as fresh calls). provider_error, interrupted and not attempted do not count as failures of Pro and are left out of its solved count; only problems both runs finished are compared. imo_1984_p6 was interrupted after 4 paid calls (0.1725 USD real, 18.1 min) and left no row. Real cost of the Pro run: 1.9407 USD (151 paid calls, all in the ledger); project ledger total: 9.3003 USD.
Pro vs Flash on the 4 problems both finished: 3.38 × the nominal cost and 3.13 × the wall time.
Project spend (ledger, all runs): main experiments (every run except the exploratory Pro run, dev included) 7.3596 USD; exploratory Pro run 1.9407 USD; total 9.3003 USD.
Discordant pairs (candidates for illustrative traces)
- C5 solved, C4 not (Critic loop): algebra_absapbon1pabsapbleqsumabsaon1pabsa (C5: solved, 24 calls, replans 1, max attempts 1); amc12_2000_p20 (C5: solved, 23 calls, replans 0, max attempts 3); amc12_2001_p9 (C5: solved, 7 calls, replans 0, max attempts 2); amc12a_2008_p15 (C5: solved, 16 calls, replans 0, max attempts 3); amc12a_2008_p25 (C5: solved, 23 calls, replans 0, max attempts 2); imo_1964_p1_2 (C5: solved, 15 calls, replans 0, max attempts 3); imo_1965_p2 (C5: solved, 14 calls, replans 0, max attempts 2); mathd_algebra_206 (C5: solved, 28 calls, replans 0, max attempts 2); numbertheory_3pow2pownm1mod2pownp3eq2pownp2 (C5: solved, 15 calls, replans 0, max attempts 3); putnam_1965_b2 (C5: solved, 17 calls, replans 0, max attempts 3)
- C4 solved, C5 not: —
- C5 solved, C3 not (RAG): amc12a_2008_p15 (C5: solved, 16 calls, replans 0, max attempts 3); mathd_numbertheory_321 (C5: solved, 1 calls, replans 0, max attempts 0); numbertheory_3pow2pownm1mod2pownp3eq2pownp2 (C5: solved, 15 calls, replans 0, max attempts 3); numbertheory_sumkmulnckeqnmul2pownm1 (C5: solved, 5 calls, replans 0, max attempts 1); putnam_1964_b2 (C5: solved, 9 calls, replans 0, max attempts 1)
- C3 solved, C5 not: imo_1993_p5 (C5: failed, 27 calls, replans 1, max attempts -)
- C5 solved, C1 not: algebra_absapbon1pabsapbleqsumabsaon1pabsa (C5: solved, 24 calls, replans 1, max attempts 1); amc12a_2008_p15 (C5: solved, 16 calls, replans 0, max attempts 3); amc12b_2002_p3 (C5: solved, 9 calls, replans 0, max attempts 1); numbertheory_sumkmulnckeqnmul2pownm1 (C5: solved, 5 calls, replans 0, max attempts 1)
- C1 solved, C5 not: —
- C2 claimed, check did not verify: amc12a_2008_p15 (C5: solved, 16 calls, replans 0, max attempts 3); amc12b_2002_p3 (C5: solved, 9 calls, replans 0, max attempts 1); imo_1964_p1_2 (C5: solved, 15 calls, replans 0, max attempts 3); imo_1965_p2 (C5: solved, 14 calls, replans 0, max attempts 2); imo_1978_p5 (C5: failed, 31 calls, replans 1, max attempts -); imo_1979_p1 (C5: failed, 28 calls, replans 1, max attempts -); imo_1984_p6 (C5: failed, 32 calls, replans 1, max attempts -); imo_1987_p6 (C5: failed, 36 calls, replans 1, max attempts -); imo_1993_p5 (C5: failed, 27 calls, replans 1, max attempts -); putnam_1965_b2 (C5: solved, 17 calls, replans 0, max attempts 3); putnam_1995_b1 (C5: failed, 25 calls, replans 1, max attempts -)
Generated data notes
- C1 merged file checked row by row: each row equals (as parsed JSON) exactly one source row; 23 from
20261105T131443Z_C1_experiment, 7 from 20261107T200233Z_C1_experiment_resume. No source row is left out.
- Kept 30 rows from
20261104T232056Z_C2_experiment_check.
- Kept 30 rows from
20261104T194056Z_C3_experiment.
- Kept 30 rows from
20261103T235556Z_C4_experiment.
- Kept 23 rows from
20261103T174646Z_C5_experiment.
- Discarded row
imo_1984_p6 (status budget_stop) of 20261103T212359Z_C5_experiment_resume, as the C5 Deviations entry says.
- Kept 2 rows from
20261103T212359Z_C5_experiment_resume.
- Kept 5 rows from
20261103T225140Z_C5_experiment_resume2.
- C1: 52 LLM responses ended at max_tokens (finish_reason length) in 9 rows.
- C1: ledger spend in no kept row = 0.09845 USD (real cost only):
20261105T131443Z_C1_experiment / imo_1987_p6: 0.09845 USD, 5 paid calls.
- C2: 0 LLM responses ended at max_tokens (finish_reason length) in 0 rows; 12 more in the claim check.
- C3: 18 LLM responses ended at max_tokens (finish_reason length) in 6 rows.
- C4: 2 LLM responses ended at max_tokens (finish_reason length) in 2 rows.
- C5: 14 LLM responses ended at max_tokens (finish_reason length) in 5 rows.
- C5: ledger spend in no kept row = 0.18153 USD (real cost only):
20261103T174646Z_C5_experiment / imo_1987_p6: 0.02309 USD, 8 paid calls; 20261103T212359Z_C5_experiment_resume / imo_1984_p6: 0.15844 USD, 15 paid calls.
Method notes (interpretations of docs/analysis_plan.md and data anomalies)
Hand-written; appended verbatim to results.md by make_tables.py. Each item is a point where the plan
had to be read literally or was silent, or a fact found in the data.
Analysis code
- Analysis code. Plan §7 places the statistics in
src/formal_mas/analysis/stats.py. The first version of
this analysis implemented them inside docs/results/make_tables.py (the file did not exist yet); on
2026-11-07, by team decision, they were moved to src/formal_mas/analysis/stats.py (Wilson, exact McNemar,
Bonferroni, bootstrap, quartiles), with offline tests against closed-form values in tests/test_stats.py.
make_tables.py imports them, and the regenerated results.md and tables.tex were checked to be byte for
byte identical to the earlier output (before Table 10 was switched to the post-test fidelity review). Exact binomial tests and Wilson intervals come from
scipy.stats.binomtest (Wilson without continuity correction). The bootstrap uses
numpy.random.default_rng. The κ classifier is imported read-only from src/formal_mas/orchestrator/policy.py.
scripts/compare.py was not written: make_tables.py produces every table of §7.
- Dev before test (plan §7). The script was first run in
--dev mode on dev runs, using stand-ins on the
same 15 dev problems, and checked there: Wilson 11/15 = [48.0, 89.1], exact McNemar p values computed by
hand, and the C1 κ classification on 20261103T161957Z_C1_experiment_c1_cost. Unit tests of the statistics
were added later (item 1).
Disclosure: before the script existed, the analyst had seen the first test row of C5 (mathd_algebra_206,
solved) and the summaries stored in the run_config.json of the C5 resume runs and of C4, while checking the
configurations. After the first test run, the script changed in three ways: (a) C2 wall time now stops at
claim_check_start (the row's wall_s includes the claim check, and the 2026-11-03 Clarification excludes
the check from C2's figures); (b) a note on responses cut at max_tokens was added; (c) LaTeX layout. No test,
comparison, correction or metric of the plan was changed.
Merges and integrity
- C5 merge follows the C5 Deviations entry exactly. It keeps 23 rows of
20261103T174646Z_C5_experiment,
imo_1987_p6 and imo_1979_p1 of ..._resume, and 5 rows of ..._resume2. The budget_stop row of
imo_1984_p6 in resume 1 is discarded; the script asserts its status. The script fails if a row is kept twice
or if a source row is neither kept nor discarded.
- C1 uses
runs/20261107T200233Z_C1_experiment_resume/C1_test_merged_results.jsonl. Every merged row equals
(as parsed JSON) exactly one source row: 23 from 20261105T131443Z_C1_experiment and 7 from the resume.
- All five configurations have exactly 30 rows with the same 30
test ids. There is no provider_error,
budget_stop, error, node_error, invalid_statement or lean_down row, and no degraded run. The plan's
re-run rule and sensitivity analysis (§5) therefore do not apply. Area and difficulty in each row match
data/eval.jsonl.
- All
run_config.json files agree on profile, seed, split, Lean toolchain, Mathlib revision and model settings.
git diff ad70ec2 2c8eae5 -- src config shows that only config/pipeline.yaml: spend.run_usd changed (the
documented spend-cap deviations). The input sha256 of the resume runs match the Deviations entry.
Anomaly: run_config.json records dirty: true for C5 resume 1, C5 resume 2, C4 and the C1 resume, with
changed_files = [docs/analysis_plan.md] only. The Deviations entry calls these trees "clean". The only
uncommitted file was the plan itself.
- The first C5 run and the first C1 run have run status
running: they were killed, as documented. Their
results files hold only finished problems.
Primary outcome and comparisons
- C2 in the SR table. Following §7.1, C2 has no verified SR. It enters through its claimed rate
|B⁺|/30 = 30/30 and FPR_B. The 19 claims that the check verified are reported as a count only, because the
plan defines no SR for them. C2 is not in any McNemar test (§3). In the per-cell and margin tables, C2 shows
claimed counts, labelled as such, because §4.1 says "for each configuration".
- McNemar: b = first configuration solved and second not; c = the reverse; two-sided exact binomial on
b | b + c with p = 1/2 (p = 1 if b + c = 0). Bonferroni-adjusted p = min(1, 3p); a test is significant if
p < 0.05/3.
- Paired bootstrap (plan: 10 000 resamples of problems, seed 2026, percentile). The generator is a fresh
numpy.random.default_rng(2026) for each statistic, so all comparisons use the same 10 000 index vectors.
The plan does not name a generator. For H2 the lower bound is exactly 0.0 pp.
- Exploratory comparisons (C3 vs C4, C1 vs C3, C1 vs C4) get b, c, ΔSR and an uncorrected exact p only. The
plan specifies no bootstrap CI for them.
FPR_B (Eq. 16, §4.2, Clarification 2026-11-03)
- B⁺ holds all 30 problems (every C2 row is
claimed_solved). Check status: 19 verified, 11 not_verified.
No check ended in budget_exhausted, provider_error, budget_stop, error or lean_down, so none is
left out. FPR_B = 11/30.
- Manual-review sample. 11 claims are not verified, which is more than 10, so 10 are drawn "with seed 2026".
The plan does not give the drawing procedure;
random.Random(2026).sample(sorted(ids), 10) was used, which
leaves out amc12a_2008_p15. The manual review itself (informal proof wrong vs. formalization failure) is a
task for the team and has not been done.
- Factual cross-reference, not a conclusion: for 5 of the 11 not-verified claims (
amc12a_2008_p15,
amc12b_2002_p3, imo_1964_p1_2, imo_1965_p2, putnam_1965_b2), C5 has a kernel-accepted proof of the
same formal statement.
Efficiency (§4.3)
- C(x) =
llm.llm_calls of the row. It includes cache hits and JSON-repair calls, as the budget counts them.
Tokens = prompt + completion (reasoning tokens are part of completion in the ledger). Nominal cost
(cost_usd_nominal) is the compared figure (§1). Real cost is shown as well.
- Real cost. The real cost of every kept row equals the ledger sum for (run_id, problem_id); no mismatch was
found. The ledger totals include spend that is in no kept row:
C1 +0.09845 USD (
imo_1987_p6 in the first C1 run: 10 calls, 5 paid and 5 cached JSON repairs), so
C1 = 2.4695 + 0.0985 = 2.5679 USD.
C5 +0.18153 USD (imo_1987_p6 in the first run: 0.02309 USD, 8 paid calls; imo_1984_p6 in resume 1:
0.15844 USD, 15 paid calls), so C5 = 1.1770 + 0.1815 = 1.3585 USD.
C2 = 0.0814 (C2 itself) + 0.6269 (claim check) = 0.7083 USD. C3 = 1.3500 USD. C4 = 0.0000 USD.
Per-problem and per-solved figures come from kept rows only.
- C2 efficiency leaves out the claim check: its calls, tokens and cost come from the row's
llm, and its
wall time runs from problem_start to claim_check_start. C̄_solved needs a "solved" count, and C2 has no
verified one, so its ratio uses claimed solutions (30).
- C4: all 132 LLM calls were cache hits (requests identical to C5's), so its real cost is 0 and its LLM
latency is near zero. Lean verification was real (5 of 259
lean_verify events were Lean-cache hits). C4's
wall time (about 0.1 min per problem) therefore does not measure LLM computation. Nominal cost is the
comparable figure (§1).
- Bootstrap CIs for the efficiency ratios: plan §4.3 gives no seed, so the same procedure as note 10 is used
(seed 2026, percentile). For Σx/#solved, resamples with no solved problem would be dropped; none occurred on
test.
- The median/IQR table (4d) and wall time by difficulty (Table 5) were requested by the team. They are
descriptive only and not in the plan.
Refinement robustness (§4.4, Eq. 14)
- Accepted plan. The trace of a solved C5 problem is split at each
plan_proposed / plan_fallback that
follows subgoal work (attempt counters restart after a replan). The accepted plan is the last segment.
"Attempts on a subgoal" is the attempts field of subgoal_solved; pre-pass subgoals count as 0. Read
literally ("its accepted proof used at most k attempts on every subgoal"), a replanned problem counts by its
accepted plan only. This affects algebra_absapbon1pabsapbleqsumabsaon1pabsa: C5 replanned once, then
solved every subgoal on its first attempt, so it counts at k = 1. It is the only problem that differs
between SR_C5(1) = 15/30 and SR_C4 = 14/30. SR_C5(3) = SR_C5 = 24/30, as expected.
- SR(0). Δ̂(1) = SR(1) − SR(0) needs SR(0), which the plan does not define. The same rule with k = 0 is
used: the accepted plan has no LLM-attempted subgoal (pre-pass or automatic closing only). This gives 9/30.
- Independence curve. The plan says p_i is "estimated from first-attempt subgoal success in C5". Each
subgoal has one first attempt, so a per-subgoal p_i would be 0 or 1. A pooled p̂ = 18/44 is used instead,
counting every LLM-attempted subgoal of every plan (including abandoned plans and failed problems). The curve
reported is Eq. 14's per-subgoal marginal gain Δ(k) = p̂(1 − p̂)^(k−1). It sits at a different level
(subgoal) from the problem-level Δ̂(k). The last two columns of Table 6 give the subgoal-level empirical
counterpart and are supplementary. The figure the plan asks for (§7.5) is
docs/results/figures/srk_curve.pdf (and .png), drawn from the same numbers: (a) SR_C5(k), k = 0..3, with
Wilson CIs and C4 at k = 1; (b) Δ̂(k) against p̂(1 − p̂)^(k−1).
Error taxonomy (§4.5)
- For C3–C5 the classes are the rows'
error_classes, one per Critic event (checked equal to the
critique events), including the k_max escalation to planning. Only kept rows count; the discarded
attempts of the merges are excluded.
- C1 attempts are classified offline with
policy.classify(Trace(reason, messages, None)), using the
stored lean_errors (at most 3 messages of 300 characters, versus 5 × 1500 in the graph). The statement end
is unknown, so misformalization cannot fire, which fits since C1 never writes statements. escalate is
not applied because C1 has no k_max or strategies. The 44 sanitize:empty attempts (classed as syntax)
happen alongside 52 C1 responses that ended at max_tokens.
- C2 has no verified attempts of its own. Its taxonomy row is the claim-check loop (labelled). The plan
does not say how C2 enters §4.5.
- Final failure reasons group stages by prefix (
verify:sg2 becomes verify).
Other data facts
- Responses that ended at
max_tokens (finish_reason = length): C1 52 (9 rows), C3 18 (6 rows),
C5 14 (5 rows), C4 2 (2 rows), C2 0 (12 in the claim check).
- C1
budget_exhausted rows (7) reached b_max = 18 calls after 9–17 recorded attempts, because JSON-repair
calls count against the budget. The 3 failed C1 rows used all 18 attempts.
- Fidelity (§6). Four solved test problems carry a fidelity note:
amc12b_2003_p6, mathd_algebra_234,
mathd_algebra_455 and putnam_1965_b2. The team reviewed them on 2026-11-07 (post_test_review in
data/curation/fidelity_review.yaml; Table 10): all four count as solved, amc12b_2003_p6 with a note in the
report. The verified SR is reported unchanged, as §6 requires.
- Supplementary, not in the plan: Table 8 (pre-pass vs LLM
solved_by) and Table 4d.
Illustrative traces (for the appendix)
amc12a_2008_p15 (Critic loop). C5 solved it on the third attempt at its only subgoal (two tactic_failure
retries with Critic feedback, 16 calls, first C5 run). C4 abandoned after the first failure. C3 failed after a
replan (30 calls). C1 failed. The C2 claim was not verified.
numbertheory_sumkmulnckeqnmul2pownm1 (RAG). C5 solved it with 5 calls on the first attempt. C3 failed after
4 missing_lemma and 2 tactic_failure attempts over two plans (30 calls).
imo_1965_p2 (FPR_B as an upper bound). C2 claimed it, and its check failed (missing_lemma,
tactic_failure, missing_lemma). C5 found a kernel-accepted proof: the pre-pass closed one subgoal and the
LLM closed the other two (one retry after missing_lemma).