File size: 51,997 Bytes
dfe1826
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
# What was tested and what came out

The single, complete results document of the project: dataset, configurations, every table of
the analysis, the SR(k) figure, the deviations from the analysis plan, the fidelity review, the
limitations, and a guide to what goes in each section of the report. It is written so that the
report can be drafted from this file alone. For how the system works, see
[how_it_works.md](how_it_works.md). The pre-registered analysis plan is
[analysis_plan.md](analysis_plan.md) (written before any `test` result was seen); the full
statistical tables are in `docs/results/`.

**Where the numbers come from.** Every table below is copied verbatim from
`docs/results/results.md`, which `docs/results/make_tables.py` generates from `runs/`
(read-only). Nothing here is recomputed. `results.md` stays the generated source; this file is the
version to read. After `make_tables.py` is re-run, `python docs/results/check_explained.py`
checks that every table and every number here still matches `results.md`. The LaTeX version of
each table is in `docs/results/tables.tex`, under the label given with each table.

**Status labels used below.**

- **Planned (confirmatory)**: a test fixed in `analysis_plan.md` before `test` was run; the only
  results that support claims.
- **Planned (exploratory)**: listed in the plan as exploratory: uncorrected, no claims.
- **Planned (descriptive)**: required by the plan, without tests.
- **Supplementary**: requested by the team after the plan, descriptive only.
- **Post hoc**: a look decided after seeing the results; descriptive only, not in the plan.
- **Integrity check**: describes the data, not the method.

Rates are in %, intervals are 95 %.

## 1. The dataset

`data/eval.jsonl` holds **45 problems**, each with its informal text and its Lean statement:

| | easy | medium | hard | total |
|---|---|---|---|---|
| `dev` (for tuning) | 5 | 5 | 5 | 15 |
| `test` (final runs only) | 10 | 10 | 10 | 30 |

- **Sources:** 40 from miniF2F (the Lean 4 port used by DeepSeek-Prover-V1.5) and 5 from
  PutnamBench (3 of them in `test`).
- **Areas in `test`:** algebra 13, number theory 13, combinatorics 4 (no easy combinatorics
  problem: that cell of the 3 × 3 grid is empty).
- Every statement was checked to compile and reviewed by hand for fidelity to the informal
  text (`data/curation/`; see §10). Prompts and settings were tuned only on `dev`; `test` was
  used once, for the final runs.

## 2. The five configurations

All five run on the same code and graph; only flags in `config/ablations/` change.

| | Name | In one line |
|---|---|---|
| **C1** | Baseline A | One LLM asked directly for a whole Lean proof; up to 18 independent tries (Lean's errors are not fed back). No agents. |
| **C2** | Baseline B | One LLM solves the problem in natural language and judges its own answer. No Lean. |
| **C3** | No RAG | Full system without the lemma search. |
| **C4** | No Critic | Full system, but one attempt per subgoal and no replanning. |
| **C5** | Full system | All agents, retries and replanning, up to 40 calls per problem. |

All runs used the same model for every role (DeepSeek `deepseek-flash`, a general-purpose
model, not a specialised prover), the same seed (2026), Lean `v4.36.0-rc1` and Mathlib
`625d891`. The exact settings of each run are in its `runs/<run>/run_config.json`.

## 3. Runs and data integrity

### 3.1 Runs used

*Integrity check · LaTeX `tab:runs`.* Which run folders feed the analysis and their recorded
configuration. C5 and C1 were interrupted and resumed (§11), so each has more than one folder;
their pieces are merged into one row per problem. "running" means the run was killed before it
could write its final status; "dirty (plan file only)" is a recording artefact explained in §11.

| Run | Config | Profile | Seed | Split | Commit | Tree | Degraded | Run status | vs first run |
|---|---|---|---|---|---|---|---|---|---|
| 20261103T174646Z_C5_experiment | C5 | experiment | 2026 | test | ad70ec2 | clean | False | running | same |
| 20261103T212359Z_C5_experiment_resume | C5 | experiment | 2026 | test | ad70ec2 | dirty (plan file only) | False | budget_stop | same |
| 20261103T225140Z_C5_experiment_resume2 | C5 | experiment | 2026 | test | 93bc701 | dirty (plan file only) | False | finished | same |
| 20261103T235556Z_C4_experiment | C4 | experiment | 2026 | test | ea4db4b | dirty (plan file only) | False | finished | same |
| 20261104T194056Z_C3_experiment | C3 | experiment | 2026 | test | 6c61d69 | clean | False | finished | same |
| 20261104T232056Z_C2_experiment_check | C2 | experiment | 2026 | test | 6c61d69 | clean | False | finished | same |
| 20261105T131443Z_C1_experiment | C1 | experiment | 2026 | test | 5ed7507 | clean | False | running | same |
| 20261107T200233Z_C1_experiment_resume | C1 | experiment | 2026 | test | 2c8eae5 | dirty (plan file only) | False | finished | same |

### 3.2 Row checks

*Integrity check · LaTeX `tab:checks`.* After merging, every configuration has exactly 30 rows
(one per `test` problem), and every row's split, configuration, area and difficulty match the
dataset.

| Config | Rows | Status counts | lean_down | split/config mismatch | area/difficulty mismatch |
|---|---|---|---|---|---|
| C1 | 30 | budget_exhausted: 7, failed: 3, solved: 20 | 0 | 0 | 0 |
| C2 | 30 | claimed_solved: 30 | 0 | 0 | 0 |
| C3 | 30 | budget_exhausted: 3, failed: 7, solved: 20 | 0 | 0 | 0 |
| C4 | 30 | failed: 16, solved: 14 | 0 | 0 | 0 |
| C5 | 30 | failed: 6, solved: 24 | 0 | 0 | 0 |

### 3.3 Infrastructure and error statuses

*Planned (plan §5 and §7 item 7) · LaTeX `tab:infra`.* No row ended because of the provider,
the spend cap, a bug, or Lean being unavailable, so the plan's re-run rule and sensitivity
analysis were never needed. Every failure in this document is a failure of the method itself.

| Config | provider_error | budget_stop | error | node_error | invalid_statement | lean_down |
|---|---|---|---|---|---|---|
| C1 | 0 | 0 | 0 | 0 | 0 | 0 |
| C2 | 0 | 0 | 0 | 0 | 0 | 0 |
| C3 | 0 | 0 | 0 | 0 | 0 | 0 |
| C4 | 0 | 0 | 0 | 0 | 0 | 0 |
| C5 | 0 | 0 | 0 | 0 | 0 | 0 |

## 4. Main result: verified success

### 4.1 Success rate (SR)

*Planned (primary outcome, plan §2 and §7 item 1) · LaTeX `tab:sr`.* A problem counts as solved
only if Lean's kernel accepts the final proof. The full system C5 solved 24/30; C1 and C3 20/30;
C4 14/30. C2 never uses Lean, so it has no SR: it appears through how often it claimed success
and how often that claim could not be confirmed (FPR_B, §6). The intervals are wide and overlap:
that is what 30 problems allow.

| Config | Solved | SR (%) | Wilson 95 % CI (%) | C2 claimed rate (%) [Wilson] | FPR_B (%) [Wilson] |
|---|---|---|---|---|---|
| C1 | 20/30 | 66.7 | [48.8, 80.8] | — | — |
| C3 | 20/30 | 66.7 | [48.8, 80.8] | — | — |
| C4 | 14/30 | 46.7 | [30.2, 63.9] | — | — |
| C5 | 24/30 | 80.0 | [62.7, 90.5] | — | — |
| C2 (Baseline B) | — | — | — | 30/30 = 100.0 [88.6, 100.0] | 11/30 = 36.7 [21.9, 54.5] |

*Note.* C2 is not verified, so it has no SR; it enters only through its claimed rate B⁺/n and FPR_B (plan §7.1). Of the 30 claimed proofs the formal check verified 19 and did not verify 11; 0 left out of FPR_B.

### 4.2 The three planned comparisons (H1–H3)

*Planned (confirmatory, plan §3) · LaTeX `tab:mcnemar`.* Exact McNemar test on the same 30
problems, Bonferroni correction for 3 tests (a comparison is significant only if its adjusted p
is below 0.05). *b* = problems only C5 solved, *c* = problems only the other configuration
solved. ΔSR is the difference in success rate in percentage points, with a paired bootstrap
interval.

| ID | Comparison | b | c | ΔSR (pp) | Paired bootstrap 95 % CI (pp) | Exact p | Bonferroni p | Reject at 0.05/3 |
|---|---|---|---|---|---|---|---|---|
| H1 | C5 vs C1 | 4 | 0 | +13.3 | [+3.3, +26.7] | 0.1250 | 0.3750 | no |
| H2 | C5 vs C3 | 5 | 1 | +13.3 | [+0.0, +30.0] | 0.2188 | 0.6562 | no |
| H3 | C5 vs C4 | 10 | 0 | +33.3 | [+16.7, +50.0] | 0.0020 | 0.0059 | yes |

*Note.* n = 30, so power is low: a non-significant result is no evidence of a difference, not evidence of no difference. H2 is Δ_RAG = SR_C5 − SR_C3; H3 is the Critic contribution SR_C5 − SR_C4.

In plain words:

- **H3: the Critic (retry and replan) makes a significant difference.** C5 solved 10 problems
  that C4 did not, and C4 solved none that C5 missed (+33.3 pp, paired bootstrap interval
  [+16.7, +50.0], Bonferroni p 0.0059).
- **H1 and H2 point in C5's favour but are not significant.** C5 vs C1: 4 problems to 0;
  C5 vs C3: 5 to 1; both +13.3 pp. With 30 problems a difference of 4 or 5 problems is
  compatible with chance. "Not significant" here means *no evidence of a difference*, not
  evidence that there is none: the test has little power at this sample size.

### 4.3 Exploratory comparisons

*Planned (exploratory, plan §3): uncorrected, no claims · LaTeX `tab:mcnemar-expl`.* The other
pairs of verified configurations. C1 and C3 solve the same number of problems but not the same
problems (3 each way).

| ID | Comparison | b | c | ΔSR (pp) | Exact p (uncorrected) |
|---|---|---|---|---|---|
| E1 | C3 vs C4 | 9 | 3 | +20.0 | 0.1460 |
| E2 | C1 vs C3 | 3 | 3 | +0.0 | 1.0000 |
| E3 | C1 vs C4 | 8 | 2 | +20.0 | 0.1094 |

## 5. Results by difficulty and area

### 5.1 Solved per cell of the 3 × 3 grid

*Planned (descriptive, plan §4.1) · LaTeX `tab:cells`.* How many problems each configuration
solved in each (difficulty, area) cell. Cells have at most 5 problems, so there are no tests
and no intervals. For C2 the column counts *claimed* solutions, not verified ones.

| Difficulty | Area | n | C1 | C3 | C4 | C5 | C2 (claimed) |
|---|---|---|---|---|---|---|---|
| easy | Algebra | 5 | 5/5 | 5/5 | 4/5 | 5/5 | 5/5 |
| easy | Number theory | 5 | 5/5 | 4/5 | 5/5 | 5/5 | 5/5 |
| easy | Combinatorics | 0 | — | — | — | — | — |
| medium | Algebra | 5 | 4/5 | 5/5 | 1/5 | 5/5 | 5/5 |
| medium | Number theory | 4 | 2/4 | 2/4 | 2/4 | 4/4 | 4/4 |
| medium | Combinatorics | 1 | 0/1 | 0/1 | 1/1 | 1/1 | 1/1 |
| hard | Algebra | 3 | 1/3 | 2/3 | 0/3 | 1/3 | 3/3 |
| hard | Number theory | 4 | 1/4 | 1/4 | 0/4 | 1/4 | 4/4 |
| hard | Combinatorics | 3 | 2/3 | 1/3 | 1/3 | 2/3 | 3/3 |

### 5.2 Success rate by difficulty and by area

*Planned (descriptive, plan §4.1) · LaTeX `tab:margins` (wide table).* The margins of the grid:
first line solved/n, second line SR % with its Wilson interval. Every verified configuration
solves almost all easy problems and 4/10 or fewer hard ones; C5's advantage is on medium
problems (10/10). C2 claims every problem, at every difficulty.

| Config | easy (n = 10) | medium (n = 10) | hard (n = 10) | Algebra (n = 13) | Number theory (n = 13) | Combinatorics (n = 4) |
|---|---|---|---|---|---|---|
| C1 | 10/10 | 6/10 | 4/10 | 10/13 | 8/13 | 2/4 |
|  | 100.0 [72.2, 100.0] | 60.0 [31.3, 83.2] | 40.0 [16.8, 68.7] | 76.9 [49.7, 91.8] | 61.5 [35.5, 82.3] | 50.0 [15.0, 85.0] |
| C3 | 9/10 | 7/10 | 4/10 | 12/13 | 7/13 | 1/4 |
|  | 90.0 [59.6, 98.2] | 70.0 [39.7, 89.2] | 40.0 [16.8, 68.7] | 92.3 [66.7, 98.6] | 53.8 [29.1, 76.8] | 25.0 [4.6, 69.9] |
| C4 | 9/10 | 4/10 | 1/10 | 5/13 | 7/13 | 2/4 |
|  | 90.0 [59.6, 98.2] | 40.0 [16.8, 68.7] | 10.0 [1.8, 40.4] | 38.5 [17.7, 64.5] | 53.8 [29.1, 76.8] | 50.0 [15.0, 85.0] |
| C5 | 10/10 | 10/10 | 4/10 | 11/13 | 10/13 | 3/4 |
|  | 100.0 [72.2, 100.0] | 100.0 [72.2, 100.0] | 40.0 [16.8, 68.7] | 84.6 [57.8, 95.7] | 76.9 [49.7, 91.8] | 75.0 [30.1, 95.4] |
| C2 (claimed) | 10/10 | 10/10 | 10/10 | 13/13 | 13/13 | 4/4 |
|  | 100.0 [72.2, 100.0] | 100.0 [72.2, 100.0] | 100.0 [72.2, 100.0] | 100.0 [77.2, 100.0] | 100.0 [77.2, 100.0] | 100.0 [51.0, 100.0] |

### 5.3 The 10 hard problems

*Post hoc (not in the analysis plan), descriptive · LaTeX `tab:hard`.* One row per hard problem,
1 = verified solved.

| Problem | Area | C1 | C3 | C4 | C5 | Solved by |
|---|---|---|---|---|---|---|
| imo_1964_p1_2 | Number theory | 1 | 1 | 0 | 1 | 3 |
| imo_1965_p2 | Algebra | 1 | 1 | 0 | 1 | 3 |
| imo_1978_p5 | Algebra | 0 | 0 | 0 | 0 | 0 |
| imo_1979_p1 | Number theory | 0 | 0 | 0 | 0 | 0 |
| imo_1984_p6 | Number theory | 0 | 0 | 0 | 0 | 0 |
| imo_1987_p6 | Number theory | 0 | 0 | 0 | 0 | 0 |
| imo_1993_p5 | Algebra | 0 | 1 | 0 | 0 | 1 |
| putnam_1964_b2 | Combinatorics | 1 | 0 | 1 | 1 | 3 |
| putnam_1965_b2 | Combinatorics | 1 | 1 | 0 | 1 | 3 |
| putnam_1995_b1 | Combinatorics | 0 | 0 | 0 | 0 | 0 |
| Solved |  | 4/10 | 4/10 | 1/10 | 4/10 |  |

*Note.* Post hoc, descriptive (not in the analysis plan): no tests, no intervals. 1 = verified solved. C1 and C5 solve the same hard problems (4 and 4); 5 hard problems are solved by no configuration (imo_1978_p5, imo_1979_p1, imo_1984_p6, imo_1987_p6, putnam_1995_b1); C3 and C5 solve 4 and 4 hard problems, but only C3 solves imo_1993_p5 and only C5 solves putnam_1964_b2.

Three things stand out. C1 and C5 solve exactly the same 4 hard problems (imo_1964_p1_2,
imo_1965_p2, putnam_1964_b2, putnam_1965_b2), so on the hard tier the whole C5 gain over C1
disappears: all 4 problems that C5 solves and C1 does not are medium (§5.5). Five hard problems
(imo_1978_p5, imo_1979_p1, imo_1984_p6, imo_1987_p6, putnam_1995_b1) are solved by no
configuration. C3 and C5 also solve 4 each but differ by one problem in each direction: only
C3 solves imo_1993_p5, only C5 solves putnam_1964_b2. Since C5 has everything C3 has plus
retrieval, a hard problem that C3 solves and C5 does not shows how much a single run can vary
from one to the next; each configuration was run once, so differences of one problem should
not be read as an effect.

### 5.4 Per-problem outcomes

*Data listing (the source of the counts above) · LaTeX `tab:per-problem` (wide table).* One row
per `test` problem: 1 = verified solved; the last column gives C2's claim and whether the Lean
check confirmed it.

| Problem | Difficulty | Area | C1 | C3 | C4 | C5 | C2 (claim, check) |
|---|---|---|---|---|---|---|---|
| algebra_absapbon1pabsapbleqsumabsaon1pabsa | medium | Algebra | 0 | 1 | 0 | 1 | claimed, verified |
| amc12_2000_p20 | medium | Algebra | 1 | 1 | 0 | 1 | claimed, verified |
| amc12_2001_p9 | medium | Algebra | 1 | 1 | 0 | 1 | claimed, verified |
| amc12a_2008_p15 | medium | Number theory | 0 | 0 | 0 | 1 | claimed, not_verified |
| amc12a_2008_p25 | medium | Algebra | 1 | 1 | 0 | 1 | claimed, verified |
| amc12b_2002_p3 | medium | Number theory | 0 | 1 | 1 | 1 | claimed, not_verified |
| amc12b_2003_p6 | medium | Algebra | 1 | 1 | 1 | 1 | claimed, verified |
| imo_1964_p1_2 | hard | Number theory | 1 | 1 | 0 | 1 | claimed, not_verified |
| imo_1965_p2 | hard | Algebra | 1 | 1 | 0 | 1 | claimed, not_verified |
| imo_1978_p5 | hard | Algebra | 0 | 0 | 0 | 0 | claimed, not_verified |
| imo_1979_p1 | hard | Number theory | 0 | 0 | 0 | 0 | claimed, not_verified |
| imo_1984_p6 | hard | Number theory | 0 | 0 | 0 | 0 | claimed, not_verified |
| imo_1987_p6 | hard | Number theory | 0 | 0 | 0 | 0 | claimed, not_verified |
| imo_1993_p5 | hard | Algebra | 0 | 1 | 0 | 0 | claimed, not_verified |
| mathd_algebra_160 | easy | Algebra | 1 | 1 | 1 | 1 | claimed, verified |
| mathd_algebra_206 | easy | Algebra | 1 | 1 | 0 | 1 | claimed, verified |
| mathd_algebra_234 | easy | Algebra | 1 | 1 | 1 | 1 | claimed, verified |
| mathd_algebra_288 | easy | Algebra | 1 | 1 | 1 | 1 | claimed, verified |
| mathd_algebra_455 | easy | Algebra | 1 | 1 | 1 | 1 | claimed, verified |
| mathd_numbertheory_321 | easy | Number theory | 1 | 0 | 1 | 1 | claimed, verified |
| mathd_numbertheory_328 | easy | Number theory | 1 | 1 | 1 | 1 | claimed, verified |
| mathd_numbertheory_345 | easy | Number theory | 1 | 1 | 1 | 1 | claimed, verified |
| mathd_numbertheory_430 | easy | Number theory | 1 | 1 | 1 | 1 | claimed, verified |
| mathd_numbertheory_64 | easy | Number theory | 1 | 1 | 1 | 1 | claimed, verified |
| numbertheory_3pow2pownm1mod2pownp3eq2pownp2 | medium | Number theory | 1 | 0 | 0 | 1 | claimed, verified |
| numbertheory_notequiv2i2jasqbsqdiv8 | medium | Number theory | 1 | 1 | 1 | 1 | claimed, verified |
| numbertheory_sumkmulnckeqnmul2pownm1 | medium | Combinatorics | 0 | 0 | 1 | 1 | claimed, verified |
| putnam_1964_b2 | hard | Combinatorics | 1 | 0 | 1 | 1 | claimed, verified |
| putnam_1965_b2 | hard | Combinatorics | 1 | 1 | 0 | 1 | claimed, not_verified |
| putnam_1995_b1 | hard | Combinatorics | 0 | 0 | 0 | 0 | claimed, not_verified |

### 5.5 Problems solved by one configuration and not the other

*Descriptive (discordant pairs of the comparisons; candidates for example traces in the
report).* For each pair, the problems behind *b* and *c* of Tables 2 and 2b, with what C5 did on
them (status, LLM calls, replans, and the largest number of attempts on any subgoal of the
accepted plan).

- C5 solved, C4 not (Critic loop): algebra_absapbon1pabsapbleqsumabsaon1pabsa (C5: solved, 24 calls, replans 1, max attempts 1); amc12_2000_p20 (C5: solved, 23 calls, replans 0, max attempts 3); amc12_2001_p9 (C5: solved, 7 calls, replans 0, max attempts 2); amc12a_2008_p15 (C5: solved, 16 calls, replans 0, max attempts 3); amc12a_2008_p25 (C5: solved, 23 calls, replans 0, max attempts 2); imo_1964_p1_2 (C5: solved, 15 calls, replans 0, max attempts 3); imo_1965_p2 (C5: solved, 14 calls, replans 0, max attempts 2); mathd_algebra_206 (C5: solved, 28 calls, replans 0, max attempts 2); numbertheory_3pow2pownm1mod2pownp3eq2pownp2 (C5: solved, 15 calls, replans 0, max attempts 3); putnam_1965_b2 (C5: solved, 17 calls, replans 0, max attempts 3)
- C4 solved, C5 not: —
- C5 solved, C3 not (RAG): amc12a_2008_p15 (C5: solved, 16 calls, replans 0, max attempts 3); mathd_numbertheory_321 (C5: solved, 1 calls, replans 0, max attempts 0); numbertheory_3pow2pownm1mod2pownp3eq2pownp2 (C5: solved, 15 calls, replans 0, max attempts 3); numbertheory_sumkmulnckeqnmul2pownm1 (C5: solved, 5 calls, replans 0, max attempts 1); putnam_1964_b2 (C5: solved, 9 calls, replans 0, max attempts 1)
- C3 solved, C5 not: imo_1993_p5 (C5: failed, 27 calls, replans 1, max attempts -)
- C5 solved, C1 not: algebra_absapbon1pabsapbleqsumabsaon1pabsa (C5: solved, 24 calls, replans 1, max attempts 1); amc12a_2008_p15 (C5: solved, 16 calls, replans 0, max attempts 3); amc12b_2002_p3 (C5: solved, 9 calls, replans 0, max attempts 1); numbertheory_sumkmulnckeqnmul2pownm1 (C5: solved, 5 calls, replans 0, max attempts 1)
- C1 solved, C5 not: —
- C2 claimed, check did not verify: amc12a_2008_p15 (C5: solved, 16 calls, replans 0, max attempts 3); amc12b_2002_p3 (C5: solved, 9 calls, replans 0, max attempts 1); imo_1964_p1_2 (C5: solved, 15 calls, replans 0, max attempts 3); imo_1965_p2 (C5: solved, 14 calls, replans 0, max attempts 2); imo_1978_p5 (C5: failed, 31 calls, replans 1, max attempts -); imo_1979_p1 (C5: failed, 28 calls, replans 1, max attempts -); imo_1984_p6 (C5: failed, 32 calls, replans 1, max attempts -); imo_1987_p6 (C5: failed, 36 calls, replans 1, max attempts -); imo_1993_p5 (C5: failed, 27 calls, replans 1, max attempts -); putnam_1965_b2 (C5: solved, 17 calls, replans 0, max attempts 3); putnam_1995_b1 (C5: failed, 25 calls, replans 1, max attempts -)

## 6. Baseline B (C2): what the LLM claimed vs what Lean confirmed

*Planned (plan §4.2, Eq. 16) · LaTeX `tab:fprb`.* C2 never uses Lean, so it cannot "solve"
anything in the verified sense. Instead it reports whether it believes its own proof is
correct. To see how often that belief is wrong, each claimed proof was handed to the Lean
pipeline (Autoformalizer + Verifier, up to 3 attempts, same lemma search as C5;
`--check` in `scripts/run_baseline_b.py`).

| Quantity | Value |
|---|---|
| Problems | 30 |
| Claimed (B⁺) | 30 |
| Not claimed | 0 |
| Check status among B⁺ | not_verified: 11, verified: 19 |
| Verified (Acc on the assembled theorem) | 19 |
| Not verified (budget_exhausted included) | 11 |
| Left out of FPR_B (provider_error, budget_stop, error, lean_down) | 0 |
| FPR_B = not verified / (verified + not verified) | 11/30 = 36.7 % [21.9, 54.5] |
| B⁺ problems for the manual review (§4.2) | 10 drawn with random.Random(2026).sample(sorted ids, 10) (listed below) |

*Note.* Manual review set (B⁺ not verified): amc12b_2002_p3, imo_1964_p1_2, imo_1965_p2, imo_1978_p5, imo_1979_p1, imo_1984_p6, imo_1987_p6, imo_1993_p5, putnam_1965_b2, putnam_1995_b1.

By difficulty (counted from Table 11):

| | easy | medium | hard | total |
|---|---|---|---|---|
| Claimed correct by the LLM | 10 | 10 | 10 | **30** |
| Confirmed by Lean | 10 | 8 | 1 | **19** |

C2 claimed to have solved every problem. Lean confirmed 19; for the other 11 the claim could
not be turned into an accepted proof. That is FPR_B = 11/30 = 36.7 % [21.9, 54.5]. Almost all
of the gap is on hard problems, where the model was sure of 10 answers and only 1 could be
confirmed. This is the main motivation of the project: a confident LLM is not a reliable judge
of its own mathematics.

FPR_B is an **upper bound** on how often C2's self-assessment is wrong: "not confirmed" means
"the pipeline could not formalise it within the budget", which does not always mean the
informal argument was wrong. Two facts bear on this:

- For 5 of the 11 (amc12a_2008_p15, amc12b_2002_p3, imo_1964_p1_2, imo_1965_p2,
  putnam_1965_b2), C5 has a Lean-verified proof of the same statement, so at least the claimed
  result is true there (whether C2's own argument was sound is a separate question).
- The plan asks for a manual review of 10 of the 11 (drawn with seed 2026; listed in the note
  of Table 1b) to separate wrong arguments from formalisation failures. The review sheet is
  `docs/results/fprb_review.md` (reviewer: Andrés). **Its verdicts are still pending.**

## 7. Efficiency and cost

### 7.1 Per solved problem

*Planned (plan §4.3, RNF-3) · LaTeX `tab:eff-solved`.* Total LLM calls, tokens, cost and
wall-clock time of a configuration divided by the problems it solved (failed problems' cost is
included in the numerator). Bootstrap intervals in brackets. **Nominal** cost prices cache hits
as fresh calls; it is the figure the plan compares, because the cache makes later runs cheaper
for reasons unrelated to the method. **Real** cost is what the provider charged.

| Config | Solved | Calls / solved | k tokens / solved | Nominal USD / solved | Real USD / solved | Wall min / solved |
|---|---|---|---|---|---|---|
| C1 | 20 | 12.85 [7.19, 22.60] | 219.7 [114.6, 394.7] | 0.1285 [0.0668, 0.2313] | 0.1235 [0.0643, 0.2218] | 17.7 [9.2, 32.0] |
| C2 (per claimed) | 30 | 2.17 [2.00, 2.40] | 5.3 [2.8, 8.6] | 0.0027 [0.0013, 0.0047] | 0.0027 [0.0013, 0.0047] | 0.4 [0.2, 0.7] |
| C3 | 20 | 26.50 [16.15, 43.67] | 150.4 [79.3, 274.6] | 0.0675 [0.0335, 0.1279] | 0.0675 [0.0335, 0.1278] | 9.2 [4.6, 17.3] |
| C4 | 14 | 9.43 [5.72, 16.67] | 70.7 [38.2, 137.8] | 0.0278 [0.0135, 0.0571] | 0.0000 [0.0000, 0.0000] | 0.2 [0.1, 0.3] |
| C5 | 24 | 16.92 [10.63, 26.21] | 148.0 [74.4, 266.4] | 0.0566 [0.0245, 0.1097] | 0.0490 [0.0236, 0.0897] | 7.6 [3.3, 14.6] |

*Note.* C2 has no verified solutions: its ratio uses claimed solutions and excludes the claim check (calls, tokens, cost and wall time up to claim_check_start).

C5 needs fewer calls per solved problem than C3 (16.92 vs 26.50) and costs less per solved
problem than C1 (nominal 0.0566 vs 0.1285 USD); C1 also takes the most wall time per solved
problem (17.7 min).

### 7.2 Per problem (means)

*Planned (plan §4.3) · LaTeX `tab:eff-problem`.* The same quantities averaged over all 30
problems, solved or not.

| Config | Calls / problem | k tokens / problem | Nominal USD / problem | Real USD / problem | Wall min / problem |
|---|---|---|---|---|---|
| C1 | 8.57 [5.83, 11.33] | 146.4 [91.1, 203.5] | 0.0857 [0.0532, 0.1192] | 0.0823 [0.0505, 0.1152] | 11.8 [7.2, 16.5] |
| C2 | 2.17 [2.00, 2.40] | 5.3 [2.8, 8.6] | 0.0027 [0.0013, 0.0047] | 0.0027 [0.0013, 0.0047] | 0.4 [0.2, 0.7] |
| C3 | 17.67 [12.97, 22.37] | 100.3 [63.8, 140.8] | 0.0450 [0.0269, 0.0658] | 0.0450 [0.0269, 0.0658] | 6.1 [3.7, 9.0] |
| C4 | 4.40 [3.37, 5.43] | 33.0 [22.8, 44.4] | 0.0130 [0.0081, 0.0186] | 0.0000 [0.0000, 0.0000] | 0.1 [0.1, 0.1] |
| C5 | 13.53 [9.57, 17.60] | 118.4 [67.9, 178.7] | 0.0453 [0.0225, 0.0737] | 0.0392 [0.0218, 0.0588] | 6.1 [3.0, 9.7] |

### 7.3 Totals, cache and real cost

*Planned (plan §4.3, with the cache and spend-ledger columns added for transparency) · LaTeX
`tab:totals` (wide table).* Totals over the kept rows. The last column is the real spend of all
the configuration's runs from `data/spend_ledger.jsonl`, which also includes the attempts
discarded when a run was interrupted (C1: 0.0985 USD; C5: 0.1815 USD) and, for C2, the claim
check.

| Config | LLM calls | Cached calls | Cached (%) | Tokens (M) | Nominal USD (rows) | Real USD (rows) | Real USD (ledger, all runs) |
|---|---|---|---|---|---|---|---|
| C1 | 257 | 47 | 18.3 | 4.393 | 2.5700 | 2.4695 | 2.5679 |
| C2 | 65 | 0 | 0.0 | 0.158 | 0.0814 | 0.0814 | 0.7083 |
| C3 | 530 | 18 | 3.4 | 3.009 | 1.3508 | 1.3500 | 1.3500 |
| C4 | 132 | 132 | 100.0 | 0.990 | 0.3887 | 0.0000 | 0.0000 |
| C5 | 406 | 34 | 8.4 | 3.551 | 1.3590 | 1.1770 | 1.3585 |

*Note.* C2 claim check (not in the C2 figures of the other columns): 77 calls (11 cached), 0.6274 nominal, 0.6269 real.

**RNF-3 (C̄_solved ≤ 20 for C5):** C̄_solved = 406/24 = 16.92 calls per solved problem: **met**.

How to read it:

- **C4 cost 0 USD** because every request it made had already been made, word for word, by C5
  (C4 is C5 with retries removed, so its first attempts are identical); the answers came from
  the cache (100.0 % cached calls). For the same reason C4's wall times measure only Lean.
- **RNF-3** (at most 20 LLM calls per solved problem for C5) is **met**: 406/24 = 16.92.
- C1 has a budget of 18 calls per problem (the average C5 used on `dev`, by a rule fixed before
  `test`); the full system has 40.
- **Project spend** (ledger, all runs, `dev` included): main experiments 7.3596 USD. Including the
  exploratory Pro run (1.9407 USD, §14): 9.3003 USD in total; the project cap was raised from 9 to
  12.5 USD for that run (§11).

### 7.4 Medians and quartiles

*Supplementary (requested by the team, not in the plan) · LaTeX `tab:medians`.* Means hide that
the distributions are very skewed: most problems take few calls and a few take many.

| Config | Calls, median [Q1, Q3] | Wall min, median [Q1, Q3] |
|---|---|---|
| C1 | 4.5 [1.0, 18.0] | 2.7 [0.6, 24.2] |
| C2 | 2.0 [2.0, 2.0] | 0.1 [0.1, 0.3] |
| C3 | 16.5 [3.2, 28.0] | 3.7 [0.6, 6.7] |
| C4 | 5.0 [1.0, 5.8] | 0.1 [0.0, 0.1] |
| C5 | 11.5 [1.0, 23.8] | 2.4 [0.2, 5.2] |

### 7.5 Wall time by difficulty

*Supplementary (requested by the team, not in the plan) · LaTeX `tab:wall-diff` (wide table).*
Minutes per problem, mean / median [quartiles]. Times are dominated by waiting for the model, and
grow with difficulty. C4's times are not comparable (cache, see above). Many easy problems take
seconds in C3–C5 because they need no LLM-written subgoal proof at all (Table 8, §9.4).

| Config | easy (n = 10): mean / median [Q1, Q3] | medium (n = 10): mean / median [Q1, Q3] | hard (n = 10): mean / median [Q1, Q3] |
|---|---|---|---|
| C1 | 0.7 / 0.4 [0.1, 1.1] | 14.3 / 16.2 [1.8, 23.2] | 20.3 / 25.3 [7.0, 32.4] |
| C2 | 0.1 / 0.1 [0.1, 0.2] | 0.1 / 0.1 [0.1, 0.2] | 1.0 / 0.5 [0.2, 1.1] |
| C3 | 0.8 / 0.1 [0.0, 1.0] | 4.0 / 2.6 [1.2, 6.0] | 13.6 / 12.3 [6.6, 19.8] |
| C4 | 0.1 / 0.0 [0.0, 0.1] | 0.1 / 0.1 [0.0, 0.1] | 0.1 / 0.1 [0.1, 0.1] |
| C5 | 0.4 / 0.0 [0.0, 0.5] | 2.7 / 2.6 [0.7, 4.5] | 15.1 / 14.1 [4.0, 21.0] |

## 8. Refinement robustness: SR(k)

*Planned (plan §4.4, Eq. 14; figure: plan §7 item 5) · LaTeX `tab:srk`; figure
`docs/results/figures/srk_curve.pdf` (`.png` below).* SR_C5(k) counts a C5 problem as solved at
k if its accepted proof needed at most k attempts on every subgoal. It shows how much each extra
attempt (with the Critic's feedback) adds. The last two columns are subgoal-level diagnostics
(supplementary).

| k | SR_C5(k) [Wilson] (%) | Δ̂(k) = SR(k) − SR(k−1) (pp) | Indep. Δ(k) = p̂(1−p̂)^(k−1) (%) | Subgoals solved at attempt k | Solved at k / reached k |
|---|---|---|---|---|---|
| 1 | 15/30 = 50.0 [33.2, 66.8] | +20.0 | 40.9 | 18/44 = 40.9 | 18/44 = 40.9 |
| 2 | 19/30 = 63.3 [45.5, 78.1] | +13.3 | 24.2 | 8/44 = 18.2 | 8/26 = 30.8 |
| 3 | 24/30 = 80.0 [62.7, 90.5] | +16.7 | 14.3 | 5/44 = 11.4 | 5/18 = 27.8 |

*Note.* SR_C5(0) = 9/30 (accepted plan with no LLM subgoal attempt). p̂ = 18/44 = 40.9 %. k = 1 from C4 (no retries, no Critic advice): 14/30 = 46.7 % [30.2, 63.9]. The last two columns are subgoal-level diagnostics (supplementary).

![SR(k) of C5 and its marginal gain against the independence curve](results/figures/srk_curve.png)

**Figure: SR(k).** (a) SR_C5(k) for k = 0…3 with Wilson intervals: 9/30 solved with no
LLM-attempted subgoal, then 15/30, 19/30 and 24/30 as up to 1, 2 and 3 attempts per subgoal are
allowed. The square is C4 (one attempt, no retries, no Critic advice): 14/30 = 46.7 %, next to
SR_C5(1) = 15/30. (b) The observed gain of each extra attempt, Δ̂(k) (+20.0, +13.3, +16.7 pp),
against the gain expected if every attempt succeeded independently with the first-attempt rate
p̂ = 18/44 = 40.9 % (40.9, 24.2, 14.3). The independence curve falls quickly; the observed gain
does not: the third attempt still adds +16.7 pp, above the 14.3 the independence model predicts.
This fits the idea that retries guided by Lean's errors are not just new random draws. Two
cautions: the two curves are at different levels (the independence curve is per subgoal, Δ̂(k)
per problem; method note 23 in `results.md`), and with n = 30 each step is a handful of problems.

## 9. Errors and failure reasons

### 9.1 Error taxonomy (κ)

*Planned (plan §4.5, Eq. 11) · LaTeX `tab:kappa` (wide table).* Every failed attempt classified
by the deterministic classifier κ, count (%). Tactic failures and syntax errors dominate. No
attempt was classified as misformalization. For C1 the attempts were classified offline with
the same κ: 44 of its syntax errors are empty answers, which happen alongside 52 responses cut
at the token limit (method note 25 in `results.md`). For C2 the row is the claim-check loop.

| Config | Failed attempts | syntax | missing lemma | tactic failure | misformalization | timeout | planning |
|---|---|---|---|---|---|---|---|
| C1 | 186 | 60 (32.3) | 50 (26.9) | 68 (36.6) | 0 (0.0) | 8 (4.3) | 0 (0.0) |
| C2 (claim check) | 45 | 14 (31.1) | 8 (17.8) | 22 (48.9) | 0 (0.0) | 1 (2.2) | 0 (0.0) |
| C3 | 82 | 20 (24.4) | 17 (20.7) | 42 (51.2) | 0 (0.0) | 2 (2.4) | 1 (1.2) |
| C4 | 16 | 5 (31.2) | 4 (25.0) | 7 (43.8) | 0 (0.0) | 0 (0.0) | 0 (0.0) |
| C5 | 57 | 19 (33.3) | 18 (31.6) | 18 (31.6) | 0 (0.0) | 2 (3.5) | 0 (0.0) |

*Note.* C3–C5: classes recorded by the Critic. C1: attempts classified offline with policy.classify (first 3 Lean messages, 300 characters each, as stored; no k_max escalation). C2 has no verified attempts of its own: its row is the claim-check loop.

### 9.2 Error taxonomy per area

*Planned (plan §4.5) · LaTeX `tab:kappa-area` (wide table).* The same counts split by area.
Missing-lemma errors are frequent in algebra; tactic failures dominate number theory.

| Config | Area | Failed attempts | syntax | missing lemma | tactic failure | misformalization | timeout | planning |
|---|---|---|---|---|---|---|---|---|
| C1 | Algebra | 77 | 14 (18.2) | 40 (51.9) | 22 (28.6) | 0 (0.0) | 1 (1.3) | 0 (0.0) |
| C1 | Number theory | 83 | 27 (32.5) | 3 (3.6) | 46 (55.4) | 0 (0.0) | 7 (8.4) | 0 (0.0) |
| C1 | Combinatorics | 26 | 19 (73.1) | 7 (26.9) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) |
| C2 (claim check) | Algebra | 14 | 5 (35.7) | 4 (28.6) | 5 (35.7) | 0 (0.0) | 0 (0.0) | 0 (0.0) |
| C2 (claim check) | Number theory | 23 | 5 (21.7) | 4 (17.4) | 13 (56.5) | 0 (0.0) | 1 (4.3) | 0 (0.0) |
| C2 (claim check) | Combinatorics | 8 | 4 (50.0) | 0 (0.0) | 4 (50.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) |
| C3 | Algebra | 18 | 3 (16.7) | 9 (50.0) | 5 (27.8) | 0 (0.0) | 0 (0.0) | 1 (5.6) |
| C3 | Number theory | 43 | 13 (30.2) | 3 (7.0) | 26 (60.5) | 0 (0.0) | 1 (2.3) | 0 (0.0) |
| C3 | Combinatorics | 21 | 4 (19.0) | 5 (23.8) | 11 (52.4) | 0 (0.0) | 1 (4.8) | 0 (0.0) |
| C4 | Algebra | 8 | 1 (12.5) | 3 (37.5) | 4 (50.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) |
| C4 | Number theory | 6 | 2 (33.3) | 1 (16.7) | 3 (50.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) |
| C4 | Combinatorics | 2 | 2 (100.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) |
| C5 | Algebra | 25 | 7 (28.0) | 8 (32.0) | 10 (40.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) |
| C5 | Number theory | 24 | 8 (33.3) | 9 (37.5) | 6 (25.0) | 0 (0.0) | 1 (4.2) | 0 (0.0) |
| C5 | Combinatorics | 8 | 4 (50.0) | 1 (12.5) | 2 (25.0) | 0 (0.0) | 1 (12.5) | 0 (0.0) |

### 9.3 Final failure reason per problem

*Planned (plan §4.5) · LaTeX `tab:final-fail`.* For every problem a configuration did not
solve, its final status and the stage where it stopped (`verify` = a subgoal proof was rejected;
`formalize` = the Lean sketch could not be assembled). C1 failures are mostly running out of its
18-call budget.

| Config | Problems | status @ stage: count |
|---|---|---|
| C1 | 10 | budget_exhausted: 7; failed: 3 |
| C2 | 30 | claimed, check verified: 19; claimed, check not_verified: 11 |
| C3 | 10 | failed @ verify: 5; budget_exhausted @ formalize: 3; failed @ formalize: 2 |
| C4 | 16 | failed @ verify: 11; failed @ formalize: 5 |
| C5 | 6 | failed @ verify: 4; failed @ formalize: 2 |

### 9.4 Subgoals closed by fixed tactics vs the LLM

*Supplementary (not in the plan) · LaTeX `tab:prepass` (wide table).* Before asking the LLM, the
graph configurations try a fixed list of tactics on each subgoal (the pre-pass). This table shows
how many subgoals of the accepted plans each closed, and how many solved problems needed no
LLM-written subgoal proof at all (8–9 per configuration). This is a shared factor of C3–C5, not
something one of them has and the others lack.

| Config | Pre-pass (accepted plans) | LLM (accepted plans) | Pre-pass (all subgoal_solved) | Solved with no LLM subgoal | Solved |
|---|---|---|---|---|---|
| C3 | 7/32 = 21.9 | 25/32 = 78.1 | 8/39 = 20.5 | 8 | 20 |
| C4 | 9/18 = 50.0 | 9/18 = 50.0 | 11/21 = 52.4 | 9 | 14 |
| C5 | 11/39 = 28.2 | 28/39 = 71.8 | 11/42 = 26.2 | 9 | 24 |

## 10. Fidelity review

Lean checks the formal statement, not the original text, so the statements were reviewed by
hand twice.

**Before any run (2026-11-01, `data/curation/fidelity_review.yaml`).** Every drawn problem's
Lean statement was compared with its informal text. Two were discarded because the formal
statement changed the mathematics (`imo_1962_p2`, `imo_1974_p5`; replaced). Seven were kept
with a note: the formal version differs from the text in a way judged harmless (for example a
sequence already reduced to one equation, or a definition written differently).

**After the `test` runs (2026-11-07, team review, `post_test_review`).** As plan §6 requires,
every problem solved on `test` by some configuration that has a note was read again against the
informal text: four problems.

*Planned (plan §6) · LaTeX `tab:fidelity`.*

| Problem | Solved by | Manual review |
|---|---|---|
| amc12b_2003_p6 | C1, C3, C4, C5 | 2026-11-07: counts as solved; report note: The formalization proves that u 0 = +-2/sqrt 3 (a necessary condition), not that -2*sqrt(3)/3 is attained; this is enough to rule out the other options. |
| mathd_algebra_234 | C1, C3, C4, C5 | 2026-11-07: counts as solved; no note |
| mathd_algebra_455 | C1, C3, C4, C5 | 2026-11-07: counts as solved; no note |
| putnam_1965_b2 | C1, C3, C5 | 2026-11-07: counts as solved; no note |

In plain words: all four count as solved and no success rate changes, so the plan's "SR with
and without" never applies. One needs a caveat in the report: **`amc12b_2003_p6`** (solved by
C1, C3, C4, C5 and confirmed for C2) asks which option is a possible first term. The
formalisation proves that the first term must be ±2/√3 (a necessary condition), not that
−2√3/3 is actually attained; that is enough to rule out the other options. The remaining notes
do not apply: `imo_2006_p3` is in `dev`, and `imo_1987_p6` and `putnam_1995_b1` were solved by
no configuration.

## 11. Deviations from the analysis plan, in plain language

The analysis itself (metrics, tests, corrections) was not changed. What happened differently
from the plan was operational, and each item is logged with date and reason in
`analysis_plan.md` (Deviations):

1. **C5 was interrupted (2026-11-03).** The machine ran low on memory and the run was killed
   after 23 of 30 problems. Those 23 rows are kept unchanged. The other 7 were run again with the
   same frozen configuration, seed and cache, in two resume runs: the first one hit its spend cap
   during its third problem (that incomplete row of `imo_1984_p6` is discarded), the second
   finished the rest. The merge rule (which row comes from which run) was written down before
   the analysis and the analysis script checks it.
2. **C4 ran before C3** (plan order: C5, C3, C4, C2, C1). For schedule and cost reasons only.
   Every configuration still ran once on the same problems, so the paired comparisons are not
   affected.
3. **The per-run spend cap was raised twice for the main runs** (to 2 USD before C3, then to
   3.5 USD before C1), by team decision, so that those runs would not stop half-way. The cap only
   decides whether a request is sent; it does not change prompts, models, call budgets or results.
   For the exploratory Pro run (§14, after all main runs) both caps were raised again: the project
   cap from 9 to 12.5 USD and the per-run cap from 3.5 to 5 USD.
4. **C1 was stopped by the team (2026-11-05)** to free the machine, after 23 of 30 problems. The
   other 7 were completed on 2026-11-07 in a resume run with the same procedure as C5, and the
   pieces were merged into one file (checked row by row).
5. **`dirty: true` in four `run_config.json` files** (found during the analysis). The run
   configuration file was rewritten at the end of each run and read the git state at that moment;
   the only changed file was `analysis_plan.md` itself, edited during the runs to log these
   incidents. Every tree was clean at launch. The recording was fixed after the `test` runs.
6. **The exploratory Pro run was stopped by the team (2026-11-08)** after 5 of its 10 problems and
   not resumed (§14). It is outside the analysis plan's results and changes none of them.

Written before any `test` run, as a clarification rather than a deviation: **how the C2 claim
check works** (plan §4.2): each claimed proof becomes a sketch for the Autoformalizer, with up to
3 attempts and Lean feedback, the same lemma search as C5, no Planner and no fixed tactics; its
calls are counted apart from C2's own.

**Where the plan was silent or had to be read literally** (full list: "Method notes" at the end
of `results.md`):

- C2 has no verified SR, so Table 1 reports its claimed rate and FPR_B instead, and C2 is in no
  McNemar test. In the grid tables C2 shows claimed counts, labelled as such.
- The plan fixes the bootstrap (10 000 resamples, seed 2026) but not the random generator;
  NumPy's default generator was used, the same for every comparison.
- The plan says "10 drawn with seed 2026" for the C2 manual review without saying how;
  Python's `random.Random(2026).sample` on the sorted ids was used.
- SR(0) and the first-attempt rate p̂ of the SR(k) analysis are not defined in the plan; the
  definitions used are in method notes 22 and 23.
- C1 attempts were classified by κ offline from the stored (shortened) Lean messages.
- Tables 4d, 5, 8 and 11b were added after the plan; they are descriptive and labelled.
- Disclosure: while checking configurations, before the analysis script existed, the analyst had
  seen the first C5 `test` row and run summaries of C5 and C4. After the first `test` run the
  script changed only in how C2's wall time is cut, an added note on truncated responses, and
  LaTeX layout. No test, comparison, correction or metric changed.

## 12. Limitations

- **Small sample.** 30 test problems (10 per difficulty, only 4 in combinatorics). Confidence
  intervals are wide, H1 and H2 have little power, and per-area numbers are descriptive only.
- **One run per configuration.** Each configuration ran once with one seed, so pass@k is
  estimated only for k = 1 (the SR). LLM output varies between runs; differences of one or two
  problems (for example C3 vs C5 on hard problems, §5.3) are within that variation.
- **One general-purpose model.** Every role used the same general model (`deepseek-flash`), not
  a model trained for Lean. Results may differ with other models. `deepseek-flash` is an
  alias with no versioned id; the run configs record what was used.
- **Possible contamination.** miniF2F and the competition problems are public, and Lean proofs
  of many miniF2F problems are online. The model may have seen some of them in training, which
  would make every configuration look better than it would on new problems. This affects all
  configurations, but not necessarily equally (C1 asks for a Lean proof directly, which is
  where a memorised proof might help most).
- **Formal vs informal statement.** Lean checks the formal statement, not the original text.
  All statements were reviewed (§10); `amc12b_2003_p6` needs the caveat above.
- **FPR_B is an upper bound** and its manual review is still pending (§6).
- **Shared tactic pre-pass.** 8–9 solved problems per graph configuration need no LLM-written
  subgoal proof (Table 8), which narrows the room in which RAG and the Critic can differ.
- **Truncated responses.** Some LLM answers hit the token limit (C1 most often: 52 responses in
  9 rows); in C1 these show up as empty attempts counted against its budget.
- **Interrupted runs.** C5 and C1 were stopped by operational incidents and resumed with the
  same frozen configuration (§11).
- **C4's cost and times** come from C5's cache and are not comparable at real cost; nominal cost
  is the comparable figure.

## 13. For the report

What to use in each section of the report, with the LaTeX label of each table in
`docs/results/tables.tex` (`\input` the file or copy the tables; it needs only `booktabs`;
tables marked "wide" are `table*`). The figure is `docs/results/figures/srk_curve.pdf`.
Keep the status label of each table in its caption or text: post hoc and supplementary tables
must say so.

### Results

| What | Table (this file) | LaTeX label | Status |
|---|---|---|---|
| Verified SR with Wilson CI; C2 claimed rate and FPR_B | Table 1 (§4.1) | `tab:sr` | Planned |
| H1–H3: b, c, ΔSR, bootstrap CI, exact and Bonferroni p | Table 2 (§4.2) | `tab:mcnemar` | Planned (confirmatory) |
| Exploratory pairs | Table 2b (§4.3) | `tab:mcnemar-expl` | Planned (exploratory) |
| Per-cell grid and margins | Tables 3, 3b (§5.1–5.2) | `tab:cells`, `tab:margins` | Planned (descriptive) |
| Baseline B claim check | Table 1b (§6) | `tab:fprb` | Planned |
| Efficiency, RNF-3 | Tables 4, 4b, 4c (§7.1–7.3) | `tab:eff-solved`, `tab:eff-problem`, `tab:totals` | Planned |
| SR(k) and the independence curve | Table 6 + figure (§8) | `tab:srk`, `srk_curve.pdf` | Planned |
| κ taxonomy and final failure reasons | Tables 7, 7b, 7c (§9.1–9.3) | `tab:kappa`, `tab:kappa-area`, `tab:final-fail` | Planned |
| Infrastructure statuses and fidelity review | Tables 9, 10 (§3.3, §10) | `tab:infra`, `tab:fidelity` | Planned |
| Appendix / supplementary material | Runs used, Row checks, Tables 4d, 5, 8, 11, 11b | `tab:runs`, `tab:checks`, `tab:medians`, `tab:wall-diff`, `tab:prepass`, `tab:per-problem`, `tab:hard` | Integrity / supplementary / post hoc |

Points to state:

- C5 solves 24/30 (80.0 %, [62.7, 90.5]); C1 and C3 20/30; C4 14/30 (Table 1).
- Only H3 (Critic) is significant: +33.3 pp, Bonferroni p 0.0059. H1 and H2 (+13.3 pp each)
  are not; report them as "no evidence of a difference", never "no difference" (Table 2).
- FPR_B = 11/30 = 36.7 % [21.9, 54.5], an upper bound (Table 1b).
- RNF-3 is met: 16.92 calls per solved problem for C5 (Table 4c).
- No provider errors, spend stops, Lean outages or degraded runs (Table 9); all four fidelity
  reviews count as solved, with the `amc12b_2003_p6` note (Table 10).

### Discussion

- **The Critic loop is what matters most** (H3, and the SR(k) figure: each extra attempt keeps
  adding solved problems, more than an independent-retry model predicts at k = 3; §8, with the
  two cautions given there).
- **C5's gain over C1 is entirely on medium problems** (C5 10/10, C1 6/10; Table 3b). On hard
  problems C1 and C5 solve the same 4 (Table 11b, post hoc), and 5 hard problems are solved by
  no configuration: hard olympiad problems remain out of reach for this setup.
- **RAG**: C5 vs C3 favours C5 (5 to 1) but is not significant; the only problem C3 solves and C5
  does not is a hard one, an example of run-to-run variation (§5.3).
- **Self-assessment is unreliable**: C2 claims 30/30; Lean confirms 19, and only 1 of the 10 hard
  claims (§6). For 5 of the 11 unconfirmed claims C5 proves the statement, which is why FPR_B is
  an upper bound; cite the manual review once it is done.
- **Cost**: C5 is cheaper per solved problem than C1 at nominal cost (Table 4); C3, without
  lemma suggestions, needs the most calls per solved problem.
- **Errors**: tactic failures and syntax errors dominate; misformalization never appears;
  missing lemmas are common in algebra (Tables 7, 7b). C1's failures are mostly budget
  exhaustion, partly driven by truncated responses (Table 7c, §9.1).
- Optional illustrative traces for an appendix are listed at the end of `results.md`
  ("Illustrative traces"; candidates in §5.5).

### Limitations

Use §12. Essential: small n and low power; one run and one seed (pass@1 only); one
general-purpose model with an unversioned alias; possible contamination of public benchmarks;
FPR_B upper bound with the manual review pending; the formal-vs-informal caveat of
`amc12b_2003_p6`; the shared pre-pass; the interrupted and resumed runs and the other deviations
of §11 (the plan requires reporting them as deviations).

### Ethics

- **Trust in AI-generated mathematics:** the central finding of Baseline B (§6) is that the
  model's confidence is not evidence of correctness; the system accepts a proof only when the
  Lean kernel does, and the report should say that LLM output should not be trusted without
  formal or human verification.
- **Data:** only public benchmarks (miniF2F, PutnamBench) were used, with no personal data; cite
  their sources and licences, and mention the contamination risk (§12).
- **Cost and resources:** Main experiments (dev + test, C1–C5): 7.36 USD. Including the
  exploratory Pro run: 9.30 USD in total; the project cap was raised from 9 to 12.5 USD for that
  run (see Deviations). A response cache avoided repeated calls; this keeps the work reproducible
  on a student budget.
- **Transparency and reproducibility:** the analysis plan was written before the `test` runs,
  every deviation is logged (§11), post hoc analyses are labelled, and each run records its
  models, seed and Lean/Mathlib versions.

## 14. Exploratory: a stronger model on the hard problems

*Post hoc (decided 2026-11-08, after all confirmatory runs), descriptive · LaTeX `tab:pro-hard`.*
This section changes no result of §4–§12. It is logged in `analysis_plan.md` ("Exploratory
analysis (post hoc, decided 2026-11-08)" and two Deviations entries of that date).

**Why.** On the hard tier the agents add nothing with DeepSeek-Flash: C1 and C5 solve exactly the
same 4 hard problems (§5.3). That suggests the hard problems are limited by the model, not by the
method. To look at that, C5 was run again on the 10 hard `test` problems with a stronger model,
DeepSeek-V4-Pro, in every role (profile `experiment_pro`); everything else (graph, prompts,
budgets, seed) was the frozen C5 configuration.

**What happened to the run.** The team stopped it after 5 of the 10 problems
(`20261108T130803Z_C5_experiment_pro_pro_hard`) and it was not resumed. Three outcomes are
**not failures of Pro** and are left out of its solved count:

- `imo_1978_p5`: **provider_error (connection drop, not a system failure)**. The 30th call of the
  problem (the Autoformalizer on subgoal 1) never reached DeepSeek: all 5 transport attempts got
  `APIConnectionError: Connection error.` with no HTTP response (15:34:50–15:37:28 UTC). The next
  problem connected normally.
- `imo_1984_p6`: **interrupted**, cut mid-problem by the stop (4 paid calls, no row).
- `imo_1964_p1_2`, `putnam_1964_b2`, `putnam_1995_b1`, `putnam_1965_b2`: **not attempted**.

Only the 4 problems that both runs finished are compared (`imo_1965_p2`, `imo_1979_p1`,
`imo_1987_p6`, `imo_1993_p5`).

### Table 12. Exploratory: C5-Flash vs C5-Pro on the hard problems (post hoc, descriptive, not in the analysis plan)

| Problem | Flash solved | Flash calls | Flash USD | Flash min | Pro outcome | Pro solved | Pro calls | Pro USD | Pro min |
|---|---|---|---|---|---|---|---|---|---|
| imo_1964_p1_2 | 1 | 15 | 0.0261 | 3.4 | not attempted | — | — | — | — |
| imo_1965_p2 | 1 | 14 | 0.0312 | 3.7 | failed | 0 | 30 | 0.3608 | 33.6 |
| imo_1978_p5 | 0 | 31 | 0.1347 | 39.6 | provider_error (connection drop, not a system failure) | — | 30 | 0.2938 | 40.7 |
| imo_1979_p1 | 0 | 28 | 0.1463 | 19.7 | failed | 0 | 24 | 0.2722 | 29.3 |
| imo_1984_p6 | 0 | 32 | 0.3366 | 27.6 | interrupted | — | — | — | — |
| imo_1987_p6 | 0 | 36 | 0.1793 | 21.4 | budget_exhausted | 0 | 40 | 0.4607 | 57.4 |
| imo_1993_p5 | 0 | 27 | 0.0790 | 10.6 | failed | 0 | 24 | 0.3807 | 53.6 |
| putnam_1964_b2 | 1 | 9 | 0.0160 | 2.3 | not attempted | — | — | — | — |
| putnam_1965_b2 | 1 | 17 | 0.0409 | 4.9 | not attempted | — | — | — | — |
| putnam_1995_b1 | 0 | 25 | 0.1268 | 17.5 | not attempted | — | — | — | — |
| Both finished (4) | 1/4 | 105 | 0.4357 | 55.5 |  | 0/4 | 118 | 1.4744 | 173.9 |
| All hard (10) | 4/10 | 234 | 1.1169 | 150.9 |  | — | — | — | — |

*Note.* Post hoc, exploratory (decided 2026-11-08, after all confirmatory runs): descriptive only, no tests, no intervals; it changes no main result. C5-Flash = the kept C5 rows of the main test result; C5-Pro = run 20261108T130803Z_C5_experiment_pro_pro_hard (profile experiment_pro: deepseek-v4-pro in every role, everything else as frozen), stopped by the team and not resumed. Cost is nominal USD (cache hits priced as fresh calls). provider_error, interrupted and not attempted do not count as failures of Pro and are left out of its solved count; only problems both runs finished are compared. imo_1984_p6 was interrupted after 4 paid calls (0.1725 USD real, 18.1 min) and left no row. Real cost of the Pro run: 1.9407 USD (151 paid calls, all in the ledger); project ledger total: 9.3003 USD.

In plain words: on the 4 problems both runs finished, Pro solved none and Flash solved one
(`imo_1965_p2`, which Pro failed after 30 calls). Pro spent about 3.4 times as much (1.47 vs
0.44 USD nominal) and took about 3.1 times as long (174 vs 56 min) on those 4. This does **not**
show that a stronger model fails to help: only 4 of the 10 problems were compared, each model
ran once, and 3 of the 4 hard problems Flash solved were never attempted by Pro. The question of
whether the hard tier is limited by the model stays open; the report can mention the run as an
exploratory, incomplete check and should not draw a conclusion from it.

**Cost.** The Pro run cost 1.9407 USD real (151 paid calls, all off-peak). With it, the project
ledger stands at 9.3003 USD; for this run the caps were raised to 12.5 USD in total and 5 USD per
run (Deviations, 2026-11-08).