File size: 51,997 Bytes
dfe1826 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 | # What was tested and what came out
The single, complete results document of the project: dataset, configurations, every table of
the analysis, the SR(k) figure, the deviations from the analysis plan, the fidelity review, the
limitations, and a guide to what goes in each section of the report. It is written so that the
report can be drafted from this file alone. For how the system works, see
[how_it_works.md](how_it_works.md). The pre-registered analysis plan is
[analysis_plan.md](analysis_plan.md) (written before any `test` result was seen); the full
statistical tables are in `docs/results/`.
**Where the numbers come from.** Every table below is copied verbatim from
`docs/results/results.md`, which `docs/results/make_tables.py` generates from `runs/`
(read-only). Nothing here is recomputed. `results.md` stays the generated source; this file is the
version to read. After `make_tables.py` is re-run, `python docs/results/check_explained.py`
checks that every table and every number here still matches `results.md`. The LaTeX version of
each table is in `docs/results/tables.tex`, under the label given with each table.
**Status labels used below.**
- **Planned (confirmatory)**: a test fixed in `analysis_plan.md` before `test` was run; the only
results that support claims.
- **Planned (exploratory)**: listed in the plan as exploratory: uncorrected, no claims.
- **Planned (descriptive)**: required by the plan, without tests.
- **Supplementary**: requested by the team after the plan, descriptive only.
- **Post hoc**: a look decided after seeing the results; descriptive only, not in the plan.
- **Integrity check**: describes the data, not the method.
Rates are in %, intervals are 95 %.
## 1. The dataset
`data/eval.jsonl` holds **45 problems**, each with its informal text and its Lean statement:
| | easy | medium | hard | total |
|---|---|---|---|---|
| `dev` (for tuning) | 5 | 5 | 5 | 15 |
| `test` (final runs only) | 10 | 10 | 10 | 30 |
- **Sources:** 40 from miniF2F (the Lean 4 port used by DeepSeek-Prover-V1.5) and 5 from
PutnamBench (3 of them in `test`).
- **Areas in `test`:** algebra 13, number theory 13, combinatorics 4 (no easy combinatorics
problem: that cell of the 3 × 3 grid is empty).
- Every statement was checked to compile and reviewed by hand for fidelity to the informal
text (`data/curation/`; see §10). Prompts and settings were tuned only on `dev`; `test` was
used once, for the final runs.
## 2. The five configurations
All five run on the same code and graph; only flags in `config/ablations/` change.
| | Name | In one line |
|---|---|---|
| **C1** | Baseline A | One LLM asked directly for a whole Lean proof; up to 18 independent tries (Lean's errors are not fed back). No agents. |
| **C2** | Baseline B | One LLM solves the problem in natural language and judges its own answer. No Lean. |
| **C3** | No RAG | Full system without the lemma search. |
| **C4** | No Critic | Full system, but one attempt per subgoal and no replanning. |
| **C5** | Full system | All agents, retries and replanning, up to 40 calls per problem. |
All runs used the same model for every role (DeepSeek `deepseek-flash`, a general-purpose
model, not a specialised prover), the same seed (2026), Lean `v4.36.0-rc1` and Mathlib
`625d891`. The exact settings of each run are in its `runs/<run>/run_config.json`.
## 3. Runs and data integrity
### 3.1 Runs used
*Integrity check · LaTeX `tab:runs`.* Which run folders feed the analysis and their recorded
configuration. C5 and C1 were interrupted and resumed (§11), so each has more than one folder;
their pieces are merged into one row per problem. "running" means the run was killed before it
could write its final status; "dirty (plan file only)" is a recording artefact explained in §11.
| Run | Config | Profile | Seed | Split | Commit | Tree | Degraded | Run status | vs first run |
|---|---|---|---|---|---|---|---|---|---|
| 20261103T174646Z_C5_experiment | C5 | experiment | 2026 | test | ad70ec2 | clean | False | running | same |
| 20261103T212359Z_C5_experiment_resume | C5 | experiment | 2026 | test | ad70ec2 | dirty (plan file only) | False | budget_stop | same |
| 20261103T225140Z_C5_experiment_resume2 | C5 | experiment | 2026 | test | 93bc701 | dirty (plan file only) | False | finished | same |
| 20261103T235556Z_C4_experiment | C4 | experiment | 2026 | test | ea4db4b | dirty (plan file only) | False | finished | same |
| 20261104T194056Z_C3_experiment | C3 | experiment | 2026 | test | 6c61d69 | clean | False | finished | same |
| 20261104T232056Z_C2_experiment_check | C2 | experiment | 2026 | test | 6c61d69 | clean | False | finished | same |
| 20261105T131443Z_C1_experiment | C1 | experiment | 2026 | test | 5ed7507 | clean | False | running | same |
| 20261107T200233Z_C1_experiment_resume | C1 | experiment | 2026 | test | 2c8eae5 | dirty (plan file only) | False | finished | same |
### 3.2 Row checks
*Integrity check · LaTeX `tab:checks`.* After merging, every configuration has exactly 30 rows
(one per `test` problem), and every row's split, configuration, area and difficulty match the
dataset.
| Config | Rows | Status counts | lean_down | split/config mismatch | area/difficulty mismatch |
|---|---|---|---|---|---|
| C1 | 30 | budget_exhausted: 7, failed: 3, solved: 20 | 0 | 0 | 0 |
| C2 | 30 | claimed_solved: 30 | 0 | 0 | 0 |
| C3 | 30 | budget_exhausted: 3, failed: 7, solved: 20 | 0 | 0 | 0 |
| C4 | 30 | failed: 16, solved: 14 | 0 | 0 | 0 |
| C5 | 30 | failed: 6, solved: 24 | 0 | 0 | 0 |
### 3.3 Infrastructure and error statuses
*Planned (plan §5 and §7 item 7) · LaTeX `tab:infra`.* No row ended because of the provider,
the spend cap, a bug, or Lean being unavailable, so the plan's re-run rule and sensitivity
analysis were never needed. Every failure in this document is a failure of the method itself.
| Config | provider_error | budget_stop | error | node_error | invalid_statement | lean_down |
|---|---|---|---|---|---|---|
| C1 | 0 | 0 | 0 | 0 | 0 | 0 |
| C2 | 0 | 0 | 0 | 0 | 0 | 0 |
| C3 | 0 | 0 | 0 | 0 | 0 | 0 |
| C4 | 0 | 0 | 0 | 0 | 0 | 0 |
| C5 | 0 | 0 | 0 | 0 | 0 | 0 |
## 4. Main result: verified success
### 4.1 Success rate (SR)
*Planned (primary outcome, plan §2 and §7 item 1) · LaTeX `tab:sr`.* A problem counts as solved
only if Lean's kernel accepts the final proof. The full system C5 solved 24/30; C1 and C3 20/30;
C4 14/30. C2 never uses Lean, so it has no SR: it appears through how often it claimed success
and how often that claim could not be confirmed (FPR_B, §6). The intervals are wide and overlap:
that is what 30 problems allow.
| Config | Solved | SR (%) | Wilson 95 % CI (%) | C2 claimed rate (%) [Wilson] | FPR_B (%) [Wilson] |
|---|---|---|---|---|---|
| C1 | 20/30 | 66.7 | [48.8, 80.8] | — | — |
| C3 | 20/30 | 66.7 | [48.8, 80.8] | — | — |
| C4 | 14/30 | 46.7 | [30.2, 63.9] | — | — |
| C5 | 24/30 | 80.0 | [62.7, 90.5] | — | — |
| C2 (Baseline B) | — | — | — | 30/30 = 100.0 [88.6, 100.0] | 11/30 = 36.7 [21.9, 54.5] |
*Note.* C2 is not verified, so it has no SR; it enters only through its claimed rate B⁺/n and FPR_B (plan §7.1). Of the 30 claimed proofs the formal check verified 19 and did not verify 11; 0 left out of FPR_B.
### 4.2 The three planned comparisons (H1–H3)
*Planned (confirmatory, plan §3) · LaTeX `tab:mcnemar`.* Exact McNemar test on the same 30
problems, Bonferroni correction for 3 tests (a comparison is significant only if its adjusted p
is below 0.05). *b* = problems only C5 solved, *c* = problems only the other configuration
solved. ΔSR is the difference in success rate in percentage points, with a paired bootstrap
interval.
| ID | Comparison | b | c | ΔSR (pp) | Paired bootstrap 95 % CI (pp) | Exact p | Bonferroni p | Reject at 0.05/3 |
|---|---|---|---|---|---|---|---|---|
| H1 | C5 vs C1 | 4 | 0 | +13.3 | [+3.3, +26.7] | 0.1250 | 0.3750 | no |
| H2 | C5 vs C3 | 5 | 1 | +13.3 | [+0.0, +30.0] | 0.2188 | 0.6562 | no |
| H3 | C5 vs C4 | 10 | 0 | +33.3 | [+16.7, +50.0] | 0.0020 | 0.0059 | yes |
*Note.* n = 30, so power is low: a non-significant result is no evidence of a difference, not evidence of no difference. H2 is Δ_RAG = SR_C5 − SR_C3; H3 is the Critic contribution SR_C5 − SR_C4.
In plain words:
- **H3: the Critic (retry and replan) makes a significant difference.** C5 solved 10 problems
that C4 did not, and C4 solved none that C5 missed (+33.3 pp, paired bootstrap interval
[+16.7, +50.0], Bonferroni p 0.0059).
- **H1 and H2 point in C5's favour but are not significant.** C5 vs C1: 4 problems to 0;
C5 vs C3: 5 to 1; both +13.3 pp. With 30 problems a difference of 4 or 5 problems is
compatible with chance. "Not significant" here means *no evidence of a difference*, not
evidence that there is none: the test has little power at this sample size.
### 4.3 Exploratory comparisons
*Planned (exploratory, plan §3): uncorrected, no claims · LaTeX `tab:mcnemar-expl`.* The other
pairs of verified configurations. C1 and C3 solve the same number of problems but not the same
problems (3 each way).
| ID | Comparison | b | c | ΔSR (pp) | Exact p (uncorrected) |
|---|---|---|---|---|---|
| E1 | C3 vs C4 | 9 | 3 | +20.0 | 0.1460 |
| E2 | C1 vs C3 | 3 | 3 | +0.0 | 1.0000 |
| E3 | C1 vs C4 | 8 | 2 | +20.0 | 0.1094 |
## 5. Results by difficulty and area
### 5.1 Solved per cell of the 3 × 3 grid
*Planned (descriptive, plan §4.1) · LaTeX `tab:cells`.* How many problems each configuration
solved in each (difficulty, area) cell. Cells have at most 5 problems, so there are no tests
and no intervals. For C2 the column counts *claimed* solutions, not verified ones.
| Difficulty | Area | n | C1 | C3 | C4 | C5 | C2 (claimed) |
|---|---|---|---|---|---|---|---|
| easy | Algebra | 5 | 5/5 | 5/5 | 4/5 | 5/5 | 5/5 |
| easy | Number theory | 5 | 5/5 | 4/5 | 5/5 | 5/5 | 5/5 |
| easy | Combinatorics | 0 | — | — | — | — | — |
| medium | Algebra | 5 | 4/5 | 5/5 | 1/5 | 5/5 | 5/5 |
| medium | Number theory | 4 | 2/4 | 2/4 | 2/4 | 4/4 | 4/4 |
| medium | Combinatorics | 1 | 0/1 | 0/1 | 1/1 | 1/1 | 1/1 |
| hard | Algebra | 3 | 1/3 | 2/3 | 0/3 | 1/3 | 3/3 |
| hard | Number theory | 4 | 1/4 | 1/4 | 0/4 | 1/4 | 4/4 |
| hard | Combinatorics | 3 | 2/3 | 1/3 | 1/3 | 2/3 | 3/3 |
### 5.2 Success rate by difficulty and by area
*Planned (descriptive, plan §4.1) · LaTeX `tab:margins` (wide table).* The margins of the grid:
first line solved/n, second line SR % with its Wilson interval. Every verified configuration
solves almost all easy problems and 4/10 or fewer hard ones; C5's advantage is on medium
problems (10/10). C2 claims every problem, at every difficulty.
| Config | easy (n = 10) | medium (n = 10) | hard (n = 10) | Algebra (n = 13) | Number theory (n = 13) | Combinatorics (n = 4) |
|---|---|---|---|---|---|---|
| C1 | 10/10 | 6/10 | 4/10 | 10/13 | 8/13 | 2/4 |
| | 100.0 [72.2, 100.0] | 60.0 [31.3, 83.2] | 40.0 [16.8, 68.7] | 76.9 [49.7, 91.8] | 61.5 [35.5, 82.3] | 50.0 [15.0, 85.0] |
| C3 | 9/10 | 7/10 | 4/10 | 12/13 | 7/13 | 1/4 |
| | 90.0 [59.6, 98.2] | 70.0 [39.7, 89.2] | 40.0 [16.8, 68.7] | 92.3 [66.7, 98.6] | 53.8 [29.1, 76.8] | 25.0 [4.6, 69.9] |
| C4 | 9/10 | 4/10 | 1/10 | 5/13 | 7/13 | 2/4 |
| | 90.0 [59.6, 98.2] | 40.0 [16.8, 68.7] | 10.0 [1.8, 40.4] | 38.5 [17.7, 64.5] | 53.8 [29.1, 76.8] | 50.0 [15.0, 85.0] |
| C5 | 10/10 | 10/10 | 4/10 | 11/13 | 10/13 | 3/4 |
| | 100.0 [72.2, 100.0] | 100.0 [72.2, 100.0] | 40.0 [16.8, 68.7] | 84.6 [57.8, 95.7] | 76.9 [49.7, 91.8] | 75.0 [30.1, 95.4] |
| C2 (claimed) | 10/10 | 10/10 | 10/10 | 13/13 | 13/13 | 4/4 |
| | 100.0 [72.2, 100.0] | 100.0 [72.2, 100.0] | 100.0 [72.2, 100.0] | 100.0 [77.2, 100.0] | 100.0 [77.2, 100.0] | 100.0 [51.0, 100.0] |
### 5.3 The 10 hard problems
*Post hoc (not in the analysis plan), descriptive · LaTeX `tab:hard`.* One row per hard problem,
1 = verified solved.
| Problem | Area | C1 | C3 | C4 | C5 | Solved by |
|---|---|---|---|---|---|---|
| imo_1964_p1_2 | Number theory | 1 | 1 | 0 | 1 | 3 |
| imo_1965_p2 | Algebra | 1 | 1 | 0 | 1 | 3 |
| imo_1978_p5 | Algebra | 0 | 0 | 0 | 0 | 0 |
| imo_1979_p1 | Number theory | 0 | 0 | 0 | 0 | 0 |
| imo_1984_p6 | Number theory | 0 | 0 | 0 | 0 | 0 |
| imo_1987_p6 | Number theory | 0 | 0 | 0 | 0 | 0 |
| imo_1993_p5 | Algebra | 0 | 1 | 0 | 0 | 1 |
| putnam_1964_b2 | Combinatorics | 1 | 0 | 1 | 1 | 3 |
| putnam_1965_b2 | Combinatorics | 1 | 1 | 0 | 1 | 3 |
| putnam_1995_b1 | Combinatorics | 0 | 0 | 0 | 0 | 0 |
| Solved | | 4/10 | 4/10 | 1/10 | 4/10 | |
*Note.* Post hoc, descriptive (not in the analysis plan): no tests, no intervals. 1 = verified solved. C1 and C5 solve the same hard problems (4 and 4); 5 hard problems are solved by no configuration (imo_1978_p5, imo_1979_p1, imo_1984_p6, imo_1987_p6, putnam_1995_b1); C3 and C5 solve 4 and 4 hard problems, but only C3 solves imo_1993_p5 and only C5 solves putnam_1964_b2.
Three things stand out. C1 and C5 solve exactly the same 4 hard problems (imo_1964_p1_2,
imo_1965_p2, putnam_1964_b2, putnam_1965_b2), so on the hard tier the whole C5 gain over C1
disappears: all 4 problems that C5 solves and C1 does not are medium (§5.5). Five hard problems
(imo_1978_p5, imo_1979_p1, imo_1984_p6, imo_1987_p6, putnam_1995_b1) are solved by no
configuration. C3 and C5 also solve 4 each but differ by one problem in each direction: only
C3 solves imo_1993_p5, only C5 solves putnam_1964_b2. Since C5 has everything C3 has plus
retrieval, a hard problem that C3 solves and C5 does not shows how much a single run can vary
from one to the next; each configuration was run once, so differences of one problem should
not be read as an effect.
### 5.4 Per-problem outcomes
*Data listing (the source of the counts above) · LaTeX `tab:per-problem` (wide table).* One row
per `test` problem: 1 = verified solved; the last column gives C2's claim and whether the Lean
check confirmed it.
| Problem | Difficulty | Area | C1 | C3 | C4 | C5 | C2 (claim, check) |
|---|---|---|---|---|---|---|---|
| algebra_absapbon1pabsapbleqsumabsaon1pabsa | medium | Algebra | 0 | 1 | 0 | 1 | claimed, verified |
| amc12_2000_p20 | medium | Algebra | 1 | 1 | 0 | 1 | claimed, verified |
| amc12_2001_p9 | medium | Algebra | 1 | 1 | 0 | 1 | claimed, verified |
| amc12a_2008_p15 | medium | Number theory | 0 | 0 | 0 | 1 | claimed, not_verified |
| amc12a_2008_p25 | medium | Algebra | 1 | 1 | 0 | 1 | claimed, verified |
| amc12b_2002_p3 | medium | Number theory | 0 | 1 | 1 | 1 | claimed, not_verified |
| amc12b_2003_p6 | medium | Algebra | 1 | 1 | 1 | 1 | claimed, verified |
| imo_1964_p1_2 | hard | Number theory | 1 | 1 | 0 | 1 | claimed, not_verified |
| imo_1965_p2 | hard | Algebra | 1 | 1 | 0 | 1 | claimed, not_verified |
| imo_1978_p5 | hard | Algebra | 0 | 0 | 0 | 0 | claimed, not_verified |
| imo_1979_p1 | hard | Number theory | 0 | 0 | 0 | 0 | claimed, not_verified |
| imo_1984_p6 | hard | Number theory | 0 | 0 | 0 | 0 | claimed, not_verified |
| imo_1987_p6 | hard | Number theory | 0 | 0 | 0 | 0 | claimed, not_verified |
| imo_1993_p5 | hard | Algebra | 0 | 1 | 0 | 0 | claimed, not_verified |
| mathd_algebra_160 | easy | Algebra | 1 | 1 | 1 | 1 | claimed, verified |
| mathd_algebra_206 | easy | Algebra | 1 | 1 | 0 | 1 | claimed, verified |
| mathd_algebra_234 | easy | Algebra | 1 | 1 | 1 | 1 | claimed, verified |
| mathd_algebra_288 | easy | Algebra | 1 | 1 | 1 | 1 | claimed, verified |
| mathd_algebra_455 | easy | Algebra | 1 | 1 | 1 | 1 | claimed, verified |
| mathd_numbertheory_321 | easy | Number theory | 1 | 0 | 1 | 1 | claimed, verified |
| mathd_numbertheory_328 | easy | Number theory | 1 | 1 | 1 | 1 | claimed, verified |
| mathd_numbertheory_345 | easy | Number theory | 1 | 1 | 1 | 1 | claimed, verified |
| mathd_numbertheory_430 | easy | Number theory | 1 | 1 | 1 | 1 | claimed, verified |
| mathd_numbertheory_64 | easy | Number theory | 1 | 1 | 1 | 1 | claimed, verified |
| numbertheory_3pow2pownm1mod2pownp3eq2pownp2 | medium | Number theory | 1 | 0 | 0 | 1 | claimed, verified |
| numbertheory_notequiv2i2jasqbsqdiv8 | medium | Number theory | 1 | 1 | 1 | 1 | claimed, verified |
| numbertheory_sumkmulnckeqnmul2pownm1 | medium | Combinatorics | 0 | 0 | 1 | 1 | claimed, verified |
| putnam_1964_b2 | hard | Combinatorics | 1 | 0 | 1 | 1 | claimed, verified |
| putnam_1965_b2 | hard | Combinatorics | 1 | 1 | 0 | 1 | claimed, not_verified |
| putnam_1995_b1 | hard | Combinatorics | 0 | 0 | 0 | 0 | claimed, not_verified |
### 5.5 Problems solved by one configuration and not the other
*Descriptive (discordant pairs of the comparisons; candidates for example traces in the
report).* For each pair, the problems behind *b* and *c* of Tables 2 and 2b, with what C5 did on
them (status, LLM calls, replans, and the largest number of attempts on any subgoal of the
accepted plan).
- C5 solved, C4 not (Critic loop): algebra_absapbon1pabsapbleqsumabsaon1pabsa (C5: solved, 24 calls, replans 1, max attempts 1); amc12_2000_p20 (C5: solved, 23 calls, replans 0, max attempts 3); amc12_2001_p9 (C5: solved, 7 calls, replans 0, max attempts 2); amc12a_2008_p15 (C5: solved, 16 calls, replans 0, max attempts 3); amc12a_2008_p25 (C5: solved, 23 calls, replans 0, max attempts 2); imo_1964_p1_2 (C5: solved, 15 calls, replans 0, max attempts 3); imo_1965_p2 (C5: solved, 14 calls, replans 0, max attempts 2); mathd_algebra_206 (C5: solved, 28 calls, replans 0, max attempts 2); numbertheory_3pow2pownm1mod2pownp3eq2pownp2 (C5: solved, 15 calls, replans 0, max attempts 3); putnam_1965_b2 (C5: solved, 17 calls, replans 0, max attempts 3)
- C4 solved, C5 not: —
- C5 solved, C3 not (RAG): amc12a_2008_p15 (C5: solved, 16 calls, replans 0, max attempts 3); mathd_numbertheory_321 (C5: solved, 1 calls, replans 0, max attempts 0); numbertheory_3pow2pownm1mod2pownp3eq2pownp2 (C5: solved, 15 calls, replans 0, max attempts 3); numbertheory_sumkmulnckeqnmul2pownm1 (C5: solved, 5 calls, replans 0, max attempts 1); putnam_1964_b2 (C5: solved, 9 calls, replans 0, max attempts 1)
- C3 solved, C5 not: imo_1993_p5 (C5: failed, 27 calls, replans 1, max attempts -)
- C5 solved, C1 not: algebra_absapbon1pabsapbleqsumabsaon1pabsa (C5: solved, 24 calls, replans 1, max attempts 1); amc12a_2008_p15 (C5: solved, 16 calls, replans 0, max attempts 3); amc12b_2002_p3 (C5: solved, 9 calls, replans 0, max attempts 1); numbertheory_sumkmulnckeqnmul2pownm1 (C5: solved, 5 calls, replans 0, max attempts 1)
- C1 solved, C5 not: —
- C2 claimed, check did not verify: amc12a_2008_p15 (C5: solved, 16 calls, replans 0, max attempts 3); amc12b_2002_p3 (C5: solved, 9 calls, replans 0, max attempts 1); imo_1964_p1_2 (C5: solved, 15 calls, replans 0, max attempts 3); imo_1965_p2 (C5: solved, 14 calls, replans 0, max attempts 2); imo_1978_p5 (C5: failed, 31 calls, replans 1, max attempts -); imo_1979_p1 (C5: failed, 28 calls, replans 1, max attempts -); imo_1984_p6 (C5: failed, 32 calls, replans 1, max attempts -); imo_1987_p6 (C5: failed, 36 calls, replans 1, max attempts -); imo_1993_p5 (C5: failed, 27 calls, replans 1, max attempts -); putnam_1965_b2 (C5: solved, 17 calls, replans 0, max attempts 3); putnam_1995_b1 (C5: failed, 25 calls, replans 1, max attempts -)
## 6. Baseline B (C2): what the LLM claimed vs what Lean confirmed
*Planned (plan §4.2, Eq. 16) · LaTeX `tab:fprb`.* C2 never uses Lean, so it cannot "solve"
anything in the verified sense. Instead it reports whether it believes its own proof is
correct. To see how often that belief is wrong, each claimed proof was handed to the Lean
pipeline (Autoformalizer + Verifier, up to 3 attempts, same lemma search as C5;
`--check` in `scripts/run_baseline_b.py`).
| Quantity | Value |
|---|---|
| Problems | 30 |
| Claimed (B⁺) | 30 |
| Not claimed | 0 |
| Check status among B⁺ | not_verified: 11, verified: 19 |
| Verified (Acc on the assembled theorem) | 19 |
| Not verified (budget_exhausted included) | 11 |
| Left out of FPR_B (provider_error, budget_stop, error, lean_down) | 0 |
| FPR_B = not verified / (verified + not verified) | 11/30 = 36.7 % [21.9, 54.5] |
| B⁺ problems for the manual review (§4.2) | 10 drawn with random.Random(2026).sample(sorted ids, 10) (listed below) |
*Note.* Manual review set (B⁺ not verified): amc12b_2002_p3, imo_1964_p1_2, imo_1965_p2, imo_1978_p5, imo_1979_p1, imo_1984_p6, imo_1987_p6, imo_1993_p5, putnam_1965_b2, putnam_1995_b1.
By difficulty (counted from Table 11):
| | easy | medium | hard | total |
|---|---|---|---|---|
| Claimed correct by the LLM | 10 | 10 | 10 | **30** |
| Confirmed by Lean | 10 | 8 | 1 | **19** |
C2 claimed to have solved every problem. Lean confirmed 19; for the other 11 the claim could
not be turned into an accepted proof. That is FPR_B = 11/30 = 36.7 % [21.9, 54.5]. Almost all
of the gap is on hard problems, where the model was sure of 10 answers and only 1 could be
confirmed. This is the main motivation of the project: a confident LLM is not a reliable judge
of its own mathematics.
FPR_B is an **upper bound** on how often C2's self-assessment is wrong: "not confirmed" means
"the pipeline could not formalise it within the budget", which does not always mean the
informal argument was wrong. Two facts bear on this:
- For 5 of the 11 (amc12a_2008_p15, amc12b_2002_p3, imo_1964_p1_2, imo_1965_p2,
putnam_1965_b2), C5 has a Lean-verified proof of the same statement, so at least the claimed
result is true there (whether C2's own argument was sound is a separate question).
- The plan asks for a manual review of 10 of the 11 (drawn with seed 2026; listed in the note
of Table 1b) to separate wrong arguments from formalisation failures. The review sheet is
`docs/results/fprb_review.md` (reviewer: Andrés). **Its verdicts are still pending.**
## 7. Efficiency and cost
### 7.1 Per solved problem
*Planned (plan §4.3, RNF-3) · LaTeX `tab:eff-solved`.* Total LLM calls, tokens, cost and
wall-clock time of a configuration divided by the problems it solved (failed problems' cost is
included in the numerator). Bootstrap intervals in brackets. **Nominal** cost prices cache hits
as fresh calls; it is the figure the plan compares, because the cache makes later runs cheaper
for reasons unrelated to the method. **Real** cost is what the provider charged.
| Config | Solved | Calls / solved | k tokens / solved | Nominal USD / solved | Real USD / solved | Wall min / solved |
|---|---|---|---|---|---|---|
| C1 | 20 | 12.85 [7.19, 22.60] | 219.7 [114.6, 394.7] | 0.1285 [0.0668, 0.2313] | 0.1235 [0.0643, 0.2218] | 17.7 [9.2, 32.0] |
| C2 (per claimed) | 30 | 2.17 [2.00, 2.40] | 5.3 [2.8, 8.6] | 0.0027 [0.0013, 0.0047] | 0.0027 [0.0013, 0.0047] | 0.4 [0.2, 0.7] |
| C3 | 20 | 26.50 [16.15, 43.67] | 150.4 [79.3, 274.6] | 0.0675 [0.0335, 0.1279] | 0.0675 [0.0335, 0.1278] | 9.2 [4.6, 17.3] |
| C4 | 14 | 9.43 [5.72, 16.67] | 70.7 [38.2, 137.8] | 0.0278 [0.0135, 0.0571] | 0.0000 [0.0000, 0.0000] | 0.2 [0.1, 0.3] |
| C5 | 24 | 16.92 [10.63, 26.21] | 148.0 [74.4, 266.4] | 0.0566 [0.0245, 0.1097] | 0.0490 [0.0236, 0.0897] | 7.6 [3.3, 14.6] |
*Note.* C2 has no verified solutions: its ratio uses claimed solutions and excludes the claim check (calls, tokens, cost and wall time up to claim_check_start).
C5 needs fewer calls per solved problem than C3 (16.92 vs 26.50) and costs less per solved
problem than C1 (nominal 0.0566 vs 0.1285 USD); C1 also takes the most wall time per solved
problem (17.7 min).
### 7.2 Per problem (means)
*Planned (plan §4.3) · LaTeX `tab:eff-problem`.* The same quantities averaged over all 30
problems, solved or not.
| Config | Calls / problem | k tokens / problem | Nominal USD / problem | Real USD / problem | Wall min / problem |
|---|---|---|---|---|---|
| C1 | 8.57 [5.83, 11.33] | 146.4 [91.1, 203.5] | 0.0857 [0.0532, 0.1192] | 0.0823 [0.0505, 0.1152] | 11.8 [7.2, 16.5] |
| C2 | 2.17 [2.00, 2.40] | 5.3 [2.8, 8.6] | 0.0027 [0.0013, 0.0047] | 0.0027 [0.0013, 0.0047] | 0.4 [0.2, 0.7] |
| C3 | 17.67 [12.97, 22.37] | 100.3 [63.8, 140.8] | 0.0450 [0.0269, 0.0658] | 0.0450 [0.0269, 0.0658] | 6.1 [3.7, 9.0] |
| C4 | 4.40 [3.37, 5.43] | 33.0 [22.8, 44.4] | 0.0130 [0.0081, 0.0186] | 0.0000 [0.0000, 0.0000] | 0.1 [0.1, 0.1] |
| C5 | 13.53 [9.57, 17.60] | 118.4 [67.9, 178.7] | 0.0453 [0.0225, 0.0737] | 0.0392 [0.0218, 0.0588] | 6.1 [3.0, 9.7] |
### 7.3 Totals, cache and real cost
*Planned (plan §4.3, with the cache and spend-ledger columns added for transparency) · LaTeX
`tab:totals` (wide table).* Totals over the kept rows. The last column is the real spend of all
the configuration's runs from `data/spend_ledger.jsonl`, which also includes the attempts
discarded when a run was interrupted (C1: 0.0985 USD; C5: 0.1815 USD) and, for C2, the claim
check.
| Config | LLM calls | Cached calls | Cached (%) | Tokens (M) | Nominal USD (rows) | Real USD (rows) | Real USD (ledger, all runs) |
|---|---|---|---|---|---|---|---|
| C1 | 257 | 47 | 18.3 | 4.393 | 2.5700 | 2.4695 | 2.5679 |
| C2 | 65 | 0 | 0.0 | 0.158 | 0.0814 | 0.0814 | 0.7083 |
| C3 | 530 | 18 | 3.4 | 3.009 | 1.3508 | 1.3500 | 1.3500 |
| C4 | 132 | 132 | 100.0 | 0.990 | 0.3887 | 0.0000 | 0.0000 |
| C5 | 406 | 34 | 8.4 | 3.551 | 1.3590 | 1.1770 | 1.3585 |
*Note.* C2 claim check (not in the C2 figures of the other columns): 77 calls (11 cached), 0.6274 nominal, 0.6269 real.
**RNF-3 (C̄_solved ≤ 20 for C5):** C̄_solved = 406/24 = 16.92 calls per solved problem: **met**.
How to read it:
- **C4 cost 0 USD** because every request it made had already been made, word for word, by C5
(C4 is C5 with retries removed, so its first attempts are identical); the answers came from
the cache (100.0 % cached calls). For the same reason C4's wall times measure only Lean.
- **RNF-3** (at most 20 LLM calls per solved problem for C5) is **met**: 406/24 = 16.92.
- C1 has a budget of 18 calls per problem (the average C5 used on `dev`, by a rule fixed before
`test`); the full system has 40.
- **Project spend** (ledger, all runs, `dev` included): main experiments 7.3596 USD. Including the
exploratory Pro run (1.9407 USD, §14): 9.3003 USD in total; the project cap was raised from 9 to
12.5 USD for that run (§11).
### 7.4 Medians and quartiles
*Supplementary (requested by the team, not in the plan) · LaTeX `tab:medians`.* Means hide that
the distributions are very skewed: most problems take few calls and a few take many.
| Config | Calls, median [Q1, Q3] | Wall min, median [Q1, Q3] |
|---|---|---|
| C1 | 4.5 [1.0, 18.0] | 2.7 [0.6, 24.2] |
| C2 | 2.0 [2.0, 2.0] | 0.1 [0.1, 0.3] |
| C3 | 16.5 [3.2, 28.0] | 3.7 [0.6, 6.7] |
| C4 | 5.0 [1.0, 5.8] | 0.1 [0.0, 0.1] |
| C5 | 11.5 [1.0, 23.8] | 2.4 [0.2, 5.2] |
### 7.5 Wall time by difficulty
*Supplementary (requested by the team, not in the plan) · LaTeX `tab:wall-diff` (wide table).*
Minutes per problem, mean / median [quartiles]. Times are dominated by waiting for the model, and
grow with difficulty. C4's times are not comparable (cache, see above). Many easy problems take
seconds in C3–C5 because they need no LLM-written subgoal proof at all (Table 8, §9.4).
| Config | easy (n = 10): mean / median [Q1, Q3] | medium (n = 10): mean / median [Q1, Q3] | hard (n = 10): mean / median [Q1, Q3] |
|---|---|---|---|
| C1 | 0.7 / 0.4 [0.1, 1.1] | 14.3 / 16.2 [1.8, 23.2] | 20.3 / 25.3 [7.0, 32.4] |
| C2 | 0.1 / 0.1 [0.1, 0.2] | 0.1 / 0.1 [0.1, 0.2] | 1.0 / 0.5 [0.2, 1.1] |
| C3 | 0.8 / 0.1 [0.0, 1.0] | 4.0 / 2.6 [1.2, 6.0] | 13.6 / 12.3 [6.6, 19.8] |
| C4 | 0.1 / 0.0 [0.0, 0.1] | 0.1 / 0.1 [0.0, 0.1] | 0.1 / 0.1 [0.1, 0.1] |
| C5 | 0.4 / 0.0 [0.0, 0.5] | 2.7 / 2.6 [0.7, 4.5] | 15.1 / 14.1 [4.0, 21.0] |
## 8. Refinement robustness: SR(k)
*Planned (plan §4.4, Eq. 14; figure: plan §7 item 5) · LaTeX `tab:srk`; figure
`docs/results/figures/srk_curve.pdf` (`.png` below).* SR_C5(k) counts a C5 problem as solved at
k if its accepted proof needed at most k attempts on every subgoal. It shows how much each extra
attempt (with the Critic's feedback) adds. The last two columns are subgoal-level diagnostics
(supplementary).
| k | SR_C5(k) [Wilson] (%) | Δ̂(k) = SR(k) − SR(k−1) (pp) | Indep. Δ(k) = p̂(1−p̂)^(k−1) (%) | Subgoals solved at attempt k | Solved at k / reached k |
|---|---|---|---|---|---|
| 1 | 15/30 = 50.0 [33.2, 66.8] | +20.0 | 40.9 | 18/44 = 40.9 | 18/44 = 40.9 |
| 2 | 19/30 = 63.3 [45.5, 78.1] | +13.3 | 24.2 | 8/44 = 18.2 | 8/26 = 30.8 |
| 3 | 24/30 = 80.0 [62.7, 90.5] | +16.7 | 14.3 | 5/44 = 11.4 | 5/18 = 27.8 |
*Note.* SR_C5(0) = 9/30 (accepted plan with no LLM subgoal attempt). p̂ = 18/44 = 40.9 %. k = 1 from C4 (no retries, no Critic advice): 14/30 = 46.7 % [30.2, 63.9]. The last two columns are subgoal-level diagnostics (supplementary).

**Figure: SR(k).** (a) SR_C5(k) for k = 0…3 with Wilson intervals: 9/30 solved with no
LLM-attempted subgoal, then 15/30, 19/30 and 24/30 as up to 1, 2 and 3 attempts per subgoal are
allowed. The square is C4 (one attempt, no retries, no Critic advice): 14/30 = 46.7 %, next to
SR_C5(1) = 15/30. (b) The observed gain of each extra attempt, Δ̂(k) (+20.0, +13.3, +16.7 pp),
against the gain expected if every attempt succeeded independently with the first-attempt rate
p̂ = 18/44 = 40.9 % (40.9, 24.2, 14.3). The independence curve falls quickly; the observed gain
does not: the third attempt still adds +16.7 pp, above the 14.3 the independence model predicts.
This fits the idea that retries guided by Lean's errors are not just new random draws. Two
cautions: the two curves are at different levels (the independence curve is per subgoal, Δ̂(k)
per problem; method note 23 in `results.md`), and with n = 30 each step is a handful of problems.
## 9. Errors and failure reasons
### 9.1 Error taxonomy (κ)
*Planned (plan §4.5, Eq. 11) · LaTeX `tab:kappa` (wide table).* Every failed attempt classified
by the deterministic classifier κ, count (%). Tactic failures and syntax errors dominate. No
attempt was classified as misformalization. For C1 the attempts were classified offline with
the same κ: 44 of its syntax errors are empty answers, which happen alongside 52 responses cut
at the token limit (method note 25 in `results.md`). For C2 the row is the claim-check loop.
| Config | Failed attempts | syntax | missing lemma | tactic failure | misformalization | timeout | planning |
|---|---|---|---|---|---|---|---|
| C1 | 186 | 60 (32.3) | 50 (26.9) | 68 (36.6) | 0 (0.0) | 8 (4.3) | 0 (0.0) |
| C2 (claim check) | 45 | 14 (31.1) | 8 (17.8) | 22 (48.9) | 0 (0.0) | 1 (2.2) | 0 (0.0) |
| C3 | 82 | 20 (24.4) | 17 (20.7) | 42 (51.2) | 0 (0.0) | 2 (2.4) | 1 (1.2) |
| C4 | 16 | 5 (31.2) | 4 (25.0) | 7 (43.8) | 0 (0.0) | 0 (0.0) | 0 (0.0) |
| C5 | 57 | 19 (33.3) | 18 (31.6) | 18 (31.6) | 0 (0.0) | 2 (3.5) | 0 (0.0) |
*Note.* C3–C5: classes recorded by the Critic. C1: attempts classified offline with policy.classify (first 3 Lean messages, 300 characters each, as stored; no k_max escalation). C2 has no verified attempts of its own: its row is the claim-check loop.
### 9.2 Error taxonomy per area
*Planned (plan §4.5) · LaTeX `tab:kappa-area` (wide table).* The same counts split by area.
Missing-lemma errors are frequent in algebra; tactic failures dominate number theory.
| Config | Area | Failed attempts | syntax | missing lemma | tactic failure | misformalization | timeout | planning |
|---|---|---|---|---|---|---|---|---|
| C1 | Algebra | 77 | 14 (18.2) | 40 (51.9) | 22 (28.6) | 0 (0.0) | 1 (1.3) | 0 (0.0) |
| C1 | Number theory | 83 | 27 (32.5) | 3 (3.6) | 46 (55.4) | 0 (0.0) | 7 (8.4) | 0 (0.0) |
| C1 | Combinatorics | 26 | 19 (73.1) | 7 (26.9) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) |
| C2 (claim check) | Algebra | 14 | 5 (35.7) | 4 (28.6) | 5 (35.7) | 0 (0.0) | 0 (0.0) | 0 (0.0) |
| C2 (claim check) | Number theory | 23 | 5 (21.7) | 4 (17.4) | 13 (56.5) | 0 (0.0) | 1 (4.3) | 0 (0.0) |
| C2 (claim check) | Combinatorics | 8 | 4 (50.0) | 0 (0.0) | 4 (50.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) |
| C3 | Algebra | 18 | 3 (16.7) | 9 (50.0) | 5 (27.8) | 0 (0.0) | 0 (0.0) | 1 (5.6) |
| C3 | Number theory | 43 | 13 (30.2) | 3 (7.0) | 26 (60.5) | 0 (0.0) | 1 (2.3) | 0 (0.0) |
| C3 | Combinatorics | 21 | 4 (19.0) | 5 (23.8) | 11 (52.4) | 0 (0.0) | 1 (4.8) | 0 (0.0) |
| C4 | Algebra | 8 | 1 (12.5) | 3 (37.5) | 4 (50.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) |
| C4 | Number theory | 6 | 2 (33.3) | 1 (16.7) | 3 (50.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) |
| C4 | Combinatorics | 2 | 2 (100.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) |
| C5 | Algebra | 25 | 7 (28.0) | 8 (32.0) | 10 (40.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) |
| C5 | Number theory | 24 | 8 (33.3) | 9 (37.5) | 6 (25.0) | 0 (0.0) | 1 (4.2) | 0 (0.0) |
| C5 | Combinatorics | 8 | 4 (50.0) | 1 (12.5) | 2 (25.0) | 0 (0.0) | 1 (12.5) | 0 (0.0) |
### 9.3 Final failure reason per problem
*Planned (plan §4.5) · LaTeX `tab:final-fail`.* For every problem a configuration did not
solve, its final status and the stage where it stopped (`verify` = a subgoal proof was rejected;
`formalize` = the Lean sketch could not be assembled). C1 failures are mostly running out of its
18-call budget.
| Config | Problems | status @ stage: count |
|---|---|---|
| C1 | 10 | budget_exhausted: 7; failed: 3 |
| C2 | 30 | claimed, check verified: 19; claimed, check not_verified: 11 |
| C3 | 10 | failed @ verify: 5; budget_exhausted @ formalize: 3; failed @ formalize: 2 |
| C4 | 16 | failed @ verify: 11; failed @ formalize: 5 |
| C5 | 6 | failed @ verify: 4; failed @ formalize: 2 |
### 9.4 Subgoals closed by fixed tactics vs the LLM
*Supplementary (not in the plan) · LaTeX `tab:prepass` (wide table).* Before asking the LLM, the
graph configurations try a fixed list of tactics on each subgoal (the pre-pass). This table shows
how many subgoals of the accepted plans each closed, and how many solved problems needed no
LLM-written subgoal proof at all (8–9 per configuration). This is a shared factor of C3–C5, not
something one of them has and the others lack.
| Config | Pre-pass (accepted plans) | LLM (accepted plans) | Pre-pass (all subgoal_solved) | Solved with no LLM subgoal | Solved |
|---|---|---|---|---|---|
| C3 | 7/32 = 21.9 | 25/32 = 78.1 | 8/39 = 20.5 | 8 | 20 |
| C4 | 9/18 = 50.0 | 9/18 = 50.0 | 11/21 = 52.4 | 9 | 14 |
| C5 | 11/39 = 28.2 | 28/39 = 71.8 | 11/42 = 26.2 | 9 | 24 |
## 10. Fidelity review
Lean checks the formal statement, not the original text, so the statements were reviewed by
hand twice.
**Before any run (2026-11-01, `data/curation/fidelity_review.yaml`).** Every drawn problem's
Lean statement was compared with its informal text. Two were discarded because the formal
statement changed the mathematics (`imo_1962_p2`, `imo_1974_p5`; replaced). Seven were kept
with a note: the formal version differs from the text in a way judged harmless (for example a
sequence already reduced to one equation, or a definition written differently).
**After the `test` runs (2026-11-07, team review, `post_test_review`).** As plan §6 requires,
every problem solved on `test` by some configuration that has a note was read again against the
informal text: four problems.
*Planned (plan §6) · LaTeX `tab:fidelity`.*
| Problem | Solved by | Manual review |
|---|---|---|
| amc12b_2003_p6 | C1, C3, C4, C5 | 2026-11-07: counts as solved; report note: The formalization proves that u 0 = +-2/sqrt 3 (a necessary condition), not that -2*sqrt(3)/3 is attained; this is enough to rule out the other options. |
| mathd_algebra_234 | C1, C3, C4, C5 | 2026-11-07: counts as solved; no note |
| mathd_algebra_455 | C1, C3, C4, C5 | 2026-11-07: counts as solved; no note |
| putnam_1965_b2 | C1, C3, C5 | 2026-11-07: counts as solved; no note |
In plain words: all four count as solved and no success rate changes, so the plan's "SR with
and without" never applies. One needs a caveat in the report: **`amc12b_2003_p6`** (solved by
C1, C3, C4, C5 and confirmed for C2) asks which option is a possible first term. The
formalisation proves that the first term must be ±2/√3 (a necessary condition), not that
−2√3/3 is actually attained; that is enough to rule out the other options. The remaining notes
do not apply: `imo_2006_p3` is in `dev`, and `imo_1987_p6` and `putnam_1995_b1` were solved by
no configuration.
## 11. Deviations from the analysis plan, in plain language
The analysis itself (metrics, tests, corrections) was not changed. What happened differently
from the plan was operational, and each item is logged with date and reason in
`analysis_plan.md` (Deviations):
1. **C5 was interrupted (2026-11-03).** The machine ran low on memory and the run was killed
after 23 of 30 problems. Those 23 rows are kept unchanged. The other 7 were run again with the
same frozen configuration, seed and cache, in two resume runs: the first one hit its spend cap
during its third problem (that incomplete row of `imo_1984_p6` is discarded), the second
finished the rest. The merge rule (which row comes from which run) was written down before
the analysis and the analysis script checks it.
2. **C4 ran before C3** (plan order: C5, C3, C4, C2, C1). For schedule and cost reasons only.
Every configuration still ran once on the same problems, so the paired comparisons are not
affected.
3. **The per-run spend cap was raised twice for the main runs** (to 2 USD before C3, then to
3.5 USD before C1), by team decision, so that those runs would not stop half-way. The cap only
decides whether a request is sent; it does not change prompts, models, call budgets or results.
For the exploratory Pro run (§14, after all main runs) both caps were raised again: the project
cap from 9 to 12.5 USD and the per-run cap from 3.5 to 5 USD.
4. **C1 was stopped by the team (2026-11-05)** to free the machine, after 23 of 30 problems. The
other 7 were completed on 2026-11-07 in a resume run with the same procedure as C5, and the
pieces were merged into one file (checked row by row).
5. **`dirty: true` in four `run_config.json` files** (found during the analysis). The run
configuration file was rewritten at the end of each run and read the git state at that moment;
the only changed file was `analysis_plan.md` itself, edited during the runs to log these
incidents. Every tree was clean at launch. The recording was fixed after the `test` runs.
6. **The exploratory Pro run was stopped by the team (2026-11-08)** after 5 of its 10 problems and
not resumed (§14). It is outside the analysis plan's results and changes none of them.
Written before any `test` run, as a clarification rather than a deviation: **how the C2 claim
check works** (plan §4.2): each claimed proof becomes a sketch for the Autoformalizer, with up to
3 attempts and Lean feedback, the same lemma search as C5, no Planner and no fixed tactics; its
calls are counted apart from C2's own.
**Where the plan was silent or had to be read literally** (full list: "Method notes" at the end
of `results.md`):
- C2 has no verified SR, so Table 1 reports its claimed rate and FPR_B instead, and C2 is in no
McNemar test. In the grid tables C2 shows claimed counts, labelled as such.
- The plan fixes the bootstrap (10 000 resamples, seed 2026) but not the random generator;
NumPy's default generator was used, the same for every comparison.
- The plan says "10 drawn with seed 2026" for the C2 manual review without saying how;
Python's `random.Random(2026).sample` on the sorted ids was used.
- SR(0) and the first-attempt rate p̂ of the SR(k) analysis are not defined in the plan; the
definitions used are in method notes 22 and 23.
- C1 attempts were classified by κ offline from the stored (shortened) Lean messages.
- Tables 4d, 5, 8 and 11b were added after the plan; they are descriptive and labelled.
- Disclosure: while checking configurations, before the analysis script existed, the analyst had
seen the first C5 `test` row and run summaries of C5 and C4. After the first `test` run the
script changed only in how C2's wall time is cut, an added note on truncated responses, and
LaTeX layout. No test, comparison, correction or metric changed.
## 12. Limitations
- **Small sample.** 30 test problems (10 per difficulty, only 4 in combinatorics). Confidence
intervals are wide, H1 and H2 have little power, and per-area numbers are descriptive only.
- **One run per configuration.** Each configuration ran once with one seed, so pass@k is
estimated only for k = 1 (the SR). LLM output varies between runs; differences of one or two
problems (for example C3 vs C5 on hard problems, §5.3) are within that variation.
- **One general-purpose model.** Every role used the same general model (`deepseek-flash`), not
a model trained for Lean. Results may differ with other models. `deepseek-flash` is an
alias with no versioned id; the run configs record what was used.
- **Possible contamination.** miniF2F and the competition problems are public, and Lean proofs
of many miniF2F problems are online. The model may have seen some of them in training, which
would make every configuration look better than it would on new problems. This affects all
configurations, but not necessarily equally (C1 asks for a Lean proof directly, which is
where a memorised proof might help most).
- **Formal vs informal statement.** Lean checks the formal statement, not the original text.
All statements were reviewed (§10); `amc12b_2003_p6` needs the caveat above.
- **FPR_B is an upper bound** and its manual review is still pending (§6).
- **Shared tactic pre-pass.** 8–9 solved problems per graph configuration need no LLM-written
subgoal proof (Table 8), which narrows the room in which RAG and the Critic can differ.
- **Truncated responses.** Some LLM answers hit the token limit (C1 most often: 52 responses in
9 rows); in C1 these show up as empty attempts counted against its budget.
- **Interrupted runs.** C5 and C1 were stopped by operational incidents and resumed with the
same frozen configuration (§11).
- **C4's cost and times** come from C5's cache and are not comparable at real cost; nominal cost
is the comparable figure.
## 13. For the report
What to use in each section of the report, with the LaTeX label of each table in
`docs/results/tables.tex` (`\input` the file or copy the tables; it needs only `booktabs`;
tables marked "wide" are `table*`). The figure is `docs/results/figures/srk_curve.pdf`.
Keep the status label of each table in its caption or text: post hoc and supplementary tables
must say so.
### Results
| What | Table (this file) | LaTeX label | Status |
|---|---|---|---|
| Verified SR with Wilson CI; C2 claimed rate and FPR_B | Table 1 (§4.1) | `tab:sr` | Planned |
| H1–H3: b, c, ΔSR, bootstrap CI, exact and Bonferroni p | Table 2 (§4.2) | `tab:mcnemar` | Planned (confirmatory) |
| Exploratory pairs | Table 2b (§4.3) | `tab:mcnemar-expl` | Planned (exploratory) |
| Per-cell grid and margins | Tables 3, 3b (§5.1–5.2) | `tab:cells`, `tab:margins` | Planned (descriptive) |
| Baseline B claim check | Table 1b (§6) | `tab:fprb` | Planned |
| Efficiency, RNF-3 | Tables 4, 4b, 4c (§7.1–7.3) | `tab:eff-solved`, `tab:eff-problem`, `tab:totals` | Planned |
| SR(k) and the independence curve | Table 6 + figure (§8) | `tab:srk`, `srk_curve.pdf` | Planned |
| κ taxonomy and final failure reasons | Tables 7, 7b, 7c (§9.1–9.3) | `tab:kappa`, `tab:kappa-area`, `tab:final-fail` | Planned |
| Infrastructure statuses and fidelity review | Tables 9, 10 (§3.3, §10) | `tab:infra`, `tab:fidelity` | Planned |
| Appendix / supplementary material | Runs used, Row checks, Tables 4d, 5, 8, 11, 11b | `tab:runs`, `tab:checks`, `tab:medians`, `tab:wall-diff`, `tab:prepass`, `tab:per-problem`, `tab:hard` | Integrity / supplementary / post hoc |
Points to state:
- C5 solves 24/30 (80.0 %, [62.7, 90.5]); C1 and C3 20/30; C4 14/30 (Table 1).
- Only H3 (Critic) is significant: +33.3 pp, Bonferroni p 0.0059. H1 and H2 (+13.3 pp each)
are not; report them as "no evidence of a difference", never "no difference" (Table 2).
- FPR_B = 11/30 = 36.7 % [21.9, 54.5], an upper bound (Table 1b).
- RNF-3 is met: 16.92 calls per solved problem for C5 (Table 4c).
- No provider errors, spend stops, Lean outages or degraded runs (Table 9); all four fidelity
reviews count as solved, with the `amc12b_2003_p6` note (Table 10).
### Discussion
- **The Critic loop is what matters most** (H3, and the SR(k) figure: each extra attempt keeps
adding solved problems, more than an independent-retry model predicts at k = 3; §8, with the
two cautions given there).
- **C5's gain over C1 is entirely on medium problems** (C5 10/10, C1 6/10; Table 3b). On hard
problems C1 and C5 solve the same 4 (Table 11b, post hoc), and 5 hard problems are solved by
no configuration: hard olympiad problems remain out of reach for this setup.
- **RAG**: C5 vs C3 favours C5 (5 to 1) but is not significant; the only problem C3 solves and C5
does not is a hard one, an example of run-to-run variation (§5.3).
- **Self-assessment is unreliable**: C2 claims 30/30; Lean confirms 19, and only 1 of the 10 hard
claims (§6). For 5 of the 11 unconfirmed claims C5 proves the statement, which is why FPR_B is
an upper bound; cite the manual review once it is done.
- **Cost**: C5 is cheaper per solved problem than C1 at nominal cost (Table 4); C3, without
lemma suggestions, needs the most calls per solved problem.
- **Errors**: tactic failures and syntax errors dominate; misformalization never appears;
missing lemmas are common in algebra (Tables 7, 7b). C1's failures are mostly budget
exhaustion, partly driven by truncated responses (Table 7c, §9.1).
- Optional illustrative traces for an appendix are listed at the end of `results.md`
("Illustrative traces"; candidates in §5.5).
### Limitations
Use §12. Essential: small n and low power; one run and one seed (pass@1 only); one
general-purpose model with an unversioned alias; possible contamination of public benchmarks;
FPR_B upper bound with the manual review pending; the formal-vs-informal caveat of
`amc12b_2003_p6`; the shared pre-pass; the interrupted and resumed runs and the other deviations
of §11 (the plan requires reporting them as deviations).
### Ethics
- **Trust in AI-generated mathematics:** the central finding of Baseline B (§6) is that the
model's confidence is not evidence of correctness; the system accepts a proof only when the
Lean kernel does, and the report should say that LLM output should not be trusted without
formal or human verification.
- **Data:** only public benchmarks (miniF2F, PutnamBench) were used, with no personal data; cite
their sources and licences, and mention the contamination risk (§12).
- **Cost and resources:** Main experiments (dev + test, C1–C5): 7.36 USD. Including the
exploratory Pro run: 9.30 USD in total; the project cap was raised from 9 to 12.5 USD for that
run (see Deviations). A response cache avoided repeated calls; this keeps the work reproducible
on a student budget.
- **Transparency and reproducibility:** the analysis plan was written before the `test` runs,
every deviation is logged (§11), post hoc analyses are labelled, and each run records its
models, seed and Lean/Mathlib versions.
## 14. Exploratory: a stronger model on the hard problems
*Post hoc (decided 2026-11-08, after all confirmatory runs), descriptive · LaTeX `tab:pro-hard`.*
This section changes no result of §4–§12. It is logged in `analysis_plan.md` ("Exploratory
analysis (post hoc, decided 2026-11-08)" and two Deviations entries of that date).
**Why.** On the hard tier the agents add nothing with DeepSeek-Flash: C1 and C5 solve exactly the
same 4 hard problems (§5.3). That suggests the hard problems are limited by the model, not by the
method. To look at that, C5 was run again on the 10 hard `test` problems with a stronger model,
DeepSeek-V4-Pro, in every role (profile `experiment_pro`); everything else (graph, prompts,
budgets, seed) was the frozen C5 configuration.
**What happened to the run.** The team stopped it after 5 of the 10 problems
(`20261108T130803Z_C5_experiment_pro_pro_hard`) and it was not resumed. Three outcomes are
**not failures of Pro** and are left out of its solved count:
- `imo_1978_p5`: **provider_error (connection drop, not a system failure)**. The 30th call of the
problem (the Autoformalizer on subgoal 1) never reached DeepSeek: all 5 transport attempts got
`APIConnectionError: Connection error.` with no HTTP response (15:34:50–15:37:28 UTC). The next
problem connected normally.
- `imo_1984_p6`: **interrupted**, cut mid-problem by the stop (4 paid calls, no row).
- `imo_1964_p1_2`, `putnam_1964_b2`, `putnam_1995_b1`, `putnam_1965_b2`: **not attempted**.
Only the 4 problems that both runs finished are compared (`imo_1965_p2`, `imo_1979_p1`,
`imo_1987_p6`, `imo_1993_p5`).
### Table 12. Exploratory: C5-Flash vs C5-Pro on the hard problems (post hoc, descriptive, not in the analysis plan)
| Problem | Flash solved | Flash calls | Flash USD | Flash min | Pro outcome | Pro solved | Pro calls | Pro USD | Pro min |
|---|---|---|---|---|---|---|---|---|---|
| imo_1964_p1_2 | 1 | 15 | 0.0261 | 3.4 | not attempted | — | — | — | — |
| imo_1965_p2 | 1 | 14 | 0.0312 | 3.7 | failed | 0 | 30 | 0.3608 | 33.6 |
| imo_1978_p5 | 0 | 31 | 0.1347 | 39.6 | provider_error (connection drop, not a system failure) | — | 30 | 0.2938 | 40.7 |
| imo_1979_p1 | 0 | 28 | 0.1463 | 19.7 | failed | 0 | 24 | 0.2722 | 29.3 |
| imo_1984_p6 | 0 | 32 | 0.3366 | 27.6 | interrupted | — | — | — | — |
| imo_1987_p6 | 0 | 36 | 0.1793 | 21.4 | budget_exhausted | 0 | 40 | 0.4607 | 57.4 |
| imo_1993_p5 | 0 | 27 | 0.0790 | 10.6 | failed | 0 | 24 | 0.3807 | 53.6 |
| putnam_1964_b2 | 1 | 9 | 0.0160 | 2.3 | not attempted | — | — | — | — |
| putnam_1965_b2 | 1 | 17 | 0.0409 | 4.9 | not attempted | — | — | — | — |
| putnam_1995_b1 | 0 | 25 | 0.1268 | 17.5 | not attempted | — | — | — | — |
| Both finished (4) | 1/4 | 105 | 0.4357 | 55.5 | | 0/4 | 118 | 1.4744 | 173.9 |
| All hard (10) | 4/10 | 234 | 1.1169 | 150.9 | | — | — | — | — |
*Note.* Post hoc, exploratory (decided 2026-11-08, after all confirmatory runs): descriptive only, no tests, no intervals; it changes no main result. C5-Flash = the kept C5 rows of the main test result; C5-Pro = run 20261108T130803Z_C5_experiment_pro_pro_hard (profile experiment_pro: deepseek-v4-pro in every role, everything else as frozen), stopped by the team and not resumed. Cost is nominal USD (cache hits priced as fresh calls). provider_error, interrupted and not attempted do not count as failures of Pro and are left out of its solved count; only problems both runs finished are compared. imo_1984_p6 was interrupted after 4 paid calls (0.1725 USD real, 18.1 min) and left no row. Real cost of the Pro run: 1.9407 USD (151 paid calls, all in the ledger); project ledger total: 9.3003 USD.
In plain words: on the 4 problems both runs finished, Pro solved none and Flash solved one
(`imo_1965_p2`, which Pro failed after 30 calls). Pro spent about 3.4 times as much (1.47 vs
0.44 USD nominal) and took about 3.1 times as long (174 vs 56 min) on those 4. This does **not**
show that a stronger model fails to help: only 4 of the 10 problems were compared, each model
ran once, and 3 of the 4 hard problems Flash solved were never attempted by Pro. The question of
whether the hard tier is limited by the model stays open; the report can mention the run as an
exploratory, incomplete check and should not draw a conclusion from it.
**Cost.** The Pro run cost 1.9407 USD real (151 paid calls, all off-peak). With it, the project
ledger stands at 9.3003 USD; for this run the caps were raised to 12.5 USD in total and 5 USD per
run (Deviations, 2026-11-08).
|