|
Download docs/results_explained.md from NicolasT08/formal-mas: direct link, hf CLI and curl.
- Browser
- Download file 52 kB
-
https://huggingface.co/NicolasT08/formal-mas/resolve/main/docs/results_explained.md
- Command line
-
hf download hf://NicolasT08/formal-mas/docs/results_explained.md
-
curl -L -o results_explained.md https://huggingface.co/NicolasT08/formal-mas/resolve/main/docs/results_explained.md
52 kB
| # What was tested and what came out | |
| The single, complete results document of the project: dataset, configurations, every table of | |
| the analysis, the SR(k) figure, the deviations from the analysis plan, the fidelity review, the | |
| limitations, and a guide to what goes in each section of the report. It is written so that the | |
| report can be drafted from this file alone. For how the system works, see | |
| [how_it_works.md](how_it_works.md). The pre-registered analysis plan is | |
| [analysis_plan.md](analysis_plan.md) (written before any `test` result was seen); the full | |
| statistical tables are in `docs/results/`. | |
| **Where the numbers come from.** Every table below is copied verbatim from | |
| `docs/results/results.md`, which `docs/results/make_tables.py` generates from `runs/` | |
| (read-only). Nothing here is recomputed. `results.md` stays the generated source; this file is the | |
| version to read. After `make_tables.py` is re-run, `python docs/results/check_explained.py` | |
| checks that every table and every number here still matches `results.md`. The LaTeX version of | |
| each table is in `docs/results/tables.tex`, under the label given with each table. | |
| **Status labels used below.** | |
| - **Planned (confirmatory)**: a test fixed in `analysis_plan.md` before `test` was run; the only | |
| results that support claims. | |
| - **Planned (exploratory)**: listed in the plan as exploratory: uncorrected, no claims. | |
| - **Planned (descriptive)**: required by the plan, without tests. | |
| - **Supplementary**: requested by the team after the plan, descriptive only. | |
| - **Post hoc**: a look decided after seeing the results; descriptive only, not in the plan. | |
| - **Integrity check**: describes the data, not the method. | |
| Rates are in %, intervals are 95 %. | |
| ## 1. The dataset | |
| `data/eval.jsonl` holds **45 problems**, each with its informal text and its Lean statement: | |
| | | easy | medium | hard | total | | |
| |---|---|---|---|---| | |
| | `dev` (for tuning) | 5 | 5 | 5 | 15 | | |
| | `test` (final runs only) | 10 | 10 | 10 | 30 | | |
| - **Sources:** 40 from miniF2F (the Lean 4 port used by DeepSeek-Prover-V1.5) and 5 from | |
| PutnamBench (3 of them in `test`). | |
| - **Areas in `test`:** algebra 13, number theory 13, combinatorics 4 (no easy combinatorics | |
| problem: that cell of the 3 × 3 grid is empty). | |
| - Every statement was checked to compile and reviewed by hand for fidelity to the informal | |
| text (`data/curation/`; see §10). Prompts and settings were tuned only on `dev`; `test` was | |
| used once, for the final runs. | |
| ## 2. The five configurations | |
| All five run on the same code and graph; only flags in `config/ablations/` change. | |
| | | Name | In one line | | |
| |---|---|---| | |
| | **C1** | Baseline A | One LLM asked directly for a whole Lean proof; up to 18 independent tries (Lean's errors are not fed back). No agents. | | |
| | **C2** | Baseline B | One LLM solves the problem in natural language and judges its own answer. No Lean. | | |
| | **C3** | No RAG | Full system without the lemma search. | | |
| | **C4** | No Critic | Full system, but one attempt per subgoal and no replanning. | | |
| | **C5** | Full system | All agents, retries and replanning, up to 40 calls per problem. | | |
| All runs used the same model for every role (DeepSeek `deepseek-flash`, a general-purpose | |
| model, not a specialised prover), the same seed (2026), Lean `v4.36.0-rc1` and Mathlib | |
| `625d891`. The exact settings of each run are in its `runs/<run>/run_config.json`. | |
| ## 3. Runs and data integrity | |
| ### 3.1 Runs used | |
| *Integrity check · LaTeX `tab:runs`.* Which run folders feed the analysis and their recorded | |
| configuration. C5 and C1 were interrupted and resumed (§11), so each has more than one folder; | |
| their pieces are merged into one row per problem. "running" means the run was killed before it | |
| could write its final status; "dirty (plan file only)" is a recording artefact explained in §11. | |
| | Run | Config | Profile | Seed | Split | Commit | Tree | Degraded | Run status | vs first run | | |
| |---|---|---|---|---|---|---|---|---|---| | |
| | 20261103T174646Z_C5_experiment | C5 | experiment | 2026 | test | ad70ec2 | clean | False | running | same | | |
| | 20261103T212359Z_C5_experiment_resume | C5 | experiment | 2026 | test | ad70ec2 | dirty (plan file only) | False | budget_stop | same | | |
| | 20261103T225140Z_C5_experiment_resume2 | C5 | experiment | 2026 | test | 93bc701 | dirty (plan file only) | False | finished | same | | |
| | 20261103T235556Z_C4_experiment | C4 | experiment | 2026 | test | ea4db4b | dirty (plan file only) | False | finished | same | | |
| | 20261104T194056Z_C3_experiment | C3 | experiment | 2026 | test | 6c61d69 | clean | False | finished | same | | |
| | 20261104T232056Z_C2_experiment_check | C2 | experiment | 2026 | test | 6c61d69 | clean | False | finished | same | | |
| | 20261105T131443Z_C1_experiment | C1 | experiment | 2026 | test | 5ed7507 | clean | False | running | same | | |
| | 20261107T200233Z_C1_experiment_resume | C1 | experiment | 2026 | test | 2c8eae5 | dirty (plan file only) | False | finished | same | | |
| ### 3.2 Row checks | |
| *Integrity check · LaTeX `tab:checks`.* After merging, every configuration has exactly 30 rows | |
| (one per `test` problem), and every row's split, configuration, area and difficulty match the | |
| dataset. | |
| | Config | Rows | Status counts | lean_down | split/config mismatch | area/difficulty mismatch | | |
| |---|---|---|---|---|---| | |
| | C1 | 30 | budget_exhausted: 7, failed: 3, solved: 20 | 0 | 0 | 0 | | |
| | C2 | 30 | claimed_solved: 30 | 0 | 0 | 0 | | |
| | C3 | 30 | budget_exhausted: 3, failed: 7, solved: 20 | 0 | 0 | 0 | | |
| | C4 | 30 | failed: 16, solved: 14 | 0 | 0 | 0 | | |
| | C5 | 30 | failed: 6, solved: 24 | 0 | 0 | 0 | | |
| ### 3.3 Infrastructure and error statuses | |
| *Planned (plan §5 and §7 item 7) · LaTeX `tab:infra`.* No row ended because of the provider, | |
| the spend cap, a bug, or Lean being unavailable, so the plan's re-run rule and sensitivity | |
| analysis were never needed. Every failure in this document is a failure of the method itself. | |
| | Config | provider_error | budget_stop | error | node_error | invalid_statement | lean_down | | |
| |---|---|---|---|---|---|---| | |
| | C1 | 0 | 0 | 0 | 0 | 0 | 0 | | |
| | C2 | 0 | 0 | 0 | 0 | 0 | 0 | | |
| | C3 | 0 | 0 | 0 | 0 | 0 | 0 | | |
| | C4 | 0 | 0 | 0 | 0 | 0 | 0 | | |
| | C5 | 0 | 0 | 0 | 0 | 0 | 0 | | |
| ## 4. Main result: verified success | |
| ### 4.1 Success rate (SR) | |
| *Planned (primary outcome, plan §2 and §7 item 1) · LaTeX `tab:sr`.* A problem counts as solved | |
| only if Lean's kernel accepts the final proof. The full system C5 solved 24/30; C1 and C3 20/30; | |
| C4 14/30. C2 never uses Lean, so it has no SR: it appears through how often it claimed success | |
| and how often that claim could not be confirmed (FPR_B, §6). The intervals are wide and overlap: | |
| that is what 30 problems allow. | |
| | Config | Solved | SR (%) | Wilson 95 % CI (%) | C2 claimed rate (%) [Wilson] | FPR_B (%) [Wilson] | | |
| |---|---|---|---|---|---| | |
| | C1 | 20/30 | 66.7 | [48.8, 80.8] | — | — | | |
| | C3 | 20/30 | 66.7 | [48.8, 80.8] | — | — | | |
| | C4 | 14/30 | 46.7 | [30.2, 63.9] | — | — | | |
| | C5 | 24/30 | 80.0 | [62.7, 90.5] | — | — | | |
| | C2 (Baseline B) | — | — | — | 30/30 = 100.0 [88.6, 100.0] | 11/30 = 36.7 [21.9, 54.5] | | |
| *Note.* C2 is not verified, so it has no SR; it enters only through its claimed rate B⁺/n and FPR_B (plan §7.1). Of the 30 claimed proofs the formal check verified 19 and did not verify 11; 0 left out of FPR_B. | |
| ### 4.2 The three planned comparisons (H1–H3) | |
| *Planned (confirmatory, plan §3) · LaTeX `tab:mcnemar`.* Exact McNemar test on the same 30 | |
| problems, Bonferroni correction for 3 tests (a comparison is significant only if its adjusted p | |
| is below 0.05). *b* = problems only C5 solved, *c* = problems only the other configuration | |
| solved. ΔSR is the difference in success rate in percentage points, with a paired bootstrap | |
| interval. | |
| | ID | Comparison | b | c | ΔSR (pp) | Paired bootstrap 95 % CI (pp) | Exact p | Bonferroni p | Reject at 0.05/3 | | |
| |---|---|---|---|---|---|---|---|---| | |
| | H1 | C5 vs C1 | 4 | 0 | +13.3 | [+3.3, +26.7] | 0.1250 | 0.3750 | no | | |
| | H2 | C5 vs C3 | 5 | 1 | +13.3 | [+0.0, +30.0] | 0.2188 | 0.6562 | no | | |
| | H3 | C5 vs C4 | 10 | 0 | +33.3 | [+16.7, +50.0] | 0.0020 | 0.0059 | yes | | |
| *Note.* n = 30, so power is low: a non-significant result is no evidence of a difference, not evidence of no difference. H2 is Δ_RAG = SR_C5 − SR_C3; H3 is the Critic contribution SR_C5 − SR_C4. | |
| In plain words: | |
| - **H3: the Critic (retry and replan) makes a significant difference.** C5 solved 10 problems | |
| that C4 did not, and C4 solved none that C5 missed (+33.3 pp, paired bootstrap interval | |
| [+16.7, +50.0], Bonferroni p 0.0059). | |
| - **H1 and H2 point in C5's favour but are not significant.** C5 vs C1: 4 problems to 0; | |
| C5 vs C3: 5 to 1; both +13.3 pp. With 30 problems a difference of 4 or 5 problems is | |
| compatible with chance. "Not significant" here means *no evidence of a difference*, not | |
| evidence that there is none: the test has little power at this sample size. | |
| ### 4.3 Exploratory comparisons | |
| *Planned (exploratory, plan §3): uncorrected, no claims · LaTeX `tab:mcnemar-expl`.* The other | |
| pairs of verified configurations. C1 and C3 solve the same number of problems but not the same | |
| problems (3 each way). | |
| | ID | Comparison | b | c | ΔSR (pp) | Exact p (uncorrected) | | |
| |---|---|---|---|---|---| | |
| | E1 | C3 vs C4 | 9 | 3 | +20.0 | 0.1460 | | |
| | E2 | C1 vs C3 | 3 | 3 | +0.0 | 1.0000 | | |
| | E3 | C1 vs C4 | 8 | 2 | +20.0 | 0.1094 | | |
| ## 5. Results by difficulty and area | |
| ### 5.1 Solved per cell of the 3 × 3 grid | |
| *Planned (descriptive, plan §4.1) · LaTeX `tab:cells`.* How many problems each configuration | |
| solved in each (difficulty, area) cell. Cells have at most 5 problems, so there are no tests | |
| and no intervals. For C2 the column counts *claimed* solutions, not verified ones. | |
| | Difficulty | Area | n | C1 | C3 | C4 | C5 | C2 (claimed) | | |
| |---|---|---|---|---|---|---|---| | |
| | easy | Algebra | 5 | 5/5 | 5/5 | 4/5 | 5/5 | 5/5 | | |
| | easy | Number theory | 5 | 5/5 | 4/5 | 5/5 | 5/5 | 5/5 | | |
| | easy | Combinatorics | 0 | — | — | — | — | — | | |
| | medium | Algebra | 5 | 4/5 | 5/5 | 1/5 | 5/5 | 5/5 | | |
| | medium | Number theory | 4 | 2/4 | 2/4 | 2/4 | 4/4 | 4/4 | | |
| | medium | Combinatorics | 1 | 0/1 | 0/1 | 1/1 | 1/1 | 1/1 | | |
| | hard | Algebra | 3 | 1/3 | 2/3 | 0/3 | 1/3 | 3/3 | | |
| | hard | Number theory | 4 | 1/4 | 1/4 | 0/4 | 1/4 | 4/4 | | |
| | hard | Combinatorics | 3 | 2/3 | 1/3 | 1/3 | 2/3 | 3/3 | | |
| ### 5.2 Success rate by difficulty and by area | |
| *Planned (descriptive, plan §4.1) · LaTeX `tab:margins` (wide table).* The margins of the grid: | |
| first line solved/n, second line SR % with its Wilson interval. Every verified configuration | |
| solves almost all easy problems and 4/10 or fewer hard ones; C5's advantage is on medium | |
| problems (10/10). C2 claims every problem, at every difficulty. | |
| | Config | easy (n = 10) | medium (n = 10) | hard (n = 10) | Algebra (n = 13) | Number theory (n = 13) | Combinatorics (n = 4) | | |
| |---|---|---|---|---|---|---| | |
| | C1 | 10/10 | 6/10 | 4/10 | 10/13 | 8/13 | 2/4 | | |
| | | 100.0 [72.2, 100.0] | 60.0 [31.3, 83.2] | 40.0 [16.8, 68.7] | 76.9 [49.7, 91.8] | 61.5 [35.5, 82.3] | 50.0 [15.0, 85.0] | | |
| | C3 | 9/10 | 7/10 | 4/10 | 12/13 | 7/13 | 1/4 | | |
| | | 90.0 [59.6, 98.2] | 70.0 [39.7, 89.2] | 40.0 [16.8, 68.7] | 92.3 [66.7, 98.6] | 53.8 [29.1, 76.8] | 25.0 [4.6, 69.9] | | |
| | C4 | 9/10 | 4/10 | 1/10 | 5/13 | 7/13 | 2/4 | | |
| | | 90.0 [59.6, 98.2] | 40.0 [16.8, 68.7] | 10.0 [1.8, 40.4] | 38.5 [17.7, 64.5] | 53.8 [29.1, 76.8] | 50.0 [15.0, 85.0] | | |
| | C5 | 10/10 | 10/10 | 4/10 | 11/13 | 10/13 | 3/4 | | |
| | | 100.0 [72.2, 100.0] | 100.0 [72.2, 100.0] | 40.0 [16.8, 68.7] | 84.6 [57.8, 95.7] | 76.9 [49.7, 91.8] | 75.0 [30.1, 95.4] | | |
| | C2 (claimed) | 10/10 | 10/10 | 10/10 | 13/13 | 13/13 | 4/4 | | |
| | | 100.0 [72.2, 100.0] | 100.0 [72.2, 100.0] | 100.0 [72.2, 100.0] | 100.0 [77.2, 100.0] | 100.0 [77.2, 100.0] | 100.0 [51.0, 100.0] | | |
| ### 5.3 The 10 hard problems | |
| *Post hoc (not in the analysis plan), descriptive · LaTeX `tab:hard`.* One row per hard problem, | |
| 1 = verified solved. | |
| | Problem | Area | C1 | C3 | C4 | C5 | Solved by | | |
| |---|---|---|---|---|---|---| | |
| | imo_1964_p1_2 | Number theory | 1 | 1 | 0 | 1 | 3 | | |
| | imo_1965_p2 | Algebra | 1 | 1 | 0 | 1 | 3 | | |
| | imo_1978_p5 | Algebra | 0 | 0 | 0 | 0 | 0 | | |
| | imo_1979_p1 | Number theory | 0 | 0 | 0 | 0 | 0 | | |
| | imo_1984_p6 | Number theory | 0 | 0 | 0 | 0 | 0 | | |
| | imo_1987_p6 | Number theory | 0 | 0 | 0 | 0 | 0 | | |
| | imo_1993_p5 | Algebra | 0 | 1 | 0 | 0 | 1 | | |
| | putnam_1964_b2 | Combinatorics | 1 | 0 | 1 | 1 | 3 | | |
| | putnam_1965_b2 | Combinatorics | 1 | 1 | 0 | 1 | 3 | | |
| | putnam_1995_b1 | Combinatorics | 0 | 0 | 0 | 0 | 0 | | |
| | Solved | | 4/10 | 4/10 | 1/10 | 4/10 | | | |
| *Note.* Post hoc, descriptive (not in the analysis plan): no tests, no intervals. 1 = verified solved. C1 and C5 solve the same hard problems (4 and 4); 5 hard problems are solved by no configuration (imo_1978_p5, imo_1979_p1, imo_1984_p6, imo_1987_p6, putnam_1995_b1); C3 and C5 solve 4 and 4 hard problems, but only C3 solves imo_1993_p5 and only C5 solves putnam_1964_b2. | |
| Three things stand out. C1 and C5 solve exactly the same 4 hard problems (imo_1964_p1_2, | |
| imo_1965_p2, putnam_1964_b2, putnam_1965_b2), so on the hard tier the whole C5 gain over C1 | |
| disappears: all 4 problems that C5 solves and C1 does not are medium (§5.5). Five hard problems | |
| (imo_1978_p5, imo_1979_p1, imo_1984_p6, imo_1987_p6, putnam_1995_b1) are solved by no | |
| configuration. C3 and C5 also solve 4 each but differ by one problem in each direction: only | |
| C3 solves imo_1993_p5, only C5 solves putnam_1964_b2. Since C5 has everything C3 has plus | |
| retrieval, a hard problem that C3 solves and C5 does not shows how much a single run can vary | |
| from one to the next; each configuration was run once, so differences of one problem should | |
| not be read as an effect. | |
| ### 5.4 Per-problem outcomes | |
| *Data listing (the source of the counts above) · LaTeX `tab:per-problem` (wide table).* One row | |
| per `test` problem: 1 = verified solved; the last column gives C2's claim and whether the Lean | |
| check confirmed it. | |
| | Problem | Difficulty | Area | C1 | C3 | C4 | C5 | C2 (claim, check) | | |
| |---|---|---|---|---|---|---|---| | |
| | algebra_absapbon1pabsapbleqsumabsaon1pabsa | medium | Algebra | 0 | 1 | 0 | 1 | claimed, verified | | |
| | amc12_2000_p20 | medium | Algebra | 1 | 1 | 0 | 1 | claimed, verified | | |
| | amc12_2001_p9 | medium | Algebra | 1 | 1 | 0 | 1 | claimed, verified | | |
| | amc12a_2008_p15 | medium | Number theory | 0 | 0 | 0 | 1 | claimed, not_verified | | |
| | amc12a_2008_p25 | medium | Algebra | 1 | 1 | 0 | 1 | claimed, verified | | |
| | amc12b_2002_p3 | medium | Number theory | 0 | 1 | 1 | 1 | claimed, not_verified | | |
| | amc12b_2003_p6 | medium | Algebra | 1 | 1 | 1 | 1 | claimed, verified | | |
| | imo_1964_p1_2 | hard | Number theory | 1 | 1 | 0 | 1 | claimed, not_verified | | |
| | imo_1965_p2 | hard | Algebra | 1 | 1 | 0 | 1 | claimed, not_verified | | |
| | imo_1978_p5 | hard | Algebra | 0 | 0 | 0 | 0 | claimed, not_verified | | |
| | imo_1979_p1 | hard | Number theory | 0 | 0 | 0 | 0 | claimed, not_verified | | |
| | imo_1984_p6 | hard | Number theory | 0 | 0 | 0 | 0 | claimed, not_verified | | |
| | imo_1987_p6 | hard | Number theory | 0 | 0 | 0 | 0 | claimed, not_verified | | |
| | imo_1993_p5 | hard | Algebra | 0 | 1 | 0 | 0 | claimed, not_verified | | |
| | mathd_algebra_160 | easy | Algebra | 1 | 1 | 1 | 1 | claimed, verified | | |
| | mathd_algebra_206 | easy | Algebra | 1 | 1 | 0 | 1 | claimed, verified | | |
| | mathd_algebra_234 | easy | Algebra | 1 | 1 | 1 | 1 | claimed, verified | | |
| | mathd_algebra_288 | easy | Algebra | 1 | 1 | 1 | 1 | claimed, verified | | |
| | mathd_algebra_455 | easy | Algebra | 1 | 1 | 1 | 1 | claimed, verified | | |
| | mathd_numbertheory_321 | easy | Number theory | 1 | 0 | 1 | 1 | claimed, verified | | |
| | mathd_numbertheory_328 | easy | Number theory | 1 | 1 | 1 | 1 | claimed, verified | | |
| | mathd_numbertheory_345 | easy | Number theory | 1 | 1 | 1 | 1 | claimed, verified | | |
| | mathd_numbertheory_430 | easy | Number theory | 1 | 1 | 1 | 1 | claimed, verified | | |
| | mathd_numbertheory_64 | easy | Number theory | 1 | 1 | 1 | 1 | claimed, verified | | |
| | numbertheory_3pow2pownm1mod2pownp3eq2pownp2 | medium | Number theory | 1 | 0 | 0 | 1 | claimed, verified | | |
| | numbertheory_notequiv2i2jasqbsqdiv8 | medium | Number theory | 1 | 1 | 1 | 1 | claimed, verified | | |
| | numbertheory_sumkmulnckeqnmul2pownm1 | medium | Combinatorics | 0 | 0 | 1 | 1 | claimed, verified | | |
| | putnam_1964_b2 | hard | Combinatorics | 1 | 0 | 1 | 1 | claimed, verified | | |
| | putnam_1965_b2 | hard | Combinatorics | 1 | 1 | 0 | 1 | claimed, not_verified | | |
| | putnam_1995_b1 | hard | Combinatorics | 0 | 0 | 0 | 0 | claimed, not_verified | | |
| ### 5.5 Problems solved by one configuration and not the other | |
| *Descriptive (discordant pairs of the comparisons; candidates for example traces in the | |
| report).* For each pair, the problems behind *b* and *c* of Tables 2 and 2b, with what C5 did on | |
| them (status, LLM calls, replans, and the largest number of attempts on any subgoal of the | |
| accepted plan). | |
| - C5 solved, C4 not (Critic loop): algebra_absapbon1pabsapbleqsumabsaon1pabsa (C5: solved, 24 calls, replans 1, max attempts 1); amc12_2000_p20 (C5: solved, 23 calls, replans 0, max attempts 3); amc12_2001_p9 (C5: solved, 7 calls, replans 0, max attempts 2); amc12a_2008_p15 (C5: solved, 16 calls, replans 0, max attempts 3); amc12a_2008_p25 (C5: solved, 23 calls, replans 0, max attempts 2); imo_1964_p1_2 (C5: solved, 15 calls, replans 0, max attempts 3); imo_1965_p2 (C5: solved, 14 calls, replans 0, max attempts 2); mathd_algebra_206 (C5: solved, 28 calls, replans 0, max attempts 2); numbertheory_3pow2pownm1mod2pownp3eq2pownp2 (C5: solved, 15 calls, replans 0, max attempts 3); putnam_1965_b2 (C5: solved, 17 calls, replans 0, max attempts 3) | |
| - C4 solved, C5 not: — | |
| - C5 solved, C3 not (RAG): amc12a_2008_p15 (C5: solved, 16 calls, replans 0, max attempts 3); mathd_numbertheory_321 (C5: solved, 1 calls, replans 0, max attempts 0); numbertheory_3pow2pownm1mod2pownp3eq2pownp2 (C5: solved, 15 calls, replans 0, max attempts 3); numbertheory_sumkmulnckeqnmul2pownm1 (C5: solved, 5 calls, replans 0, max attempts 1); putnam_1964_b2 (C5: solved, 9 calls, replans 0, max attempts 1) | |
| - C3 solved, C5 not: imo_1993_p5 (C5: failed, 27 calls, replans 1, max attempts -) | |
| - C5 solved, C1 not: algebra_absapbon1pabsapbleqsumabsaon1pabsa (C5: solved, 24 calls, replans 1, max attempts 1); amc12a_2008_p15 (C5: solved, 16 calls, replans 0, max attempts 3); amc12b_2002_p3 (C5: solved, 9 calls, replans 0, max attempts 1); numbertheory_sumkmulnckeqnmul2pownm1 (C5: solved, 5 calls, replans 0, max attempts 1) | |
| - C1 solved, C5 not: — | |
| - C2 claimed, check did not verify: amc12a_2008_p15 (C5: solved, 16 calls, replans 0, max attempts 3); amc12b_2002_p3 (C5: solved, 9 calls, replans 0, max attempts 1); imo_1964_p1_2 (C5: solved, 15 calls, replans 0, max attempts 3); imo_1965_p2 (C5: solved, 14 calls, replans 0, max attempts 2); imo_1978_p5 (C5: failed, 31 calls, replans 1, max attempts -); imo_1979_p1 (C5: failed, 28 calls, replans 1, max attempts -); imo_1984_p6 (C5: failed, 32 calls, replans 1, max attempts -); imo_1987_p6 (C5: failed, 36 calls, replans 1, max attempts -); imo_1993_p5 (C5: failed, 27 calls, replans 1, max attempts -); putnam_1965_b2 (C5: solved, 17 calls, replans 0, max attempts 3); putnam_1995_b1 (C5: failed, 25 calls, replans 1, max attempts -) | |
| ## 6. Baseline B (C2): what the LLM claimed vs what Lean confirmed | |
| *Planned (plan §4.2, Eq. 16) · LaTeX `tab:fprb`.* C2 never uses Lean, so it cannot "solve" | |
| anything in the verified sense. Instead it reports whether it believes its own proof is | |
| correct. To see how often that belief is wrong, each claimed proof was handed to the Lean | |
| pipeline (Autoformalizer + Verifier, up to 3 attempts, same lemma search as C5; | |
| `--check` in `scripts/run_baseline_b.py`). | |
| | Quantity | Value | | |
| |---|---| | |
| | Problems | 30 | | |
| | Claimed (B⁺) | 30 | | |
| | Not claimed | 0 | | |
| | Check status among B⁺ | not_verified: 11, verified: 19 | | |
| | Verified (Acc on the assembled theorem) | 19 | | |
| | Not verified (budget_exhausted included) | 11 | | |
| | Left out of FPR_B (provider_error, budget_stop, error, lean_down) | 0 | | |
| | FPR_B = not verified / (verified + not verified) | 11/30 = 36.7 % [21.9, 54.5] | | |
| | B⁺ problems for the manual review (§4.2) | 10 drawn with random.Random(2026).sample(sorted ids, 10) (listed below) | | |
| *Note.* Manual review set (B⁺ not verified): amc12b_2002_p3, imo_1964_p1_2, imo_1965_p2, imo_1978_p5, imo_1979_p1, imo_1984_p6, imo_1987_p6, imo_1993_p5, putnam_1965_b2, putnam_1995_b1. | |
| By difficulty (counted from Table 11): | |
| | | easy | medium | hard | total | | |
| |---|---|---|---|---| | |
| | Claimed correct by the LLM | 10 | 10 | 10 | **30** | | |
| | Confirmed by Lean | 10 | 8 | 1 | **19** | | |
| C2 claimed to have solved every problem. Lean confirmed 19; for the other 11 the claim could | |
| not be turned into an accepted proof. That is FPR_B = 11/30 = 36.7 % [21.9, 54.5]. Almost all | |
| of the gap is on hard problems, where the model was sure of 10 answers and only 1 could be | |
| confirmed. This is the main motivation of the project: a confident LLM is not a reliable judge | |
| of its own mathematics. | |
| FPR_B is an **upper bound** on how often C2's self-assessment is wrong: "not confirmed" means | |
| "the pipeline could not formalise it within the budget", which does not always mean the | |
| informal argument was wrong. Two facts bear on this: | |
| - For 5 of the 11 (amc12a_2008_p15, amc12b_2002_p3, imo_1964_p1_2, imo_1965_p2, | |
| putnam_1965_b2), C5 has a Lean-verified proof of the same statement, so at least the claimed | |
| result is true there (whether C2's own argument was sound is a separate question). | |
| - The plan asks for a manual review of 10 of the 11 (drawn with seed 2026; listed in the note | |
| of Table 1b) to separate wrong arguments from formalisation failures. The review sheet is | |
| `docs/results/fprb_review.md` (reviewer: Andrés). **Its verdicts are still pending.** | |
| ## 7. Efficiency and cost | |
| ### 7.1 Per solved problem | |
| *Planned (plan §4.3, RNF-3) · LaTeX `tab:eff-solved`.* Total LLM calls, tokens, cost and | |
| wall-clock time of a configuration divided by the problems it solved (failed problems' cost is | |
| included in the numerator). Bootstrap intervals in brackets. **Nominal** cost prices cache hits | |
| as fresh calls; it is the figure the plan compares, because the cache makes later runs cheaper | |
| for reasons unrelated to the method. **Real** cost is what the provider charged. | |
| | Config | Solved | Calls / solved | k tokens / solved | Nominal USD / solved | Real USD / solved | Wall min / solved | | |
| |---|---|---|---|---|---|---| | |
| | C1 | 20 | 12.85 [7.19, 22.60] | 219.7 [114.6, 394.7] | 0.1285 [0.0668, 0.2313] | 0.1235 [0.0643, 0.2218] | 17.7 [9.2, 32.0] | | |
| | C2 (per claimed) | 30 | 2.17 [2.00, 2.40] | 5.3 [2.8, 8.6] | 0.0027 [0.0013, 0.0047] | 0.0027 [0.0013, 0.0047] | 0.4 [0.2, 0.7] | | |
| | C3 | 20 | 26.50 [16.15, 43.67] | 150.4 [79.3, 274.6] | 0.0675 [0.0335, 0.1279] | 0.0675 [0.0335, 0.1278] | 9.2 [4.6, 17.3] | | |
| | C4 | 14 | 9.43 [5.72, 16.67] | 70.7 [38.2, 137.8] | 0.0278 [0.0135, 0.0571] | 0.0000 [0.0000, 0.0000] | 0.2 [0.1, 0.3] | | |
| | C5 | 24 | 16.92 [10.63, 26.21] | 148.0 [74.4, 266.4] | 0.0566 [0.0245, 0.1097] | 0.0490 [0.0236, 0.0897] | 7.6 [3.3, 14.6] | | |
| *Note.* C2 has no verified solutions: its ratio uses claimed solutions and excludes the claim check (calls, tokens, cost and wall time up to claim_check_start). | |
| C5 needs fewer calls per solved problem than C3 (16.92 vs 26.50) and costs less per solved | |
| problem than C1 (nominal 0.0566 vs 0.1285 USD); C1 also takes the most wall time per solved | |
| problem (17.7 min). | |
| ### 7.2 Per problem (means) | |
| *Planned (plan §4.3) · LaTeX `tab:eff-problem`.* The same quantities averaged over all 30 | |
| problems, solved or not. | |
| | Config | Calls / problem | k tokens / problem | Nominal USD / problem | Real USD / problem | Wall min / problem | | |
| |---|---|---|---|---|---| | |
| | C1 | 8.57 [5.83, 11.33] | 146.4 [91.1, 203.5] | 0.0857 [0.0532, 0.1192] | 0.0823 [0.0505, 0.1152] | 11.8 [7.2, 16.5] | | |
| | C2 | 2.17 [2.00, 2.40] | 5.3 [2.8, 8.6] | 0.0027 [0.0013, 0.0047] | 0.0027 [0.0013, 0.0047] | 0.4 [0.2, 0.7] | | |
| | C3 | 17.67 [12.97, 22.37] | 100.3 [63.8, 140.8] | 0.0450 [0.0269, 0.0658] | 0.0450 [0.0269, 0.0658] | 6.1 [3.7, 9.0] | | |
| | C4 | 4.40 [3.37, 5.43] | 33.0 [22.8, 44.4] | 0.0130 [0.0081, 0.0186] | 0.0000 [0.0000, 0.0000] | 0.1 [0.1, 0.1] | | |
| | C5 | 13.53 [9.57, 17.60] | 118.4 [67.9, 178.7] | 0.0453 [0.0225, 0.0737] | 0.0392 [0.0218, 0.0588] | 6.1 [3.0, 9.7] | | |
| ### 7.3 Totals, cache and real cost | |
| *Planned (plan §4.3, with the cache and spend-ledger columns added for transparency) · LaTeX | |
| `tab:totals` (wide table).* Totals over the kept rows. The last column is the real spend of all | |
| the configuration's runs from `data/spend_ledger.jsonl`, which also includes the attempts | |
| discarded when a run was interrupted (C1: 0.0985 USD; C5: 0.1815 USD) and, for C2, the claim | |
| check. | |
| | Config | LLM calls | Cached calls | Cached (%) | Tokens (M) | Nominal USD (rows) | Real USD (rows) | Real USD (ledger, all runs) | | |
| |---|---|---|---|---|---|---|---| | |
| | C1 | 257 | 47 | 18.3 | 4.393 | 2.5700 | 2.4695 | 2.5679 | | |
| | C2 | 65 | 0 | 0.0 | 0.158 | 0.0814 | 0.0814 | 0.7083 | | |
| | C3 | 530 | 18 | 3.4 | 3.009 | 1.3508 | 1.3500 | 1.3500 | | |
| | C4 | 132 | 132 | 100.0 | 0.990 | 0.3887 | 0.0000 | 0.0000 | | |
| | C5 | 406 | 34 | 8.4 | 3.551 | 1.3590 | 1.1770 | 1.3585 | | |
| *Note.* C2 claim check (not in the C2 figures of the other columns): 77 calls (11 cached), 0.6274 nominal, 0.6269 real. | |
| **RNF-3 (C̄_solved ≤ 20 for C5):** C̄_solved = 406/24 = 16.92 calls per solved problem: **met**. | |
| How to read it: | |
| - **C4 cost 0 USD** because every request it made had already been made, word for word, by C5 | |
| (C4 is C5 with retries removed, so its first attempts are identical); the answers came from | |
| the cache (100.0 % cached calls). For the same reason C4's wall times measure only Lean. | |
| - **RNF-3** (at most 20 LLM calls per solved problem for C5) is **met**: 406/24 = 16.92. | |
| - C1 has a budget of 18 calls per problem (the average C5 used on `dev`, by a rule fixed before | |
| `test`); the full system has 40. | |
| - **Project spend** (ledger, all runs, `dev` included): main experiments 7.3596 USD. Including the | |
| exploratory Pro run (1.9407 USD, §14): 9.3003 USD in total; the project cap was raised from 9 to | |
| 12.5 USD for that run (§11). | |
| ### 7.4 Medians and quartiles | |
| *Supplementary (requested by the team, not in the plan) · LaTeX `tab:medians`.* Means hide that | |
| the distributions are very skewed: most problems take few calls and a few take many. | |
| | Config | Calls, median [Q1, Q3] | Wall min, median [Q1, Q3] | | |
| |---|---|---| | |
| | C1 | 4.5 [1.0, 18.0] | 2.7 [0.6, 24.2] | | |
| | C2 | 2.0 [2.0, 2.0] | 0.1 [0.1, 0.3] | | |
| | C3 | 16.5 [3.2, 28.0] | 3.7 [0.6, 6.7] | | |
| | C4 | 5.0 [1.0, 5.8] | 0.1 [0.0, 0.1] | | |
| | C5 | 11.5 [1.0, 23.8] | 2.4 [0.2, 5.2] | | |
| ### 7.5 Wall time by difficulty | |
| *Supplementary (requested by the team, not in the plan) · LaTeX `tab:wall-diff` (wide table).* | |
| Minutes per problem, mean / median [quartiles]. Times are dominated by waiting for the model, and | |
| grow with difficulty. C4's times are not comparable (cache, see above). Many easy problems take | |
| seconds in C3–C5 because they need no LLM-written subgoal proof at all (Table 8, §9.4). | |
| | Config | easy (n = 10): mean / median [Q1, Q3] | medium (n = 10): mean / median [Q1, Q3] | hard (n = 10): mean / median [Q1, Q3] | | |
| |---|---|---|---| | |
| | C1 | 0.7 / 0.4 [0.1, 1.1] | 14.3 / 16.2 [1.8, 23.2] | 20.3 / 25.3 [7.0, 32.4] | | |
| | C2 | 0.1 / 0.1 [0.1, 0.2] | 0.1 / 0.1 [0.1, 0.2] | 1.0 / 0.5 [0.2, 1.1] | | |
| | C3 | 0.8 / 0.1 [0.0, 1.0] | 4.0 / 2.6 [1.2, 6.0] | 13.6 / 12.3 [6.6, 19.8] | | |
| | C4 | 0.1 / 0.0 [0.0, 0.1] | 0.1 / 0.1 [0.0, 0.1] | 0.1 / 0.1 [0.1, 0.1] | | |
| | C5 | 0.4 / 0.0 [0.0, 0.5] | 2.7 / 2.6 [0.7, 4.5] | 15.1 / 14.1 [4.0, 21.0] | | |
| ## 8. Refinement robustness: SR(k) | |
| *Planned (plan §4.4, Eq. 14; figure: plan §7 item 5) · LaTeX `tab:srk`; figure | |
| `docs/results/figures/srk_curve.pdf` (`.png` below).* SR_C5(k) counts a C5 problem as solved at | |
| k if its accepted proof needed at most k attempts on every subgoal. It shows how much each extra | |
| attempt (with the Critic's feedback) adds. The last two columns are subgoal-level diagnostics | |
| (supplementary). | |
| | k | SR_C5(k) [Wilson] (%) | Δ̂(k) = SR(k) − SR(k−1) (pp) | Indep. Δ(k) = p̂(1−p̂)^(k−1) (%) | Subgoals solved at attempt k | Solved at k / reached k | | |
| |---|---|---|---|---|---| | |
| | 1 | 15/30 = 50.0 [33.2, 66.8] | +20.0 | 40.9 | 18/44 = 40.9 | 18/44 = 40.9 | | |
| | 2 | 19/30 = 63.3 [45.5, 78.1] | +13.3 | 24.2 | 8/44 = 18.2 | 8/26 = 30.8 | | |
| | 3 | 24/30 = 80.0 [62.7, 90.5] | +16.7 | 14.3 | 5/44 = 11.4 | 5/18 = 27.8 | | |
| *Note.* SR_C5(0) = 9/30 (accepted plan with no LLM subgoal attempt). p̂ = 18/44 = 40.9 %. k = 1 from C4 (no retries, no Critic advice): 14/30 = 46.7 % [30.2, 63.9]. The last two columns are subgoal-level diagnostics (supplementary). | |
|  | |
| **Figure: SR(k).** (a) SR_C5(k) for k = 0…3 with Wilson intervals: 9/30 solved with no | |
| LLM-attempted subgoal, then 15/30, 19/30 and 24/30 as up to 1, 2 and 3 attempts per subgoal are | |
| allowed. The square is C4 (one attempt, no retries, no Critic advice): 14/30 = 46.7 %, next to | |
| SR_C5(1) = 15/30. (b) The observed gain of each extra attempt, Δ̂(k) (+20.0, +13.3, +16.7 pp), | |
| against the gain expected if every attempt succeeded independently with the first-attempt rate | |
| p̂ = 18/44 = 40.9 % (40.9, 24.2, 14.3). The independence curve falls quickly; the observed gain | |
| does not: the third attempt still adds +16.7 pp, above the 14.3 the independence model predicts. | |
| This fits the idea that retries guided by Lean's errors are not just new random draws. Two | |
| cautions: the two curves are at different levels (the independence curve is per subgoal, Δ̂(k) | |
| per problem; method note 23 in `results.md`), and with n = 30 each step is a handful of problems. | |
| ## 9. Errors and failure reasons | |
| ### 9.1 Error taxonomy (κ) | |
| *Planned (plan §4.5, Eq. 11) · LaTeX `tab:kappa` (wide table).* Every failed attempt classified | |
| by the deterministic classifier κ, count (%). Tactic failures and syntax errors dominate. No | |
| attempt was classified as misformalization. For C1 the attempts were classified offline with | |
| the same κ: 44 of its syntax errors are empty answers, which happen alongside 52 responses cut | |
| at the token limit (method note 25 in `results.md`). For C2 the row is the claim-check loop. | |
| | Config | Failed attempts | syntax | missing lemma | tactic failure | misformalization | timeout | planning | | |
| |---|---|---|---|---|---|---|---| | |
| | C1 | 186 | 60 (32.3) | 50 (26.9) | 68 (36.6) | 0 (0.0) | 8 (4.3) | 0 (0.0) | | |
| | C2 (claim check) | 45 | 14 (31.1) | 8 (17.8) | 22 (48.9) | 0 (0.0) | 1 (2.2) | 0 (0.0) | | |
| | C3 | 82 | 20 (24.4) | 17 (20.7) | 42 (51.2) | 0 (0.0) | 2 (2.4) | 1 (1.2) | | |
| | C4 | 16 | 5 (31.2) | 4 (25.0) | 7 (43.8) | 0 (0.0) | 0 (0.0) | 0 (0.0) | | |
| | C5 | 57 | 19 (33.3) | 18 (31.6) | 18 (31.6) | 0 (0.0) | 2 (3.5) | 0 (0.0) | | |
| *Note.* C3–C5: classes recorded by the Critic. C1: attempts classified offline with policy.classify (first 3 Lean messages, 300 characters each, as stored; no k_max escalation). C2 has no verified attempts of its own: its row is the claim-check loop. | |
| ### 9.2 Error taxonomy per area | |
| *Planned (plan §4.5) · LaTeX `tab:kappa-area` (wide table).* The same counts split by area. | |
| Missing-lemma errors are frequent in algebra; tactic failures dominate number theory. | |
| | Config | Area | Failed attempts | syntax | missing lemma | tactic failure | misformalization | timeout | planning | | |
| |---|---|---|---|---|---|---|---|---| | |
| | C1 | Algebra | 77 | 14 (18.2) | 40 (51.9) | 22 (28.6) | 0 (0.0) | 1 (1.3) | 0 (0.0) | | |
| | C1 | Number theory | 83 | 27 (32.5) | 3 (3.6) | 46 (55.4) | 0 (0.0) | 7 (8.4) | 0 (0.0) | | |
| | C1 | Combinatorics | 26 | 19 (73.1) | 7 (26.9) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | | |
| | C2 (claim check) | Algebra | 14 | 5 (35.7) | 4 (28.6) | 5 (35.7) | 0 (0.0) | 0 (0.0) | 0 (0.0) | | |
| | C2 (claim check) | Number theory | 23 | 5 (21.7) | 4 (17.4) | 13 (56.5) | 0 (0.0) | 1 (4.3) | 0 (0.0) | | |
| | C2 (claim check) | Combinatorics | 8 | 4 (50.0) | 0 (0.0) | 4 (50.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | | |
| | C3 | Algebra | 18 | 3 (16.7) | 9 (50.0) | 5 (27.8) | 0 (0.0) | 0 (0.0) | 1 (5.6) | | |
| | C3 | Number theory | 43 | 13 (30.2) | 3 (7.0) | 26 (60.5) | 0 (0.0) | 1 (2.3) | 0 (0.0) | | |
| | C3 | Combinatorics | 21 | 4 (19.0) | 5 (23.8) | 11 (52.4) | 0 (0.0) | 1 (4.8) | 0 (0.0) | | |
| | C4 | Algebra | 8 | 1 (12.5) | 3 (37.5) | 4 (50.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | | |
| | C4 | Number theory | 6 | 2 (33.3) | 1 (16.7) | 3 (50.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | | |
| | C4 | Combinatorics | 2 | 2 (100.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | | |
| | C5 | Algebra | 25 | 7 (28.0) | 8 (32.0) | 10 (40.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | | |
| | C5 | Number theory | 24 | 8 (33.3) | 9 (37.5) | 6 (25.0) | 0 (0.0) | 1 (4.2) | 0 (0.0) | | |
| | C5 | Combinatorics | 8 | 4 (50.0) | 1 (12.5) | 2 (25.0) | 0 (0.0) | 1 (12.5) | 0 (0.0) | | |
| ### 9.3 Final failure reason per problem | |
| *Planned (plan §4.5) · LaTeX `tab:final-fail`.* For every problem a configuration did not | |
| solve, its final status and the stage where it stopped (`verify` = a subgoal proof was rejected; | |
| `formalize` = the Lean sketch could not be assembled). C1 failures are mostly running out of its | |
| 18-call budget. | |
| | Config | Problems | status @ stage: count | | |
| |---|---|---| | |
| | C1 | 10 | budget_exhausted: 7; failed: 3 | | |
| | C2 | 30 | claimed, check verified: 19; claimed, check not_verified: 11 | | |
| | C3 | 10 | failed @ verify: 5; budget_exhausted @ formalize: 3; failed @ formalize: 2 | | |
| | C4 | 16 | failed @ verify: 11; failed @ formalize: 5 | | |
| | C5 | 6 | failed @ verify: 4; failed @ formalize: 2 | | |
| ### 9.4 Subgoals closed by fixed tactics vs the LLM | |
| *Supplementary (not in the plan) · LaTeX `tab:prepass` (wide table).* Before asking the LLM, the | |
| graph configurations try a fixed list of tactics on each subgoal (the pre-pass). This table shows | |
| how many subgoals of the accepted plans each closed, and how many solved problems needed no | |
| LLM-written subgoal proof at all (8–9 per configuration). This is a shared factor of C3–C5, not | |
| something one of them has and the others lack. | |
| | Config | Pre-pass (accepted plans) | LLM (accepted plans) | Pre-pass (all subgoal_solved) | Solved with no LLM subgoal | Solved | | |
| |---|---|---|---|---|---| | |
| | C3 | 7/32 = 21.9 | 25/32 = 78.1 | 8/39 = 20.5 | 8 | 20 | | |
| | C4 | 9/18 = 50.0 | 9/18 = 50.0 | 11/21 = 52.4 | 9 | 14 | | |
| | C5 | 11/39 = 28.2 | 28/39 = 71.8 | 11/42 = 26.2 | 9 | 24 | | |
| ## 10. Fidelity review | |
| Lean checks the formal statement, not the original text, so the statements were reviewed by | |
| hand twice. | |
| **Before any run (2026-11-01, `data/curation/fidelity_review.yaml`).** Every drawn problem's | |
| Lean statement was compared with its informal text. Two were discarded because the formal | |
| statement changed the mathematics (`imo_1962_p2`, `imo_1974_p5`; replaced). Seven were kept | |
| with a note: the formal version differs from the text in a way judged harmless (for example a | |
| sequence already reduced to one equation, or a definition written differently). | |
| **After the `test` runs (2026-11-07, team review, `post_test_review`).** As plan §6 requires, | |
| every problem solved on `test` by some configuration that has a note was read again against the | |
| informal text: four problems. | |
| *Planned (plan §6) · LaTeX `tab:fidelity`.* | |
| | Problem | Solved by | Manual review | | |
| |---|---|---| | |
| | amc12b_2003_p6 | C1, C3, C4, C5 | 2026-11-07: counts as solved; report note: The formalization proves that u 0 = +-2/sqrt 3 (a necessary condition), not that -2*sqrt(3)/3 is attained; this is enough to rule out the other options. | | |
| | mathd_algebra_234 | C1, C3, C4, C5 | 2026-11-07: counts as solved; no note | | |
| | mathd_algebra_455 | C1, C3, C4, C5 | 2026-11-07: counts as solved; no note | | |
| | putnam_1965_b2 | C1, C3, C5 | 2026-11-07: counts as solved; no note | | |
| In plain words: all four count as solved and no success rate changes, so the plan's "SR with | |
| and without" never applies. One needs a caveat in the report: **`amc12b_2003_p6`** (solved by | |
| C1, C3, C4, C5 and confirmed for C2) asks which option is a possible first term. The | |
| formalisation proves that the first term must be ±2/√3 (a necessary condition), not that | |
| −2√3/3 is actually attained; that is enough to rule out the other options. The remaining notes | |
| do not apply: `imo_2006_p3` is in `dev`, and `imo_1987_p6` and `putnam_1995_b1` were solved by | |
| no configuration. | |
| ## 11. Deviations from the analysis plan, in plain language | |
| The analysis itself (metrics, tests, corrections) was not changed. What happened differently | |
| from the plan was operational, and each item is logged with date and reason in | |
| `analysis_plan.md` (Deviations): | |
| 1. **C5 was interrupted (2026-11-03).** The machine ran low on memory and the run was killed | |
| after 23 of 30 problems. Those 23 rows are kept unchanged. The other 7 were run again with the | |
| same frozen configuration, seed and cache, in two resume runs: the first one hit its spend cap | |
| during its third problem (that incomplete row of `imo_1984_p6` is discarded), the second | |
| finished the rest. The merge rule (which row comes from which run) was written down before | |
| the analysis and the analysis script checks it. | |
| 2. **C4 ran before C3** (plan order: C5, C3, C4, C2, C1). For schedule and cost reasons only. | |
| Every configuration still ran once on the same problems, so the paired comparisons are not | |
| affected. | |
| 3. **The per-run spend cap was raised twice for the main runs** (to 2 USD before C3, then to | |
| 3.5 USD before C1), by team decision, so that those runs would not stop half-way. The cap only | |
| decides whether a request is sent; it does not change prompts, models, call budgets or results. | |
| For the exploratory Pro run (§14, after all main runs) both caps were raised again: the project | |
| cap from 9 to 12.5 USD and the per-run cap from 3.5 to 5 USD. | |
| 4. **C1 was stopped by the team (2026-11-05)** to free the machine, after 23 of 30 problems. The | |
| other 7 were completed on 2026-11-07 in a resume run with the same procedure as C5, and the | |
| pieces were merged into one file (checked row by row). | |
| 5. **`dirty: true` in four `run_config.json` files** (found during the analysis). The run | |
| configuration file was rewritten at the end of each run and read the git state at that moment; | |
| the only changed file was `analysis_plan.md` itself, edited during the runs to log these | |
| incidents. Every tree was clean at launch. The recording was fixed after the `test` runs. | |
| 6. **The exploratory Pro run was stopped by the team (2026-11-08)** after 5 of its 10 problems and | |
| not resumed (§14). It is outside the analysis plan's results and changes none of them. | |
| Written before any `test` run, as a clarification rather than a deviation: **how the C2 claim | |
| check works** (plan §4.2): each claimed proof becomes a sketch for the Autoformalizer, with up to | |
| 3 attempts and Lean feedback, the same lemma search as C5, no Planner and no fixed tactics; its | |
| calls are counted apart from C2's own. | |
| **Where the plan was silent or had to be read literally** (full list: "Method notes" at the end | |
| of `results.md`): | |
| - C2 has no verified SR, so Table 1 reports its claimed rate and FPR_B instead, and C2 is in no | |
| McNemar test. In the grid tables C2 shows claimed counts, labelled as such. | |
| - The plan fixes the bootstrap (10 000 resamples, seed 2026) but not the random generator; | |
| NumPy's default generator was used, the same for every comparison. | |
| - The plan says "10 drawn with seed 2026" for the C2 manual review without saying how; | |
| Python's `random.Random(2026).sample` on the sorted ids was used. | |
| - SR(0) and the first-attempt rate p̂ of the SR(k) analysis are not defined in the plan; the | |
| definitions used are in method notes 22 and 23. | |
| - C1 attempts were classified by κ offline from the stored (shortened) Lean messages. | |
| - Tables 4d, 5, 8 and 11b were added after the plan; they are descriptive and labelled. | |
| - Disclosure: while checking configurations, before the analysis script existed, the analyst had | |
| seen the first C5 `test` row and run summaries of C5 and C4. After the first `test` run the | |
| script changed only in how C2's wall time is cut, an added note on truncated responses, and | |
| LaTeX layout. No test, comparison, correction or metric changed. | |
| ## 12. Limitations | |
| - **Small sample.** 30 test problems (10 per difficulty, only 4 in combinatorics). Confidence | |
| intervals are wide, H1 and H2 have little power, and per-area numbers are descriptive only. | |
| - **One run per configuration.** Each configuration ran once with one seed, so pass@k is | |
| estimated only for k = 1 (the SR). LLM output varies between runs; differences of one or two | |
| problems (for example C3 vs C5 on hard problems, §5.3) are within that variation. | |
| - **One general-purpose model.** Every role used the same general model (`deepseek-flash`), not | |
| a model trained for Lean. Results may differ with other models. `deepseek-flash` is an | |
| alias with no versioned id; the run configs record what was used. | |
| - **Possible contamination.** miniF2F and the competition problems are public, and Lean proofs | |
| of many miniF2F problems are online. The model may have seen some of them in training, which | |
| would make every configuration look better than it would on new problems. This affects all | |
| configurations, but not necessarily equally (C1 asks for a Lean proof directly, which is | |
| where a memorised proof might help most). | |
| - **Formal vs informal statement.** Lean checks the formal statement, not the original text. | |
| All statements were reviewed (§10); `amc12b_2003_p6` needs the caveat above. | |
| - **FPR_B is an upper bound** and its manual review is still pending (§6). | |
| - **Shared tactic pre-pass.** 8–9 solved problems per graph configuration need no LLM-written | |
| subgoal proof (Table 8), which narrows the room in which RAG and the Critic can differ. | |
| - **Truncated responses.** Some LLM answers hit the token limit (C1 most often: 52 responses in | |
| 9 rows); in C1 these show up as empty attempts counted against its budget. | |
| - **Interrupted runs.** C5 and C1 were stopped by operational incidents and resumed with the | |
| same frozen configuration (§11). | |
| - **C4's cost and times** come from C5's cache and are not comparable at real cost; nominal cost | |
| is the comparable figure. | |
| ## 13. For the report | |
| What to use in each section of the report, with the LaTeX label of each table in | |
| `docs/results/tables.tex` (`\input` the file or copy the tables; it needs only `booktabs`; | |
| tables marked "wide" are `table*`). The figure is `docs/results/figures/srk_curve.pdf`. | |
| Keep the status label of each table in its caption or text: post hoc and supplementary tables | |
| must say so. | |
| ### Results | |
| | What | Table (this file) | LaTeX label | Status | | |
| |---|---|---|---| | |
| | Verified SR with Wilson CI; C2 claimed rate and FPR_B | Table 1 (§4.1) | `tab:sr` | Planned | | |
| | H1–H3: b, c, ΔSR, bootstrap CI, exact and Bonferroni p | Table 2 (§4.2) | `tab:mcnemar` | Planned (confirmatory) | | |
| | Exploratory pairs | Table 2b (§4.3) | `tab:mcnemar-expl` | Planned (exploratory) | | |
| | Per-cell grid and margins | Tables 3, 3b (§5.1–5.2) | `tab:cells`, `tab:margins` | Planned (descriptive) | | |
| | Baseline B claim check | Table 1b (§6) | `tab:fprb` | Planned | | |
| | Efficiency, RNF-3 | Tables 4, 4b, 4c (§7.1–7.3) | `tab:eff-solved`, `tab:eff-problem`, `tab:totals` | Planned | | |
| | SR(k) and the independence curve | Table 6 + figure (§8) | `tab:srk`, `srk_curve.pdf` | Planned | | |
| | κ taxonomy and final failure reasons | Tables 7, 7b, 7c (§9.1–9.3) | `tab:kappa`, `tab:kappa-area`, `tab:final-fail` | Planned | | |
| | Infrastructure statuses and fidelity review | Tables 9, 10 (§3.3, §10) | `tab:infra`, `tab:fidelity` | Planned | | |
| | Appendix / supplementary material | Runs used, Row checks, Tables 4d, 5, 8, 11, 11b | `tab:runs`, `tab:checks`, `tab:medians`, `tab:wall-diff`, `tab:prepass`, `tab:per-problem`, `tab:hard` | Integrity / supplementary / post hoc | | |
| Points to state: | |
| - C5 solves 24/30 (80.0 %, [62.7, 90.5]); C1 and C3 20/30; C4 14/30 (Table 1). | |
| - Only H3 (Critic) is significant: +33.3 pp, Bonferroni p 0.0059. H1 and H2 (+13.3 pp each) | |
| are not; report them as "no evidence of a difference", never "no difference" (Table 2). | |
| - FPR_B = 11/30 = 36.7 % [21.9, 54.5], an upper bound (Table 1b). | |
| - RNF-3 is met: 16.92 calls per solved problem for C5 (Table 4c). | |
| - No provider errors, spend stops, Lean outages or degraded runs (Table 9); all four fidelity | |
| reviews count as solved, with the `amc12b_2003_p6` note (Table 10). | |
| ### Discussion | |
| - **The Critic loop is what matters most** (H3, and the SR(k) figure: each extra attempt keeps | |
| adding solved problems, more than an independent-retry model predicts at k = 3; §8, with the | |
| two cautions given there). | |
| - **C5's gain over C1 is entirely on medium problems** (C5 10/10, C1 6/10; Table 3b). On hard | |
| problems C1 and C5 solve the same 4 (Table 11b, post hoc), and 5 hard problems are solved by | |
| no configuration: hard olympiad problems remain out of reach for this setup. | |
| - **RAG**: C5 vs C3 favours C5 (5 to 1) but is not significant; the only problem C3 solves and C5 | |
| does not is a hard one, an example of run-to-run variation (§5.3). | |
| - **Self-assessment is unreliable**: C2 claims 30/30; Lean confirms 19, and only 1 of the 10 hard | |
| claims (§6). For 5 of the 11 unconfirmed claims C5 proves the statement, which is why FPR_B is | |
| an upper bound; cite the manual review once it is done. | |
| - **Cost**: C5 is cheaper per solved problem than C1 at nominal cost (Table 4); C3, without | |
| lemma suggestions, needs the most calls per solved problem. | |
| - **Errors**: tactic failures and syntax errors dominate; misformalization never appears; | |
| missing lemmas are common in algebra (Tables 7, 7b). C1's failures are mostly budget | |
| exhaustion, partly driven by truncated responses (Table 7c, §9.1). | |
| - Optional illustrative traces for an appendix are listed at the end of `results.md` | |
| ("Illustrative traces"; candidates in §5.5). | |
| ### Limitations | |
| Use §12. Essential: small n and low power; one run and one seed (pass@1 only); one | |
| general-purpose model with an unversioned alias; possible contamination of public benchmarks; | |
| FPR_B upper bound with the manual review pending; the formal-vs-informal caveat of | |
| `amc12b_2003_p6`; the shared pre-pass; the interrupted and resumed runs and the other deviations | |
| of §11 (the plan requires reporting them as deviations). | |
| ### Ethics | |
| - **Trust in AI-generated mathematics:** the central finding of Baseline B (§6) is that the | |
| model's confidence is not evidence of correctness; the system accepts a proof only when the | |
| Lean kernel does, and the report should say that LLM output should not be trusted without | |
| formal or human verification. | |
| - **Data:** only public benchmarks (miniF2F, PutnamBench) were used, with no personal data; cite | |
| their sources and licences, and mention the contamination risk (§12). | |
| - **Cost and resources:** Main experiments (dev + test, C1–C5): 7.36 USD. Including the | |
| exploratory Pro run: 9.30 USD in total; the project cap was raised from 9 to 12.5 USD for that | |
| run (see Deviations). A response cache avoided repeated calls; this keeps the work reproducible | |
| on a student budget. | |
| - **Transparency and reproducibility:** the analysis plan was written before the `test` runs, | |
| every deviation is logged (§11), post hoc analyses are labelled, and each run records its | |
| models, seed and Lean/Mathlib versions. | |
| ## 14. Exploratory: a stronger model on the hard problems | |
| *Post hoc (decided 2026-11-08, after all confirmatory runs), descriptive · LaTeX `tab:pro-hard`.* | |
| This section changes no result of §4–§12. It is logged in `analysis_plan.md` ("Exploratory | |
| analysis (post hoc, decided 2026-11-08)" and two Deviations entries of that date). | |
| **Why.** On the hard tier the agents add nothing with DeepSeek-Flash: C1 and C5 solve exactly the | |
| same 4 hard problems (§5.3). That suggests the hard problems are limited by the model, not by the | |
| method. To look at that, C5 was run again on the 10 hard `test` problems with a stronger model, | |
| DeepSeek-V4-Pro, in every role (profile `experiment_pro`); everything else (graph, prompts, | |
| budgets, seed) was the frozen C5 configuration. | |
| **What happened to the run.** The team stopped it after 5 of the 10 problems | |
| (`20261108T130803Z_C5_experiment_pro_pro_hard`) and it was not resumed. Three outcomes are | |
| **not failures of Pro** and are left out of its solved count: | |
| - `imo_1978_p5`: **provider_error (connection drop, not a system failure)**. The 30th call of the | |
| problem (the Autoformalizer on subgoal 1) never reached DeepSeek: all 5 transport attempts got | |
| `APIConnectionError: Connection error.` with no HTTP response (15:34:50–15:37:28 UTC). The next | |
| problem connected normally. | |
| - `imo_1984_p6`: **interrupted**, cut mid-problem by the stop (4 paid calls, no row). | |
| - `imo_1964_p1_2`, `putnam_1964_b2`, `putnam_1995_b1`, `putnam_1965_b2`: **not attempted**. | |
| Only the 4 problems that both runs finished are compared (`imo_1965_p2`, `imo_1979_p1`, | |
| `imo_1987_p6`, `imo_1993_p5`). | |
| ### Table 12. Exploratory: C5-Flash vs C5-Pro on the hard problems (post hoc, descriptive, not in the analysis plan) | |
| | Problem | Flash solved | Flash calls | Flash USD | Flash min | Pro outcome | Pro solved | Pro calls | Pro USD | Pro min | | |
| |---|---|---|---|---|---|---|---|---|---| | |
| | imo_1964_p1_2 | 1 | 15 | 0.0261 | 3.4 | not attempted | — | — | — | — | | |
| | imo_1965_p2 | 1 | 14 | 0.0312 | 3.7 | failed | 0 | 30 | 0.3608 | 33.6 | | |
| | imo_1978_p5 | 0 | 31 | 0.1347 | 39.6 | provider_error (connection drop, not a system failure) | — | 30 | 0.2938 | 40.7 | | |
| | imo_1979_p1 | 0 | 28 | 0.1463 | 19.7 | failed | 0 | 24 | 0.2722 | 29.3 | | |
| | imo_1984_p6 | 0 | 32 | 0.3366 | 27.6 | interrupted | — | — | — | — | | |
| | imo_1987_p6 | 0 | 36 | 0.1793 | 21.4 | budget_exhausted | 0 | 40 | 0.4607 | 57.4 | | |
| | imo_1993_p5 | 0 | 27 | 0.0790 | 10.6 | failed | 0 | 24 | 0.3807 | 53.6 | | |
| | putnam_1964_b2 | 1 | 9 | 0.0160 | 2.3 | not attempted | — | — | — | — | | |
| | putnam_1965_b2 | 1 | 17 | 0.0409 | 4.9 | not attempted | — | — | — | — | | |
| | putnam_1995_b1 | 0 | 25 | 0.1268 | 17.5 | not attempted | — | — | — | — | | |
| | Both finished (4) | 1/4 | 105 | 0.4357 | 55.5 | | 0/4 | 118 | 1.4744 | 173.9 | | |
| | All hard (10) | 4/10 | 234 | 1.1169 | 150.9 | | — | — | — | — | | |
| *Note.* Post hoc, exploratory (decided 2026-11-08, after all confirmatory runs): descriptive only, no tests, no intervals; it changes no main result. C5-Flash = the kept C5 rows of the main test result; C5-Pro = run 20261108T130803Z_C5_experiment_pro_pro_hard (profile experiment_pro: deepseek-v4-pro in every role, everything else as frozen), stopped by the team and not resumed. Cost is nominal USD (cache hits priced as fresh calls). provider_error, interrupted and not attempted do not count as failures of Pro and are left out of its solved count; only problems both runs finished are compared. imo_1984_p6 was interrupted after 4 paid calls (0.1725 USD real, 18.1 min) and left no row. Real cost of the Pro run: 1.9407 USD (151 paid calls, all in the ledger); project ledger total: 9.3003 USD. | |
| In plain words: on the 4 problems both runs finished, Pro solved none and Flash solved one | |
| (`imo_1965_p2`, which Pro failed after 30 calls). Pro spent about 3.4 times as much (1.47 vs | |
| 0.44 USD nominal) and took about 3.1 times as long (174 vs 56 min) on those 4. This does **not** | |
| show that a stronger model fails to help: only 4 of the 10 problems were compared, each model | |
| ran once, and 3 of the 4 hard problems Flash solved were never attempted by Pro. The question of | |
| whether the hard tier is limited by the model stays open; the report can mention the run as an | |
| exploratory, incomplete check and should not draw a conclusion from it. | |
| **Cost.** The Pro run cost 1.9407 USD real (151 paid calls, all off-peak). With it, the project | |
| ledger stands at 9.3003 USD; for this run the caps were raised to 12.5 USD in total and 5 USD per | |
| run (Deviations, 2026-11-08). | |