RUNLOG.md

RUNLOG

One line per completed step, appended by the drivers themselves so it stays true without a live session. Format: TIMESTAMP | PHASE | STEP | RESULT | COMMIT.

Pre-committed decision rules in force (applied without asking, logged here):

• R1 CEILING — cells with honest accuracy >= 0.93 at f=0 are HEADROOM-LOW: kept in cache, excluded from headline figures, retained in appendix tables. Headline benchmarks: math500, medqa, mmlu, arc. GSM8K/BoolQ headline claims restricted to weak/mid tiers.
• R2 AUC POWER — confidence AUC interpreted only where n_wrong >= 30; otherwise underpowered (n=..) with no direction-of-effect language.
• R3 SELF-REPORT — reasoning models failing to emit a self-reported number on multiple choice is a FINDING, not a prompt bug. Never patched.
• R4 BUDGET — 10 GPU-hours remaining cap. Cut order: T=0.7 stragglers -> big-3 on non-headline benchmarks -> arc adversarial cells.
• R5 ATTACK PRIORITY — math500 > medqa > mmlu > arc; gsm8k/boolq only where headroom exists.
• R6 LOGGING — this file.

Hard stops: (1) a headline claim fails; (2) a budget cap is exceeded after the cut order; (3) tests red after one fix attempt.

timestamp	phase	step	result	commit
2026-09-02T10:15Z	0	env rebuild + roster preflight	PASS 7/7 models fit, ungated	3008b87
2026-09-02T10:20Z	0	roster revision -> 7 models / 7 labs	PASS; FP8 + base-model traps caught	9deb2df
2026-09-02T10:28Z	1a	boolq/medqa/arc + 1500-task list	PASS 401 tests	51e1f6f
2026-09-02T10:57Z	1	multimodal media budget = 0	FIX ministral3_14b loads	1deeae1
2026-09-02T11:20Z	1	qwen38 determinism 1/30 -> KEEP	RULE applied; gate d now binding	174ec6c
2026-09-02T11:25Z	1	T=0.7 driver, all 7, resample_kind	PASS backfilled 18 cells	9f2f300
2026-09-02T11:35Z	1	swarm-headroom column	PASS	8ad65ab
2026-09-02T17:14Z	1	gemma4 per-layer geometry override	FIX loads 58.9GiB, generates	60510f4
2026-09-02T17:23Z	2	derive benchmark roster from config	FIX 4 scripts, chance-q bug caught	18c5b8a
2026-09-02T17:25Z	1	gate report generator (R1+R2 encoded)	PASS 30/42 cells so far	60a58f5
2026-09-02T17:32Z	phase1_tail	smoke	ok in 0.0m	-
2026-09-02T17:32Z	phase2	smoke	ok in 0.0m	-
2026-09-02T17:32Z	phase3	smoke	ok in 0.0m	-
2026-09-02T17:32Z	phase4	smoke	ok in 0.0m	-
2026-09-02T17:32Z	phase5	SKIPPED (needs agent)	E1/E2 and the LAACS-style reputation-decay defender are not implemented yet	-
2026-09-02T17:32Z	phase6	smoke	ok in 0.0m	-
2026-09-02T17:32Z	run_all	COMPLETE_WITH_SKIPS	5 ran, 1 skipped	8dc6e79
2026-09-02T17:33Z	phase2	skip	already complete (stamp)	-
2026-09-02T17:38Z	phase1_tail	smoke	ok in 0.0m	-
2026-09-02T17:38Z	phase2	smoke	ok in 0.0m	-
2026-09-02T17:38Z	phase3	smoke	ok in 0.0m	-
2026-09-02T17:38Z	phase4	smoke	ok in 0.0m	-
2026-09-02T17:38Z	phase5	SKIPPED (needs agent)	E1/E2 and the LAACS-style reputation-decay defender are not implemented yet	-
2026-09-02T17:38Z	phase6	smoke	ok in 0.0m	-
2026-09-02T17:38Z	run_all	COMPLETE_WITH_SKIPS	5 ran, 1 skipped	6539875
2026-09-02T17:39Z	phase1_tail	FAILED	exit 1 from: /home/ubuntu/Liars_Are_Information/scripts/run_phase1.sh olmo3_32b_think after 0.3m	-
[2026-09-02T17:39Z] phase1_tail: Phase 1 tail: olmo3 cache, T=0.7 second pass, gate report
STOPPED at phase1_tail: exit 1 from: /home/ubuntu/Liars_Are_Information/scripts/run_phase1.sh olmo3_32b_think (log: /home/ubuntu/Liars_Are_Information/results/logs/phase1_tail.log)
2026-09-02T19:59Z	phase1_tail	FAILED	exit 1 from: bash /home/ubuntu/Liars_Are_Information/scripts/run_phase1_t07.sh after 100.3m	-
[2026-09-02T18:19Z] phase1_tail: Phase 1 tail: olmo3 cache, T=0.7 second pass, gate report
STOPPED at phase1_tail: exit 1 from: bash /home/ubuntu/Liars_Are_Information/scripts/run_phase1_t07.sh (log: /home/ubuntu/Liars_Are_Information/results/logs/phase1_tail.log)
2026-09-02T20:53Z	phase1_tail	complete	ok in 41.0m	c6187d3
2026-09-02T21:34Z	phase2	complete	ok in 41.4m	0b0b621
2026-09-02T21:42Z	phase3	FAILED	exit 1 from: /home/ubuntu/Liars_Are_Information/.venv/bin/python /home/ubuntu/Liars_Are_Information/scripts/phase_c_aggregate.py after 7.7m	-
[2026-09-02T20:12Z] phase1_tail: Phase 1 tail: olmo3 cache, T=0.7 second pass, gate report

ok in 41.0m  c6187d3

[2026-09-02T20:53Z] phase2: Phase 2: correlation, multiclass law, honest-q v3 recalibration ok in 41.4m 0b0b621 [2026-09-02T21:34Z] phase3: Phase 3: aggregation sweep, N-scaling, parity audit, minimax regret STOPPED at phase3: exit 1 from: /home/ubuntu/Liars_Are_Information/.venv/bin/python /home/ubuntu/Liars_Are_Information/scripts/phase_c_aggregate.py (log: /home/ubuntu/Liars_Are_Information/results/logs/phase3.log) | 2026-09-03T03:03Z | phase3 | complete | ok in 56.0m | 397fdb4 | | 2026-09-03T03:23Z | phase4 | FAILED | exit 1 from: /home/ubuntu/Liars_Are_Information/.venv/bin/python /home/ubuntu/Liars_Are_Information/scripts/phase_d_figures.py after 20.3m | - | [2026-09-03T02:07Z] phase3: Phase 3: aggregation sweep, N-scaling, parity audit, minimax regret ok in 56.0m 397fdb4 [2026-09-03T03:03Z] phase4: Phase 4: real-LLM adversaries, pilot gate then full sweep STOPPED at phase4: exit 1 from: /home/ubuntu/Liars_Are_Information/.venv/bin/python /home/ubuntu/Liars_Are_Information/scripts/phase_d_figures.py (log: /home/ubuntu/Liars_Are_Information/results/logs/phase4.log) | 2026-09-03T06:06Z | phase2 | complete | ok in 63.0m | 823c1ac | | 2026-09-03T06:06Z | phase3 | skip | already complete (stamp) | - | | 2026-09-03T06:06Z | phase4 | FAILED | exit 1 from: /home/ubuntu/Liars_Are_Information/.venv/bin/python /home/ubuntu/Liars_Are_Information/scripts/phase_d_adversarial.py after 0.0m | - | [2026-09-03T05:03Z] phase2: Phase 2: correlation, multiclass law, honest-q v3 recalibration ok in 63.0m 823c1ac skip phase3 (already complete) [2026-09-03T06:06Z] phase4: Phase 4: real-LLM adversaries, pilot gate then full sweep STOPPED at phase4: exit 1 from: /home/ubuntu/Liars_Are_Information/.venv/bin/python /home/ubuntu/Liars_Are_Information/scripts/phase_d_adversarial.py (log: /home/ubuntu/Liars_Are_Information/results/logs/phase4.log) | 2026-09-03T08:41Z | phase4 | complete | ok in 101.0m | 4d67445 | | 2026-09-03T08:41Z | phase5 | SKIPPED (needs agent) | E1/E2 and the LAACS-style reputation-decay defender are not implemented yet | - | | 2026-09-03T08:41Z | phase6 | FAILED | exit 1 from: /home/ubuntu/Liars_Are_Information/.venv/bin/python /home/ubuntu/Liars_Are_Information/scripts/phase_e_claims.py after 0.0m | - | [2026-09-03T07:00Z] phase4: Phase 4: real-LLM adversaries, pilot gate then full sweep ok in 101.0m 4d67445 SKIP phase5: E1/E2 and the LAACS-style reputation-decay defender are not implemented yet [2026-09-03T08:41Z] phase6: Phase 6: ledger v2, numbers.tex, claims map, CI checks STOPPED at phase6: exit 1 from: /home/ubuntu/Liars_Are_Information/.venv/bin/python /home/ubuntu/Liars_Are_Information/scripts/phase_e_claims.py (log: /home/ubuntu/Liars_Are_Information/results/logs/phase6.log) | 2026-09-03T11:10Z | phase4 | complete | ok in 76.2m | 15b1d80 | | 2026-09-03T11:10Z | phase5 | SKIPPED (needs agent) | E1/E2 and the LAACS-style reputation-decay defender are not implemented yet | - | | 2026-09-03T11:12Z | phase6 | FAILED | exit 1 from: /home/ubuntu/Liars_Are_Information/.venv/bin/python /home/ubuntu/Liars_Are_Information/scripts/phase_e_tables.py after 2.0m | - | [2026-09-03T09:54Z] phase4: Phase 4: real-LLM adversaries, pilot gate then full sweep ok in 76.2m 15b1d80 SKIP phase5: E1/E2 and the LAACS-style reputation-decay defender are not implemented yet [2026-09-03T11:10Z] phase6: Phase 6: ledger v2, numbers.tex, claims map, CI checks STOPPED at phase6: exit 1 from: /home/ubuntu/Liars_Are_Information/.venv/bin/python /home/ubuntu/Liars_Are_Information/scripts/phase_e_tables.py (log: /home/ubuntu/Liars_Are_Information/results/logs/phase6.log) | 2026-09-03T12:06Z | phase5 | complete | ok in 1.4m | 3e0dc74 | | 2026-09-03T12:08Z | phase6 | FAILED | exit 1 from: /home/ubuntu/Liars_Are_Information/.venv/bin/python /home/ubuntu/Liars_Are_Information/scripts/make_numbers.py after 2.0m | - | [2026-09-03T12:05Z] phase5: Phase 5: sleeper (E1) and Sybil (E2) ok in 1.4m 3e0dc74 [2026-09-03T12:06Z] phase6: Phase 6: ledger v2, numbers.tex, claims map, CI checks STOPPED at phase6: exit 1 from: /home/ubuntu/Liars_Are_Information/.venv/bin/python /home/ubuntu/Liars_Are_Information/scripts/make_numbers.py (log: /home/ubuntu/Liars_Are_Information/results/logs/phase6.log) | 2026-09-03T16:22Z | phase6 | complete | ok in 2.0m | 3482175 | | 2026-09-03T16:22Z | run_all | COMPLETE | 1 ran, 0 skipped | 3482175 | [2026-09-03T16:20Z] phase6: Phase 6: ledger v2, numbers.tex, claims map, CI checks ok in 2.0m 3482175 COMPLETE: 1 ran, 0 skipped

2026-09-04 — Task 6: the pending-claims audit

scripts/task6_audit_pending.py re-derives H2, F1, B1, G2 and M2 from their parquets and compares every effect against the per-benchmark measurement floor. results/claims.md no longer has a pending section, and tests/test_paper.py now fails if one comes back.

id	outcome
H2	split three ways. True only for the symbolic adversary and deployable methods. Oracle Krum holds 0.815 on BoolQ; under real-LLM adversaries nothing reaches zero and AIP loses to majority vote on BoolQ (−0.290).
F1	CONFIRMED. AIP moves by exactly 0.000 (identity: the gate never reads the self-report); the self-report reader loses 0.104–0.234 on 4 of 6.
B1	CONFIRMED. Pooled 0.470 → windowed 0.965 at f=0.7, 7.3× floor, undetectable cost on the stationary control.
G2	REFUTED as worded — coordinatability is highest on closed label spaces (binary 0.557 vs open text 0.069). Restated as G2′: the margin over honest agreement is what collapses.
M2	split. The confound is real (discard family exactly parity-invariant over 3024 cells); "AIP does not exploit it" is REFUTED at f=0.5 (25/54 cells over floor, max 0.840) and undetectable elsewhere.

Also found and fixed while auditing: four L4-era constants survived in paper/numbers.tex where no test could see them — 4 models, 100 tasks per benchmark, "three benchmarks", "3–9B". All four are now derived from configs/models.yaml (new arm: field), configs/task_lists.json and configs/decision_rules.yaml. Ledger U2 corrected from 11 models to 9.

2026-09-04 — Task 7: the evasion story re-anchored at f = 0.5

New figure paper/figures/evasion_band.pdf (scripts/fig_evasion_band.py): three panels at f = 0.5 only, markers with no interpolation, honest-coherence rug with underpowered pairs drawn separately, the chance line 1/(C−1) and the calibrated ceiling both marked, M1–M3 on the same axes.

Numbers moved to the anchor:

quantity	was	now (f = 0.5)
attacker gain, MedQA	0.92 (f = 0.7, confounded)	0.796 — 8.0× floor
attacker gain, MMLU	—	0.529 — 5.3× floor
M1 gain, evasion band	0.350 → 0.563 (all f)	0.463 → 0.623 (+0.160, 1.6× floor)
M1 gain, whole attack space	—	0.457 → 0.557 (+0.100, exactly at the floor)
M2 vs hard gate	"actively harmful", −0.059	−0.013 whole space, +0.011 in band — UNDETECTABLE both ways
N_eff best heterogeneous	2.31	2.46, measured basis only

Two corrections the floor rule forced. The old D2 verdict called M2 "actively harmful" on a −0.059 difference that was itself below the 0.100 floor; it is now UNDETECTABLE in both directions. The old numNeffHetMax was 6.86 — an upper bound computed from cross-model ρ alone on ARC, a demoted benchmark, quoted beside homogeneous numbers computed the other way. The macros are now restricted to the measured within-model basis and the upper bound is labelled as one.

\numEvasionAnchorF is now a macro and a test asserts it is 0.5, so the f = 0.7 anchor cannot come back silently.

2026-09-04 — Task 8: full prose

Twelve numbered sections plus one appendix, replacing the scaffold. mechanism.tex folded into §3, conclusion.tex into §12; three new sections written from nothing (§5 measurement, §7 real-LLM adversaries, §8 gate-aware, §9 limits, §10 prior art revisited). ~10,200 words of prose, 231-word abstract.

New artifacts: paper/figures/sybil_accuracy.pdf (scripts/fig_sybil.py) and the attack-class table in §3. All nine cited figures exist; no dangling \ref.

Writing the prose surfaced two more mismatched-basis comparisons, both now fixed in the ledger and the macros: the weak-arm comparison (I3, six benchmarks against three) and the H1 fall (no reproducible slice). Both are recorded in §11 as refutations rather than dropped.

Test counts, stated exactly. The default pytest invocation carries -m "not slow", so it runs 377 of the 424 collected tests; the commit messages for Tasks 6-8 quote the collected total. The 47 slow tests (pytest -m slow) pass as well. No GPU-marked test was run in this pass.

2026-09-04 — Hugging Face packaging, and an L4 leak the packaging found

Preparing the artifact for the Hub surfaced a RULE 1 violation that had been live the whole regeneration: seven of the manuscript's figures were L4-era PDFs from the initial upload. The generators defaulted to figures/ while LaTeX read paper/figures/, so the two directories drifted and nothing detected it — the files existed and the references resolved.

Fixed structurally, not by copying:

• one figure directory, paper/figures/, which every generator now writes to;
• src/aip/analysis/provenance.py stamps paper/figures/MANIFEST.json with the script, sources, git commit and timestamp for each figure;
• tests/test_paper.py::TestFigureProvenance fails if a cited figure is missing, unstamped, or if a second figures directory reappears;
• run_all.py phase 6 now regenerates every cited figure.

Two generators had to be repaired before they could produce anything: phase_b_correlation.py globbed the cache directory for models and crashed on the weak-arm models, which have no T=0.7 resample — it now reads the frozen roster from configs/models.yaml (arm: frozen) and fails loudly if one is missing; phase_c_figures.py hardcoded subplots(1, 3) against a six-benchmark list — the tenth instance of the three-benchmark assumption — and aborted before writing the gating-ablation figure. The regenerated correlation parquets are byte-identical, so the roster restriction changed no result.

Hub packaging: README.md is now the dataset card (YAML front matter with seven viewer configs), .gitattributes declares LFS patterns, LICENSE states what MIT does and does not cover, scripts/hf_push.py uploads with a token-shape check and a completeness refusal, and tests/test_readme.py checks the card parses, every viewer config resolves to real files, every path named exists, and 27 pinned numbers still match paper/numbers.tex. STATUS.json and FINAL_REPORT.md moved into results/; data/benchmarks/ is excluded from the upload (ARC and BoolQ are CC BY-SA) and the offline chain was verified to run with that directory removed.

2026-09-04 — Task 9 and the paper restructure

9a, cost of defence (scripts/phase_e3_cost.py). Not recoverable from the phase logs: those carry one timestamp per benchmark-composition block covering all fourteen rules, so no per-method figure can be extracted and a block average would have been a fabrication. Measured directly by timing each aggregator over the cache. µs per receiver-decision at f=0.5: majority 3.2, krum 20.9, trimmed mean 68.5, coord median 75.6, AIP trust-only 77.8, AIP-gated 79.6, multi-Krum 92.9, geometric median 671.6. The gate adds +1.7 µs (2.2%) over its own inversion-free ablation — almost all of AIP's cost is channel bookkeeping the ablation also pays. Generation is ≈5073× the slowest rule.

9b, composition (scripts/phase_composition.py). All 35 three-model and 21 five-model subsets, with a new subset_swarm_rho_bar generalising the two-model helper over the real pair census, and every distinct assignment of the surplus agent enumerated and averaged — a 4/3/3 split lets the heavier model win any three-way disagreement, so allocating the surplus by name would have made the answer depend on alphabetical order.

Result: the intended framing is refuted. N_eff correlates −0.958 with mean roster accuracy (it is close to a restatement of roster weakness) and −0.702 with swarm accuracy — the wrong sign for a design objective. What survives partialling out the mean is accuracy spread (partial ≈ +0.7). And no composition beats its own best single member detectably on any benchmark; on math500 every subset is worse by ≥0.097. Family diversity cannot be assessed: all 7 models are 7 lineages, so the predictor is constant.

Protocol finding (scripts/phase_confidence_auc.py). The cited AUC 0.358 has no counterpart in this regime — the only 0.358 in the tree is a correlation CI bound in phase2.log — so it was recomputed. Frozen roster, 22 of 42 cells powered under R2: logprob 0.724, self-report 0.661, self-report falsified 0.339, below chance on 21 of 22 cells. Close to the L4 figure, but now measured here. R3 also bites: the self-report is absent on up to 34.0% of rows.

Paper. Thirteen sections. New §5 "Swarm-level design laws" (three laws, 9b table), new §7 "The confidence protocol", 9a as a table in §6. Title now reads "…in Multi-Agent LLM Swarms, and Its Limit".

Deterrence removed entirely. The condition fixed before the audit was that it enters only if G2 passed Task 6 re-verification. G2 did not pass — refuted as worded, restated as G2′. So the section, the figure, the forward reference in related work and the claim are all gone; the bandit data stays in the release and is cited nowhere. If the intent was "if G2′ holds" rather than "if G2 passes", say so and it comes back in one commit.

2026-09-04 — published to the Hugging Face Hub

Dhruv1000/liars-are-information, dataset repo, private. 576 files, 42.1 MB, verified file-by-file against the local plan with nothing missing. Pushed with --include-benchmark-cache, so the frozen task items ship too; the README and LICENSE were corrected first, because both claimed that directory was not redistributed and that would have been a false licence statement on the page (ARC and BoolQ are CC BY-SA and are not covered by this repository's MIT).

Two problems surfaced during the push, both now fixed at the source.

A bare *.log in .gitignore was hiding every GPU run log. Unanchored LaTeX build patterns excluded all 13 phase logs — the raw provenance of ~10 GPU-hours — and none had ever been committed. The Hub honours .gitignore, so it accepted the commit and discarded the paths: the files 404'd while the commit title said they were added. Patterns are now anchored to paper/, the logs are tracked, and hf_push.py publishes results/logs/.

The .gitattributes written for the Hub had already destroyed the working tree (see the entry above): *.parquet filter=lfs turned all 143 parquets into 131-byte pointers and every analysis script failed with an Arrow "magic bytes" error naming no cause. Restored from .git/lfs/objects, LFS dropped entirely, and tests/test_artifact.py now fails if a parquet is a pointer or if such a filter is declared again.

The repo is private. --private off, or the Hub's settings page, makes it public; the manuscript still says "affiliation withheld for review", so that is the author's call and not a default.

2026-09-04 — §5 thesis reframing, and the manuscript compiles

§5 reframed around one thesis, now also in the abstract and the introduction: an LLM swarm's ceiling on honest error is correlation, not competence. Law 1 (replication is not diversity), S1 as a negative result (no composition beats its best single member; MATH-500 every subset ≥0.097 worse), and the honest attractors together say honest pooling is capped. The consequence is the paper's spine: the only structure in the broadcast set that honest agents do not already share is adversarial coherence, so a disciplined liar is a swarm's one informative contributor and inversion is the mechanism that spends it. This is why the positive result is about robustness rather than accuracy, and why an adversary that declines to coordinate defeats the mechanism by leaving it nothing to spend. The 9b table stays in §5.

Compiled with tectonic 0.15.0 (single binary, no TeX Live). 17 pages. 0 undefined references, 0 undefined citations, 0 multiply-defined labels, 0 overfull boxes, 4 underfull boxes (loose lines in narrow columns).

Three build problems fixed on the way:

• IEEEtran asks for Times in T1; tectonic runs XeTeX whose TU encoding has no ptm, so every Times shape fell back to Computer Modern — 170 font warnings and a page count inflated by the wider face. Added fontenc+newtx.
• newtxmath and amssymb both define \Bbbk, which halted the build with "No pages of output". amssymb/amsfonts dropped; newtxmath supplies them.
• One 6.9pt overfull box in the attack-class table, fixed by shrinking the type and abbreviating the widest cell rather than letting it hang into the gutter.

Five font warnings remain and are not fixable from the preamble: IEEEtran.cls probes ptm at class-load time, before any \DeclareFontShape can exist. The aliases added do resolve every later request, so body text is set in Times.

paper/main.pdf is now committed and shipped — a reader should not need a LaTeX toolchain to see what the numbers render into. paper/TODO_BIB.md exports the placeholder bibliography: 12 of 16 entries, 10 of them cited.

2026-09-11 — synced to the public model repo

Dhruv1000/Liars_Are_Information (model repo, public) now carries the full artifact: 576 files, 42.4 MB, verified file-by-file with nothing missing and nothing stale.

That repo already existed and was five commits behind — its last content push was 2026-09-04, so it had none of Task 9, the §5 repositioning, the 12-page cut or the verified bibliography. Two pruning passes were needed because upload_folder only adds and overwrites:

• four merged-away section files (protocol.tex never reached it, but real_adversaries.tex, prior_art_revisited.tex, measurement.tex and appendix_indistinguishability.tex were sitting beside the sections that absorbed them);
• 32 files that are ignored locally but had been pushed earlier — data/cache_dryrun/* (smoke-test output) and results/.stamps/*.done. The stamps mattered more than tidiness: in a clone they tell run_all.py that every phase is already complete.

scripts/hf_push.py gained --repo-type and --prune for this. Prune deletes anything remote that is not in the local plan, including ignored paths, since a file excluded from the artifact should not persist on the Hub.

The separate private dataset repo Dhruv1000/liars-are-information still exists and is now the older of the two.