releases/dgx-research-20260912-v2/provenance/v1/README.md

metadata

license: mit
language:
  - en
pretty_name: Liars Are Information — Byzantine-robust LLM swarm artifact
size_categories:
  - 10K<n<100K
task_categories:
  - question-answering
tags:
  - byzantine-fault-tolerance
  - multi-agent
  - llm-agents
  - adversarial-robustness
  - error-correlation
  - reproducibility
  - research-artifact
configs:
  - config_name: cache_greedy
    description: Honest broadcasts, greedy decoding. One row per (task, model).
    data_files:
      - split: train
        path: data/cache/*/*.parquet
  - config_name: cache_resample
    description: >-
      Second sample of the same tasks at T=0.7, for within-model error
      correlation.
    data_files:
      - split: train
        path: data/cache_t07/*/*.parquet
  - config_name: cache_adversarial
    description: Prompt-injected adversarial broadcasts over the same task ids.
    data_files:
      - split: train
        path: data/cache_adversarial/*/*/*.parquet
  - config_name: sweep_symbolic
    description: Phase 3 aggregation sweep against a symbolic coherent adversary.
    data_files:
      - split: train
        path: results/aggregation/sweep.parquet
  - config_name: sweep_adversarial
    description: Phase 4 sweep against seven real-LLM attack classes.
    data_files:
      - split: train
        path: results/adversarial/adversarial_sweep.parquet
  - config_name: gate_aware
    description: The gate-aware adversary and the three pre-registered mitigations.
    data_files:
      - split: train
        path: results/adversarial/gate_aware_mitigations.parquet
  - config_name: correlation
    description: Pairwise within- and cross-model error correlation.
    data_files:
      - split: train
        path: results/correlation/pairwise_estimates.parquet

Liars Are Information — DGX verification release

This release audits the original AIP artifact and adds a reproducible, low-resource multi-agent replay study. It contains corrected code, three explicit contribution groups, saved simulations, statistical comparisons, and measured DGX resource use. Honest model outputs are replayed from existing caches; no new LLM inference was performed.

Start with the measured replay report, the cache audit, and the three-contribution specification.

Contribution	Concrete deliverable	Evidence
1. Inversion mechanism and fair comparison	Receiver-local inversion ablations and a full confusion-matrix Dawid–Skene baseline that predicts unseen tasks	Paired comparisons, baseline code
2. Adaptive attack boundary	Corrected adversary-coherence law, qualified mathematical claims, validation-selected attacks against each defender	Selected attacks, corrected note
3. Reproducible DGX evaluation	Fixed task splits, three swarm seeds, partial observation, diversity controls, cache analysis, saved traces, bounded execution	Run manifest, resource receipt, agent review

These are three contribution groups in this verification release. The original claims-ledger identifiers C1/C2/C3 retain their existing meanings. See the contribution document for assumptions and limits; benchmark superiority and general robustness are not implied by the grouping.

What ran

The canonical run completed in 75.76 seconds, with 504.20 MiB peak sampled RSS, two-core affinity, one computational thread, and no GPU inference. The full test suite passed 516 tests. Two fresh smoke processes produced identical scientific records and traces; see reproducibility.json.

At 50% coherent Byzantine agents and full visibility, mean AIP test accuracy was 89.9% on MMLU, 48.8% on MATH-500, and 0% on BoolQ. The corresponding honest receiver-only accuracies were 83.7%, 58.6%, and 89.3%. These results support a conditional MMLU gain and also show substantial failure regimes; they do not support universal majority robustness. Paired intervals and analysis.

At 70% Byzantine agents, the validation-selected stationary gate-aware attack reduced MMLU hard-gated AIP test accuracy to 0%. This empirical failure is part of Contribution 2, alongside the corrected mathematical account.

The standard profile contains 144 simulation worlds: 72 mechanism/visibility conditions, 12 homogeneous replay controls, and 60 stationary adaptive-attack conditions. Each world has ten agent identities. The mixed population uses the seven frozen model identities; the cache audit separately covers all nine cached models across six benchmarks. The replay evaluates MMLU, BoolQ, and MATH-500.

Each benchmark supplies 200 cached tasks, split into 80 history, 40 validation, and 80 test tasks. Each receiver fits only its own history. Byzantine identities, gold labels, model names, and attack labels are withheld from the defense. Paired confidence intervals resample tasks after averaging swarm seeds. Receivers and repeated cache copies do not increase the benchmark sample size.

MATH-500 uses the repository's approximate math-equivalence scorer. Equivalent answer strings are excluded from the symbolic wrong-answer pool. Exact-string accuracy is also saved as a sensitivity column. Aggregation still uses the original answer strings, so semantic agreement is not an input oracle.

Historical calibration overlaps some cached task lists. The new split holds out tasks from new channel fitting and attack selection, but cannot establish independence from prior threshold calibration. Adaptive candidates each have their own stationary attack-specific history; they do not test a changing attacker against one shared pretrained state. Binary inversion, open-answer generality, correlated replicas, and heuristic gate calibration remain material limits.

The additional analysis report includes paired uncertainty, gate decisions, scoring sensitivity, and composition comparisons.

Run with low resources

Python 3.12 was used on an aarch64 DGX with NVIDIA GB10. The offline environment has no PyTorch, vLLM, or model-weight requirement. The lock file records the actual scientific Python environment.

python3 -m venv .venv
.venv/bin/python -m pip install -r requirements-dgx.lock.txt
PYTHONPATH=src .venv/bin/python -m pytest -q
.venv/bin/python scripts/dgx_run_bounded.py --profile smoke --out results/dgx/my_smoke --timeout 120
.venv/bin/python scripts/dgx_run_bounded.py --profile standard --out results/dgx/my_standard --timeout 600
.venv/bin/python scripts/dgx_cache_audit.py --out results/dgx/my_cache_audit --resamples 200 --seed 20260912

Choose a new output directory for every replay. The launcher pins two available CPU cores, sets one native computational thread, uses nice 10, hides CUDA, and monitors a 2 GiB RSS budget and wall timeout. Both a completed run manifest and a successful resource receipt establish completion. RSS monitoring is sampled, not a hard memory reservation.

Download from Hugging Face

The supplied location is a Hugging Face storage bucket, which has no Git commit history or dataset viewer. Use the bucket API/CLI rather than git clone or datasets.load_dataset on the bucket address. Hugging Face bucket documentation.

hf buckets sync hf://buckets/Nabidnur/Liars_Are_Information-bucket/releases/dgx-verification-20260912-v1 ./aip-dgx

The dated release is additive and leaves the original bucket objects intact. Its file manifest fingerprints the release. The scientific output files contain no authentication credential.

Historical artifacts and corrections

README_UPSTREAM.md, the original result tables, and the historical manuscript preserve the source artifact. They describe earlier H100 runs and have not been revalidated in full by this DGX replay. In particular, the historical gate-aware theoretical q axis is incorrect for the implemented sampler; claims that rely on that axis or infer indistinguishability from mean q alone require revision. Historical paper integrity tests verify structure and macro consistency, not scientific correctness.

The correct expected pair collision for the shared/uniform-wrong-pool mixture is q = p² + (1 − p²)/K, where K is the actual wrong-candidate pool size. Pair collision differs from the receiver-conditioned gate statistic. Dawid–Skene can model systematic anti-expertise and should not be characterized as simply discarding liars.

The release also fixes static AIP task alignment under partial visibility, process-stable threshold randomness, jitter ordering, and roster truncation. The old windowed path and broad original sweeps are outside the new validation claim. Independent work and review record.

License and provenance

The original authors' LICENSE is preserved. It distinguishes the code and derived artifacts from benchmark corpora and model outputs, whose upstream terms still apply. This release claims no new authorship of the original algorithm, caches, or manuscript.