Zeno Fragility Beta

An experimental fragility score: which confident forecasts look most likely to break.

Most forecasting tools add one more forecast. Zeno does something different: it is a model of forecasters. It watches existing forecasters — weather models, epidemic hubs, economists, markets, AI forecasters — and learns when a confident forecast is about to break, by comparing it with how similar forecasts failed before. Surprise is always measured against a named reference: the crowd, the forecaster's own past, or its peers.

This is our first public release: a compact, fast, explainable model that runs on any laptop (no GPU), built to grow as more forecast history arrives.

Highlights

Prediction markets: the 10% of confident forecasts Zeno flags as most fragile fail 1.3× more often than the average confident forecast (warning score 0.67 vs 0.37 for distance-from-crowd).
Crypto: the 10% of confident forecasts Zeno flags as most fragile fail 1.8× more often than the average confident forecast (warning score 0.57 vs 0.44 for distance-from-crowd).
RSV: the 10% of confident forecasts Zeno flags as most fragile fail 2.3× more often than the average confident forecast (warning score 0.64 vs 0.54 for distance-from-crowd).
Planted-signal tests pass: Zeno finds a hidden skilled forecaster (+7.5%), corrects an overconfident crowd (+4.6%), and correctly does nothing when there is no signal.
Blind spots made visible: in physical sensing, 21–24 weather models behave like only about 4 independent forecasters — exactly the shared blind spots where confident failures come from.
Built on 2.0M forecast records across 9 sources, with data frozen and fingerprinted before training, time-ordered splits and leak checks.

Fragility warning by source

Score from 0 to 1 = how well Zeno ranks confident forecasts that later fail (0.5 is a coin flip). "≥" is the low end of the 90% likely range. "Top-10% lift" = how much more often the most-flagged forecasts fail than the average confident forecast.

Source	Zeno warning	Crowd-distance rule	Top-10% lift	Status
Prediction markets	0.67 (≥ 0.60)	0.37	1.3×	ranks better than both simple rules
Crypto	0.57 (≥ 0.55)	0.44	1.8×	beats the crowd-distance rule
RSV	0.64 (≥ 0.59)	0.54	2.3×	beats the crowd-distance rule
ECB economist surveys	0.41 (≥ 0.38)	0.18	0.3×	not better than chance yet
ForecastBench questions	0.82 (≥ 0.56)	0.70	5.2×	candidate

Weather, flu and COVID join in the next release, now that their peer-relative labels are available.

Crowd corrector (served only where it beats the plain average on unseen confirmation data)

Source	Gain over the plain crowd average
Physical sensors (air, urban, sea, agriculture)	+18.6% (90% range +15.5% to +22.2%)

A safety rule keeps the plain crowd average for every other source until the corrector beats it on both test and confirmation data.

Model details


Type	Per-source gradient-boosted trees + logistic fallback (scikit-learn)
Parameters	62,880 learned numbers (tree split thresholds + leaf values + logistic weights, 6 sources) — stored in `model.safetensors`
Formats	ONNX (`onnx/<source>.onnx`, verified identical to the original to 1e-7), safetensors, scikit-learn pickle
Inputs (6)	crowd distance, long and recent track record vs crowd, crowd spread, crowd size, own confidence
Output	experimental fragility score, 0–1 (higher = more fragile; a ranking, not a calibrated probability)
Fitting rows	38,459 confident forecasts (served models)
Training data	Offdiagonal forecast corpus, 2.0M records, 9 sources, frozen snapshot `ep-20261001T0533Z-v035`
Run	`v038-full-20261001T1537Z` (CPU only)

Download and use

pip install huggingface_hub scikit-learn numpy
hf download ZenoDivergent/zeno-fragility-beta --local-dir zeno-fragility-beta   # or: python -c "from huggingface_hub import snapshot_download as d; d('ZenoDivergent/zeno-fragility-beta', local_dir='zeno-fragility-beta')"

import sys; sys.path.insert(0, "zeno-fragility-beta")
from zeno_fragility import load, crowd_features, warn

m = load("market")   # available: crypto, ecb, forecastbench, health_rsv, market, physical
f = crowd_features(own=0.82, crowd=[0.55, 0.60, 0.48, 0.82, 0.58], own_confidence=0.64,
                   record_long=(12.1, 14.0), record_recent=(3.0, 3.1), m0=m["prior_track_record_m0"])
print(f"fragility score: {warn(m, f):.2f}")

Run with ONNX (any language, no scikit-learn needed):

import numpy as np, onnxruntime as ort
from huggingface_hub import hf_hub_download
sess = ort.InferenceSession(hf_hub_download("ZenoDivergent/zeno-fragility-beta", "onnx/market.logistic.onnx"))   # each source's selected model: see reports/per_source.json → served
# features in this order: crowd_distance, log_record_long, log_record_recent, crowd_spread, log_crowd_size, own_confidence
x = np.array([[2.1, -0.15, -0.03, 0.9, 1.8, 0.64]], dtype=np.float32)
print(sess.run(None, {"features": x})[1][0, 1])   # fragility score

Why is it so small?

The served models were fitted on 38,459 confident forecasts with known outcomes across 6 sources, reading 6 signals per forecast. Small means fast, explainable and hard to overfit. The next model reads whole forecasting events — every forecaster's range, track record and shared blind spots — and this beta is the baseline it has to beat.

Files: onnx/<source>.onnx (onnx/<source>.logistic.onnx where logistic is selected), model.safetensors, config.json, warning/<source>.pkl (fragility model per source), correctors/<source>.pkl (served correctors), zeno_fragility.py (helper), reports/per_source.json (every number, including the ones that didn't pass).

Limits

Research beta, early stage; scores are per source and are rankings, not calibrated probabilities.
Each source uses the model selected on held-out data (stored as selected in each file; warn() uses it by default).

ZenoDivergent
/

zeno-fragility-beta