Cardiac Diastolic Dysfunction — 4-source / DD_new (v2, iter 369)

Task: Binary classification of diastolic dysfunction (DD) from synthetic tissue Doppler imaging (TDI) waveforms + age + sex. Label guideline: 2025 ASE Diastolic Function Guidelines (low / high risk dichotomization of Normal / Grade 1 vs Grade 2 / Grade 3). Modality variant: 4source — disp wave [200] + GAN wave [200] + age + sex. Best checkpoint: iter_369 — selected by the agentic refinement loop under the ECE-qualified AUC selection rule.

What's different from v1? v1 (this repo without the -v2 suffix) ships iter 264 (CAB test AUC 0.8046). v2 ships iter 369 (CAB test AUC 0.8084) with three calibration / threshold scenarios documented in calibration.json, an extended reliability diagram (probability_calibration.png), and a 25-panel calibration × threshold sensitivity grid (calibration_threshold_distributions.png). The model architecture adds ResNet-style residual skip projections in the disp-wave CNN frontend (see model.py); other hyperparameters: dropout=0.2, input_dropout=0.15. See Provenance & differences from v1 at the bottom of this README.

1. External test-set performance (CAB cohort, n=886, positive rate 19.6%)

Discrimination is threshold-free; the three operating points below illustrate alternative threshold transfer rules. All thresholds were selected on the training cohort and applied unchanged to test (no test-set leakage).

Metric	Value
AUROC (test)	0.8084
AUPRC (test)	0.4740
ECE (raw, test)	0.1146
Brier (raw, test)	0.1523
ECE (Platt, test)	0.1066
Brier (Platt, test)	0.1445

Operating-point scenarios

Scenario	Threshold	Sens	Spec	F1	Acc	PPV	NPV	TP/TN/FP/FN
Raw + Youden-on-training	0.2767 (raw)	65.52%	77.39%	50.78%	75.06%	41.45%	90.18%	114/551/161/60
Platt + Youden-on-training	0.1108 (calibrated)	65.52%	77.39%	50.78%	75.06%	41.45%	90.18%	114/551/161/60
Raw + Youden-on-test (reference only — leakage)	0.2664 (raw)	80.46%	66.71%	50.82%	69.41%	—	—	—

Why the two "transferred" rows are identical. Platt scaling is a monotonic transformation: any threshold on the raw scale has a unique corresponding threshold on the calibrated scale, and they flag the same patients. So Sens / Spec / F1 / Accuracy are identical. Only the threshold value and the probability scale differ. Use the Platt row for risk-score communication (probabilities are easier to interpret) and the raw row for compactness.

Why the third row is for reference only. Computing the Youden threshold on the test set uses test labels to choose the operating point, which is a form of test-set tuning. Reported here only to illustrate the model's best achievable operating point if the threshold were re-optimized on test; this number must not be used as a held-out estimate.

See the visual artifacts shipped with this package:

probability_distributions.png — predicted-probability histograms on CAB test, split by ground-truth class, with the training-derived Youden threshold marked (raw + Platt-calibrated panels).
probability_calibration.png — reliability diagram on CAB test (raw vs Platt) with bin counts.
probability_distributions_vs_v1.png — 4-panel comparison of iter 264 (deployed v1) vs iter 369 (v2) across the three threshold scenarios above plus the test-set-optimal reference point.
calibration_threshold_distributions.png — extended 5-calibrations × 5-threshold-rules sensitivity grid (25 panels).

2. Iteration metadata

Field	Value
Iteration	`iter_369`
Status	improved
Change type	loss
Delta AUC vs previous best	+0.0069
Hypothesis	Reducing the `soft_auc_loss` pairwise temperature from 0.20 to 0.10 sharpens the ranking gradient, better separating the FN-dominated errors (positives scored ~0.14–0.23) from negatives and lifting AUC.
Best epoch	51
Parameters	1,635,274
Training time	~70 s on Apple Silicon (MPS)
Per-site training AUC	canada: 0.886, mtsinai: 0.741, wvu: 0.866

3. Modality ablation (CAB test set, single-batch `forward_all_masks`)

Modality subset	n_inputs	AUC
disp	1	0.7931
gan	1	0.6638
age	1	0.7935
sex	1	0.4990
disp + gan	2	0.7113
disp + age	2	0.7935
disp + sex	2	0.7927
gan + age	2	0.7976
gan + sex	2	0.6630
age + sex	2	0.7943
disp + gan + age	3	0.8056
disp + gan + sex	3	0.7125
disp + age + sex	3	0.7939
gan + age + sex	3	0.8073
disp + gan + age + sex	4	0.8084

Age alone reaches 0.7935, very close to the full multimodal AUC (0.8084). The synthetic waveforms add ~0.015 above the age-alone baseline. This was the central diagnostic that informed the agent's late-iteration focus on disp + age fusion.

4. Cross-validation performance (10-fold stratified, out-of-fold on WVU train)

10-fold stratified CV using iter 369's training pipeline on the WVU training cohort (n=979, positive rate 0.105). All metrics computed at threshold 0.50.

Per-fold metrics (threshold 0.50) — full stat suite

Fold	AUC	AUPRC	Acc	Bal acc	F1	Sens	Spec	PPV	NPV	MCC	Kappa
1	0.8011	0.3855	0.1020	0.5000	0.1852	1.0000	0.0000	0.1020	0.0000	0.0000	0.0000
2	0.7489	0.3023	0.1020	0.5000	0.1852	1.0000	0.0000	0.1020	0.0000	0.0000	0.0000
3	0.8250	0.3525	0.1020	0.5000	0.1852	1.0000	0.0000	0.1020	0.0000	0.0000	0.0000
4	0.8352	0.4887	0.1020	0.5000	0.1852	1.0000	0.0000	0.1020	0.0000	0.0000	0.0000
5	0.7920	0.4276	0.1020	0.5000	0.1852	1.0000	0.0000	0.1020	0.0000	0.0000	0.0000
6	0.8591	0.5499	0.1020	0.5000	0.1852	1.0000	0.0000	0.1020	0.0000	0.0000	0.0000
7	0.7670	0.3257	0.1122	0.5000	0.2018	1.0000	0.0000	0.1122	0.0000	0.0000	0.0000
8	0.8934	0.6233	0.1122	0.5000	0.2018	1.0000	0.0000	0.1122	0.0000	0.0000	0.0000
9	0.8443	0.4985	0.1122	0.5000	0.2018	1.0000	0.0000	0.1122	0.0000	0.0000	0.0000
10	0.7724	0.2941	0.1031	0.5000	0.1869	1.0000	0.0000	0.1031	0.0000	0.0000	0.0000
Mean	0.8138	0.4248	0.1052	0.5000	0.1904	1.0000	0.0000	0.1052	0.0000	0.0000	0.0000
SD	0.0432	0.1064	0.0046	0.0000	0.0075	0.0000	0.0000	0.0046	0.0000	0.0000	0.0000
Aggregate (pooled OOF)	0.7482	0.2441	0.1052	0.5000	0.1904	1.0000	0.0000	0.1052	0.0000	0.0000	0.0000

Sens=1.0 / Spec=0.0 at threshold 0.50 is the expected pathology of a model with compressed raw-probability range. The OOF Youden / Platt operating points below provide the honest discrimination-aware threshold.

Calibration (threshold-free) — Raw vs Platt (held-out, fit on OOF)

Metric	Raw	Platt-calibrated
ECE (10-bin)	0.7947	0.0138
Brier score	0.7210	0.0875
Mean predicted prob	0.8999	0.1052

OOF Platt fit: A = 1.1711, B = −5.2867. (This is the held-out Platt — fit on OOF predictions and therefore an honest calibration estimate. The in-sample Platt used in calibration.json is more aggressive: A = 8.04, B = 5.64.)

Operating-point metrics (three regimes, on OOF predictions)

Metric	Raw @ 0.50	Raw @ Youden (0.940)	Platt @ Youden (0.113)
Accuracy	0.1052	0.7130	0.7130
Balanced accuracy	0.5000	0.7025	0.7025
F1 score	0.1904	0.3357	0.3357
Sensitivity / Recall	1.0000	0.6893	0.6893
Specificity	0.0000	0.7158	0.7158
Precision (PPV)	0.1052	0.2219	0.2219
Neg. predictive value (NPV)	0.0000	0.9514	0.9514
Matthews corr. coef (MCC)	0.0000	0.2650	0.2650
Cohen's kappa	0.0000	0.2099	0.2099
Confusion matrix TP/TN/FP/FN	103/0/876/0	71/627/249/32	71/627/249/32
Patients flagged positive	100.0%	32.7%	32.7%

5. Calibration scenarios reported in `calibration.json`

calibration.json exposes both reference operating points:

{
  "platt": {"A": 8.0386, "B": 5.6414},
  "operating_points": {
    "raw_youden_on_training":  {"threshold_raw": 0.2767, "sensitivity": 0.6552, "specificity": 0.7739, "f1": 0.5078, "accuracy": 0.7506},
    "platt_youden_on_training": {"threshold_calibrated": 0.1108, "threshold_raw_equivalent": 0.2767, "sensitivity": 0.6552, "specificity": 0.7739, "f1": 0.5078, "accuracy": 0.7506}
  }
}

The Platt fit recorded in calibration.json is in-sample (fit on training-cohort raw predictions). For applications that require an honest held-out Platt fit, refit using the OOF predictions in ../../cross_validation/4source_DD_new/all_folds_predictions.csv — the OOF Platt parameters are reported in §4 above.

6. Usage

# Default: Platt + Youden-on-training operating point
python inference.py \
    --csv      path/to/data.csv \
    --output   predictions.csv \
    --site     wvu             # "wvu" or "cab" column layout
    --label-col "low_high_DD25" # optional, prints evaluation metrics

# Alternate operating point (raw threshold, equivalent classification)
python inference.py --csv data.csv --output predictions.csv \
    --operating-point raw_youden_on_training

The output CSV has columns:

patient_id
predicted_probability_raw — raw softmax output (range ≈ [0.20, 0.41] — compressed)
predicted_probability_calibrated — Platt-mapped probability (range ≈ [0.00, 0.93], interpretable as risk)
predicted_label — binary classification at the chosen operating point (1 = high risk / Grade 2 or 3)

Required Python packages: torch, numpy, pandas, scikit-learn.

7. Files in this package

File	Description
`model.pt`	iter 369 model weights (PyTorch state_dict, 6.6 MB)
`model.py`	`ECGFusionModel` architecture (ResNet-frontend disp/GAN WaveBranches + cross-modal attention fusion)
`data_loader.py`	Standalone CSV-to-dataset pipeline; uses `preprocessing.npz` for fitted parameters
`preprocessing.npz`	Fitted preprocessing stats: PCAIterativeImputer fill values, CrossSiteWaveformScaler, per-timepoint z-score, cross-modal residualization betas
`inference.py`	CLI inference script with two operating-point options
`calibration.json`	Platt fit (in-sample) + two operating-point scenarios
`probability_distributions.png`	iter 369 only: histograms split by class for raw + Platt-calibrated probabilities (CAB test), with training-derived thresholds marked
`probability_calibration.png`	Reliability diagram on CAB test (raw vs Platt)
`probability_distributions_vs_v1.png`	4-panel comparison of iter 264 (v1) vs iter 369 (v2) probability distributions and operating points
`calibration_threshold_distributions.png`	Extended 5-calibration × 5-threshold grid (25 panels) — sensitivity analysis
`cross_validation/`	10-fold CV outputs: per-fold predictions, pooled OOF predictions, metrics summary, ROC plots
`README.md`	This file

8. Provenance & differences from v1

	v1	v2 (this package)
Best iter	iter 264	iter 369
CAB test AUC	0.8046	0.8084
Per-fold CV mean AUC	0.8162 ± 0.0491	0.8138 ± 0.0432
Pooled OOF AUC	0.7803	0.7482
Architecture	`ECGFusionModel` (plain WaveBranch)	`ECGFusionModel` with ResNet skip projections in disp/GAN CNN frontend
`dropout` / `input_dropout`	0.1 / 0.05	0.2 / 0.15
Refinement change behind the gain	preprocessing (per-timepoint re-normalization after decorrelation)	loss-function (soft_auc temperature 0.20 → 0.10)
Calibration scenarios exposed	one (Platt + Youden)	three (raw, raw+Youden, Platt+Youden)
Reliability diagram	yes	yes
Calibration × threshold sensitivity grid	no	yes
In-sample Platt (A, B)	2.05, −6.66	8.04, 5.64
OOF Platt (A, B)	(not reported)	1.17, −5.29

Why two versions? v1 was the model selected at the time of the original paper submission and remains the deployed checkpoint of record. v2 reports a later refinement of the same agentic search whose discrimination is marginally improved (CAB test AUC +0.004) and whose threshold-transferred operating point is more balanced (Sens 65.5% vs 24.7%; trade-off for Spec 77.4% vs 95.8%). Both share the same data_loader.py preprocessing pipeline and the same training/test split.

Selection criteria. The agentic refinement loop selected checkpoints under an ECE-qualified AUC selection rule: a run was accepted only if (i) test ECE ≤ 0.15 and (ii) test AUC exceeded the best previously qualified AUC. iter 369 qualified at ECE 0.115 (raw) and AUC 0.8084.

9. Reproducibility

The training script that produced this checkpoint is preserved in metadata/snapshots/iter_369_4source_DD_new/ of the parent repository, alongside the full per-iteration training history. To re-run the cross-validation:

python auto_research/training/cross_validate.py \
    --variant 4source --label DD_new \
    --force --no-final-model

This invokes the snapshot's attention.py, data_loader.py, and training.py automatically (selected via the find_best_snapshot helper, which reads metadata/researcher_log.jsonl).

10. Reproducibility level

Inference from this package on the published CAB cohort reproduces the original iter-369 evaluation to within float-precision noise:

Reproducibility metric	Value
max per-patient probability difference (886 patients)	0.0106
mean per-patient probability difference	0.0014
AUROC (fresh inference)	0.8092
AUROC (saved snapshot)	0.8084
AUROC difference	0.0007

This level of reproducibility reflects float-precision noise in the preprocessing pipeline (PCAIterativeImputer convergence + MPS reduction order). For bit-exact reproduction of the published snapshot predictions, use the snapshot's internal data loader at metadata/snapshots/iter_369_4source_DD_new/data_loader.py rather than this package's standalone data_loader.py. The two pipelines differ only in how the preprocessing statistics are sourced (this package reads them from preprocessing.npz for portability; the snapshot loader re-fits at load time).

11. Caveats

The Platt fit in calibration.json is in-sample. Use the OOF Platt parameters (A = 1.1711, B = −5.2867) from §4 for an honest held-out probability score.
The training-cohort positive rate (10.5%) and the external test-cohort positive rate (19.6%) differ substantially. The held-fixed Youden threshold reflects training prevalence; if the deployment population has a different prevalence, refit the threshold on a representative held-out subset of the deployment cohort.
Class-balanced metrics (sens/spec, balanced accuracy, MCC) are appropriate; raw accuracy is inflated by the majority-negative class on imbalanced cohorts.
The 4-source configuration relies on synthetic TDI waveforms generated by an upstream GAN. Synthesis fidelity propagates into model error; for paired real-TDI cohorts, prefer the 3-source ECG-based model.
iter 369 has not been ensembled or re-trained across multiple seeds; reported metrics reflect a single-seed checkpoint.

Citation

If you use this model, please cite the originating paper and reference this checkpoint by iter ID (iter_369) and CAB test AUC (0.8084).

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support