ShantanuT01/vanguard-ai-text-detector

Vanguard — AI-Generated Text Detector

This model is a fine-tuned version of ModernBERT-large for binary classification of human-written vs. AI-generated text. It outputs a single probability, P(AI), indicating the likelihood that a given input was generated by a language model. Vanguard AI-text detector was part of the 2nd-place system at PAN-CLEF 2026's Voight-Kampff Generative AI Detection leaderboard.

Model Details

• Base model: ModernBERT-large
• Architecture: ModernBERT-large with a single classification head (binary, sigmoid output)
• Output: A single scalar P(AI) in [0, 1]; a decision threshold of 0.5 is used by default, where P(AI) > 0.5 indicates AI-generated text
• Language: English
• License: MIT License

Training Data

This model was trained on approximately 1.1 million texts from three datasets: DACTYL 2.0, LLMTrace, and MAGA-Bench.

Evaluation

The model was evaluated against two leading open-source AI text detectors, Fakespot and Desklib, across ten benchmark datasets. Three of these (dactyl-v2.0, llm-trace-eng, maga) are in-distribution with respect to this model's training data; the remaining seven (beemo, coconuts, detectrl, dolly-cosmopedia, originalityai, realdet, uchicago) are out-of-distribution and were not seen during training.

All F1 scores are macro-averaged and computed at a decision threshold of P(AI) = 0.5.

Results

Dataset	AUROC	Macro-F1
dactyl-v2.0 †	0.9903	0.9709
llm-trace-eng †	0.9909	0.9661
maga †	0.9992	0.9901
beemo	0.8560	0.6973
coconuts	0.9792	0.8690
detectrl	0.9370	0.8849
dolly-cosmopedia	0.9958	0.8884
originalityai	0.9322	0.7776
realdet	0.9540	0.9177
uchicago	0.9784	0.9105

† In-distribution (training data overlap)

Out-of-distribution averages: AUROC 0.9475, Macro-F1 0.8493

Comparison to Baselines (OOD average)

Model	AUROC	Macro-F1
Fakespot	0.9315	0.8029
Desklib	0.9213	0.7837
Vanguard	0.9475	0.8493

Notes on Evaluation

• Fakespot and Desklib use a two-head softmax architecture and were evaluated using their native argmax decision rule, which is mathematically equivalent to thresholding P(AI) at 0.5.
• This model outperforms both baselines on most out-of-distribution datasets, with the exception of coconuts (Desklib) and originalityai (Fakespot), where the baselines hold an edge.
• In-distribution performance is substantially higher than out-of-distribution performance, which is expected and should be taken into account when interpreting the headline averages; OOD results are more representative of expected real-world generalization.

Limitations and Out-of-Scope Use

This model should not be used as the sole basis for high-stakes decisions such as academic penalties or employment actions, given the false positive/negative rates documented below. Performance also degrades on text distributions not represented in training data; see evaluation results.

Citation

@article{thorat2026panclef,
  title={Team DACTYL at PAN 2026: Bayesian Data Mixing and Empirical X-risk Minimization for AI-text Detection},
  author={Thorat, Shantanu},
  journal={Working Notes of CLEF},
  year={2026}
}