oneryalcin/finepdfs-purpose-neomme-frozen-head

FinePDFs document purpose — frozen NeoMME embeddings + logistic head

This is a frozen-body baseline, not a contrastively fine-tuned SetFit model. NeoMME weights are untouched; only a scikit-learn logistic-regression head (C=100, no class weighting) was fit on L2-normalised page embeddings. It is packaged as a SetFitModel so it loads and predicts with the standard API.

Task: classify a PDF by its primary purpose from the rendered first page only (800 px JPEG), using the six labels of Daniel van Strien's FinePDFs-Edu purpose classifier: administration_policy, exercise_assessment, instruction_reference, news_promotion, research_analysis, other.

Data

oneryalcin/finepdfs-purpose-pages-v2: PDFs sampled from davanstrien/finepdfs-edu-purpose, re-fetched from their source URLs, page 1 rendered with pdftoppm. Labels are not the text classifier's predictions: every page was relabelled by DeepSeek-V4.1-Flash given the page image plus a start/middle/end text excerpt and the purpose guide (the judge agreed with the text classifier's label on 67% of documents). 4,972 train / 702 test.

Results (702 held-out pages, judge labels as reference)

model	input	accuracy	macro-F1 (6 classes)
davanstrien/finepdfs-edu-purpose-classifier	text excerpt	0.679	0.570
this model	page-1 image	0.694	0.634

Per-class F1: administration_policy 0.61, exercise_assessment 0.74, instruction_reference 0.70, news_promotion 0.69, research_analysis 0.75, other 0.32. The 702-document test set gives roughly ±3.5 points; treat the two models as on par. The head's C was chosen on this same test split.

Usage

from PIL import Image
from setfit import SetFitModel  # needs the image-input patch: https://github.com/huggingface/setfit/pull/653
model = SetFitModel.from_pretrained("oneryalcin/finepdfs-purpose-neomme-frozen-head", task="document")
model.predict([Image.open("page-1.jpg")])

Requires sentence-transformers>=6, transformers>=5.16 (NeoMME). Runs at ~20 pages/s on an Apple M-series GPU.