HC3 Human / ChatGPT Answer Classifier

Educational binary sequence classifier fine-tuned from sentence-transformers/all-MiniLM-L6-v2. Labels: 0 = human, 1 = ChatGPT. Input is answer text only.

Data and training

English full-answer HC3, revision 4d0ff18143b5a7e1b1e79beb540c04549d1e59d3. The supplied preparation keeps the first nonempty answer per class for each eligible question, removes empty questions, missing/identical pairs and repeated questions, then splits questions 80/10/10 with seed 42 before flattening the paired answers. Answer counts: {'train': 37334, 'validation': 4666, 'test': 4668}.

All transformer parameters and the classification head are updated using AdamW (learning rate 2e-05, weight decay 0.01), batch size 32, maximum length 256 tokens, and 5 epochs. The final epoch is saved. Validation is monitored each epoch; the test split is held out from training.

Evaluation

Model	Test accuracy	Test macro F1
Frozen embeddings + logistic regression	0.844901	0.844882
Fine-tuned classifier	0.989717	0.989716

Mean training-batch loss: 0.022114. Additional metrics and confusion matrices are in metrics.json.

Usage

from transformers import pipeline
detector = pipeline("text-classification", model="./hw1-hc3-detector")
print(detector("An answer to classify.", truncation=True, max_length=256))

After publishing, replace the local model path with the public repository ID.

Limitations

This historical benchmark measures discrimination within HC3, not reliable authorship detection for current models or new domains. Dataset style, source, and formatting can act as shortcuts. Inputs longer than 256 tokens are truncated. Softmax scores are not calibrated authorship probabilities. Do not use this model as evidence of academic misconduct or for consequential decisions about people.