HC3 Answer Classifier

This homework model classifies English HC3 answers as human (0) or ChatGPT (1). The encoder and classification head were trained together for 5 epochs. Inputs use only answer text, with truncation at 256 tokens.

Data and training

HC3 revision: 4d0ff18143b5a7e1b1e79beb540c04549d1e59d3. Each eligible question contributes its first nonempty human answer and first nonempty ChatGPT answer. Empty questions, missing or identical answer pairs, and duplicate normalized questions are excluded. Questions are split 80/10/10 with seed 42 before answers are flattened. Train/validation/test answer counts: 37334/4666/4668. All splits are balanced, and no normalized question is shared between splits. Optimizer: PyTorch AdamW, learning rate 2e-05, batch size 32, five epochs. Validation metrics were logged each epoch; the final fifth-epoch model is saved. The test set was used for final evaluation, not for model selection.

Results on the held-out HC3 test split

Model	Accuracy	Macro precision	Macro recall	Macro F1
Frozen MiniLM + logistic regression	0.845116	0.845294	0.845116	0.845096
Fine-tuned MiniLM classifier	0.990788	0.990955	0.990788	0.990788

Confusion matrices use rows=true labels, columns=predicted labels, in the order [human, ChatGPT]. Baseline: [[1946, 388], [335, 1999]]. Fine-tuned: [[2291, 43], [0, 2334]]. Full metrics, validation history, and library versions are in metrics.json.

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_path = "skyyyyks/hw1-hc3-detector"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForSequenceClassification.from_pretrained(model_path)
model.eval()
inputs = tokenizer("An example answer.", return_tensors="pt", truncation=True, max_length=256)
with torch.no_grad():
    logits = model(**inputs).logits
label = model.config.id2label[logits.argmax(dim=-1).item()]
print(label)

Limitations

These scores describe a historical, balanced HC3 benchmark. They do not establish reliability for modern generators, unfamiliar domains, paraphrased text, edited answers, or mixed human/AI writing. Dataset formatting and source differences can provide shortcuts. Long answers lose content after truncation. False positives remain possible; predictions should not be used as evidence of student misconduct.