ModernBERT-ecoRouter

ModernBERT-ecoRouter is a classifier designed to route chatbot prompts to the most appropriate model based on task complexity. It predicts whether a user input should be handled by a small or large language model, helping reduce unnecessary compute for simple tasks.

This model is used in the “Do I really need a huge LLM?” demo, where a chatbot interface reveals which model responded to each user message based on routing decisions from this classifier.

Intended use

The model takes a single user prompt as input and returns a label:

• small if the input can likely be handled well by a small model (e.g. TinyLlama, Phi)
• large if it would likely benefit from a larger model (e.g. Mistral, GPT-4)

This enables dynamic model selection in chatbot or API systems where cost and latency matter.

Training data

The model was fine-tuned on a mix of samples from:

• ShareGPT
• WildChat

A Mistral model was used to classify prompts into small or large categories based on expected complexity of the task for an LLM.

Performance

On a held-out test set, the classifier achieves:

• ~80% accuracy

Limitations

• The model is trained on single prompts, not full conversations
• Some prompts are ambiguous without user history
• It may overpredict large in borderline cases