---
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
  - mixture-of-experts
  - moe
  - corm
---

# CoRM-182M (Top-1)

Contrastive Routing Mixture-of-Experts (CoRM). Checkpoint for the paper [Beyond Magnitude: Contrastive Routing for Modular Mixture-of-Experts](https://arxiv.org/abs/2609.01100).

Part of the [ilsp/CoRM](https://huggingface.co/ilsp/CoRM) collection.

| Model | Active Params | Total Params | Routing |
|---|---|---|---|
| [CoRM-182M-top1](https://huggingface.co/ilsp/CoRM-182M-top1) | 182M | 777M | Top-1 |

## Architecture

| | |
|---|---|
| Hidden size | 768 |
| Layers | 12 |
| Attention heads | 12 (4 KV heads, GQA) |
| Intermediate size | 3072 |
| Experts | 8 |
| Experts per token | 1 |
| Vocab size | 51200 |
| Max position embeddings | 1024 |
| Dtype | bfloat16 |

## Usage

This model uses custom modeling code, so `trust_remote_code=True` is required.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ilsp/CoRM-182M-top1"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)

inputs = tokenizer("The capital of Greece is", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=32)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Training

| | |
|---|---|
| Data | [The Pile](https://huggingface.co/datasets/monology/pile-uncopyrighted), GPT-2 BPE tokenizer |
| Tokens | ~31B (60k steps × 512 sequences × 1024 tokens) |
| Learning rate | 5e-4, cosine decay to 5e-5, 1% warmup |
| Load-balancing loss coefficient | 1e-2 |
| Precision / hardware | bf16, 4 GPUs |

Training code and launch scripts: [github.com/athena-ilsp/CoRM](https://github.com/athena-ilsp/CoRM).

## Evaluation

See the [paper](https://arxiv.org/abs/2609.01100) for zero-shot results on nine reasoning benchmarks.

## Limitations

This is a base model pretrained on The Pile; it is not instruction-tuned. The Hugging Face implementation does not use a KV cache and ignores `attention_mask`, so pass unpadded sequences.

## Citation

```bibtex
@misc{xiros2026magnitudecontrastiveroutingmodular,
      title={Beyond Magnitude: Contrastive Routing for Modular Mixture-of-Experts},
      author={Nikolaos Xiros and Dimitrios Damianos and Maria-Eleni Zoumpoulidi and Leon Voukoutis and Vassilis Katsouros and Georgios Paraskevopoulos},
      year={2026},
      eprint={2609.01100},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2609.01100},
}
```