---
library_name: transformers
pipeline_tag: text-generation
license: apache-2.0
base_model:
- mistralai/Mistral-7B-Instruct-v0.2
- openchat/openchat_3.5
- openbmb/Eurus-7b-kto
tags:
- mixtral
- moe
- mergekit
---

# MOEmoe-3x7B-BF16 (v2 rebuild)

Architecture: Built from 3 models: [M]  `MixtralForCausalLM` (`model_type: mixtral`),  1 expert [MINSTRAL] Minstrel Is Lee personality and tool department followed by [O] `openchat/openchat-3.5 Whose role is the logic and reasoning, Proceeded by:[E] openbmb/Eurus-7b-kto Who is head of the math department for a total of a 19B M.O.E. Named: "MOEmoe" Which is my very first M.O.E. Or as I like to say: "Mistral of Exports 😎. Next I will then be creating a couple of this in new specialized custom quants thus reducing it down to an estimated 7GB With your loss intelligence or abilities Stay tuned

  Goerge Roby & 'theCREW' [DR Studios] 

---

## The name
**MOEmoe = "Mistral of Experts".** Three Mistral-family 7B models whose initials spell **M-O-E**:
- **M** = Mistral-7B-Instruct-v0.2, the main talker ("Minstrel" in the original note was a spelling slip for Mistral)
- **O** = OpenChat-3.5, logic and reasoning
- **E** = Eurus-7b-kto, math

They share the same tokenizer and tensor shapes, so they frankenmerge cleanly into one Mixtral-style MoE.

## Why v2 exists
The original upload [kirikir13/MOEmoe-3x7B-BF16](https://huggingface.co/kirikir13/MOEmoe-3x7B-BF16) (and its bucket copies) is **incomplete**: 19.3 GB of the ~37.0 GB model. It only has layers 0–15 and part of layer 16. Layers 16–31, all 32 router (`gate`) weights, `model.norm` and `lm_head` are missing (upload/hard-drive failure). The original is left untouched. This repo is a full rebuild from the same source models with mergekit-moe.

## Architecture
| | |
|---|---|
| Class | `MixtralForCausalLM` (`model_type: mixtral`) |
| Experts | 3 (`num_local_experts: 3`), top-2 routing (`num_experts_per_tok: 2`) |
| Layers / hidden / FFN | 32 / 4096 / 14336 |
| Attention | 32 heads, 8 KV heads (GQA), head_dim 128 |
| RoPE theta / context | 1,000,000 / 32768 (from Mistral-7B-Instruct-v0.2), no sliding window |
| Vocab | 32000 (Mistral SentencePiece), bos `<s>`=1, eos `</s>`=2 |
| Params / size | ~18.5B total (~12.9B active per token with top-2), 37.0 GB BF16, 8 shards |

| Expert | Source model | Role |
|---|---|---|
| 0 | mistralai/Mistral-7B-Instruct-v0.2 | **Main talker**: personality, chat, tools (also supplies attention, embeddings, norms, lm_head) |
| 1 | openchat/openchat_3.5 | Logic and reasoning |
| 2 | openbmb/Eurus-7b-kto | Math |

The source weights came from George's bucket `kirikir13/M-O-E_moe`. The original config's `_name_or_path` was Mistral-7B-Instruct-v0.2, so that is the base. "Minstrel" in the notes above is Mistral-7B-Instruct-v0.2.

Router: `gate_mode: cheap_embed`. Each expert's router vector comes from the base model's token embeddings of its positive prompts (see `mergekit_moe_config.yml`). `hidden` mode needs a full forward pass of the base model, which did not fit in the 15 GB RAM build box. The router is a sensible starting point, not a trained one. Fine-tuning the gates (or a `hidden` rebuild on a bigger machine) will sharpen the routing.

## Tokenizer and chat template
The tokenizer is Mistral-7B-Instruct-v0.2's (32000 tokens). It matches `vocab_size` and the `embed_tokens`/`lm_head` shape `[32000, 4096]` exactly. The embeddings come from the Mistral base. OpenChat's tokenizer has 32002 tokens (`<|end_of_turn|>`, `<|pad_0|>`), which this model has no embeddings for, so the OpenChat "GPT4 Correct User" template is **not** used. The chat template is Mistral `[INST]`:
```
<s> [INST] {system}\n\n{user} [/INST] {assistant}</s>
```

## Usage (transformers)
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
repo = "kirikir13/MOEmoe-3x7B-BF16-v2"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")
msgs = [{"role": "user", "content": "Solve 3x + 5 = 20 and explain."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=256)
print(tok.decode(out[0][ids.shape[-1]:], skip_special_tokens=True))
```
You need about 38 GB RAM/VRAM in BF16. Use 4-bit (`bitsandbytes`) or a GGUF quant for smaller machines.

### "Main talker only" switch
- **1 expert per token:** set `"num_experts_per_tok": 1` in `config.json`, or `config.num_experts_per_tok = 1` before loading. This is faster, but the router still picks per token.
- **Talker only:** there is no exact gate-only switch, because router scores depend on the token. If you want only the talker, run expert 0's source model (Mistral-7B-Instruct-v0.2) directly. A future build can give the talker a stronger prior by adding more and broader positive prompts to expert 0.

## llama.cpp / GGUF
The weights use the classic Mixtral tensor names (`block_sparse_moe.experts.N.w1/w2/w3`, `block_sparse_moe.gate`), so llama.cpp converts them directly:
```bash
python convert_hf_to_gguf.py ./MOEmoe-3x7B-BF16-v2 --outtype bf16 --outfile moemoe-3x7b-bf16.gguf
./llama-quantize moemoe-3x7b-bf16.gguf moemoe-3x7b-Q4_K_M.gguf Q4_K_M   # ~11 GB
./llama-server -m moemoe-3x7b-Q4_K_M.gguf --chat-template mistral -c 8192
```
To run on a single expert in llama.cpp, add `--override-kv llama.expert_used_count=int:1`.

## Files
- `model-0000{1..8}-of-00008.safetensors` and `model.safetensors.index.json`: 515 tensors, 37,032,828,928 bytes, BF16
- `config.json`, `generation_config.json` (bos 1, eos 2)
- `tokenizer.json`, `tokenizer.model`, `tokenizer_config.json`, `special_tokens_map.json`, `chat_template.jinja`
- `mergekit_moe_config.yml`: the exact merge recipe

## Changelog
- **Oct 23, 2026**: v2 full rebuild. Diagnosed the v1 upload as truncated (missing layers 16–31, routers, final norm, lm_head). Rebuilt with mergekit 0.1.5 (`mergekit-moe --lazy-unpickle`, CPU, `gate_mode: cheap_embed`, top-2) from Mistral-7B-Instruct-v0.2 + openchat_3.5 + Eurus-7b-kto. Converted to classic Mixtral tensor names for llama.cpp/vLLM/transformers-4.x compatibility. Added the tokenizer, `[INST]` chat template, generation config and index. Verified: the index matches all shard headers; transformers 5.20.2 loads it with 0 missing / 0 unexpected / 0 mismatched keys; a 2-layer forward pass gives finite logits and a working router.