kirikir13/MOEmoe-3x7B-BF16-v2

MOEmoe-3x7B-BF16 (v2 rebuild)

Architecture: Built from 3 models: [M] MixtralForCausalLM (model_type: mixtral), 1 expert [MINSTRAL] Minstrel Is Lee personality and tool department followed by [O] `openchat/openchat-3.5 Whose role is the logic and reasoning, Proceeded by:[E] openbmb/Eurus-7b-kto Who is head of the math department for a total of a 19B M.O.E. Named: "MOEmoe" Which is my very first M.O.E. Or as I like to say: "Mistral of Exports 😎. Next I will then be creating a couple of this in new specialized custom quants thus reducing it down to an estimated 7GB With your loss intelligence or abilities Stay tuned

Goerge Roby & 'theCREW' [DR Studios]

---

The name

MOEmoe = "Mistral of Experts". Three Mistral-family 7B models whose initials spell M-O-E:

• M = Mistral-7B-Instruct-v0.2, the main talker ("Minstrel" in the original note was a spelling slip for Mistral)
• O = OpenChat-3.5, logic and reasoning
• E = Eurus-7b-kto, math

They share the same tokenizer and tensor shapes, so they frankenmerge cleanly into one Mixtral-style MoE.

Why v2 exists

The original upload kirikir13/MOEmoe-3x7B-BF16 (and its bucket copies) is incomplete: 19.3 GB of the ~37.0 GB model. It only has layers 0–15 and part of layer 16. Layers 16–31, all 32 router (gate) weights, model.norm and lm_head are missing (upload/hard-drive failure). The original is left untouched. This repo is a full rebuild from the same source models with mergekit-moe.

Architecture

Class	MixtralForCausalLM (model_type: mixtral)
Experts	3 (num_local_experts: 3), top-2 routing (num_experts_per_tok: 2)
Layers / hidden / FFN	32 / 4096 / 14336
Attention	32 heads, 8 KV heads (GQA), head_dim 128
RoPE theta / context	1,000,000 / 32768 (from Mistral-7B-Instruct-v0.2), no sliding window
Vocab	32000 (Mistral SentencePiece), bos <s>=1, eos </s>=2
Params / size	18.5B total (12.9B active per token with top-2), 37.0 GB BF16, 8 shards

Expert	Source model	Role
0	mistralai/Mistral-7B-Instruct-v0.2	Main talker: personality, chat, tools (also supplies attention, embeddings, norms, lm_head)
1	openchat/openchat_3.5	Logic and reasoning
2	openbmb/Eurus-7b-kto	Math

The source weights came from George's bucket kirikir13/M-O-E_moe. The original config's _name_or_path was Mistral-7B-Instruct-v0.2, so that is the base. "Minstrel" in the notes above is Mistral-7B-Instruct-v0.2.

Router: gate_mode: cheap_embed. Each expert's router vector comes from the base model's token embeddings of its positive prompts (see mergekit_moe_config.yml). hidden mode needs a full forward pass of the base model, which did not fit in the 15 GB RAM build box. The router is a sensible starting point, not a trained one. Fine-tuning the gates (or a hidden rebuild on a bigger machine) will sharpen the routing.

Tokenizer and chat template

The tokenizer is Mistral-7B-Instruct-v0.2's (32000 tokens). It matches vocab_size and the embed_tokens/lm_head shape [32000, 4096] exactly. The embeddings come from the Mistral base. OpenChat's tokenizer has 32002 tokens (<|end_of_turn|>, <|pad_0|>), which this model has no embeddings for, so the OpenChat "GPT4 Correct User" template is not used. The chat template is Mistral [INST]:

<s> [INST] {system}\n\n{user} [/INST] {assistant}</s>

Usage (transformers)

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
repo = "kirikir13/MOEmoe-3x7B-BF16-v2"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")
msgs = [{"role": "user", "content": "Solve 3x + 5 = 20 and explain."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=256)
print(tok.decode(out[0][ids.shape[-1]:], skip_special_tokens=True))

You need about 38 GB RAM/VRAM in BF16. Use 4-bit (bitsandbytes) or a GGUF quant for smaller machines.

"Main talker only" switch

• 1 expert per token: set "num_experts_per_tok": 1 in config.json, or config.num_experts_per_tok = 1 before loading. This is faster, but the router still picks per token.
• Talker only: there is no exact gate-only switch, because router scores depend on the token. If you want only the talker, run expert 0's source model (Mistral-7B-Instruct-v0.2) directly. A future build can give the talker a stronger prior by adding more and broader positive prompts to expert 0.

llama.cpp / GGUF

The weights use the classic Mixtral tensor names (block_sparse_moe.experts.N.w1/w2/w3, block_sparse_moe.gate), so llama.cpp converts them directly:

python convert_hf_to_gguf.py ./MOEmoe-3x7B-BF16-v2 --outtype bf16 --outfile moemoe-3x7b-bf16.gguf
./llama-quantize moemoe-3x7b-bf16.gguf moemoe-3x7b-Q4_K_M.gguf Q4_K_M   # ~11 GB
./llama-server -m moemoe-3x7b-Q4_K_M.gguf --chat-template mistral -c 8192

To run on a single expert in llama.cpp, add --override-kv llama.expert_used_count=int:1.

Files

• model-0000{1..8}-of-00008.safetensors and model.safetensors.index.json: 515 tensors, 37,032,828,928 bytes, BF16
• config.json, generation_config.json (bos 1, eos 2)
• tokenizer.json, tokenizer.model, tokenizer_config.json, special_tokens_map.json, chat_template.jinja
• mergekit_moe_config.yml: the exact merge recipe

Changelog

• Oct 23, 2026: v2 full rebuild. Diagnosed the v1 upload as truncated (missing layers 16–31, routers, final norm, lm_head). Rebuilt with mergekit 0.1.5 (mergekit-moe --lazy-unpickle, CPU, gate_mode: cheap_embed, top-2) from Mistral-7B-Instruct-v0.2 + openchat_3.5 + Eurus-7b-kto. Converted to classic Mixtral tensor names for llama.cpp/vLLM/transformers-4.x compatibility. Added the tokenizer, [INST] chat template, generation config and index. Verified: the index matches all shard headers; transformers 5.20.2 loads it with 0 missing / 0 unexpected / 0 mismatched keys; a 2-layer forward pass gives finite logits and a working router.