best_bvv_ru

This repository contains the model and associated resources from the papers

📚 Paper (Emergent Semantics Beyond Token Embeddings: Transformer LMs with Frozen Visual Unicode Representations) -

📚 Paper (Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate) -

💻 Code

Proof-of-concept Transformer LM with frozen, non-semantic token embeddings trained on a small English-Russian corpus.

This model is part of a series of models designed to demonstrate:

• The viability of transformer language models where the embedding layer is precomputed from non-semantic (Unicode/visual) features and entirely frozen during training.
• The possibility of modular/federated model fusion (MoE) by combining models with a shared token embedding matrix, without any additional retraining or alignment.

Model facts

• Parameters: 0.5B
• Architecture: 16-layer transformer, rotary attention, 1024 context, 32 heads.
• Embedding: Precomputed, frozen visual/Unicode-based.
• Training corpus: Small-scale, <10B tokens, ~10% SFT-mixed (for metric tracking, not strong performance).
• Languages: Russian, English.
• MoE compatibility: Embedding space is shared with other bvv models (e.g. Bochkov/best_bvv_zh) enabling seamless MoE or model fusion at output head level.

Key points

This model was trained on a small corpus and is intended only to demonstrate the viability of frozen, visual/Unicode-derived embeddings for training and transfer between languages.

Performance is not comparable to SOTA but shows competitive compositional skills versus a fully trainable embedding baseline.

For direct benchmarking, see also [Bochkov/best_bvv_unfrozen_ru] — an identical architecture and dataset, but with standard trainable token embeddings. Enables seamless fusion/MoE with Bochkov/best_bvv_zh and Bochkov/best_bvv_moe (merged model) due to shared embedding space. Main evaluation MMLU avg: 22.3% ±0.1 ARC-e: 23.0% ARC-c: 24.6% CommonsenseQA: 20.1% SQUAD: 14.8% BLEU [en-ru]: 6.4% BLEU [ru-en]: 8.8%

🧑‍🔬 Citation & Concept

If you use or build upon this demo, please cite:

@article{
      bochkov2025emergent,
      title={Emergent Semantics Beyond Token Embeddings: Transformer {LM}s with Frozen Visual Unicode Representations},
      author={Andrey Bochkov},
      journal={Transactions on Machine Learning Research},
      issn={2835-8856},
      year={2025},
      url={https://openreview.net/forum?id=Odh8IynO1o},
      note={}
}

@misc{bochkov2025growingtransformersmodularcomposition,
      title={Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate}, 
      author={A. Bochkov},
      year={2025},
      eprint={2507.07129},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2507.07129}, 
}

This work demonstrates that transformer blocks, not token embeddings, carry the semantic burden in LLMs — a step toward modular, fusable, multilingual LMs.

Example Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model = AutoModelForCausalLM.from_pretrained('Bochkov/best_bvv_ru', trust_remote_code=True).to('cuda')
tokenizer = AutoTokenizer.from_pretrained('Bochkov/best_bvv_ru')
inputs = tokenizer("Hello, мир! ", return_tensors="pt").to('cuda')
outputs = model.generate(
    **inputs, 
    max_new_tokens=100, 
    temperature=0.8, 
    top_k=50, 
    top_p=0.95, 
    do_sample=True
)
print(tokenizer.decode(outputs[0]))