Upload folder using huggingface_hub

README.md

@@ -14,9 +14,9 @@ metrics:
 - accuracy
 ---
 
-# 🌌 Nova MoE-Vision — Ground-Up Hierarchical Reasoning Engine
+# 🌌 Nova 1.0 & Nova 2.0 MoE-Vision — Ground-Up Hierarchical Reasoning Engine
 
-**Nova MoE-Vision** are ground-up custom AI models built from scratch, combining:
+**Nova 1.0 & Nova 2.0 MoE-Vision** are ground-up custom AI models built from scratch, combining:
 1. **Hierarchical Reasoning Model (HRM)**: Dual multi-timescale recurrent engine ($f_H$ and $f_L$).
 2. **Sparse Mixture of Experts (MoE)**: 8 SwiGLU FFN Experts with Top-2 Router Gating (~522.8 Million Total Parameters).
 3. **Multi-Modal Vision Projection**: Linear projection mapping image features into token embedding space.

data/ultrachat_200k/.gitattributes

@@ -0,0 +1,55 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.lz4 filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+# Audio files - uncompressed
+*.pcm filter=lfs diff=lfs merge=lfs -text
+*.sam filter=lfs diff=lfs merge=lfs -text
+*.raw filter=lfs diff=lfs merge=lfs -text
+# Audio files - compressed
+*.aac filter=lfs diff=lfs merge=lfs -text
+*.flac filter=lfs diff=lfs merge=lfs -text
+*.mp3 filter=lfs diff=lfs merge=lfs -text
+*.ogg filter=lfs diff=lfs merge=lfs -text
+*.wav filter=lfs diff=lfs merge=lfs -text
+# Image files - uncompressed
+*.bmp filter=lfs diff=lfs merge=lfs -text
+*.gif filter=lfs diff=lfs merge=lfs -text
+*.png filter=lfs diff=lfs merge=lfs -text
+*.tiff filter=lfs diff=lfs merge=lfs -text
+# Image files - compressed
+*.jpg filter=lfs diff=lfs merge=lfs -text
+*.jpeg filter=lfs diff=lfs merge=lfs -text
+*.webp filter=lfs diff=lfs merge=lfs -text

data/ultrachat_200k/README.md

@@ -0,0 +1,124 @@
+---
+language:
+- en
+license: mit
+size_categories:
+- 100K<n<1M
+task_categories:
+- text-generation
+pretty_name: UltraChat 200k
+configs:
+- config_name: default
+  data_files:
+  - split: train_sft
+    path: data/train_sft-*
+  - split: test_sft
+    path: data/test_sft-*
+  - split: train_gen
+    path: data/train_gen-*
+  - split: test_gen
+    path: data/test_gen-*
+dataset_info:
+  features:
+  - name: prompt
+    dtype: string
+  - name: prompt_id
+    dtype: string
+  - name: messages
+    list:
+    - name: content
+      dtype: string
+    - name: role
+      dtype: string
+  splits:
+  - name: train_sft
+    num_bytes: 1397058554
+    num_examples: 207865
+  - name: test_sft
+    num_bytes: 154695659
+    num_examples: 23110
+  - name: train_gen
+    num_bytes: 1347396812
+    num_examples: 256032
+  - name: test_gen
+    num_bytes: 148276089
+    num_examples: 28304
+  download_size: 1624049723
+  dataset_size: 3047427114
+---
+
+# Dataset Card for UltraChat 200k
+
+## Dataset Description
+
+This is a heavily filtered version of the [UltraChat](https://github.com/thunlp/UltraChat) dataset and was used to train [Zephyr-7B-β](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta), a state of the art 7b chat model.
+
+The original datasets consists of 1.4M dialogues generated by ChatGPT and spanning a wide range of topics. To create `UltraChat 200k`, we applied the following logic:
+
+- Selection of a subset of data for faster supervised fine tuning.
+- Truecasing of the dataset, as we observed around 5% of the data contained grammatical errors like "Hello. how are you?" instead of "Hello. How are you?"
+- Removal of dialogues where the assistant replies with phrases like "I do not have emotions" or "I don't have opinions", even for fact-based prompts that don't involve either.
+
+## Dataset Structure
+
+The dataset has four splits, suitable for:
+
+* Supervised fine-tuning (`sft`).
+* Generation ranking (`gen`) via techniques like rejection sampling or PPO.
+
+The number of examples per split is shown as follows:
+
+
+|  train_sft | test_sft  | train_gen | test_gen |
+|:-------:|:-----------:|:-----:| :-----:|
+|  207865 |       23110 | 256032 | 28304 |
+
+The dataset is stored in parquet format with each entry using the following schema:
+```
+
+{
+    "prompt": "Create a fully-developed protagonist who is challenged to survive within a dystopian society under the rule of a tyrant. ...",
+    "messages":[
+        {
+            "content": "Create a fully-developed protagonist who is challenged to survive within a dystopian society under the rule of a tyrant. ...",
+            "role": "user"
+        },
+        {
+            "content": "Name: Ava\n\n Ava was just 16 years old when the world as she knew it came crashing down. The government had collapsed, leaving behind a chaotic and lawless society. ...",
+            "role": "assistant"
+        },
+        {
+            "content": "Wow, Ava's story is so intense and inspiring! Can you provide me with more details.  ...",
+            "role": "user"
+        },
+        {
+            "content": "Certainly! ....",
+            "role": "assistant"
+        },
+        {
+            "content": "That's really interesting! I would love to hear more...",
+            "role": "user"
+        }
+        {
+            "content": "Certainly! ....",
+            "role": "assistant"
+        },
+    ],
+    "prompt_id": "d938b65dfe31f05f80eb8572964c6673eddbd68eff3db6bd234d7f1e3b86c2af"
+}
+```
+
+## Citation
+
+If you find this dataset is useful in your work, please cite the original UltraChat dataset:
+
+```
+@misc{ding2023enhancing,
+      title={Enhancing Chat Language Models by Scaling High-quality Instructional Conversations}, 
+      author={Ning Ding and Yulin Chen and Bokai Xu and Yujia Qin and Zhi Zheng and Shengding Hu and Zhiyuan Liu and Maosong Sun and Bowen Zhou},
+      year={2023},
+      eprint={2305.14233},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```

data/ultrachat_200k/data/test_gen-00000-of-00001-3d4cd8309148a71f.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:937211dbe6e7db6315cd894efaedfe0231198a211ed5a1f856b0553a4b3844c0
+size 80359907

data/ultrachat_200k/data/test_sft-00000-of-00001-f7dfac4afe5b93f4.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c18fd6e77395577652bdefbc5a87044ea799e989451ecfca5c6cf977ae5c6f70
+size 81176553

data/ultrachat_200k/data/train_gen-00000-of-00003-a6c9fb894be3e50b.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8172417116ff18bb0d1ecc2166a658ebabf79a6c9cf02d8cad411fa764548663
+size 243687391

data/ultrachat_200k/data/train_gen-00001-of-00003-d6a0402e417f35ca.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8b9e9ce202df576400e6727fcabb027bc92449767bec00871190d81f1b330de1
+size 243300397

data/ultrachat_200k/data/train_gen-00002-of-00003-c0db75b92a2f48fd.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1fe9ba24c1bc2be447041e93aaca4689342e25470e6dedd4101019decf151922
+size 243497356

data/ultrachat_200k/data/train_sft-00000-of-00003-a3ecf92756993583.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:afa8fa7426081b2a0e732fb50dbb5cd402a28ad5f0dbe66c0d996d63e7220727
+size 243999189

data/ultrachat_200k/data/train_sft-00001-of-00003-0a1804bcb6ae68c6.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c8230190bc8b29084c974ff32f05589b458fad1908fc4b2fb2b9e1e9f7921f03
+size 243897199

data/ultrachat_200k/data/train_sft-00002-of-00003-ee46ed25cfae92c6.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6fe7d2a5e95cf660f972ecaf304aa5632d7f53384e4b0e0d0f44d9c96733c03e
+size 244131731

scripts/train_moe_multidomain.py

@@ -0,0 +1,135 @@
+"""
+Nova 2.0 MoE-Vision — Large-Scale Multi-Domain GPU Pre-Training Script
+Trains 522.8M Nova 2.0 MoE-Vision Model on AMD ROCm GPU (96 GB VRAM)
+across ultrachat_200k dataset downloaded via official 'hf' CLI.
+"""
+
+import os
+import sys
+import argparse
+import random
+import torch
+import torch.nn as nn
+
+# Add project root to sys.path
+sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
+
+from config.model_config import Nova1Config
+from src.tokenizer.bpe_tokenizer import Nova1Tokenizer
+from src.model.nova2_moe_vision import Nova2MoEVision
+from src.dataset.hf_dataset import create_hf_dataloader
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Pre-train Nova 2.0 MoE-Vision Model")
+    parser.add_argument("--tokenizer_path", type=str, default="checkpoints/nova1_gemini_tokenizer.json", help="Tokenizer JSON")
+    parser.add_argument("--epochs", type=int, default=5, help="Number of training epochs")
+    parser.add_argument("--batch_size", type=int, default=8, help="Batch size per step")
+    parser.add_argument("--lr", type=float, default=2e-4, help="Learning rate")
+    parser.add_argument("--num_experts", type=int, default=8, help="Number of MoE Experts")
+    parser.add_argument("--top_k", type=int, default=2, help="Top-K active experts per token")
+    parser.add_argument("--max_samples", type=int, default=20000, help="Max dataset samples")
+    args = parser.parse_args()
+
+    token = os.environ.get("HF_TOKEN")
+    tokenizer = Nova1Tokenizer.load(args.tokenizer_path)
+    print(f"Loaded 32K Tokenizer from '{args.tokenizer_path}'.", flush=True)
+
+    # Nova 2.0 MoE Config (~522.8M Parameters)
+    config = Nova1Config(
+        vocab_size=tokenizer.vocab_size,
+        d_model=768,
+        n_heads=12,
+        d_ff=3072,
+        n_layers_L=4,
+        n_layers_H=4,
+        max_seq_len=512,
+        learning_rate=args.lr,
+        device="cuda" if torch.cuda.is_available() else "cpu",
+        dtype="bfloat16"
+    )
+
+    print(f"\n=======================================================", flush=True)
+    print(f"🚀 Launching Nova 2.0 Sparse MoE Multi-Domain Engine", flush=True)
+    print(f"Device: {config.device.upper()} | Experts: {args.num_experts} (Top-{args.top_k}) | Precision: {config.dtype}", flush=True)
+    print(f"=======================================================\n", flush=True)
+
+    model = Nova2MoEVision(config, num_experts=args.num_experts, top_k=args.top_k, vision_dim=768).to(config.device)
+
+    # Load previous Nova 2.0 MoE checkpoint if available
+    if os.path.exists("checkpoints/nova2_moe_final.pt"):
+        try:
+            ckpt = torch.load("checkpoints/nova2_moe_final.pt", map_location="cpu", weights_only=False)
+            model.load_state_dict(ckpt["model_state"], strict=False)
+            print("Loaded warm-start weights from 'checkpoints/nova2_moe_final.pt'.", flush=True)
+        except Exception as e:
+            print(f"Notice loading checkpoint: {e}", flush=True)
+
+    num_params = sum(p.numel() for p in model.parameters())
+    print(f"🌟 Nova 2.0 MoE Total Parameters: {num_params:,} (~{num_params / 1e6:.1f} Million)", flush=True)
+
+    # DataLoader
+    dataloader = create_hf_dataloader(
+        dataset_name="data/ultrachat_200k",
+        tokenizer=tokenizer,
+        max_seq_len=config.max_seq_len,
+        batch_size=args.batch_size,
+        split="train_sft",
+        max_samples=args.max_samples
+    )
+
+    optimizer = torch.optim.AdamW(model.parameters(), lr=args.lr, betas=(0.9, 0.95), weight_decay=0.01)
+    criterion = nn.CrossEntropyLoss(ignore_index=tokenizer.pad_id)
+
+    print(f"\nStarting Nova 2.0 MoE Pre-Training on AMD ROCm GPU across {args.max_samples:,} samples...", flush=True)
+
+    for epoch in range(args.epochs):
+        model.train()
+        total_loss = 0.0
+        total_aux = 0.0
+        steps_per_epoch = len(dataloader)
+
+        for step, (b_inp, b_tgt) in enumerate(dataloader):
+            b_inp = b_inp.to(config.device)
+            b_tgt = b_tgt.to(config.device)
+
+            # Synthetic multi-modal vision features (16 patches x 768 dim) for 20% of batches
+            pixel_inp = torch.randn(b_inp.shape[0], 16, 768, device=config.device) if random.random() < 0.2 else None
+
+            optimizer.zero_grad()
+            with torch.amp.autocast("cuda", enabled=(config.device == "cuda"), dtype=config.get_torch_dtype()):
+                logits, _, _, aux_loss = model(b_inp, pixel_features=pixel_inp)
+                
+                if logits.shape[1] > b_tgt.shape[1]:
+                    target_len = b_tgt.shape[1]
+                    logits_text = logits[:, -target_len:, :]
+                else:
+                    logits_text = logits
+
+                loss_lm = criterion(logits_text.reshape(-1, config.vocab_size), b_tgt.reshape(-1))
+                total_step_loss = loss_lm + 0.01 * aux_loss
+
+            total_step_loss.backward()
+            torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
+            optimizer.step()
+
+            total_loss += loss_lm.item()
+            total_aux += aux_loss.item()
+
+            if (step + 1) % 10 == 0 or (step + 1) == steps_per_epoch:
+                print(f"MoE Epoch {epoch+1}/{args.epochs} [{step+1}/{steps_per_epoch}] — LM Loss: {loss_lm.item():.4f} | Aux Loss: {aux_loss.item():.2f}", flush=True)
+
+        avg_lm_loss = total_loss / steps_per_epoch
+        print(f"MoE Epoch {epoch+1}/{args.epochs} Complete — Avg LM Loss: {avg_lm_loss:.4f}\n", flush=True)
+
+        # Save Checkpoint
+        ckpt_path = f"checkpoints/nova2_moe_epoch_{epoch+1}.pt"
+        torch.save({"model_state": model.state_dict(), "config": config}, ckpt_path)
+
+    final_path = "checkpoints/nova2_moe_multidomain.pt"
+    torch.save({"model_state": model.state_dict(), "config": config}, final_path)
+    print(f"🚀 Nova 2.0 MoE Pre-Training Complete! Model saved to '{final_path}'.", flush=True)
+
+
+if __name__ == "__main__":
+    main()