BenevolenceMessiah
/

Yi-Coder-9B-Chat-Instruct-TIES-MoE-v1.0

@@ -6,6 +6,8 @@ tags:
 - mergekit
 - merge
 license: apache-2.0
 ---
 # merge
@@ -90,7 +92,7 @@ This resulted in the creation of [BenevolenceMessiah/Yi-Coder-9B-Chat-8x-MoE](ht
 5. **Efficient Model Scaling**:
    Sparse upcycling allows for **scaling** the model in a way that adds capacity without a proportional increase in computation costs. This is ideal for applications where high capacity is needed, but hardware or energy constraints make it impractical to compute all parameters for every input.
-Resultantly, this model went from 8.83B parameters to 53.B parameters:
 - **8 experts**: Each expert is another copy of the base model (9B each), leading to \( 8 \times 9B = 72B \) additional
    parameters. The total number of parameters is therefore around **81B** (9B + 72B). However, since only a fraction of the experts are used for each token (due to sparse activation), the model does not use all 54.3B parameters during every inference step. This makes the model scalable without requiring computation over the entire parameter space every time.
 - **Sparse Upcycling and Gating Mechanism**:
@@ -113,15 +115,35 @@ Our innovative training regimen involved configuring distinct experts within the
 By integrating advanced tools and technologies such as Docker, Kubernetes, and Prometheus within the prompts, the experts were trained to produce code aligned with modern DevOps practices. The merge methods combined the strengths of the individual models while mitigating their limitations, resulting in a model capable of handling a wide array of complex coding tasks across multiple programming languages.
-Model Tree and Resources
 The development process can be visualized through the following model hierarchy:
 01-ai/Yi-Coder-9B (Base Model)
-01-ai/Yi-Coder-9B-Chat (Chat-Fine-Tuned Model)
-BenevolenceMessiah/Yi-Coder-9B-Chat-Instruct-TIES (Result of TIES Merge)
-BenevolenceMessiah/Yi-Coder-9B-Chat-8x-MoE (Result of Sparse Upcycling Merge)
-BenevolenceMessiah/Yi-Coder-9B-Chat-Instruct-TIES-MoE-v1.0 (Final Integrated Model)
 ### Configuration
@@ -413,18 +435,16 @@ experts:
 - Maintainability and Scalability: Promotes the creation of modular, scalable, and maintainable codebases, essential for large-scale applications.
 Coming Soon:
-BenevolenceMessiah/Yi-Coder-9B-Chat-Instruct-TIES-MoE-v1.0-DELLA (DELLA pruned version of this Final Integrated Model)
 #### Intended Outcomes
 The primary objectives of this project are:
 - `Versatility and Reliability`: Develop a highly versatile MoE model capable of generating production-ready code across numerous programming languages and complex scenarios.
 - `Enhanced Efficiency`: Utilize sparse activation and the MoE architecture to increase model capacity without a proportional increase in computational costs.
 - `Code Quality and Security`: Ensure the generated code adheres to industry best practices, is secure, and is optimized for performance and scalability.
 - `Accelerated Development`: Reduce the time and effort required for software development by providing comprehensive, deployable code snippets that integrate seamlessly into various development pipelines.
 #### Conclusion

 - mergekit
 - merge
 license: apache-2.0
+metrics:
+- code_eval
 ---
 # merge
 5. **Efficient Model Scaling**:
    Sparse upcycling allows for **scaling** the model in a way that adds capacity without a proportional increase in computation costs. This is ideal for applications where high capacity is needed, but hardware or energy constraints make it impractical to compute all parameters for every input.
+Resultantly, this model went from 8.83B parameters to 54.3B parameters:
 - **8 experts**: Each expert is another copy of the base model (9B each), leading to \( 8 \times 9B = 72B \) additional
    parameters. The total number of parameters is therefore around **81B** (9B + 72B). However, since only a fraction of the experts are used for each token (due to sparse activation), the model does not use all 54.3B parameters during every inference step. This makes the model scalable without requiring computation over the entire parameter space every time.
 - **Sparse Upcycling and Gating Mechanism**:
 By integrating advanced tools and technologies such as Docker, Kubernetes, and Prometheus within the prompts, the experts were trained to produce code aligned with modern DevOps practices. The merge methods combined the strengths of the individual models while mitigating their limitations, resulting in a model capable of handling a wide array of complex coding tasks across multiple programming languages.
+#### Model Tree and Resources
 The development process can be visualized through the following model hierarchy:
+1. `01-ai/Yi-Coder-9B` (Base Model)
+2. `01-ai/Yi-Coder-9B-Chat` (Chat-Fine-Tuned Model)
+3. `BenevolenceMessiah/Yi-Coder-9B-Chat-Instruct-TIES` (Result of TIES Merge)
+4. `BenevolenceMessiah/Yi-Coder-9B-Chat-8x-MoE` (Result of Sparse Upcycling Merge)
+5. `BenevolenceMessiah/Yi-Coder-9B-Chat-Instruct-TIES-MoE-v1.0` (Final Integrated (This)Model)
+````
+Yi-Coder-9B Model Family Tree
+=============================
 01-ai/Yi-Coder-9B (Base Model)
+│
+├── 01-ai/Yi-Coder-9B-Chat (Fine-Tuned Model)
+│   │
+│   ├── BenevolenceMessiah/Yi-Coder-9B-Chat-Instruct-TIES (TIES Merged Model)
+│   │   │
+│   │   └── BenevolenceMessiah/Yi-Coder-9B-Chat-Instruct-TIES-MoE-v1.0 (Final Integrated Model)
+│   │
+│   └── BenevolenceMessiah/Yi-Coder-9B-Chat-8x-MoE (Sparse Upcycling Merge)
+│       │
+│       └── BenevolenceMessiah/Yi-Coder-9B-Chat-Instruct-TIES-MoE-v1.0 (Final Integrated Model)
+│
+└── BenevolenceMessiah/Yi-Coder-9B-Chat-Instruct-TIES-MoE-v1.0 (Final Integrated Model)
+````
 ### Configuration
 - Maintainability and Scalability: Promotes the creation of modular, scalable, and maintainable codebases, essential for large-scale applications.
 Coming Soon:
+- BenevolenceMessiah/Yi-Coder-9B-Chat-Instruct-TIES-MoE-v1.0-DELLA (DELLA pruned version of this Final Integrated Model)
+- BenevolenceMessiah/Yi-Coder-9B-Chat-Instruct-TIES-MoE-v2.X (Can we get any experimental? You bet! Did we break down the barriers and mechanics between training, finetuning, and merging?? Maybe...)
 #### Intended Outcomes
 The primary objectives of this project are:
 - `Versatility and Reliability`: Develop a highly versatile MoE model capable of generating production-ready code across numerous programming languages and complex scenarios.
 - `Enhanced Efficiency`: Utilize sparse activation and the MoE architecture to increase model capacity without a proportional increase in computational costs.
 - `Code Quality and Security`: Ensure the generated code adheres to industry best practices, is secure, and is optimized for performance and scalability.
 - `Accelerated Development`: Reduce the time and effort required for software development by providing comprehensive, deployable code snippets that integrate seamlessly into various development pipelines.
 #### Conclusion