Anserwise commited on
Commit
a8b2e24
·
verified ·
1 Parent(s): 1808112

docs: Add VIDRAFT Darwin platform breeding/evolution description

Browse files
Files changed (1) hide show
  1. README.md +112 -20
README.md CHANGED
@@ -10,8 +10,12 @@ tags:
10
  - think
11
  - gemma
12
  - gemma-4
 
13
  - reasoning
14
  - distillation
 
 
 
15
  - ko
16
  - en
17
  base_model:
@@ -45,69 +49,157 @@ model-index:
45
 
46
  # AWAXIS-Think-31B
47
 
48
- **AWAXIS-Think-31B** is a 31B-parameter Korean/English reasoning model built via the Darwin V8 FFN-crossbreed merge engine.
49
 
50
- ## Build recipe (honest disclosure)
51
 
52
- - **Mother (kept full)**: [TeichAI/gemma-4-31B-it-Claude-Opus-Distill-v2](https://huggingface.co/TeichAI/gemma-4-31B-it-Claude-Opus-Distill-v2) ??reasoning-distill base, retained 100% (incl. `<think>` chain-of-thought style)
53
- - **Father (FFN donor)**: [google/gemma-4-31B-it](https://huggingface.co/google/gemma-4-31B-it) ??base Gemma-4 FFN tensors blended at **慣 = 0.1**
54
- - **Method**: per-layer FFN blend `w = w_mother*(1-慣) + w_father*慣` on `mlp.{gate,up,down}_proj` + `pre/post_feedforward_layernorm` for all 60 language-model layers; grid search 慣??0.1, 0.2, 0.3, 0.4} on CLIcK-50 ??best 慣=0.1 (CLIcK-200 = 86.0%)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
55
  - **Architecture**: `Gemma4ForConditionalGeneration` (multimodal wrapper; text generation primary)
56
  - **Tokenizer**: Gemma-4 (vocab 262,144)
57
 
58
- ## Measured benchmarks
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
59
 
60
  | Benchmark | Setting | Result |
61
  |-----------|---------|--------|
62
  | GPQA Diamond 20Q (seed 42) | greedy, max_new_tokens=**4096**, 2-way DP | **12/20 = 60.0%** (16/20 still hit token cap, 0 null) |
63
- | GPQA Diamond 20Q (seed 42) | greedy, max_new_tokens=**2048** | 9/20 = 45.0% (16/20 truncated, 2 null) ??*truncation artifact, included for transparency* |
64
- | CLIcK (Korean) 200Q | greedy -grid winner | 86.0% |
65
 
66
- ### Honest caveats
67
- - GPQA 60% is from **n=20** (small sample). 16/20 still hit the 4096-token cap ??real ceiling may be higher with longer generation budget.
68
- - Comparison to random baseline: GPQA random 25% ??+35pp clear learning signal.
69
  - The full GPQA Diamond (198Q) and other broad suites have not yet been measured for this exact merged artifact.
70
- - The model retains the **Mother's `<think>...</think>` reasoning template** ??strip via post-processing if undesired.
71
 
72
- ## Intended use
 
 
73
 
74
  - Korean/English step-by-step reasoning, instruction following, knowledge QA
75
  - The `Think` suffix reflects the inherited Opus-distilled chain-of-thought behavior
 
 
 
76
 
77
- ## Out-of-scope / limitations
 
 
78
 
79
- - Not a final clinical/legal advisor; outputs may be confidently wrong on hard graduate-level questions (40% wrong on the GPQA-20 set).
80
- - Inherits Gemma-4 base limitations (multimodal wrapper retained; image inputs not the primary use-case here).
81
- - Subject to Gemma Terms of Use; see parent model cards for derivative-use clauses.
82
 
83
  ## Inference
84
 
85
  ```python
86
  from transformers import AutoTokenizer, AutoModelForCausalLM
87
  import torch
 
88
  tok = AutoTokenizer.from_pretrained("Anserwise/AWAXIS-Think-31B", trust_remote_code=True)
89
  model = AutoModelForCausalLM.from_pretrained(
90
  "Anserwise/AWAXIS-Think-31B",
91
  dtype=torch.bfloat16,
92
  device_map="auto",
93
  trust_remote_code=True,
94
- attn_implementation="eager", # required for the Gemma4 multimodal wrapper
95
  )
96
- msgs = [{"role": "user", "content": "?쒓뎅?대줈 ?먯떊???뚭컻??二쇱꽭??"}]
 
97
  text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
98
  inp = tok(text, return_tensors="pt").to(model.device)
99
  out = model.generate(**inp, max_new_tokens=2048, do_sample=False)
100
  print(tok.decode(out[0][inp["input_ids"].shape[-1]:], skip_special_tokens=True))
101
  ```
102
 
 
 
103
  ## License
104
 
105
  Gemma Terms of Use (inherited from base). Use of this model is bound by [Google Gemma Terms](https://ai.google.dev/gemma/terms).
106
 
107
  ## Acknowledgements
108
 
 
109
  - TeichAI for the Opus-Distill base
110
  - Google DeepMind for Gemma-4
111
 
112
  ---
113
- *Built with Darwin V8 FFN-crossbreed merge engine. Measured numbers above are exact; nothing inflated.*
 
 
10
  - think
11
  - gemma
12
  - gemma-4
13
+ - gemma4
14
  - reasoning
15
  - distillation
16
+ - darwin-derived
17
+ - vidraft
18
+ - darwin-crossbreed
19
  - ko
20
  - en
21
  base_model:
 
49
 
50
  # AWAXIS-Think-31B
51
 
52
+ ## Overview
53
 
54
+ **AWAXIS-Think-31B** is a 31B-parameter Korean/English reasoning model created through the **[VIDRAFT](https://huggingface.co/VIDraft) Darwin AI Model Breeding/Evolution Platform**. This model was produced using Darwin's proprietary **FFN-crossbreed merge engine (V8)**, which emulates biological crossbreeding between AI models to create offspring with combined strengths of both parents.
55
 
56
+ > AWAXIS-Think-31B은 **VIDRAFT Darwin AI 모델 교배/진화 플랫폼**을 통해 생성된 31B 파라미터 한국어/영어 추론 모델입니다.
57
+
58
+ ---
59
+
60
+ ## VIDRAFT Darwin Platform
61
+
62
+ **[VIDRAFT Darwin](https://huggingface.co/VIDraft)**은 AI 모델의 **교배(Crossbreeding)와 진화(Evolution)**를 통해 새로운 고성능 모델을 자동 생성하는 플랫폼입니다. 생물학적 유전 원리에서 영감을 받아, 두 개 이상의 부모 모델에서 각각의 장점을 선택적으로 결합하여 자식 모델을 탄생시킵니다.
63
+
64
+ ### Darwin 교배/진화 핵심 기술
65
+
66
+ | 기술 | 설명 |
67
+ |------|------|
68
+ | **FFN Crossbreed Engine (V8)** | 부모 모델의 Feed-Forward Network(FFN) 레이어를 선택적으로 교차 결합하는 핵심 엔진. 어텐션·임베딩은 어머니(Mother)에서, FFN 시그널은 아버지(Father)에서 추출하여 블렌딩 |
69
+ | **Smart MRI (Model Resonance Imaging)** | 두 모델 간 레이어별 유사도·호환성을 분석하여 최적 교배 비율(alpha)을 자동 탐색하는 기술 |
70
+ | **Alpha Grid Search** | 교배 비율 alpha를 체계적으로 탐색(0.1~0.4)하여 벤치마크 성능이 최대화되는 최적점을 발견 |
71
+ | **Multi-Generation Breeding** | 1세대 교배 결과물을 다시 부모로 삼아 2세대, 3세대 교배를 수행하는 다세대 진화 |
72
+
73
+ ### Darwin 교배 프로세스 (이 모델의 생성 과정)
74
+
75
+ ```
76
+ [Step 1] 부모 선정 (Parent Selection)
77
+ - Mother: TeichAI/gemma-4-31B-it-Claude-Opus-Distill-v2 (추론 능력 기반)
78
+ - Father: google/gemma-4-31B-it (Gemma-4 원본 FFN 기여)
79
+
80
+ [Step 2] Smart MRI 호환성 분석
81
+ - 두 모델의 60개 레이어별 FFN 텐서 유사도 스캔
82
+ - 아키텍처 호환성 확인 (동일 Gemma-4 family = 100% 호환)
83
+
84
+ [Step 3] FFN Crossbreed (교배 실행)
85
+ - 어머니의 어텐션, 임베딩, 라우팅 = 100% 보존
86
+ - 아버지의 FFN (gate_proj, up_proj, down_proj) = alpha 비율로 블렌딩
87
+ - 수식: w_child = w_mother * (1 - alpha) + w_father * alpha
88
+
89
+ [Step 4] Alpha Grid Search (최적 교배 비율 탐색)
90
+ - alpha = {0.1, 0.2, 0.3, 0.4} 4종 생성
91
+ - CLIcK-50 벤치마크로 각 alpha 평가
92
+ - 최적: alpha = 0.1 (CLIcK-200 = 86.0%)
93
+
94
+ [Step 5] 검증 및 출시
95
+ - GPQA Diamond, CLIcK 등 벤치마크 검증
96
+ - HuggingFace 모델 허브 공개
97
+ ```
98
+
99
+ ### 왜 Darwin 교배인가?
100
+
101
+ 기존 모델 합성 방식(단순 가중치 평균, SLERP, TIES 등)과 달리, Darwin 교배는:
102
+
103
+ 1. **생물학적 유전 모방**: 어머니/아버지 역할을 명확히 분리하여 각 부모의 핵심 능력만 선택적으로 상속
104
+ 2. **FFN 선택적 주입**: 어텐션(문맥 이해)은 어머니에서 100% 보존하고, FFN(지식·추론 패턴)만 아버지에서 교차 → 능력 충돌 최소화
105
+ 3. **벤치마크 기반 자연선택**: alpha grid search로 여러 자식 후보를 생성한 뒤, 실측 벤치마크로 최적 개체를 선택 (= 자연선택 시뮬레이션)
106
+ 4. **다세대 진화 가능**: 이 모델(AWAXIS-Think-31B)이 다시 AWAXIS-KR-31B의 아버지가 되어 2세대 교배 수행 → 능력 누적 진화
107
+
108
+ ---
109
+
110
+ ## Build Recipe (Honest Disclosure)
111
+
112
+ - **Mother (kept full)**: [TeichAI/gemma-4-31B-it-Claude-Opus-Distill-v2](https://huggingface.co/TeichAI/gemma-4-31B-it-Claude-Opus-Distill-v2) — reasoning-distill base, retained 100% (incl. `<think>` chain-of-thought style)
113
+ - **Father (FFN donor)**: [google/gemma-4-31B-it](https://huggingface.co/google/gemma-4-31B-it) — base Gemma-4 FFN tensors blended at **alpha = 0.1**
114
+ - **Method**: Darwin V8 FFN-crossbreed — per-layer FFN blend `w = w_mother*(1-alpha) + w_father*alpha` on `mlp.{gate,up,down}_proj` + `pre/post_feedforward_layernorm` for all 60 language-model layers; grid search alpha in {0.1, 0.2, 0.3, 0.4} on CLIcK-50 — best alpha=0.1 (CLIcK-200 = 86.0%)
115
+ - **Platform**: **VIDRAFT Darwin AI Model Breeding/Evolution Platform** ([VIDraft on HuggingFace](https://huggingface.co/VIDraft))
116
  - **Architecture**: `Gemma4ForConditionalGeneration` (multimodal wrapper; text generation primary)
117
  - **Tokenizer**: Gemma-4 (vocab 262,144)
118
 
119
+ ---
120
+
121
+ ## Model Lineage (Genealogy)
122
+
123
+ ```
124
+ AWAXIS-Think-31B (this model -- Darwin V8 FFN-crossbreed)
125
+ |
126
+ +-- Mother (kept full, 100%)
127
+ | TeichAI/gemma-4-31B-it-Claude-Opus-Distill-v2
128
+ | -- Claude Opus reasoning distill base
129
+ |
130
+ +-- Father (FFN donor, alpha=0.1)
131
+ google/gemma-4-31B-it
132
+ -- Gemma-4 base FFN tensors
133
+ ```
134
+
135
+ **Common ancestor**: Google **Gemma-4** architecture.
136
+
137
+ ---
138
+
139
+ ## Measured Benchmarks
140
 
141
  | Benchmark | Setting | Result |
142
  |-----------|---------|--------|
143
  | GPQA Diamond 20Q (seed 42) | greedy, max_new_tokens=**4096**, 2-way DP | **12/20 = 60.0%** (16/20 still hit token cap, 0 null) |
144
+ | GPQA Diamond 20Q (seed 42) | greedy, max_new_tokens=**2048** | 9/20 = 45.0% (16/20 truncated, 2 null) *truncation artifact, included for transparency* |
145
+ | CLIcK (Korean) 200Q | greedy alpha-grid winner | 86.0% |
146
 
147
+ ### Honest Caveats
148
+ - GPQA 60% is from **n=20** (small sample). 16/20 still hit the 4096-token cap real ceiling may be higher with longer generation budget.
149
+ - Comparison to random baseline: GPQA random 25% +35pp clear learning signal.
150
  - The full GPQA Diamond (198Q) and other broad suites have not yet been measured for this exact merged artifact.
151
+ - The model retains the **Mother's `<think>...</think>` reasoning template** strip via post-processing if undesired.
152
 
153
+ ---
154
+
155
+ ## Intended Use
156
 
157
  - Korean/English step-by-step reasoning, instruction following, knowledge QA
158
  - The `Think` suffix reflects the inherited Opus-distilled chain-of-thought behavior
159
+ - **2nd-generation breeding parent**: This model served as the Father for [AWAXIS-KR-31B](https://huggingface.co/Anserwise/AWAXIS-KR-31B), demonstrating Darwin's multi-generation evolution capability
160
+
161
+ ## Out-of-Scope / Limitations
162
 
163
+ - Not a final clinical/legal advisor; outputs may be confidently wrong on hard graduate-level questions
164
+ - Inherits Gemma-4 base limitations (multimodal wrapper retained; image inputs not the primary use-case here)
165
+ - Subject to Gemma Terms of Use; see parent model cards for derivative-use clauses
166
 
167
+ ---
 
 
168
 
169
  ## Inference
170
 
171
  ```python
172
  from transformers import AutoTokenizer, AutoModelForCausalLM
173
  import torch
174
+
175
  tok = AutoTokenizer.from_pretrained("Anserwise/AWAXIS-Think-31B", trust_remote_code=True)
176
  model = AutoModelForCausalLM.from_pretrained(
177
  "Anserwise/AWAXIS-Think-31B",
178
  dtype=torch.bfloat16,
179
  device_map="auto",
180
  trust_remote_code=True,
181
+ attn_implementation="eager",
182
  )
183
+
184
+ msgs = [{"role": "user", "content": "양자역학의 불확정성 원리를 자세히 설명해 주세요."}]
185
  text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
186
  inp = tok(text, return_tensors="pt").to(model.device)
187
  out = model.generate(**inp, max_new_tokens=2048, do_sample=False)
188
  print(tok.decode(out[0][inp["input_ids"].shape[-1]:], skip_special_tokens=True))
189
  ```
190
 
191
+ ---
192
+
193
  ## License
194
 
195
  Gemma Terms of Use (inherited from base). Use of this model is bound by [Google Gemma Terms](https://ai.google.dev/gemma/terms).
196
 
197
  ## Acknowledgements
198
 
199
+ - **[VIDRAFT](https://huggingface.co/VIDraft)** — Darwin AI Model Breeding/Evolution Platform
200
  - TeichAI for the Opus-Distill base
201
  - Google DeepMind for Gemma-4
202
 
203
  ---
204
+
205
+ *Built with the **VIDRAFT Darwin AI Model Breeding/Evolution Platform** — FFN-crossbreed V8 engine. This model was generated through Darwin's automated crossbreeding process, which selectively combines the strengths of parent models using biologically-inspired genetic algorithms. Measured numbers above are exact; nothing inflated.*