update en&ja README

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tech-dev.png filter=lfs diff=lfs merge=lfs -text

README-ja.md

@@ -0,0 +1,109 @@
+---
+license: apache-2.0
+license_link: https://huggingface.co/Qwen/Qwen2.5-32B-Instruct/blob/main/LICENSE
+language:
+- en
+pipeline_tag: text-generation
+base_model: Qwen/Qwen2.5-32B
+tags:
+- chat
+library_name: transformers
+---
+
+# Flux-Japanese-Qwen2.5-32B-Instruct-V1.0
+[[English](./README.md)] [**Japanese**]
+Flux-Japanese-Qwen2.5-32B-Instruct-V1.0は、320億個のパラメータを持つオープンウェイトの大規模言語モデルです。日本語に関する深い知識と、高度な推論能力および言語能力を特長としています。Qwen2.5‑32B‑Instruct をベースとしてトレーニングされ、Apache 2.0 オープンソースライセンスの下で提供されています。
+
+# 🏆 Open-Japanese-LLM-Leaderboard 第1位
+[Open LLM Japanese LLM Leaderboard](https://huggingface.co/spaces/deep-analysis-research/open-japanese-llm-leaderboard) において、以下の結果となりました。
+-	ベースモデル：「Qwen2.5‑32B‑Instruct 」（平均スコア：0.6553）
+-	元評価トップのモデル：「D2IL‑Japanese‑Qwen2.5‑32B‑Instruct‑v0.1 」（平均スコア： 0.7100）
+-	本モデル：「Flux‑Japanese‑Qwen2.5‑32B‑V1.0 」（平均スコア：0.7417）
+ベースモデルである Qwen2.5‑32B‑Instruct と比較すると、本モデルはほとんどのタスクで性能が大幅に向上し、特に FA（Fundamental Analysis／基礎分析）、SUM（Summarization／要約）、CG（Code Generation／コード生成） において顕著な改善が見られます。
+
+| Tasks                              | Qwen2.5-32B-Instruct | D2IL-Japanese-Qwen2.5-32B-Instruct-v0.1 | Flux-Japanese-Qwen2.5-32B-Instruct-V1.0 |
+|------------------------------------|----------------------|-----------------------------------------|------------------------------------------|
+| NLI - 自然言語推論                  | 0.8106               | 0.8793                                  | 0.8846 (+0.0740)                         |
+| QA - 質問応答                       | 0.541                | 0.5897                                  | 0.5965 (+0.0555)                         |
+| RC - 読解力                         | 0.9047               | 0.9005                                  | 0.9261 (+0.0214)                         |
+| MC - 多肢選択式質問応答              | 0.8966               | 0.9139                                  | 0.9128 (+0.0162)                         |
+| EL - エンティティリンキン            | 0.5894               | 0.6782                                  | 0.6975 (+0.1081)                         |
+| FA - 基礎分析                       | 0.2737               | 0.4321                                  | 0.5185 (+0.2448)                         |
+| MR - 数学的推論                     | 0.944                | 0.938                                   | 0.9420 (-0.0020)                         |
+| MT - 機械翻訳                       | 0.8479               | 0.7954                                  | 0.8389 (-0.0090)                         |
+| HE - 試験問題                       | 0.7757               | 0.7902                                  | 0.7987 (+0.0230)                         |
+| CG - コード生成                     | 0.5281               | 0.6084                                  | 0.7610 (+0.2329)                         |
+| SUM - 要約                          | 0.097                | 0.2843                                  | 0.2827 (+0.1857)                         |
+| **Average**                         | **0.6553**           | **0.71**                                | **0.7417 (+0.0864)**                     |
+
+
+# 🚀 一貫した汎用性能
+Flux‑Japanese‑Qwen2.5‑32B‑Instruct‑V1.0は学習により日本語能力が大幅に改善した一方で、汎用的なタスクや英語でのタスク遂行能力においても性能を維持しており、ベースモデルである「Qwen2.5-32B-Instruct」と比較して、1%以下というごく僅かな差にとどまっています。評価は [simple-evals](https://github.com/deep-analysis-research/simple-evals) に基づいて行われています。
+
+| Tasks               | Dataset        | Qwen2.5-32B-Instruct | Flux-Japanese-Qwen2.5-32B-Instruct-V1.0 |
+|---------------------|----------------|----------------------|------------------------------------------|
+| General Tasks       | MMLU-redux     | 80.37                | 80.03 (-0.34)                            |
+|                     | GPQGA-Diamond  | 46.11                | 47.32 (+1.21)                            |
+|                     | MMLU           | 82.84                | 83.39 (+0.55)                            |
+| Math Tasks          | MATH-500       | 78.14                | 78.50 (+0.36)                            |
+|                     | AIME24         | 17.06                | 17.92 (+0.86)                            |
+|                     | AIME25         | 16.25                | 14.58 (-1.67)                            |
+|                     | MT-AIME24      | 12.73                | 12.97 (+0.24)                            |
+| Multilingual Tasks  | Multi-IF       | 71.85                | 63.45 (-8.40)                            |
+|                     | INCLUDE        | 65.16                | 64.64 (-0.52)                            |
+|                     | MMMLU          | 73.43                | 74.08 (+0.65)                            |
+| Coding Tasks        | HumanEval      | 87.93                | 86.51 (-1.42)                            |
+| Alignment Tasks     | IFEval         | 78.37                | 77.46 (-0.91)                            |
+| **Average**             |                | **59.17**                | **58.40 (-0.77)**                            |
+
+
+# ⚙️ 技術開発
+
+![](./tech-dev.png)
+
+- **Phase 1: Interpretability Analysis & Pinpoint Tuning** — 日本語能力（知識・推論・言語）向上のために、Mechanistic Interpretability (MI)技術を活用することで、特定のパスと回路を識別し、僅か5%のパラメータに対してPinpoint Tuningを実施する方法を採用しました。これにより日本語の知識・推論・言語のそれぞれの能力に特化した3つのエキスパートモデルを作成しました。
+- **Phase 2: Pinpoint Merging** — 3つのエキスパートモデルに対し、Pinpoint Merging を実施し、日本語の知識・推論・言語の各分野でエキスパートレベルの性能を備えた”日本語特化”統合モデルを構築します。 [[Pinpoint Merging のコード](https://github.com/deep-analysis-research/SLTA)].
+
+# 🚩 クイックスタート
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+device = "cuda" # the device to load the model onto
+
+model = AutoModelForCausalLM.from_pretrained(
+    "Deep-Analysis-Research/Flux-Japanese-Qwen2.5-32B-V1.0",
+    torch_dtype="auto",
+    device_map="auto"
+)
+tokenizer = AutoTokenizer.from_pretrained("Deep-Analysis-Research/Flux-Japanese-Qwen2.5-32B-V1.0")
+
+prompt = "大規模言語モデルについて簡単に紹介してください。"
+messages = [
+    {"role": "user", "content": prompt}
+]
+text = tokenizer.apply_chat_template(
+    messages,
+    tokenize=False,
+    add_generation_prompt=True
+)
+model_inputs = tokenizer([text], return_tensors="pt").to(device)
+
+generated_ids = model.generate(
+    model_inputs.input_ids,
+    max_new_tokens=512
+)
+generated_ids = [
+    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
+]
+
+response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
+```
+# 💡 利用規約
+本モデルは、バイアスや有害な応答をはじめとする各種リスクを低減するため、様々な技術的アプローチを用いて、その安全性と信頼性の向上に努めております。
+しかし、本モデルを含む全ての大規模言語モデル（LLM）には、不正確な情報、誤解を招く内容、あるいは偏見を反映した、意図しない応答を生成する可能性が依然として存在します。本モデルをダウンロード、利用、または対話形式で使用することにより、利用者は以下の事項を理解し、これに同意したものとみなされます。
+1. 禁止事項
+   - 利用者は、本モデルを、詐欺、濫用、嫌がらせ、プライバシー侵害、悪意のあるコンテンツの作成／拡散などを含む、違法、または法令に違反する悪質な活動に利用することは禁止します。
+2. 利用者の責任
+   - 本モデルの使用方法及びその利用に起因して生じるすべての事象については、利用者自身が単独で責任を負うこととします。
+   - 本モデルの公開に関与した作成者・公開者及び機関は、その使用に起因して生じるいかなる結果についても、一切の責任を負いません。
+3. 保証
+   - 本モデルは現状の状態で提供され、いかなる保証も伴いません。

README.md

@@ -10,58 +10,69 @@ tags:
 library_name: transformers
 ---
 
-# Qwen2.5-32B-Instruct
-
-## Introduction
-
-Qwen2.5 is the latest series of Qwen large language models. For Qwen2.5, we release a number of base language models and instruction-tuned language models ranging from 0.5 to 72 billion parameters. Qwen2.5 brings the following improvements upon Qwen2:
-
-- Significantly **more knowledge** and has greatly improved capabilities in **coding** and **mathematics**, thanks to our specialized expert models in these domains.
-- Significant improvements in **instruction following**, **generating long texts** (over 8K tokens), **understanding structured data** (e.g, tables), and **generating structured outputs** especially JSON. **More resilient to the diversity of system prompts**, enhancing role-play implementation and condition-setting for chatbots.
-- **Long-context Support** up to 128K tokens and can generate up to 8K tokens.
-- **Multilingual support** for over 29 languages, including Chinese, English, French, Spanish, Portuguese, German, Italian, Russian, Japanese, Korean, Vietnamese, Thai, Arabic, and more. 
-
-**This repo contains the instruction-tuned 32B Qwen2.5 model**, which has the following features:
-- Type: Causal Language Models
-- Training Stage: Pretraining & Post-training
-- Architecture: transformers with RoPE, SwiGLU, RMSNorm, and Attention QKV bias
-- Number of Parameters: 32.5B
-- Number of Paramaters (Non-Embedding): 31.0B
-- Number of Layers: 64
-- Number of Attention Heads (GQA): 40 for Q and 8 for KV
-- Context Length: Full 131,072 tokens and generation 8192 tokens
-  - Please refer to [this section](#processing-long-texts) for detailed instructions on how to deploy Qwen2.5 for handling long texts.
-
-For more details, please refer to our [blog](https://qwenlm.github.io/blog/qwen2.5/), [GitHub](https://github.com/QwenLM/Qwen2.5), and [Documentation](https://qwen.readthedocs.io/en/latest/).
-
-## Requirements
-
-The code of Qwen2.5 has been in the latest Hugging face `transformers` and we advise you to use the latest version of `transformers`.
-
-With `transformers<4.37.0`, you will encounter the following error:
-```
-KeyError: 'qwen2'
-```
-
-## Quickstart
-
-Here provides a code snippet with `apply_chat_template` to show you how to load the tokenizer and model and how to generate contents.
-
+# Flux-Japanese-Qwen2.5-32B-Instruct-V1.0
+[**English**] [[Japanese](./README-ja.md)]
+Flux-Japanese-Qwen2.5-32B-Instruct-V1.0 is a 32 billion parameter open-weights models with strong performance in Japanese knowledge, reasoning and language. It is trained based on the Qwen2.5-32B-Instruct and under the Apache 2.0 opensource licence
+
+# 🏆 Open-Japanese-LLM-Leaderboard Rank-1
+On the [Open LLM Japanese LLM Leaderboard](https://huggingface.co/spaces/deep-analysis-research/open-japanese-llm-leaderboard), Qwen2.5-32B-Instruct scores 0.6553, compared to the former top-ranked D2IL-Japanese-Qwen2.5-32B-Instruct-v0.1 at 0.7100, and Flux-Japanese-Qwen2.5-32B-V1.0 at 0.7417. Compared with the original Qwen2.5-32B-Instruct, Flux-Japanese-Qwen2.5-32B-v1.0 demonstrates significant gains across most tasks, with especially strong improvements in FA (Fundamental Analysis, 基礎分析), SUM (Summarization, 要約), and CG (Code Generation, コード生成).
+| Tasks                              | Qwen2.5-32B-Instruct | D2IL-Japanese-Qwen2.5-32B-Instruct-v0.1 | Flux-Japanese-Qwen2.5-32B-Instruct-V1.0 |
+|------------------------------------|----------------------|-----------------------------------------|------------------------------------------|
+| NLI - 自然言語推論                  | 0.8106               | 0.8793                                  | 0.8846 (+0.0740)                         |
+| QA - 質問応答                       | 0.541                | 0.5897                                  | 0.5965 (+0.0555)                         |
+| RC - 読解力                         | 0.9047               | 0.9005                                  | 0.9261 (+0.0214)                         |
+| MC - 多肢選択式質問応答              | 0.8966               | 0.9139                                  | 0.9128 (+0.0162)                         |
+| EL - エンティティリンキン            | 0.5894               | 0.6782                                  | 0.6975 (+0.1081)                         |
+| FA - 基礎分析                       | 0.2737               | 0.4321                                  | 0.5185 (+0.2448)                         |
+| MR - 数学的推論                     | 0.944                | 0.938                                   | 0.9420 (-0.0020)                         |
+| MT - 機械翻訳                       | 0.8479               | 0.7954                                  | 0.8389 (-0.0090)                         |
+| HE - 試験問題                       | 0.7757               | 0.7902                                  | 0.7987 (+0.0230)                         |
+| CG - コード生成                     | 0.5281               | 0.6084                                  | 0.7610 (+0.2329)                         |
+| SUM - 要約                          | 0.097                | 0.2843                                  | 0.2827 (+0.1857)                         |
+| **Average**                         | **0.6553**           | **0.71**                                | **0.7417 (+0.0864)**                     |
+
+
+# 🚀 Consistent General Performance
+While Flux-Japanese-Qwen2.5-32B-Instruct-V1.0 has been specifically tuned for Japanese, its performance on general capabilities and English tasks remains within 1% of Qwen2.5-32B-Instruct, indicating negligible impact. The evaluation is based on [simple-evals](https://github.com/deep-analysis-research/simple-evals).
+
+| Tasks               | Dataset        | Qwen2.5-32B-Instruct | Flux-Japanese-Qwen2.5-32B-Instruct-V1.0 |
+|---------------------|----------------|----------------------|------------------------------------------|
+| General Tasks       | MMLU-redux     | 80.37                | 80.03 (-0.34)                            |
+|                     | GPQGA-Diamond  | 46.11                | 47.32 (+1.21)                            |
+|                     | MMLU           | 82.84                | 83.39 (+0.55)                            |
+| Math Tasks          | MATH-500       | 78.14                | 78.50 (+0.36)                            |
+|                     | AIME24         | 17.06                | 17.92 (+0.86)                            |
+|                     | AIME25         | 16.25                | 14.58 (-1.67)                            |
+|                     | MT-AIME24      | 12.73                | 12.97 (+0.24)                            |
+| Multilingual Tasks  | Multi-IF       | 71.85                | 63.45 (-8.40)                            |
+|                     | INCLUDE        | 65.16                | 64.64 (-0.52)                            |
+|                     | MMMLU          | 73.43                | 74.08 (+0.65)                            |
+| Coding Tasks        | HumanEval      | 87.93                | 86.51 (-1.42)                            |
+| Alignment Tasks     | IFEval         | 78.37                | 77.46 (-0.91)                            |
+| **Average**             |                | **59.17**                | **58.40 (-0.77)**                            |
+
+
+# ⚙️ Technical Development
+
+![](./tech-dev.png)
+
+- **Phase 1: Interpretability Analysis & Pinpoint Tuning** — For Japanese Knowledge, Reasoning, and Language, leverage mechanistic interpretability techniques to identify independent pathways/circuits, and apply targeted pinpoint tuning to only 5% of the parameters. This produces three expert models specialized respectively in Japanese knowledge, reasoning, and language.
+- **Phase 2: Pinpoint Merging** — Perform pinpoint parameter merging on the three expert models to obtain a unified model that reaches expert-level performance across Japanese knowledge, reasoning, and language [[Code of Pinpoint Merging](https://github.com/deep-analysis-research/SLTA)].
+
+# 🚩 Quickstart
 ```python
 from transformers import AutoModelForCausalLM, AutoTokenizer
-
-model_name = "Qwen/Qwen2.5-32B-Instruct"
+device = "cuda" # the device to load the model onto
 
 model = AutoModelForCausalLM.from_pretrained(
-    model_name,
+    "Deep-Analysis-Research/Flux-Japanese-Qwen2.5-32B-V1.0",
     torch_dtype="auto",
     device_map="auto"
 )
-tokenizer = AutoTokenizer.from_pretrained(model_name)
+tokenizer = AutoTokenizer.from_pretrained("Deep-Analysis-Research/Flux-Japanese-Qwen2.5-32B-V1.0")
 
-prompt = "Give me a short introduction to large language model."
+prompt = "大規模言語モデルについて簡単に紹介してください。"
 messages = [
-    {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."},
     {"role": "user", "content": prompt}
 ]
 text = tokenizer.apply_chat_template(
@@ -69,10 +80,10 @@ text = tokenizer.apply_chat_template(
     tokenize=False,
     add_generation_prompt=True
 )
-model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
+model_inputs = tokenizer([text], return_tensors="pt").to(device)
 
 generated_ids = model.generate(
-    **model_inputs,
+    model_inputs.input_ids,
     max_new_tokens=512
 )
 generated_ids = [
@@ -81,52 +92,12 @@ generated_ids = [
 
 response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
 ```
-
-### Processing Long Texts
-
-The current `config.json` is set for context length up to 32,768 tokens.
-To handle extensive inputs exceeding 32,768 tokens, we utilize [YaRN](https://arxiv.org/abs/2309.00071), a technique for enhancing model length extrapolation, ensuring optimal performance on lengthy texts.
-
-For supported frameworks, you could add the following to `config.json` to enable YaRN:
-```json
-{
-  ...,
-  "rope_scaling": {
-    "factor": 4.0,
-    "original_max_position_embeddings": 32768,
-    "type": "yarn"
-  }
-}
-```
-
-For deployment, we recommend using vLLM. 
-Please refer to our [Documentation](https://qwen.readthedocs.io/en/latest/deployment/vllm.html) for usage if you are not familar with vLLM.
-Presently, vLLM only supports static YARN, which means the scaling factor remains constant regardless of input length, **potentially impacting performance on shorter texts**. 
-We advise adding the `rope_scaling` configuration only when processing long contexts is required.
-
-## Evaluation & Performance
-
-Detailed evaluation results are reported in this [📑 blog](https://qwenlm.github.io/blog/qwen2.5/).
-
-For requirements on GPU memory and the respective throughput, see results [here](https://qwen.readthedocs.io/en/latest/benchmark/speed_benchmark.html).
-
-## Citation
-
-If you find our work helpful, feel free to give us a cite.
-
-```
-@misc{qwen2.5,
-    title = {Qwen2.5: A Party of Foundation Models},
-    url = {https://qwenlm.github.io/blog/qwen2.5/},
-    author = {Qwen Team},
-    month = {September},
-    year = {2024}
-}
-
-@article{qwen2,
-      title={Qwen2 Technical Report}, 
-      author={An Yang and Baosong Yang and Binyuan Hui and Bo Zheng and Bowen Yu and Chang Zhou and Chengpeng Li and Chengyuan Li and Dayiheng Liu and Fei Huang and Guanting Dong and Haoran Wei and Huan Lin and Jialong Tang and Jialin Wang and Jian Yang and Jianhong Tu and Jianwei Zhang and Jianxin Ma and Jin Xu and Jingren Zhou and Jinze Bai and Jinzheng He and Junyang Lin and Kai Dang and Keming Lu and Keqin Chen and Kexin Yang and Mei Li and Mingfeng Xue and Na Ni and Pei Zhang and Peng Wang and Ru Peng and Rui Men and Ruize Gao and Runji Lin and Shijie Wang and Shuai Bai and Sinan Tan and Tianhang Zhu and Tianhao Li and Tianyu Liu and Wenbin Ge and Xiaodong Deng and Xiaohuan Zhou and Xingzhang Ren and Xinyu Zhang and Xipin Wei and Xuancheng Ren and Yang Fan and Yang Yao and Yichang Zhang and Yu Wan and Yunfei Chu and Yuqiong Liu and Zeyu Cui and Zhenru Zhang and Zhihao Fan},
-      journal={arXiv preprint arXiv:2407.10671},
-      year={2024}
-}
-```
+# 💡 Terms of Use
+We have employed various techniques to reduce bias, harmful outputs, and other risks in the model. While these efforts help improve safety and reliability, the model, like all Large Language Models, may still generate inaccurate, misleading, biased, or otherwise undesirable content. By downloading, using, or interacting with this model, you acknowledge these limitations and agree to the following:
+1. Prohibited Uses
+   - You may NOT use this model for any illegal, unlawful, or harmful activities, including but not limited to fraud, abuse, harassment, privacy violations, or the creation/dissemination of malicious content.
+2. User Responsibility
+   - You are solely responsible for how you use the model and for any outcomes that result from its use.
+   - The authors and institutions involved in releasing this model do NOT accept liability for any consequences arising from its use.
+3. No Warranty
+   - The model is provided “as is” without any warranties or guarantees.