Upload 2 files

README.md

@@ -41,7 +41,103 @@
 - **Model Scoring Dataset**: [HPC-LLM-8k](https://huggingface.co/datasets/ECNU-InnoSpark/HPC-LLM-8k)
 - **Human Scoring Dataset**: [HPC-Human-8k](https://huggingface.co/datasets/ECNU-InnoSpark/HPC-Human-8k)
 
-## 🚀 Core Features
+## 🚀 Quickstart
+
+Here provides a code snippet with `apply_chat_template` to show you how to load the tokenizer and model and how to generate contents.
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+device = "cuda" # the device to load the model onto
+
+model = AutoModelForCausalLM.from_pretrained(
+    "sii-research/InnoSpark-72B-0710",
+    torch_dtype="auto",
+    device_map="auto"
+)
+tokenizer = AutoTokenizer.from_pretrained("sii-research/InnoSpark-72B-0710")
+
+prompt = "Introduce yourself in detail."
+messages = [
+    {"role": "system", "content": "You are InnoSpark（启创）, created by Shanghai Innovation Institute （上海创智学院） and East China Normal University(华东师范大学). You are a helpful assistant."},
+    {"role": "user", "content": prompt}
+]
+text = tokenizer.apply_chat_template(
+    messages,
+    tokenize=False,
+    add_generation_prompt=True
+)
+model_inputs = tokenizer([text], return_tensors="pt").to(device)
+
+generated_ids = model.generate(
+    model_inputs.input_ids,
+    max_new_tokens=512
+)
+generated_ids = [
+    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
+]
+
+response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
+```
+
+### VLLM
+
+We recommend deploying our model using 4 A100 GPUs. You can run the vllm server-side with the following code in terminal:
+
+```python
+python -m vllm.entrypoints.openai.api_server --served-model-name InnoSpark --model path/to/InnoSpark --gpu-memory-utilization 0.98 --tensor-parallel-size 4 --port 6000
+```
+
+Then, you can use the following code to deploy client-side:
+
+```python
+import requests
+import json
+
+def Innospark_stream(inputs,history):
+    url = 'http://loaclhost:6000/v1/chat/completions'
+
+    history+=[{"role": "user", "content": inputs},]
+
+    headers = {"User-Agent": "vLLM Client"}
+
+    pload = {
+        "model": "InnoSpark",
+        "stream": True,
+        "messages": history
+    }
+    response = requests.post(url,
+                             headers=headers,
+                             json=pload,
+                             stream=True)
+
+    for chunk in response.iter_lines(chunk_size=1,
+                                     decode_unicode=False,
+                                     delimiter=b"\n"):
+        if chunk:
+            string_data = chunk.decode("utf-8")
+            try:
+                json_data = json.loads(string_data[6:])
+                delta_content = json_data["choices"][0]["delta"]["content"]
+                assistant_reply+=delta_content
+                yield delta_content
+            except KeyError as e:
+                delta_content = json_data["choices"][0]["delta"]["role"]
+            except json.JSONDecodeError as e:
+                history+=[{
+                        "role": "assistant",
+                        "content": assistant_reply,
+                        "tool_calls": []
+                    },]
+                delta_content='[DONE]'
+                assert '[DONE]'==chunk.decode("utf-8")[6:]
+
+inputs='hi'
+history=[]
+for response_text in Innospark_stream(inputs,history):
+    print(response_text,end='')
+```
+
+## 🌟 Core Features
 
 ### 🎯 Open Source Product Matrix
 

README_zh.md

@@ -41,7 +41,103 @@
 - **模型打分数据集**: [HPC-LLM-8k](https://huggingface.co/datasets/ECNU-InnoSpark/HPC-LLM-8k)
 - **人工打分数据集**: [HPC-Human-8k](https://huggingface.co/datasets/ECNU-InnoSpark/HPC-Human-8k)
 
-## 🚀 核心特性
+## 🚀 快速开始
+
+这里提供了一个使用 `apply_chat_template` 的代码示例，展示如何加载分词器和模型以及如何生成内容。
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+device = "cuda" # 加载模型的设备
+
+model = AutoModelForCausalLM.from_pretrained(
+    "sii-research/InnoSpark-72B-0710",
+    torch_dtype="auto",
+    device_map="auto"
+)
+tokenizer = AutoTokenizer.from_pretrained("sii-research/InnoSpark-72B-0710")
+
+prompt = "详细介绍一下你自己。"
+messages = [
+    {"role": "system", "content": "You are InnoSpark（启创）, created by Shanghai Innovation Institute （上海创智学院） and East China Normal University(华东师范大学). You are a helpful assistant."},
+    {"role": "user", "content": prompt}
+]
+text = tokenizer.apply_chat_template(
+    messages,
+    tokenize=False,
+    add_generation_prompt=True
+)
+model_inputs = tokenizer([text], return_tensors="pt").to(device)
+
+generated_ids = model.generate(
+    model_inputs.input_ids,
+    max_new_tokens=512
+)
+generated_ids = [
+    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
+]
+
+response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
+```
+
+### VLLM 部署
+
+我们推荐使用 4 块 A100 GPU 部署我们的模型。您可以在终端中使用以下代码运行 vllm 服务端：
+
+```python
+python -m vllm.entrypoints.openai.api_server --served-model-name InnoSpark --model path/to/InnoSpark --gpu-memory-utilization 0.98 --tensor-parallel-size 4 --port 6000
+```
+
+然后，您可以使用以下代码部署客户端：
+
+```python
+import requests
+import json
+
+def Innospark_stream(inputs,history):
+    url = 'http://loaclhost:6000/v1/chat/completions'
+
+    history+=[{"role": "user", "content": inputs},]
+
+    headers = {"User-Agent": "vLLM Client"}
+
+    pload = {
+        "model": "InnoSpark",
+        "stream": True,
+        "messages": history
+    }
+    response = requests.post(url,
+                             headers=headers,
+                             json=pload,
+                             stream=True)
+
+    for chunk in response.iter_lines(chunk_size=1,
+                                     decode_unicode=False,
+                                     delimiter=b"\n"):
+        if chunk:
+            string_data = chunk.decode("utf-8")
+            try:
+                json_data = json.loads(string_data[6:])
+                delta_content = json_data["choices"][0]["delta"]["content"]
+                assistant_reply+=delta_content
+                yield delta_content
+            except KeyError as e:
+                delta_content = json_data["choices"][0]["delta"]["role"]
+            except json.JSONDecodeError as e:
+                history+=[{
+                        "role": "assistant",
+                        "content": assistant_reply,
+                        "tool_calls": []
+                    },]
+                delta_content='[DONE]'
+                assert '[DONE]'==chunk.decode("utf-8")[6:]
+
+inputs='hi'
+history=[]
+for response_text in Innospark_stream(inputs,history):
+    print(response_text,end='')
+```
+
+## 🌟 核心特性
 
 ### 🎯 开源产品矩阵