plamo-2-8b / README.md

yhirokawa

Update README.md (#10)

82c58b9 verified 12 days ago

preview code

raw

history blame contribute delete

12.6 kB

metadata

license: other
license_name: plamo-community-license
license_link: https://plamo.preferredai.jp/info/plamo-community-license-en
language:
  - en
  - ja
pipeline_tag: text-generation
library_name: transformers
extra_gated_heading: PLaMo community license to download PLaMo 2 8B
extra_gated_description: >-
  To download PLaMo 2 8B, you have to agree to our license. PLaMo 2 8B is
  released PLaMo community license. For commercial use, please contact us via
  this [form](https://forms.gle/mTL8tBLrMYXKNZD56).
extra_gated_button_content: agree to PLaMo community license
extra_gated_prompt: >-

  ### PLaMo Community License Agreement

  The PLaMo Community License Agreement outlines the terms and conditions for
  utilizing the large-scale language foundation model PLaMo and its derivatives
  provided by Preferred Networks, Inc., as well as any obligations the USER must
  observe. The USER’s access to the PLaMo model and its derivatives is subject
  to this AGREEMENT. By agreeing to or utilizing this MODEL, the USER shall be
  bound by the terms of this AGREEMENT.


  #### Article 1 (Definitions)

  (1) "AGREEMENT" refers to the PLaMo Community License Agreement.

  (2) "PFN" refers to Preferred Networks, Inc.

  (3) "MODEL" denotes the model weights, model code, tokenizer, training
  scripts, and other supplementary materials provided by PFN under the name
  "PLaMo" as separately specified.

  (4) "USER" means an individual or legal entity utilizing the MODEL.

  (5) "DERIVED MODEL" refers to models created through modification or use of
  the MODEL, including their model weights, code, and other associated
  materials.

  (6) "OUTPUT" means the results produced by the MODEL or DERIVED MODEL.

  (7) " MODEL MATERIALS" collectively refers to the MODEL, DERIVED MODEL, and
  OUTPUT.

  (8) "LICENSE" means PFN's authorization to grant the USER permission to use
  the MODEL MATERIALS under this AGREEMENT.

  (9) "COMMERCIAL PURPOSE" means any use beyond personal or academic use,
  including business applications or commercial purposes. Notably, COMMERCIAL
  PURPOSE also includes usage for developing, modifying, or providing products,
  services, or business operations (including through hosting services or APIs),
  and usage within the USER's organizational frameworks.


  #### Article 2 (User Requirements)

  The USER must be 18 years of age or older, or have reached the age of legal
  capacity in the USER’s country of residence, unless the USER's legal guardian
  or legal representative has consented to this AGREEMENT on the USER’s behalf.


  #### Article 3 (This License)

  (1) PFN grants the USER permission to use the MODEL MATERIALS under the terms
  and conditions and scope specified in this AGREEMENT, provided that the USER
  agrees to and comply with this AGREEMENT.

  (2) This LICENSE is non-exclusive, worldwide, non-transferable, and
  royalty-free.

  (3) The USER may utilize the MODEL MATERIALS for COMMERCIAL PURPOSE only if
  the USER satisfies all of the following conditions. If the USER fails to meet
  any of these conditions, the USER may no longer use the MODEL MATERIALS for
  COMMERCIAL PURPOSE and must obtain a new commercial license from PFN if the
  USER wishes to use the MODEL MATERIALS for COMMERCIAL PURPOSE.

    (i) Prior registration on PFN's official registration page (https://forms.gle/mTL8tBLrMYXKNZD56).

    (ii) The USER's or the USER’s affiliated company's annual revenue or sales must not exceed 1 billion yen (converted to the local currency of the USER's jurisdiction).


  #### Article 4 (Redistribution and Attribution Requirements)

  (1) When providing the MODEL MATERIALS to third parties, the USER must meet
  the following requirements:

    (i) Provide a copy of this AGREEMENT and ensure the recipients comply with its terms.

    (ii) Explicitly indicate "Built with PLaMo" in related websites, user interfaces, blog posts, product information pages, or product documentation.

    (iii) Include "PLaMo" in the names of AI models created using the MODEL MATERIALS.


  #### Article 5 (Usage of Outputs)

  (1) The USER may publicly distribute the OUTPUT provided that the USER clearly
  indicates that it is the output of the MODEL or DERIVED MODEL.

  (2) If models are trained using the OUTPUT, those models shall be treated as
  DERIVED MODEL subject to this AGREEMENT's terms, and may only be used,
  distributed, and commercialized under the same license conditions.


  #### Article 6 (Additional Usage Terms)

  The USER must not engage in any of the following activities regarding the use
  of the MODEL MATERIALS:

  (1) Actions violating laws or public order and morals

  (2) Criminal activities or any actions that threaten, participate in, or
  facilitate such activities

  (3) Actions infringing upon the rights or interests of PFN or third parties

  (4) Actions that damage the reputation or credibility of PFN or third parties

  (5) Dissemination of information that could create the false impression that
  the OUTPUT represents PFN's official views, etc.

  (6) Dissemination of false information

  (7) Any other actions that PFN reasonably determines to be inappropriate


  #### Article 7 (Warranty Disclaimer)

  (1) The MODEL and the OUTPUT are provided "as is" without any warranties from
  PFN regarding accuracy, truthfulness, merchantability, quality, performance,
  fitness for specific purposes, non-infringement, or any other matter.

  (2) The USER must not use the OUTPUT as the sole evidence, evaluation, or
  opinion regarding legal matters, medical diagnoses, financial decisions, or
  important personal assessments.

  (3) The USER assumes all responsibility for the use of the MODEL MATERIALS and
  any consequences arising therefrom.


  #### Article 8 (Limitation of Liability)

  (1) Regardless of whether liability arises from contract, tort, product
  liability, or other legal grounds, PFN's maximum liability to the USER
  regarding this AGREEMENT and the MODEL MATERIALS is limited to direct and
  ordinary damages only (including loss of profits, special damages, indirect
  damages, and other damages, regardless of foreseeability). The maximum
  liability amount shall be 500 yen. This limitation does not apply if PFN is
  found to have acted with intent or gross negligence.

  (2) Notwithstanding the preceding paragraph, if the USER utilizes the MODEL
  MATERIALS for business purposes, PFN shall assume no liability whatsoever,
  including any damages or other obligations, regarding this AGREEMENT and the
  MODEL MATERIALS.


  #### Article 9 (User Responsibility)

  (1) The USER shall comply with all applicable laws and regulations (including
  those related to import/export and trade) and this AGREEMENT regarding the
  acquisition and use of the MODEL MATERIALS.

  (2) If the USER causes PFN any damages due to breach of this AGREEMENT or use
  of the MODEL MATERIALS, the USER must compensate PFN for those damages.

  (3) If PFN receives any claims for damages or other demands from third parties
  due to USER's use of the MODEL MATERIALS, the USER shall indemnify PFN against
  such claims and ensure PFN is not harmed.


  #### Article 10 (Ownership Rights)

  (1) All rights to the MODEL belong to PFN or any third party licensed by PFN
  to license the MODEL.

  (2) For DERIVED MODEL, rights pertaining to the portions modified by the USER
  belong to the USER, while all other rights belong to PFN.

  (3) All rights to the OUTPUT belong to the USER.


  #### Article 11 (Contract Term and Termination)

  (1) This AGREEMENT shall be effective from the time the USER agrees to this
  AGREEMENT or access the MODEL MATERIALS and continue until termination.

  (2) If the USER violates any terms of this AGREEMENT, PFN may immediately
  terminate this AGREEMENT, and the USER must delete all copies of the MODEL
  MATERIALS and cease usage immediately.


  #### Article 12 (Contract Modification)

  PFN may modify this AGREEMENT (including rules and regulations related to the
  MODEL MATERIALS; the same applies hereinafter). When modifying this AGREEMENT,
  PFN shall notify the USER of the modification details and the effective date
  of the change through the method specified by PFN by the effective date.


  #### Article 13 (Governing Law and Jurisdiction)

  (1) The laws of Japan shall govern this AGREEMENT.

  (2) Any disputes arising from the MODEL MATERIALS or AGREEMENT shall be
  subject to the exclusive jurisdiction of the Tokyo District Court.

PLaMo 2 8B

Model Description

PLaMo 2 8B is a 8B model pre-trained on English and Japanese datasets, developed by Preferred Elements, Inc.

PLaMo 2 models adapt the hybrid architecture like Samba rather than the Transformer architecture. Samba integrates Mamba, a selective State Space Model (SSM), with sliding window attention, combining their strengths for improved efficiency and performance. The major differences between Samba and PLaMo 2 are 1) adding normalization layers to improve training stability, and 2) using Mamba2 kernel for computational efficiency.

PLaMo 2 8B is released under PLaMo community license. Please check the following license and agree to this before downloading.

NOTE: This model has NOT been instruction-tuned for chat dialog or other downstream tasks.

For commercial users

Please check the PLaMo community license and contact us via the following form to use commercial purpose.

(EN/JA) https://forms.gle/mTL8tBLrMYXKNZD56

Usage

Requirements

numpy>=1.26.4
numba>=0.60.0
torch<=2.5.1
transformers>=4.44.2,<=4.57.1
mamba_ssm>=2.2.2,<=2.2.4
causal_conv1d>=1.4.0

Use a pipeline as a high-level helper

import transformers
pipeline = transformers.pipeline("text-generation", model="pfnet/plamo-2-8b", trust_remote_code=True)
print(pipeline("The future of artificial intelligence technology is ", max_new_tokens=32))

Load model directly

from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("pfnet/plamo-2-8b", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("pfnet/plamo-2-8b", trust_remote_code=True)
text = "これからの人工知能技術は"
input_ids = tokenizer(text, return_tensors="pt").input_ids
generated_tokens = model.generate(
    inputs=input_ids,
    max_new_tokens=32,
    do_sample=True,
    top_k=50,
    top_p=0.95,
    temperature=1.0,
)[0]
generated_text = tokenizer.decode(generated_tokens)
print(generated_text)

Model Details

Model size: 8B
Trained tokens: 6T tokens
Developed by: Preferred Elements, Inc.
Model type: Causal decoder-only
Language(s): English, Japanese
License: PLaMo community license

Training Dataset

We trained PLaMo 2 8B in two phases, phase 1 with 5.25T tokens and phase 2 with 0.75T tokens. The percentage of datasets in each phase is shown in the following table.

	5.25T (phase 1)	0.75T (phase 2)	Tokens
English	45 %	35 %	2.625 T
Japanese	30 %	40 %	1.875 T
Coding	15 %	15 %	0.9 T
Other	10 %	10 %	0.6 T

Tokenizer

PLaMo 2 8B tokenizer is optimized by numba, which is JIT compiler for numerical functions. The tokenizer is trained on a subset of the datasets for model pre-training.

Tech Blog

Bias, Risks, and Limitations

PLaMo 2 8B is a new technology that carries risks with use. Testing conducted to date has been in English and Japanese, and has not covered, nor could it cover all scenarios. For these reasons, as with all LLMs, PLaMo 2 8B’s potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate, biased or other objectionable responses to user prompts. Therefore, before deploying any applications of PLaMo 2 8B, developers should perform safety testing and tuning tailored to their specific applications of the model.

Acknowledgement

This model is trained under the project, “Research and Development Project of the Enhanced Infrastructures for Post 5G Information and Communication System” (JPNP 20017), subsidized by the New Energy and Industrial Technology Development Organization (NEDO).

pfnet
/

plamo-2-8b