baseline-model-instruct-32k-yarn

This model is ft-llm-team-mkj/baseline-model-instruct with YaRN (Yet another RoPE extensioN) applied to extend the context window.

Model Details

• Base Model: ft-llm-team-mkj/baseline-model-instruct
• Original Context Length: 4096 tokens
• Extended Context Length: 32768 tokens
• Extension Method: YaRN
• Scaling Factor: 8.0

YaRN Configuration

{
  "rope_type": "yarn",
  "factor": 8.0,
  "original_max_position_embeddings": 4096,
  "beta_fast": 32,
  "beta_slow": 1
}

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "DeL-TaiseiOzaki/baseline-model-instruct-32k-yarn",
    torch_dtype="bfloat16",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("DeL-TaiseiOzaki/baseline-model-instruct-32k-yarn")

Note

This model has YaRN applied but has not been fine-tuned on long-context data yet. For optimal performance on long sequences, fine-tuning on long documents is recommended.

References

• YaRN Paper (ICLR 2024)
• YaRN GitHub