Darkhn/M3.2-24B-Animus-V5.1-Pro

Buy me a coffee, or help me get that NVIDIA RTX PRO 6000 Blackwell, so i can train even more models

Model Name - MS-3.2-24B-Animus-V5

Character Card & Lore Book

For the best roleplaying experience, it is highly recommended to use the provided character card and lore book, which have been updated for V5. These files help guide the model's persona and provide rich, in-universe context.

Download the Character Card and Lore Book here

Link to the GGUF

Model Description

This is Version 5 of the fine-tuned anthracite-core/Mistral-Small-3.2-24B-Instruct-2506-Text-Only, specialized for roleplaying and instruction-following within the Wings of Fire universe. V5 builds upon its predecessors with a refined training methodology focused on data quality and character consistency, resulting in a more coherent and immersive roleplaying experience.

The model was first adapted on a highly cleaned dataset extracted from the Wings of Fire book series. It was then fine-tuned for 2 epochs on a larger, more potent dataset designed to teach character persistence and advanced roleplay.

The goal of this model is to provide a high-quality, immersive, and lore-accurate conversational experience. It can adopt character personas, answer questions about the world, engage in creative storytelling, portray multiple characters at once, and handle more mature themes from the series with improved logical consistency.

Training Details

Training Hardware

The model was fine-tuned on a single NVIDIA RTX PRO 6000 Blackwell, Provided by @Quͫaͦcͦk on discord.

Training Procedure

A QLoRA (Quantized Low-Rank Adaptation) approach was used for efficient fine-tuning, with an optimized process configured using Axolotl.

Chat Template

This model uses the Mistral_V7_tekken chat template. It is crucial to format your prompts using this template for the model to function as intended.

Training Data

The training process involved two main stages with a strong emphasis on data quality:

1. Refined Domain Adaptation (Pre-training): The base model was adapted using the Darkhn/WOF_Pretraining dataset. This dataset was meticulously cleaned to remove extraneous information such as chapter markers, image links (.jpg), and other formatting artifacts. This provides a purer textual foundation for the model to learn from.

2. Instruction & Chat Fine-tuning: The model was fine-tuned for 2 epochs on an expanded, high-quality dataset.

   • Enhanced Dataset: The roleplay dataset was expanded to 2,200 high-quality roleplay examples. The dataset was generated using Gemini Pro, leading to more nuanced, creative, and diverse scenarios.
   • Character Persistence Workflow: The V5-Pro dataset continues the successful methodology of reusing characters across multiple scenarios. This method trains the model to maintain a character's personality, traits, and history consistently, significantly reducing the logical mistakes and contradictions seen in previous versions.
   • OOC Commentaries: A key new feature in V5-Pro is the inclusion of Out-of-Character (OOC) commentaries in the training data. This helps the model better understand conversational context, user intent, and the distinction between in-character actions and meta-commentary.
   • The dataset continues to feature multi-turn scenarios, portrayal of multiple characters, and the more mature or 'darker' themes present in the book series.

Intended Use & Limitations

• Intended Use: This model is intended for creative and roleplaying purposes within the Wings of Fire universe. It is designed for fans of the series and is not a general-purpose chatbot.

• Limitations & Quirks:

  • Performance on tasks outside of its training domain (general knowledge, coding, etc.) is not guaranteed and will likely be poor.
  • Character Consistency: While all models can have logical lapses, V5's training was specifically designed to improve character persistence. It should be noticeably better at remembering character traits and history across a conversation compared to previous versions.
  • The model may "hallucinate" or generate plausible but non-canonical information.
  • Content: The roleplay training data includes more mature and darker themes from the Wings of Fire series, such as character death, conflict, and moral ambiguity. The model is capable of generating content reflecting these themes. It can generate gratuitous or explicit content, as always it's up to the user what they do with it.
  • Formatting: The training data was cleaned to remove formatting artifacts like asterisks (*...*) for single word emphasis. The model should now produce cleaner, more narrative-style prose.
  • Safety: This model has not undergone additional safety alignment beyond what was included in its base Mistral-Small-3.2 model. Standard responsible AI practices should be followed.

Recommended Sampler Settings

For optimal performance that balances creativity and coherence, the following default sampler settings are recommended.

• Temperature: 0.7
• Min_P: 0.035
• DRY Sampler:
  • Multiplier: 0.8
  • Allowed Length: 4
  • Base: 1.75

Acknowledgements

• Credit to Mistral AI for the powerful Mistral-Small-3.2 architecture.
• Credit to Evan Armstrong for Augmentoolkit
• Credit to Google for Gemini Pro to generate the dataset.
• Credit to @Quͫaͦcͦk for letting me borrow his NVIDIA RTX PRO 6000 Blackwell.