ilahgel/llama-8-b

See axolotl config

axolotl version: 0.8.0.dev0

base_model: AliMaatouk/LLama-3-8B-Tele
# optionally might have model_type or tokenizer_type
model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer
# Automatically upload checkpoint and final model to HF
# hub_model_id: username/custom_model_name

load_in_8bit: true
load_in_4bit: false

datasets:
  - path: ilahgel/train_dataset
    type: completion
    field: Statement        # <-- colonne question / entrée
    response_field: Answer  # <-- colonne réponse / cible


dataset_prepared_path:
val_set_size: 0.05
output_dir: ./outputs/lora-out

sequence_len: 4096
sample_packing: false


adapter: lora
lora_model_dir:
lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true

wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:

gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 6
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0001

bf16: auto
tf32: false

gradient_checkpointing: true
resume_from_checkpoint:
logging_steps: 1
flash_attention: true

warmup_ratio: 0.1
evals_per_epoch: 4
saves_per_epoch: 1
weight_decay: 0.0

special_tokens:
   pad_token: <|end_of_text|>
   eos_token: <|end_of_text|>


# save_first_step: true  # uncomment this to validate checkpoint saving works with your config

outputs/lora-out

This model is a fine-tuned version of AliMaatouk/LLama-3-8B-Tele on the ilahgel/train_dataset dataset. It achieves the following results on the evaluation set:

• Loss: 2.0893

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 8
• optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 647
• num_epochs: 6.0

Training results

Training Loss	Epoch	Step	Validation Loss
4.4515	0.0009	1	5.3572
1.106	0.2502	270	1.6036
0.818	0.5003	540	1.5649
1.0485	0.7505	810	1.4442
1.4447	1.0	1080	1.4195
0.9753	1.2502	1350	1.5863
0.7307	1.5003	1620	1.4133
0.6976	1.7505	1890	1.4350
0.7043	2.0	2160	1.5619
0.5593	2.2502	2430	1.5128
0.6806	2.5003	2700	1.5410
0.5136	2.7505	2970	1.5258
0.6413	3.0	3240	1.5198
0.5054	3.2502	3510	1.6675
0.4392	3.5003	3780	1.6507
0.511	3.7505	4050	1.7479
0.4954	4.0	4320	1.6402
0.396	4.2502	4590	1.8868
0.4947	4.5003	4860	1.8823
0.4051	4.7505	5130	1.8906
0.4297	5.0	5400	1.8975
0.3414	5.2502	5670	2.0067
0.377	5.5003	5940	2.0694
0.3128	5.7505	6210	2.0893

Framework versions

• PEFT 0.14.0
• Transformers 4.49.0
• Pytorch 2.5.1+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0