Llama2-7B-lora-r-32-generic-step-1200-lr-1e-5-labels_40.0-2

This model is a fine-tuned version of meta-llama/Llama-2-7b-hf on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 2.8431

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 50
• training_steps: 1200
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
5.491	0.9132	50	4.8940
4.2692	1.8265	100	3.8713
3.6673	2.7397	150	3.5427
3.3905	3.6530	200	3.3337
3.2068	4.5662	250	3.1931
3.0651	5.4795	300	3.0953
2.9385	6.3927	350	3.0245
2.8496	7.3059	400	2.9731
2.7728	8.2192	450	2.9357
2.7166	9.1324	500	2.9062
2.6951	10.0457	550	nan
2.6518	10.9589	600	2.8699
2.5752	11.8721	650	2.8596
2.5952	12.7854	700	2.8510
2.5826	13.6986	750	2.8485
2.531	14.6119	800	2.8446
2.5043	15.5251	850	2.8426
2.4706	16.4384	900	2.8423
2.5093	17.3516	950	2.8409
2.4448	18.2648	1000	2.8401
2.5033	19.1781	1050	2.8402
2.4672	20.0913	1100	2.8423
2.428	21.0046	1150	2.8429
2.4223	21.9178	1200	2.8431

Framework versions

• PEFT 0.15.2
• Transformers 4.45.2
• Pytorch 2.5.0+cu121
• Datasets 3.2.0
• Tokenizers 0.20.3