Llama2-7B-lora-r-32-generic-step-1200-lr-1e-5-labels_40.0-4

This model is a fine-tuned version of meta-llama/Llama-2-7b-hf on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 2.8857

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 50
• training_steps: 1200
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
5.4809	0.9132	50	4.9196
4.2651	1.8265	100	3.8889
3.6786	2.7397	150	3.5590
3.3958	3.6530	200	3.3616
3.1947	4.5662	250	3.2255
3.0476	5.4795	300	3.1306
2.9537	6.3927	350	3.0624
2.8502	7.3059	400	3.0163
2.7741	8.2192	450	2.9761
2.7038	9.1324	500	2.9509
2.6587	10.0457	550	2.9295
2.6098	10.9589	600	2.9147
2.5678	11.8721	650	2.9058
2.5428	12.7854	700	2.8959
2.5852	13.6986	750	2.8937
2.5312	14.6119	800	2.8897
2.4594	15.5251	850	2.8861
2.4514	16.4384	900	2.8875
2.4842	17.3516	950	2.8851
2.4602	18.2648	1000	2.8845
2.498	19.1781	1050	2.8843
2.4345	20.0913	1100	2.8851
2.4501	21.0046	1150	2.8855
2.4594	21.9178	1200	2.8857

Framework versions

• PEFT 0.15.2
• Transformers 4.45.2
• Pytorch 2.5.0+cu121
• Datasets 3.2.0
• Tokenizers 0.20.3