logs/qwen_lora_3677815.err

Lmod has detected the following error: The following module(s) are unknown:
"Miniconda3/24.7.1-0"

Please check the spelling or version number. Also try "module spider ..."
It is also possible your cache file is out-of-date; it may help to try:
  $ module --ignore_cache load "Miniconda3/24.7.1-0"

Also make sure that all modulefiles written in TCL start with the string
#%Module



Lmod has detected the following error: The following module(s) are unknown:
"PyTorch/2.1.2-foss-2023a"

Please check the spelling or version number. Also try "module spider ..."
It is also possible your cache file is out-of-date; it may help to try:
  $ module --ignore_cache load "PyTorch/2.1.2-foss-2023a"

Also make sure that all modulefiles written in TCL start with the string
#%Module



/cm/local/apps/slurm/var/spool/job3677815/slurm_script: line 20: /cm/local/apps/Miniconda3/24.7.1-0/etc/profile.d/conda.sh: No such file or directory

CondaError: Run 'conda init' before 'conda activate'

The `load_in_4bit` and `load_in_8bit` arguments are deprecated and will be removed in the future versions. Please, pass a `BitsAndBytesConfig` object in `quantization_config` argument instead.

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files:  25%|██▌       | 1/4 [00:43<02:11, 43.70s/it]
Fetching 4 files: 100%|██████████| 4/4 [00:43<00:00, 10.92s/it]

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:11<00:33, 11.12s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:22<00:22, 11.23s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:33<00:11, 11.34s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:45<00:00, 11.35s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:45<00:00, 11.32s/it]

Map:   0%|          | 0/473 [00:00<?, ? examples/s]
Map: 100%|██████████| 473/473 [00:00<00:00, 2341.28 examples/s]
Map: 100%|██████████| 473/473 [00:00<00:00, 2222.38 examples/s]
/home/gabrielp24/InsurAI/sft_lora_finetuner.py:125: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `Trainer.__init__`. Use `processing_class` instead.
  trainer = Trainer(
The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}.

  0%|          | 0/180 [00:00<?, ?it/s]
  1%|          | 1/180 [00:03<09:21,  3.14s/it]
  1%|          | 2/180 [00:03<04:49,  1.63s/it]
  2%|▏         | 3/180 [00:04<03:22,  1.14s/it]
  2%|▏         | 4/180 [00:04<02:40,  1.09it/s]
  3%|▎         | 5/180 [00:05<02:18,  1.26it/s]
  3%|▎         | 6/180 [00:05<02:04,  1.40it/s]
  4%|▍         | 7/180 [00:06<01:56,  1.49it/s]
  4%|▍         | 8/180 [00:07<01:50,  1.56it/s]
  5%|▌         | 9/180 [00:07<01:45,  1.62it/s]
  6%|▌         | 10/180 [00:08<01:42,  1.65it/s]
                                                

  6%|▌         | 10/180 [00:08<01:42,  1.65it/s]
  6%|▌         | 11/180 [00:08<01:40,  1.68it/s]
  7%|▋         | 12/180 [00:09<01:38,  1.70it/s]
  7%|▋         | 13/180 [00:09<01:37,  1.71it/s]
  8%|▊         | 14/180 [00:10<01:35,  1.73it/s]
  8%|▊         | 15/180 [00:11<01:34,  1.75it/s]
  9%|▉         | 16/180 [00:11<01:32,  1.77it/s]
  9%|▉         | 17/180 [00:12<01:31,  1.78it/s]
 10%|█         | 18/180 [00:12<01:30,  1.78it/s]
 11%|█         | 19/180 [00:13<01:30,  1.79it/s]
 11%|█         | 20/180 [00:13<01:29,  1.79it/s]
                                                

 11%|█         | 20/180 [00:13<01:29,  1.79it/s]
 12%|█▏        | 21/180 [00:14<01:28,  1.79it/s]
 12%|█▏        | 22/180 [00:15<01:27,  1.80it/s]
 13%|█▎        | 23/180 [00:15<01:27,  1.80it/s]
 13%|█▎        | 24/180 [00:16<01:27,  1.78it/s]
 14%|█▍        | 25/180 [00:16<01:27,  1.78it/s]
 14%|█▍        | 26/180 [00:17<01:27,  1.77it/s]
 15%|█▌        | 27/180 [00:17<01:26,  1.77it/s]
 16%|█▌        | 28/180 [00:18<01:26,  1.76it/s]
 16%|█▌        | 29/180 [00:18<01:25,  1.76it/s]
 17%|█▋        | 30/180 [00:19<01:25,  1.76it/s]
                                                

 17%|█▋        | 30/180 [00:19<01:25,  1.76it/s]
 17%|█▋        | 31/180 [00:20<01:24,  1.76it/s]
 18%|█▊        | 32/180 [00:20<01:23,  1.77it/s]
 18%|█▊        | 33/180 [00:21<01:22,  1.77it/s]
 19%|█▉        | 34/180 [00:21<01:22,  1.78it/s]
 19%|█▉        | 35/180 [00:22<01:21,  1.78it/s]
 20%|██        | 36/180 [00:22<01:20,  1.79it/s]
 21%|██        | 37/180 [00:23<01:19,  1.79it/s]
 21%|██        | 38/180 [00:24<01:19,  1.79it/s]
 22%|██▏       | 39/180 [00:24<01:18,  1.79it/s]
 22%|██▏       | 40/180 [00:25<01:17,  1.80it/s]
                                                

 22%|██▏       | 40/180 [00:25<01:17,  1.80it/s]
 23%|██▎       | 41/180 [00:25<01:17,  1.80it/s]
 23%|██▎       | 42/180 [00:26<01:16,  1.80it/s]
 24%|██▍       | 43/180 [00:26<01:16,  1.80it/s]
 24%|██▍       | 44/180 [00:27<01:16,  1.79it/s]
 25%|██▌       | 45/180 [00:27<01:15,  1.78it/s]
 26%|██▌       | 46/180 [00:28<01:15,  1.77it/s]
 26%|██▌       | 47/180 [00:29<01:15,  1.77it/s]
 27%|██▋       | 48/180 [00:29<01:14,  1.77it/s]
 27%|██▋       | 49/180 [00:30<01:13,  1.77it/s]
 28%|██▊       | 50/180 [00:30<01:13,  1.77it/s]
                                                

 28%|██▊       | 50/180 [00:30<01:13,  1.77it/s]
 28%|██▊       | 51/180 [00:31<01:12,  1.77it/s]
 29%|██▉       | 52/180 [00:31<01:12,  1.77it/s]
 29%|██▉       | 53/180 [00:32<01:11,  1.78it/s]
 30%|███       | 54/180 [00:33<01:10,  1.78it/s]
 31%|███       | 55/180 [00:33<01:09,  1.79it/s]
 31%|███       | 56/180 [00:34<01:09,  1.79it/s]
 32%|███▏      | 57/180 [00:34<01:08,  1.80it/s]
 32%|███▏      | 58/180 [00:35<01:07,  1.80it/s]
 33%|███▎      | 59/180 [00:35<01:07,  1.80it/s]
 33%|███▎      | 60/180 [00:36<00:54,  2.18it/s]
                                                

 33%|███▎      | 60/180 [00:36<00:54,  2.18it/s]
 34%|███▍      | 61/180 [00:37<01:28,  1.34it/s]
 34%|███▍      | 62/180 [00:37<01:21,  1.45it/s]
 35%|███▌      | 63/180 [00:38<01:16,  1.53it/s]
 36%|███▌      | 64/180 [00:39<01:12,  1.59it/s]
 36%|███▌      | 65/180 [00:39<01:10,  1.64it/s]
 37%|███▋      | 66/180 [00:40<01:08,  1.67it/s]
 37%|███▋      | 67/180 [00:40<01:06,  1.71it/s]
 38%|███▊      | 68/180 [00:41<01:04,  1.73it/s]
 38%|███▊      | 69/180 [00:41<01:03,  1.74it/s]
 39%|███▉      | 70/180 [00:42<01:02,  1.75it/s]
                                                

 39%|███▉      | 70/180 [00:42<01:02,  1.75it/s]
 39%|███▉      | 71/180 [00:43<01:02,  1.75it/s]
 40%|████      | 72/180 [00:43<01:01,  1.76it/s]
 41%|████      | 73/180 [00:44<01:00,  1.77it/s]
 41%|████      | 74/180 [00:44<00:59,  1.78it/s]
 42%|████▏     | 75/180 [00:45<00:58,  1.79it/s]
 42%|████▏     | 76/180 [00:45<00:57,  1.80it/s]
 43%|████▎     | 77/180 [00:46<00:57,  1.80it/s]
 43%|████▎     | 78/180 [00:46<00:56,  1.80it/s]
 44%|████▍     | 79/180 [00:47<00:55,  1.81it/s]
 44%|████▍     | 80/180 [00:48<00:55,  1.81it/s]
                                                

 44%|████▍     | 80/180 [00:48<00:55,  1.81it/s]
 45%|████▌     | 81/180 [00:48<00:54,  1.81it/s]
 46%|████▌     | 82/180 [00:49<00:54,  1.79it/s]
 46%|████▌     | 83/180 [00:49<00:54,  1.78it/s]
 47%|████▋     | 84/180 [00:50<00:54,  1.78it/s]
 47%|████▋     | 85/180 [00:50<00:53,  1.78it/s]
 48%|████▊     | 86/180 [00:51<00:52,  1.78it/s]
 48%|████▊     | 87/180 [00:52<00:52,  1.77it/s]
 49%|████▉     | 88/180 [00:52<00:51,  1.77it/s]
 49%|████▉     | 89/180 [00:53<00:51,  1.77it/s]
 50%|█████     | 90/180 [00:53<00:50,  1.77it/s]
                                                

 50%|█████     | 90/180 [00:53<00:50,  1.77it/s]
 51%|█████     | 91/180 [00:54<00:50,  1.77it/s]
 51%|█████     | 92/180 [00:54<00:49,  1.78it/s]
 52%|█████▏    | 93/180 [00:55<00:48,  1.79it/s]
 52%|█████▏    | 94/180 [00:55<00:47,  1.80it/s]
 53%|█████▎    | 95/180 [00:56<00:47,  1.80it/s]
 53%|█████▎    | 96/180 [00:57<00:46,  1.80it/s]
 54%|█████▍    | 97/180 [00:57<00:45,  1.81it/s]
 54%|█████▍    | 98/180 [00:58<00:45,  1.81it/s]
 55%|█████▌    | 99/180 [00:58<00:44,  1.81it/s]
 56%|█████▌    | 100/180 [00:59<00:44,  1.81it/s]
                                                 

 56%|█████▌    | 100/180 [00:59<00:44,  1.81it/s]
 56%|█████▌    | 101/180 [00:59<00:43,  1.80it/s]
 57%|█████▋    | 102/180 [01:00<00:43,  1.80it/s]
 57%|█████▋    | 103/180 [01:00<00:43,  1.79it/s]
 58%|█████▊    | 104/180 [01:01<00:42,  1.79it/s]
 58%|█████▊    | 105/180 [01:02<00:41,  1.79it/s]
 59%|█████▉    | 106/180 [01:02<00:41,  1.78it/s]
 59%|█████▉    | 107/180 [01:03<00:41,  1.78it/s]
 60%|██████    | 108/180 [01:03<00:40,  1.77it/s]
 61%|██████    | 109/180 [01:04<00:40,  1.77it/s]
 61%|██████    | 110/180 [01:04<00:39,  1.77it/s]
                                                 

 61%|██████    | 110/180 [01:04<00:39,  1.77it/s]
 62%|██████▏   | 111/180 [01:05<00:38,  1.77it/s]
 62%|██████▏   | 112/180 [01:06<00:38,  1.78it/s]
 63%|██████▎   | 113/180 [01:06<00:37,  1.79it/s]
 63%|██████▎   | 114/180 [01:07<00:36,  1.80it/s]
 64%|██████▍   | 115/180 [01:07<00:36,  1.80it/s]
 64%|██████▍   | 116/180 [01:08<00:35,  1.81it/s]
 65%|██████▌   | 117/180 [01:08<00:34,  1.81it/s]
 66%|██████▌   | 118/180 [01:09<00:34,  1.81it/s]
 66%|██████▌   | 119/180 [01:09<00:33,  1.81it/s]
 67%|██████▋   | 120/180 [01:10<00:26,  2.26it/s]
                                                 

 67%|██████▋   | 120/180 [01:10<00:26,  2.26it/s]
 67%|██████▋   | 121/180 [01:11<00:51,  1.15it/s]
 68%|██████▊   | 122/180 [01:12<00:44,  1.29it/s]
 68%|██████▊   | 123/180 [01:13<00:40,  1.41it/s]
 69%|██████▉   | 124/180 [01:13<00:37,  1.51it/s]
 69%|██████▉   | 125/180 [01:14<00:34,  1.58it/s]
 70%|███████   | 126/180 [01:14<00:33,  1.63it/s]
 71%|███████   | 127/180 [01:15<00:31,  1.67it/s]
 71%|███████   | 128/180 [01:15<00:30,  1.70it/s]
 72%|███████▏  | 129/180 [01:16<00:29,  1.72it/s]
 72%|███████▏  | 130/180 [01:16<00:28,  1.74it/s]
                                                 

 72%|███████▏  | 130/180 [01:16<00:28,  1.74it/s]
 73%|███████▎  | 131/180 [01:17<00:27,  1.76it/s]
 73%|███████▎  | 132/180 [01:18<00:26,  1.78it/s]
 74%|███████▍  | 133/180 [01:18<00:26,  1.79it/s]
 74%|███████▍  | 134/180 [01:19<00:25,  1.79it/s]
 75%|███████▌  | 135/180 [01:19<00:24,  1.80it/s]
 76%|███████▌  | 136/180 [01:20<00:24,  1.81it/s]
 76%|███████▌  | 137/180 [01:20<00:23,  1.81it/s]
 77%|███████▋  | 138/180 [01:21<00:23,  1.79it/s]
 77%|███████▋  | 139/180 [01:21<00:22,  1.79it/s]
 78%|███████▊  | 140/180 [01:22<00:22,  1.78it/s]
                                                 

 78%|███████▊  | 140/180 [01:22<00:22,  1.78it/s]
 78%|███████▊  | 141/180 [01:23<00:21,  1.79it/s]
 79%|███████▉  | 142/180 [01:23<00:21,  1.79it/s]
 79%|███████▉  | 143/180 [01:24<00:20,  1.79it/s]
 80%|████████  | 144/180 [01:24<00:20,  1.79it/s]
 81%|████████  | 145/180 [01:25<00:19,  1.78it/s]
 81%|████████  | 146/180 [01:25<00:19,  1.78it/s]
 82%|████████▏ | 147/180 [01:26<00:18,  1.77it/s]
 82%|████████▏ | 148/180 [01:27<00:18,  1.77it/s]
 83%|████████▎ | 149/180 [01:27<00:17,  1.77it/s]
 83%|████████▎ | 150/180 [01:28<00:16,  1.79it/s]
                                                 

 83%|████████▎ | 150/180 [01:28<00:16,  1.79it/s]
 84%|████████▍ | 151/180 [01:28<00:16,  1.79it/s]
 84%|████████▍ | 152/180 [01:29<00:15,  1.80it/s]
 85%|████████▌ | 153/180 [01:29<00:14,  1.81it/s]
 86%|████████▌ | 154/180 [01:30<00:14,  1.81it/s]
 86%|████████▌ | 155/180 [01:30<00:13,  1.81it/s]
 87%|████████▋ | 156/180 [01:31<00:13,  1.81it/s]
 87%|████████▋ | 157/180 [01:31<00:12,  1.82it/s]
 88%|████████▊ | 158/180 [01:32<00:12,  1.82it/s]
 88%|████████▊ | 159/180 [01:33<00:11,  1.82it/s]
 89%|████████▉ | 160/180 [01:33<00:10,  1.82it/s]
                                                 

 89%|████████▉ | 160/180 [01:33<00:10,  1.82it/s]
 89%|████████▉ | 161/180 [01:34<00:10,  1.82it/s]
 90%|█████████ | 162/180 [01:34<00:09,  1.82it/s]
 91%|█████████ | 163/180 [01:35<00:09,  1.82it/s]
 91%|█████████ | 164/180 [01:35<00:08,  1.82it/s]
 92%|█████████▏| 165/180 [01:36<00:08,  1.81it/s]
 92%|█████████▏| 166/180 [01:36<00:07,  1.79it/s]
 93%|█████████▎| 167/180 [01:37<00:07,  1.78it/s]
 93%|█████████▎| 168/180 [01:38<00:06,  1.78it/s]
 94%|█████████▍| 169/180 [01:38<00:06,  1.77it/s]
 94%|█████████▍| 170/180 [01:39<00:05,  1.77it/s]
                                                 

 94%|█████████▍| 170/180 [01:39<00:05,  1.77it/s]
 95%|█████████▌| 171/180 [01:39<00:05,  1.78it/s]
 96%|█████████▌| 172/180 [01:40<00:04,  1.79it/s]
 96%|█████████▌| 173/180 [01:40<00:03,  1.80it/s]
 97%|█████████▋| 174/180 [01:41<00:03,  1.81it/s]
 97%|█████████▋| 175/180 [01:41<00:02,  1.81it/s]
 98%|█████████▊| 176/180 [01:42<00:02,  1.81it/s]
 98%|█████████▊| 177/180 [01:43<00:01,  1.81it/s]
 99%|█████████▉| 178/180 [01:43<00:01,  1.82it/s]
 99%|█████████▉| 179/180 [01:44<00:00,  1.82it/s]
100%|██████████| 180/180 [01:44<00:00,  2.28it/s]
                                                 

100%|██████████| 180/180 [01:44<00:00,  2.28it/s]
                                                 

100%|██████████| 180/180 [01:45<00:00,  2.28it/s]
100%|██████████| 180/180 [01:45<00:00,  1.71it/s]