logs/qwen_lora_3695261.err

Lmod has detected the following error: The following module(s) are unknown:
"Miniconda3/24.7.1-0"

Please check the spelling or version number. Also try "module spider ..."
It is also possible your cache file is out-of-date; it may help to try:
  $ module --ignore_cache load "Miniconda3/24.7.1-0"

Also make sure that all modulefiles written in TCL start with the string
#%Module



Lmod has detected the following error: The following module(s) are unknown:
"PyTorch/2.1.2-foss-2023a"

Please check the spelling or version number. Also try "module spider ..."
It is also possible your cache file is out-of-date; it may help to try:
  $ module --ignore_cache load "PyTorch/2.1.2-foss-2023a"

Also make sure that all modulefiles written in TCL start with the string
#%Module



/cm/local/apps/slurm/var/spool/job3695261/slurm_script: line 20: /cm/local/apps/Miniconda3/24.7.1-0/etc/profile.d/conda.sh: No such file or directory

CondaError: Run 'conda init' before 'conda activate'

`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:07<00:22,  7.46s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:14<00:14,  7.17s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:21<00:07,  7.22s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:28<00:00,  6.86s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:28<00:00,  7.00s/it]

Map:   0%|          | 0/473 [00:00<?, ? examples/s]
Map: 100%|██████████| 473/473 [00:00<00:00, 3294.26 examples/s]
Map: 100%|██████████| 473/473 [00:00<00:00, 3227.75 examples/s]
/home/gabrielp24/InsurAI/sft_lora_finetuner.py:125: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `Trainer.__init__`. Use `processing_class` instead.
  trainer = Trainer(
The model is already on multiple devices. Skipping the move to device specified in `args`.
The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}.

  0%|          | 0/180 [00:00<?, ?it/s]
  1%|          | 1/180 [00:05<14:59,  5.03s/it]
  1%|          | 2/180 [00:05<06:49,  2.30s/it]
  2%|▏         | 3/180 [00:05<04:11,  1.42s/it]
  2%|▏         | 4/180 [00:06<02:57,  1.01s/it]
  3%|▎         | 5/180 [00:06<02:17,  1.28it/s]
  3%|▎         | 6/180 [00:06<01:52,  1.55it/s]
  4%|▍         | 7/180 [00:07<01:36,  1.79it/s]
  4%|▍         | 8/180 [00:07<01:26,  2.00it/s]
  5%|▌         | 9/180 [00:08<01:19,  2.16it/s]
  6%|▌         | 10/180 [00:08<01:14,  2.28it/s]
                                                

  6%|▌         | 10/180 [00:08<01:14,  2.28it/s]
  6%|▌         | 11/180 [00:08<01:11,  2.38it/s]
  7%|▋         | 12/180 [00:09<01:08,  2.45it/s]
  7%|▋         | 13/180 [00:09<01:06,  2.50it/s]
  8%|▊         | 14/180 [00:09<01:05,  2.55it/s]
  8%|▊         | 15/180 [00:10<01:04,  2.57it/s]
  9%|▉         | 16/180 [00:10<01:03,  2.59it/s]
  9%|▉         | 17/180 [00:11<01:02,  2.60it/s]
 10%|█         | 18/180 [00:11<01:01,  2.62it/s]
 11%|█         | 19/180 [00:11<01:01,  2.62it/s]
 11%|█         | 20/180 [00:12<01:00,  2.63it/s]
                                                

 11%|█         | 20/180 [00:12<01:00,  2.63it/s]
 12%|█▏        | 21/180 [00:12<01:00,  2.64it/s]
 12%|█▏        | 22/180 [00:13<00:59,  2.63it/s]
 13%|█▎        | 23/180 [00:13<00:59,  2.64it/s]
 13%|█▎        | 24/180 [00:13<00:59,  2.64it/s]
 14%|█▍        | 25/180 [00:14<00:58,  2.64it/s]
 14%|█▍        | 26/180 [00:14<00:58,  2.64it/s]
 15%|█▌        | 27/180 [00:14<00:57,  2.64it/s]
 16%|█▌        | 28/180 [00:15<00:57,  2.65it/s]
 16%|█▌        | 29/180 [00:15<00:57,  2.65it/s]
 17%|█▋        | 30/180 [00:16<00:56,  2.64it/s]
                                                

 17%|█▋        | 30/180 [00:16<00:56,  2.64it/s]
 17%|█▋        | 31/180 [00:16<00:56,  2.64it/s]
 18%|█▊        | 32/180 [00:16<00:56,  2.64it/s]
 18%|█▊        | 33/180 [00:17<00:55,  2.64it/s]
 19%|█▉        | 34/180 [00:17<00:55,  2.64it/s]
 19%|█▉        | 35/180 [00:17<00:54,  2.64it/s]
 20%|██        | 36/180 [00:18<00:54,  2.64it/s]
 21%|██        | 37/180 [00:18<00:54,  2.64it/s]
 21%|██        | 38/180 [00:19<00:53,  2.64it/s]
 22%|██▏       | 39/180 [00:19<00:53,  2.64it/s]
 22%|██▏       | 40/180 [00:19<00:52,  2.64it/s]
                                                

 22%|██▏       | 40/180 [00:19<00:52,  2.64it/s]
 23%|██▎       | 41/180 [00:20<00:52,  2.64it/s]
 23%|██▎       | 42/180 [00:20<00:52,  2.64it/s]
 24%|██▍       | 43/180 [00:20<00:51,  2.65it/s]
 24%|██▍       | 44/180 [00:21<00:51,  2.65it/s]
 25%|██▌       | 45/180 [00:21<00:50,  2.65it/s]
 26%|██▌       | 46/180 [00:22<00:50,  2.66it/s]
 26%|██▌       | 47/180 [00:22<00:49,  2.66it/s]
 27%|██▋       | 48/180 [00:22<00:49,  2.66it/s]
 27%|██▋       | 49/180 [00:23<00:49,  2.65it/s]
 28%|██▊       | 50/180 [00:23<00:48,  2.66it/s]
                                                

 28%|██▊       | 50/180 [00:23<00:48,  2.66it/s]
 28%|██▊       | 51/180 [00:23<00:48,  2.66it/s]
 29%|██▉       | 52/180 [00:24<00:48,  2.66it/s]
 29%|██▉       | 53/180 [00:24<00:47,  2.66it/s]
 30%|███       | 54/180 [00:25<00:47,  2.66it/s]
 31%|███       | 55/180 [00:25<00:47,  2.66it/s]
 31%|███       | 56/180 [00:25<00:46,  2.66it/s]
 32%|███▏      | 57/180 [00:26<00:46,  2.67it/s]
 32%|███▏      | 58/180 [00:26<00:45,  2.67it/s]
 33%|███▎      | 59/180 [00:26<00:45,  2.67it/s]
 33%|███▎      | 60/180 [00:27<00:35,  3.37it/s]
                                                

 33%|███▎      | 60/180 [00:27<00:35,  3.37it/s]
 34%|███▍      | 61/180 [00:28<01:00,  1.96it/s]
 34%|███▍      | 62/180 [00:28<00:55,  2.13it/s]
 35%|███▌      | 63/180 [00:28<00:51,  2.27it/s]
 36%|███▌      | 64/180 [00:29<00:48,  2.37it/s]
 36%|███▌      | 65/180 [00:29<00:46,  2.46it/s]
 37%|███▋      | 66/180 [00:29<00:45,  2.52it/s]
 37%|███▋      | 67/180 [00:30<00:44,  2.56it/s]
 38%|███▊      | 68/180 [00:30<00:43,  2.58it/s]
 38%|███▊      | 69/180 [00:31<00:42,  2.61it/s]
 39%|███▉      | 70/180 [00:31<00:41,  2.63it/s]
                                                

 39%|███▉      | 70/180 [00:31<00:41,  2.63it/s]
 39%|███▉      | 71/180 [00:31<00:41,  2.64it/s]
 40%|████      | 72/180 [00:32<00:40,  2.65it/s]
 41%|████      | 73/180 [00:32<00:40,  2.66it/s]
 41%|████      | 74/180 [00:32<00:39,  2.66it/s]
 42%|████▏     | 75/180 [00:33<00:39,  2.66it/s]
 42%|████▏     | 76/180 [00:33<00:39,  2.66it/s]
 43%|████▎     | 77/180 [00:34<00:38,  2.67it/s]
 43%|████▎     | 78/180 [00:34<00:38,  2.67it/s]
 44%|████▍     | 79/180 [00:34<00:37,  2.67it/s]
 44%|████▍     | 80/180 [00:35<00:37,  2.67it/s]
                                                

 44%|████▍     | 80/180 [00:35<00:37,  2.67it/s]
 45%|████▌     | 81/180 [00:35<00:37,  2.67it/s]
 46%|████▌     | 82/180 [00:35<00:36,  2.67it/s]
 46%|████▌     | 83/180 [00:36<00:36,  2.67it/s]
 47%|████▋     | 84/180 [00:36<00:35,  2.68it/s]
 47%|████▋     | 85/180 [00:37<00:35,  2.68it/s]
 48%|████▊     | 86/180 [00:37<00:35,  2.67it/s]
 48%|████▊     | 87/180 [00:37<00:34,  2.67it/s]
 49%|████▉     | 88/180 [00:38<00:34,  2.67it/s]
 49%|████▉     | 89/180 [00:38<00:34,  2.67it/s]
 50%|█████     | 90/180 [00:38<00:33,  2.68it/s]
                                                

 50%|█████     | 90/180 [00:38<00:33,  2.68it/s]
 51%|█████     | 91/180 [00:39<00:33,  2.68it/s]
 51%|█████     | 92/180 [00:39<00:32,  2.68it/s]
 52%|█████▏    | 93/180 [00:40<00:32,  2.67it/s]
 52%|█████▏    | 94/180 [00:40<00:32,  2.67it/s]
 53%|█████▎    | 95/180 [00:40<00:31,  2.67it/s]
 53%|█████▎    | 96/180 [00:41<00:31,  2.66it/s]
 54%|█████▍    | 97/180 [00:41<00:31,  2.66it/s]
 54%|█████▍    | 98/180 [00:41<00:30,  2.66it/s]
 55%|█████▌    | 99/180 [00:42<00:30,  2.66it/s]
 56%|█████▌    | 100/180 [00:42<00:30,  2.66it/s]
                                                 

 56%|█████▌    | 100/180 [00:42<00:30,  2.66it/s]
 56%|█████▌    | 101/180 [00:43<00:29,  2.66it/s]
 57%|█████▋    | 102/180 [00:43<00:29,  2.66it/s]
 57%|█████▋    | 103/180 [00:43<00:28,  2.66it/s]
 58%|█████▊    | 104/180 [00:44<00:28,  2.65it/s]
 58%|█████▊    | 105/180 [00:44<00:28,  2.66it/s]
 59%|█████▉    | 106/180 [00:44<00:27,  2.66it/s]
 59%|█████▉    | 107/180 [00:45<00:27,  2.66it/s]
 60%|██████    | 108/180 [00:45<00:27,  2.66it/s]
 61%|██████    | 109/180 [00:46<00:26,  2.66it/s]
 61%|██████    | 110/180 [00:46<00:26,  2.66it/s]
                                                 

 61%|██████    | 110/180 [00:46<00:26,  2.66it/s]
 62%|██████▏   | 111/180 [00:46<00:25,  2.66it/s]
 62%|██████▏   | 112/180 [00:47<00:25,  2.66it/s]
 63%|██████▎   | 113/180 [00:47<00:25,  2.66it/s]
 63%|██████▎   | 114/180 [00:47<00:24,  2.66it/s]
 64%|██████▍   | 115/180 [00:48<00:24,  2.66it/s]
 64%|██████▍   | 116/180 [00:48<00:24,  2.66it/s]
 65%|██████▌   | 117/180 [00:49<00:23,  2.66it/s]
 66%|██████▌   | 118/180 [00:49<00:23,  2.66it/s]
 66%|██████▌   | 119/180 [00:49<00:22,  2.66it/s]
                                                 

 67%|██████▋   | 120/180 [00:49<00:22,  2.66it/s]
 67%|██████▋   | 121/180 [00:50<00:24,  2.40it/s]
 68%|██████▊   | 122/180 [00:51<00:23,  2.46it/s]
 68%|██████▊   | 123/180 [00:51<00:22,  2.51it/s]
 69%|██████▉   | 124/180 [00:51<00:21,  2.55it/s]
 69%|██████▉   | 125/180 [00:52<00:21,  2.58it/s]
 70%|███████   | 126/180 [00:52<00:20,  2.60it/s]
 71%|███████   | 127/180 [00:53<00:20,  2.62it/s]
 71%|███████   | 128/180 [00:53<00:19,  2.63it/s]
 72%|███████▏  | 129/180 [00:53<00:19,  2.63it/s]
 72%|███████▏  | 130/180 [00:54<00:18,  2.65it/s]
                                                 

 72%|███████▏  | 130/180 [00:54<00:18,  2.65it/s]
 73%|███████▎  | 131/180 [00:54<00:18,  2.65it/s]
 73%|███████▎  | 132/180 [00:54<00:18,  2.65it/s]
 74%|███████▍  | 133/180 [00:55<00:17,  2.65it/s]
 74%|███████▍  | 134/180 [00:55<00:17,  2.65it/s]
 75%|███████▌  | 135/180 [00:56<00:16,  2.65it/s]
 76%|███████▌  | 136/180 [00:56<00:16,  2.66it/s]
 76%|███████▌  | 137/180 [00:56<00:16,  2.66it/s]
 77%|███████▋  | 138/180 [00:57<00:15,  2.66it/s]
 77%|███████▋  | 139/180 [00:57<00:15,  2.66it/s]
 78%|███████▊  | 140/180 [00:57<00:15,  2.66it/s]
                                                 

 78%|███████▊  | 140/180 [00:57<00:15,  2.66it/s]
 78%|███████▊  | 141/180 [00:58<00:14,  2.66it/s]
 79%|███████▉  | 142/180 [00:58<00:14,  2.66it/s]
 79%|███████▉  | 143/180 [00:59<00:13,  2.67it/s]
 80%|████████  | 144/180 [00:59<00:13,  2.66it/s]
 81%|████████  | 145/180 [00:59<00:13,  2.66it/s]
 81%|████████  | 146/180 [01:00<00:12,  2.66it/s]
 82%|████████▏ | 147/180 [01:00<00:12,  2.66it/s]
 82%|████████▏ | 148/180 [01:00<00:12,  2.66it/s]
 83%|████████▎ | 149/180 [01:01<00:11,  2.66it/s]
 83%|████████▎ | 150/180 [01:01<00:11,  2.66it/s]
                                                 

 83%|████████▎ | 150/180 [01:01<00:11,  2.66it/s]
 84%|████████▍ | 151/180 [01:02<00:10,  2.66it/s]
 84%|████████▍ | 152/180 [01:02<00:10,  2.66it/s]
 85%|████████▌ | 153/180 [01:02<00:10,  2.66it/s]
 86%|████████▌ | 154/180 [01:03<00:09,  2.66it/s]
 86%|████████▌ | 155/180 [01:03<00:09,  2.66it/s]
 87%|████████▋ | 156/180 [01:03<00:08,  2.67it/s]
 87%|████████▋ | 157/180 [01:04<00:08,  2.66it/s]
 88%|████████▊ | 158/180 [01:04<00:08,  2.66it/s]
 88%|████████▊ | 159/180 [01:05<00:07,  2.66it/s]
 89%|████████▉ | 160/180 [01:05<00:07,  2.66it/s]
                                                 

 89%|████████▉ | 160/180 [01:05<00:07,  2.66it/s]
 89%|████████▉ | 161/180 [01:05<00:07,  2.66it/s]
 90%|█████████ | 162/180 [01:06<00:06,  2.67it/s]
 91%|█████████ | 163/180 [01:06<00:06,  2.66it/s]
 91%|█████████ | 164/180 [01:06<00:06,  2.66it/s]
 92%|█████████▏| 165/180 [01:07<00:05,  2.66it/s]
 92%|█████████▏| 166/180 [01:07<00:05,  2.66it/s]
 93%|█████████▎| 167/180 [01:08<00:04,  2.66it/s]
 93%|█████████▎| 168/180 [01:08<00:04,  2.66it/s]
 94%|█████████▍| 169/180 [01:08<00:04,  2.67it/s]
 94%|█████████▍| 170/180 [01:09<00:03,  2.67it/s]
                                                 

 94%|█████████▍| 170/180 [01:09<00:03,  2.67it/s]
 95%|█████████▌| 171/180 [01:09<00:03,  2.66it/s]
 96%|█████████▌| 172/180 [01:09<00:03,  2.66it/s]
 96%|█████████▌| 173/180 [01:10<00:02,  2.66it/s]
 97%|█████████▋| 174/180 [01:10<00:02,  2.66it/s]
 97%|█████████▋| 175/180 [01:11<00:01,  2.67it/s]
 98%|█████████▊| 176/180 [01:11<00:01,  2.66it/s]
 98%|█████████▊| 177/180 [01:11<00:01,  2.66it/s]
 99%|█████████▉| 178/180 [01:12<00:00,  2.66it/s]
 99%|█████████▉| 179/180 [01:12<00:00,  2.66it/s]
                                                 

100%|██████████| 180/180 [01:12<00:00,  2.66it/s]
                                                 

100%|██████████| 180/180 [01:13<00:00,  2.66it/s]
100%|██████████| 180/180 [01:13<00:00,  2.46it/s]