***** Running training *****
Training 1/1 epoch:   0%|          | 0/125 [00:00<?, ?it/s]`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
Training 1/1 epoch (loss 2.6420): 100%|██████████| 125/125 [03:09<00:00,  1.52s/it]
[2025-05-27 21:24:24,343] [INFO] [logging.py:107:log_dist] [Rank 0] step=10, skipped=0, lr=[1e-05, 1e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2025-05-27 21:24:24,343] [INFO] [timer.py:264:stop] epoch=0/micro_step=80/global_step=10, RunningAvgSamplesPerSec=5.752954441199185, CurrSamplesPerSec=5.316102543402453, MemAllocated=3.04GB, MaxMemAllocated=4.24GB
Saving model to "/aifs4su/hansirui_1st/jiayi/setting3-imdb/tinyllama-3T/tinyllama-3T-s3-Q1-2000-Q2-1000" ...
tokenizer config file saved in /aifs4su/hansirui_1st/jiayi/setting3-imdb/tinyllama-3T/tinyllama-3T-s3-Q1-2000-Q2-1000/tokenizer_config.json
Special tokens file saved in /aifs4su/hansirui_1st/jiayi/setting3-imdb/tinyllama-3T/tinyllama-3T-s3-Q1-2000-Q2-1000/special_tokens_map.json
Saving 16-bit model...
[2025-05-27 21:25:30,023] [INFO] [logging.py:107:log_dist] [Rank 0] [Torch] Checkpoint global_step15 is about to be saved!
[2025-05-27 21:25:30,023] [INFO] [engine.py:3860:save_16bit_model] Saving model weights to /aifs4su/hansirui_1st/jiayi/setting3-imdb/tinyllama-3T/tinyllama-3T-s3-Q1-2000-Q2-1000/pytorch_model.bin, tag: global_step15
[2025-05-27 21:25:30,023] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving /aifs4su/hansirui_1st/jiayi/setting3-imdb/tinyllama-3T/tinyllama-3T-s3-Q1-2000-Q2-1000/pytorch_model.bin...
[2025-05-27 21:25:32,278] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved /aifs4su/hansirui_1st/jiayi/setting3-imdb/tinyllama-3T/tinyllama-3T-s3-Q1-2000-Q2-1000/pytorch_model.bin.
[2025-05-27 21:25:32,278] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step15 is ready now!
Model saved!