nla-gemma4e2b-research-checkpoints / L23_clean_cont_train.log

Add files using upload-large-folder tool

73ce672 verified 4 months ago

11.3 kB

C:\Users\caleb\deception-nanochat-sae-research\.venv-gemma4\Lib\site-packages\transformers\modeling_utils.py:4832: UserWarning: expandable_segments not supported on this platform (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\pytorch\c10/cuda/CUDAAllocatorConfig.h:39.)
  _ = torch.empty(int(byte_count // 2), dtype=torch.float16, device=device, requires_grad=False)

Loading weights:   0%|          | 0/1951 [00:00<?, ?it/s]
Loading weights:   0%|          | 3/1951 [00:00<06:21,  5.10it/s]
Loading weights:   1%|1         | 29/1951 [00:00<00:35, 54.71it/s]
Loading weights:   3%|3         | 65/1951 [00:00<00:15, 118.15it/s]
Loading weights:   5%|4         | 88/1951 [00:00<00:13, 142.87it/s]
Loading weights:   6%|6         | 122/1951 [00:01<00:09, 187.37it/s]
Loading weights:   8%|7         | 148/1951 [00:01<00:09, 192.49it/s]
Loading weights:   9%|9         | 185/1951 [00:01<00:07, 237.67it/s]
Loading weights:  11%|#         | 214/1951 [00:01<00:07, 231.47it/s]
Loading weights:  13%|#2        | 249/1951 [00:01<00:06, 261.88it/s]
Loading weights:  14%|#4        | 278/1951 [00:01<00:06, 246.86it/s]
Loading weights:  16%|#6        | 313/1951 [00:01<00:06, 268.98it/s]
Loading weights:  18%|#7        | 342/1951 [00:01<00:05, 273.25it/s]
Loading weights:  19%|#9        | 375/1951 [00:01<00:05, 279.26it/s]
Loading weights:  21%|##        | 406/1951 [00:02<00:05, 275.54it/s]
Loading weights:  23%|##2       | 442/1951 [00:02<00:05, 283.97it/s]
Loading weights:  24%|##4       | 477/1951 [00:02<00:04, 294.83it/s]
Loading weights:  26%|##5       | 507/1951 [00:02<00:04, 291.28it/s]
Loading weights:  28%|##7       | 539/1951 [00:02<00:04, 294.08it/s]
Loading weights:  29%|##9       | 569/1951 [00:02<00:04, 291.19it/s]
Loading weights:  31%|###       | 601/1951 [00:02<00:04, 294.50it/s]
Loading weights:  32%|###2      | 631/1951 [00:02<00:04, 288.45it/s]
Loading weights:  34%|###3      | 663/1951 [00:02<00:04, 296.03it/s]
Loading weights:  36%|###5      | 693/1951 [00:03<00:04, 295.63it/s]
Loading weights:  37%|###7      | 723/1951 [00:03<00:04, 293.13it/s]
Loading weights:  39%|###8      | 753/1951 [00:03<00:04, 284.32it/s]
Loading weights:  39%|###8      | 753/1951 [00:20<00:04, 284.32it/s]
Loading weights:  39%|###8      | 755/1951 [00:23<05:32,  3.60it/s] 
Loading weights:  39%|###8      | 759/1951 [00:23<05:10,  3.84it/s]
Loading weights:  40%|###9      | 780/1951 [00:23<03:25,  5.71it/s]
Loading weights:  41%|####      | 797/1951 [00:24<02:30,  7.69it/s]
Loading weights:  42%|####1     | 810/1951 [00:24<01:58,  9.67it/s]
Loading weights:  42%|####2     | 821/1951 [00:24<01:33, 12.02it/s]
Loading weights:  43%|####2     | 831/1951 [00:24<01:19, 14.05it/s]
Loading weights:  43%|####3     | 839/1951 [00:24<01:06, 16.73it/s]
Loading weights:  43%|####3     | 847/1951 [00:25<00:59, 18.67it/s]
Loading weights:  44%|####4     | 860/1951 [00:25<00:43, 25.23it/s]
Loading weights:  44%|####4     | 867/1951 [00:25<00:39, 27.19it/s]
Loading weights:  45%|####4     | 877/1951 [00:25<00:32, 32.88it/s]
Loading weights:  45%|####5     | 883/1951 [00:25<00:31, 33.40it/s]
Loading weights:  46%|####5     | 894/1951 [00:26<00:25, 41.01it/s]
Loading weights:  46%|####6     | 900/1951 [00:26<00:26, 39.52it/s]
Loading weights:  47%|####6     | 911/1951 [00:26<00:22, 46.40it/s]
Loading weights:  47%|####7     | 917/1951 [00:26<00:24, 42.94it/s]
Loading weights:  47%|####7     | 924/1951 [00:26<00:22, 45.27it/s]
Loading weights:  48%|####7     | 930/1951 [00:26<00:27, 37.71it/s]
Loading weights:  48%|####8     | 942/1951 [00:27<00:19, 52.03it/s]
Loading weights:  49%|####8     | 949/1951 [00:27<00:23, 42.95it/s]
Loading weights:  49%|####9     | 962/1951 [00:27<00:19, 51.83it/s]
Loading weights:  50%|####9     | 968/1951 [00:27<00:21, 46.51it/s]
Loading weights:  50%|#####     | 979/1951 [00:27<00:18, 51.22it/s]
Loading weights:  50%|#####     | 985/1951 [00:28<00:21, 45.83it/s]
Loading weights:  51%|#####1    | 996/1951 [00:28<00:18, 51.88it/s]
Loading weights:  51%|#####1    | 1002/1951 [00:28<00:20, 46.86it/s]
Loading weights:  52%|#####1    | 1009/1951 [00:28<00:18, 50.52it/s]
Loading weights:  52%|#####2    | 1015/1951 [00:28<00:30, 31.09it/s]
Loading weights:  53%|#####2    | 1027/1951 [00:29<00:24, 37.06it/s]
Loading weights:  53%|#####2    | 1032/1951 [00:29<00:33, 27.23it/s]
Loading weights:  53%|#####3    | 1041/1951 [00:29<00:31, 28.70it/s]
Loading weights:  54%|#####3    | 1045/1951 [00:30<00:39, 22.85it/s]
Loading weights:  54%|#####4    | 1055/1951 [00:30<00:32, 27.41it/s]
Loading weights:  54%|#####4    | 1059/1951 [00:30<00:38, 22.94it/s]
Loading weights:  55%|#####4    | 1069/1951 [00:30<00:31, 27.96it/s]
Loading weights:  55%|#####4    | 1073/1951 [00:31<00:38, 22.76it/s]
Loading weights:  55%|#####5    | 1080/1951 [00:31<00:31, 28.03it/s]
Loading weights:  56%|#####5    | 1084/1951 [00:31<00:39, 22.07it/s]
Loading weights:  56%|#####5    | 1087/1951 [00:31<00:41, 20.92it/s]
Loading weights:  56%|#####6    | 1097/1951 [00:32<00:31, 27.54it/s]
Loading weights:  56%|#####6    | 1101/1951 [00:32<00:39, 21.68it/s]
Loading weights:  57%|#####6    | 1111/1951 [00:32<00:31, 26.47it/s]
Loading weights:  57%|#####7    | 1114/1951 [00:33<00:40, 20.84it/s]
Loading weights:  58%|#####7    | 1125/1951 [00:33<00:29, 28.00it/s]
Loading weights:  58%|#####7    | 1129/1951 [00:33<00:35, 23.33it/s]
Loading weights:  58%|#####8    | 1139/1951 [00:33<00:27, 29.19it/s]
Loading weights:  59%|#####8    | 1143/1951 [00:34<00:34, 23.49it/s]
Loading weights:  59%|#####8    | 1150/1951 [00:34<00:27, 28.63it/s]
Loading weights:  59%|#####9    | 1154/1951 [00:34<00:36, 22.13it/s]
Loading weights:  59%|#####9    | 1157/1951 [00:34<00:37, 21.27it/s]
Loading weights:  60%|#####9    | 1165/1951 [00:34<00:25, 30.32it/s]
Loading weights:  60%|#####9    | 1170/1951 [00:35<00:38, 20.41it/s]
Loading weights:  61%|######    | 1181/1951 [00:35<00:29, 26.54it/s]
Loading weights:  61%|######    | 1185/1951 [00:35<00:34, 21.95it/s]
Loading weights:  61%|######1   | 1195/1951 [00:36<00:29, 25.78it/s]
Loading weights:  61%|######1   | 1199/1951 [00:36<00:36, 20.54it/s]
Loading weights:  62%|######1   | 1209/1951 [00:36<00:28, 25.74it/s]
Loading weights:  62%|######2   | 1212/1951 [00:37<00:35, 21.03it/s]
Loading weights:  63%|######2   | 1220/1951 [00:37<00:26, 27.54it/s]
Loading weights:  63%|######2   | 1224/1951 [00:37<00:32, 22.14it/s]
Loading weights:  63%|######2   | 1227/1951 [00:37<00:34, 21.05it/s]
Loading weights:  63%|######3   | 1237/1951 [00:37<00:25, 27.79it/s]
Loading weights:  64%|######3   | 1241/1951 [00:38<00:32, 22.16it/s]
Loading weights:  64%|######4   | 1251/1951 [00:38<00:24, 28.35it/s]
Loading weights:  64%|######4   | 1255/1951 [00:38<00:28, 24.04it/s]
Loading weights:  65%|######4   | 1265/1951 [00:38<00:22, 29.84it/s]
Loading weights:  65%|######5   | 1269/1951 [00:39<00:28, 24.08it/s]
Loading weights:  66%|######5   | 1279/1951 [00:39<00:23, 29.07it/s]
Loading weights:  66%|######5   | 1283/1951 [00:39<00:29, 22.93it/s]
Loading weights:  66%|######6   | 1290/1951 [00:39<00:23, 27.69it/s]
Loading weights:  66%|######6   | 1294/1951 [00:40<00:22, 28.90it/s]
Loading weights:  67%|######7   | 1316/1951 [00:40<00:10, 61.85it/s]
Loading weights:  69%|######8   | 1343/1951 [00:40<00:06, 101.22it/s]
Loading weights:  70%|#######   | 1373/1951 [00:40<00:04, 141.47it/s]
Loading weights:  72%|#######1  | 1396/1951 [00:40<00:03, 161.70it/s]
Loading weights:  73%|#######2  | 1420/1951 [00:40<00:02, 180.58it/s]
Loading weights:  74%|#######4  | 1444/1951 [00:40<00:02, 195.66it/s]
Loading weights:  75%|#######5  | 1471/1951 [00:40<00:02, 206.58it/s]
Loading weights:  77%|#######7  | 1507/1951 [00:40<00:01, 244.32it/s]
Loading weights:  79%|#######9  | 1543/1951 [00:41<00:01, 276.19it/s]
Loading weights:  81%|########  | 1578/1951 [00:41<00:01, 290.42it/s]
Loading weights:  82%|########2 | 1608/1951 [00:41<00:01, 277.92it/s]
Loading weights:  84%|########3 | 1637/1951 [00:41<00:01, 262.93it/s]
Loading weights:  85%|########5 | 1666/1951 [00:41<00:01, 263.32it/s]
Loading weights:  87%|########6 | 1696/1951 [00:41<00:00, 267.59it/s]
Loading weights:  88%|########8 | 1724/1951 [00:41<00:00, 252.38it/s]
Loading weights:  90%|########9 | 1753/1951 [00:41<00:00, 248.62it/s]
Loading weights:  92%|#########1| 1789/1951 [00:41<00:00, 270.67it/s]
Loading weights:  93%|#########3| 1819/1951 [00:42<00:00, 277.27it/s]
Loading weights:  95%|#########4| 1847/1951 [00:42<00:00, 272.08it/s]
Loading weights:  96%|#########6| 1876/1951 [00:42<00:00, 273.68it/s]
Loading weights:  98%|#########8| 1912/1951 [00:42<00:00, 284.18it/s]
Loading weights:  99%|#########9| 1941/1951 [00:42<00:00, 280.47it/s]
Loading weights: 100%|##########| 1951/1951 [00:42<00:00, 45.79it/s] 
[pd] RESUMED from experiments/v8_nla_local/checkpoints/av_L23_clean/step_000400 (extending grokking horizon)
[pd] data=experiments/v8_nla_local/data/stage3_balanced/av_sft_balanced.parquet rows=1356 uniform=True gamma=1.0 corpus_mean_weight=1.000
[pd] DOMAIN-AWARE negs: avg same-dom=16.0 diff-dom=16.0 per anchor (15 domains)
[pd] contrastive ON: beta=1.0 K=2 temp=1.0 domain_aware=True token_dropout=0.0 (InfoNCE over hard-neg activations)
[pd] INJECT at residual layer 23 (native scale); embed pass-through
[pd] inject_layer=23 inject_mode=raw
[pd] optimizer=adamw8bit lr=5e-05 wd=0.01 steps=600 ga=8 trainable_params=12.08M
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
C:\Users\caleb\deception-nanochat-sae-research\.venv-gemma4\Lib\site-packages\torch\_dynamo\eval_frame.py:1181: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. Starting in PyTorch 2.9, calling checkpoint without use_reentrant will raise an exception. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  return fn(*args, **kwargs)
C:\Users\caleb\deception-nanochat-sae-research\experiments\v8_nla_local\train_prior_deviation_av.py:325: UserWarning: Converting a tensor with requires_grad=True to a scalar may lead to unexpected behavior.
Consider using tensor.detach() first. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\pytorch\torch\csrc\autograd\generated\python_variable_methods.cpp:837.)
  (loss / ga).backward(); tot += float(loss) / ga
[pd] step 1/600 wloss=3.759 skipped=0
[pd] step 10/600 wloss=4.033 skipped=0
[pd] step 20/600 wloss=3.353 skipped=0
[pd] step 30/600 wloss=3.700 skipped=0
[pd] step 40/600 wloss=3.672 skipped=0
[pd] step 50/600 wloss=3.679 skipped=0
[pd] step 60/600 wloss=4.049 skipped=0
[pd] step 70/600 wloss=3.437 skipped=0
[pd] step 80/600 wloss=3.949 skipped=0
[pd] step 90/600 wloss=3.907 skipped=0
[pd] step 100/600 wloss=3.877 skipped=0
C:\Users\caleb\deception-nanochat-sae-research\.venv-gemma4\Lib\site-packages\peft\utils\save_and_load.py:372: UserWarning: Could not find a config file in google/gemma-4-E2B - will assume that the vocabulary was not modified.
  warnings.warn(
[pd] saved experiments/v8_nla_local/checkpoints/av_L23_clean_cont\step_000100
[pd] step 110/600 wloss=3.954 skipped=0
[pd] step 120/600 wloss=3.498 skipped=0

	C:\Users\caleb\deception-nanochat-sae-research\.venv-gemma4\Lib\site-packages\transformers\modeling_utils.py:4832: UserWarning: expandable_segments not supported on this platform (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\pytorch\c10/cuda/CUDAAllocatorConfig.h:39.)
	_ = torch.empty(int(byte_count // 2), dtype=torch.float16, device=device, requires_grad=False)
	Loading weights: 0%\| \| 0/1951 [00:00<?, ?it/s] Loading weights: 0%\| \| 3/1951 [00:00<06:21, 5.10it/s] Loading weights: 1%\|1 \| 29/1951 [00:00<00:35, 54.71it/s] Loading weights: 3%\|3 \| 65/1951 [00:00<00:15, 118.15it/s] Loading weights: 5%\|4 \| 88/1951 [00:00<00:13, 142.87it/s] Loading weights: 6%\|6 \| 122/1951 [00:01<00:09, 187.37it/s] Loading weights: 8%\|7 \| 148/1951 [00:01<00:09, 192.49it/s] Loading weights: 9%\|9 \| 185/1951 [00:01<00:07, 237.67it/s] Loading weights: 11%\|# \| 214/1951 [00:01<00:07, 231.47it/s] Loading weights: 13%\|#2 \| 249/1951 [00:01<00:06, 261.88it/s] Loading weights: 14%\|#4 \| 278/1951 [00:01<00:06, 246.86it/s] Loading weights: 16%\|#6 \| 313/1951 [00:01<00:06, 268.98it/s] Loading weights: 18%\|#7 \| 342/1951 [00:01<00:05, 273.25it/s] Loading weights: 19%\|#9 \| 375/1951 [00:01<00:05, 279.26it/s] Loading weights: 21%\|## \| 406/1951 [00:02<00:05, 275.54it/s] Loading weights: 23%\|##2 \| 442/1951 [00:02<00:05, 283.97it/s] Loading weights: 24%\|##4 \| 477/1951 [00:02<00:04, 294.83it/s] Loading weights: 26%\|##5 \| 507/1951 [00:02<00:04, 291.28it/s] Loading weights: 28%\|##7 \| 539/1951 [00:02<00:04, 294.08it/s] Loading weights: 29%\|##9 \| 569/1951 [00:02<00:04, 291.19it/s] Loading weights: 31%\|### \| 601/1951 [00:02<00:04, 294.50it/s] Loading weights: 32%\|###2 \| 631/1951 [00:02<00:04, 288.45it/s] Loading weights: 34%\|###3 \| 663/1951 [00:02<00:04, 296.03it/s] Loading weights: 36%\|###5 \| 693/1951 [00:03<00:04, 295.63it/s] Loading weights: 37%\|###7 \| 723/1951 [00:03<00:04, 293.13it/s] Loading weights: 39%\|###8 \| 753/1951 [00:03<00:04, 284.32it/s] Loading weights: 39%\|###8 \| 753/1951 [00:20<00:04, 284.32it/s] Loading weights: 39%\|###8 \| 755/1951 [00:23<05:32, 3.60it/s] Loading weights: 39%\|###8 \| 759/1951 [00:23<05:10, 3.84it/s] Loading weights: 40%\|###9 \| 780/1951 [00:23<03:25, 5.71it/s] Loading weights: 41%\|#### \| 797/1951 [00:24<02:30, 7.69it/s] Loading weights: 42%\|####1 \| 810/1951 [00:24<01:58, 9.67it/s] Loading weights: 42%\|####2 \| 821/1951 [00:24<01:33, 12.02it/s] Loading weights: 43%\|####2 \| 831/1951 [00:24<01:19, 14.05it/s] Loading weights: 43%\|####3 \| 839/1951 [00:24<01:06, 16.73it/s] Loading weights: 43%\|####3 \| 847/1951 [00:25<00:59, 18.67it/s] Loading weights: 44%\|####4 \| 860/1951 [00:25<00:43, 25.23it/s] Loading weights: 44%\|####4 \| 867/1951 [00:25<00:39, 27.19it/s] Loading weights: 45%\|####4 \| 877/1951 [00:25<00:32, 32.88it/s] Loading weights: 45%\|####5 \| 883/1951 [00:25<00:31, 33.40it/s] Loading weights: 46%\|####5 \| 894/1951 [00:26<00:25, 41.01it/s] Loading weights: 46%\|####6 \| 900/1951 [00:26<00:26, 39.52it/s] Loading weights: 47%\|####6 \| 911/1951 [00:26<00:22, 46.40it/s] Loading weights: 47%\|####7 \| 917/1951 [00:26<00:24, 42.94it/s] Loading weights: 47%\|####7 \| 924/1951 [00:26<00:22, 45.27it/s] Loading weights: 48%\|####7 \| 930/1951 [00:26<00:27, 37.71it/s] Loading weights: 48%\|####8 \| 942/1951 [00:27<00:19, 52.03it/s] Loading weights: 49%\|####8 \| 949/1951 [00:27<00:23, 42.95it/s] Loading weights: 49%\|####9 \| 962/1951 [00:27<00:19, 51.83it/s] Loading weights: 50%\|####9 \| 968/1951 [00:27<00:21, 46.51it/s] Loading weights: 50%\|##### \| 979/1951 [00:27<00:18, 51.22it/s] Loading weights: 50%\|##### \| 985/1951 [00:28<00:21, 45.83it/s] Loading weights: 51%\|#####1 \| 996/1951 [00:28<00:18, 51.88it/s] Loading weights: 51%\|#####1 \| 1002/1951 [00:28<00:20, 46.86it/s] Loading weights: 52%\|#####1 \| 1009/1951 [00:28<00:18, 50.52it/s] Loading weights: 52%\|#####2 \| 1015/1951 [00:28<00:30, 31.09it/s] Loading weights: 53%\|#####2 \| 1027/1951 [00:29<00:24, 37.06it/s] Loading weights: 53%\|#####2 \| 1032/1951 [00:29<00:33, 27.23it/s] Loading weights: 53%\|#####3 \| 1041/1951 [00:29<00:31, 28.70it/s] Loading weights: 54%\|#####3 \| 1045/1951 [00:30<00:39, 22.85it/s] Loading weights: 54%\|#####4 \| 1055/1951 [00:30<00:32, 27.41it/s] Loading weights: 54%\|#####4 \| 1059/1951 [00:30<00:38, 22.94it/s] Loading weights: 55%\|#####4 \| 1069/1951 [00:30<00:31, 27.96it/s] Loading weights: 55%\|#####4 \| 1073/1951 [00:31<00:38, 22.76it/s] Loading weights: 55%\|#####5 \| 1080/1951 [00:31<00:31, 28.03it/s] Loading weights: 56%\|#####5 \| 1084/1951 [00:31<00:39, 22.07it/s] Loading weights: 56%\|#####5 \| 1087/1951 [00:31<00:41, 20.92it/s] Loading weights: 56%\|#####6 \| 1097/1951 [00:32<00:31, 27.54it/s] Loading weights: 56%\|#####6 \| 1101/1951 [00:32<00:39, 21.68it/s] Loading weights: 57%\|#####6 \| 1111/1951 [00:32<00:31, 26.47it/s] Loading weights: 57%\|#####7 \| 1114/1951 [00:33<00:40, 20.84it/s] Loading weights: 58%\|#####7 \| 1125/1951 [00:33<00:29, 28.00it/s] Loading weights: 58%\|#####7 \| 1129/1951 [00:33<00:35, 23.33it/s] Loading weights: 58%\|#####8 \| 1139/1951 [00:33<00:27, 29.19it/s] Loading weights: 59%\|#####8 \| 1143/1951 [00:34<00:34, 23.49it/s] Loading weights: 59%\|#####8 \| 1150/1951 [00:34<00:27, 28.63it/s] Loading weights: 59%\|#####9 \| 1154/1951 [00:34<00:36, 22.13it/s] Loading weights: 59%\|#####9 \| 1157/1951 [00:34<00:37, 21.27it/s] Loading weights: 60%\|#####9 \| 1165/1951 [00:34<00:25, 30.32it/s] Loading weights: 60%\|#####9 \| 1170/1951 [00:35<00:38, 20.41it/s] Loading weights: 61%\|###### \| 1181/1951 [00:35<00:29, 26.54it/s] Loading weights: 61%\|###### \| 1185/1951 [00:35<00:34, 21.95it/s] Loading weights: 61%\|######1 \| 1195/1951 [00:36<00:29, 25.78it/s] Loading weights: 61%\|######1 \| 1199/1951 [00:36<00:36, 20.54it/s] Loading weights: 62%\|######1 \| 1209/1951 [00:36<00:28, 25.74it/s] Loading weights: 62%\|######2 \| 1212/1951 [00:37<00:35, 21.03it/s] Loading weights: 63%\|######2 \| 1220/1951 [00:37<00:26, 27.54it/s] Loading weights: 63%\|######2 \| 1224/1951 [00:37<00:32, 22.14it/s] Loading weights: 63%\|######2 \| 1227/1951 [00:37<00:34, 21.05it/s] Loading weights: 63%\|######3 \| 1237/1951 [00:37<00:25, 27.79it/s] Loading weights: 64%\|######3 \| 1241/1951 [00:38<00:32, 22.16it/s] Loading weights: 64%\|######4 \| 1251/1951 [00:38<00:24, 28.35it/s] Loading weights: 64%\|######4 \| 1255/1951 [00:38<00:28, 24.04it/s] Loading weights: 65%\|######4 \| 1265/1951 [00:38<00:22, 29.84it/s] Loading weights: 65%\|######5 \| 1269/1951 [00:39<00:28, 24.08it/s] Loading weights: 66%\|######5 \| 1279/1951 [00:39<00:23, 29.07it/s] Loading weights: 66%\|######5 \| 1283/1951 [00:39<00:29, 22.93it/s] Loading weights: 66%\|######6 \| 1290/1951 [00:39<00:23, 27.69it/s] Loading weights: 66%\|######6 \| 1294/1951 [00:40<00:22, 28.90it/s] Loading weights: 67%\|######7 \| 1316/1951 [00:40<00:10, 61.85it/s] Loading weights: 69%\|######8 \| 1343/1951 [00:40<00:06, 101.22it/s] Loading weights: 70%\|####### \| 1373/1951 [00:40<00:04, 141.47it/s] Loading weights: 72%\|#######1 \| 1396/1951 [00:40<00:03, 161.70it/s] Loading weights: 73%\|#######2 \| 1420/1951 [00:40<00:02, 180.58it/s] Loading weights: 74%\|#######4 \| 1444/1951 [00:40<00:02, 195.66it/s] Loading weights: 75%\|#######5 \| 1471/1951 [00:40<00:02, 206.58it/s] Loading weights: 77%\|#######7 \| 1507/1951 [00:40<00:01, 244.32it/s] Loading weights: 79%\|#######9 \| 1543/1951 [00:41<00:01, 276.19it/s] Loading weights: 81%\|######## \| 1578/1951 [00:41<00:01, 290.42it/s] Loading weights: 82%\|########2 \| 1608/1951 [00:41<00:01, 277.92it/s] Loading weights: 84%\|########3 \| 1637/1951 [00:41<00:01, 262.93it/s] Loading weights: 85%\|########5 \| 1666/1951 [00:41<00:01, 263.32it/s] Loading weights: 87%\|########6 \| 1696/1951 [00:41<00:00, 267.59it/s] Loading weights: 88%\|########8 \| 1724/1951 [00:41<00:00, 252.38it/s] Loading weights: 90%\|########9 \| 1753/1951 [00:41<00:00, 248.62it/s] Loading weights: 92%\|#########1\| 1789/1951 [00:41<00:00, 270.67it/s] Loading weights: 93%\|#########3\| 1819/1951 [00:42<00:00, 277.27it/s] Loading weights: 95%\|#########4\| 1847/1951 [00:42<00:00, 272.08it/s] Loading weights: 96%\|#########6\| 1876/1951 [00:42<00:00, 273.68it/s] Loading weights: 98%\|#########8\| 1912/1951 [00:42<00:00, 284.18it/s] Loading weights: 99%\|#########9\| 1941/1951 [00:42<00:00, 280.47it/s] Loading weights: 100%\|##########\| 1951/1951 [00:42<00:00, 45.79it/s]
	[pd] RESUMED from experiments/v8_nla_local/checkpoints/av_L23_clean/step_000400 (extending grokking horizon)
	[pd] data=experiments/v8_nla_local/data/stage3_balanced/av_sft_balanced.parquet rows=1356 uniform=True gamma=1.0 corpus_mean_weight=1.000
	[pd] DOMAIN-AWARE negs: avg same-dom=16.0 diff-dom=16.0 per anchor (15 domains)
	[pd] contrastive ON: beta=1.0 K=2 temp=1.0 domain_aware=True token_dropout=0.0 (InfoNCE over hard-neg activations)
	[pd] INJECT at residual layer 23 (native scale); embed pass-through
	[pd] inject_layer=23 inject_mode=raw
	[pd] optimizer=adamw8bit lr=5e-05 wd=0.01 steps=600 ga=8 trainable_params=12.08M
	`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
	C:\Users\caleb\deception-nanochat-sae-research\.venv-gemma4\Lib\site-packages\torch\_dynamo\eval_frame.py:1181: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. Starting in PyTorch 2.9, calling checkpoint without use_reentrant will raise an exception. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
	return fn(args, *kwargs)
	C:\Users\caleb\deception-nanochat-sae-research\experiments\v8_nla_local\train_prior_deviation_av.py:325: UserWarning: Converting a tensor with requires_grad=True to a scalar may lead to unexpected behavior.
	Consider using tensor.detach() first. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\pytorch\torch\csrc\autograd\generated\python_variable_methods.cpp:837.)
	(loss / ga).backward(); tot += float(loss) / ga
	[pd] step 1/600 wloss=3.759 skipped=0
	[pd] step 10/600 wloss=4.033 skipped=0
	[pd] step 20/600 wloss=3.353 skipped=0
	[pd] step 30/600 wloss=3.700 skipped=0
	[pd] step 40/600 wloss=3.672 skipped=0
	[pd] step 50/600 wloss=3.679 skipped=0
	[pd] step 60/600 wloss=4.049 skipped=0
	[pd] step 70/600 wloss=3.437 skipped=0
	[pd] step 80/600 wloss=3.949 skipped=0
	[pd] step 90/600 wloss=3.907 skipped=0
	[pd] step 100/600 wloss=3.877 skipped=0
	C:\Users\caleb\deception-nanochat-sae-research\.venv-gemma4\Lib\site-packages\peft\utils\save_and_load.py:372: UserWarning: Could not find a config file in google/gemma-4-E2B - will assume that the vocabulary was not modified.
	warnings.warn(
	[pd] saved experiments/v8_nla_local/checkpoints/av_L23_clean_cont\step_000100
	[pd] step 110/600 wloss=3.954 skipped=0
	[pd] step 120/600 wloss=3.498 skipped=0