2024-04-02 17:57:20,893 - ==> Logging on master GPU: 0 2024-04-02 17:57:20,893 - ==> Running Trainer: MAMBAADTrainer 2024-04-02 17:57:20,893 - ==> Using GPU: [0] for Training 2024-04-02 17:57:20,893 - ==> Building model 2024-04-02 17:57:21,287 - Loaded from checkpoint 'model/pretrain/resnet34-43635321.pth' 2024-04-02 17:57:39,708 - ------------------------------------ MAMBAAD ------------------------------------ | module | #parameters or shape | #flops | |:------------------------------------------------------------|:-----------------------|:-------------| | model | 23.099M | 7.537G | | net_t | 8.17M | 3.955G | | net_t.conv1 | 9.408K | 0.154G | | net_t.conv1.weight | (64, 3, 7, 7) | | | net_t.bn1 | 0.128K | 2.097M | | net_t.bn1.weight | (64,) | | | net_t.bn1.bias | (64,) | | | net_t.layer1 | 0.222M | 0.909G | | net_t.layer1.0 | 73.984K | 0.303G | | net_t.layer1.0.conv1 | 36.864K | 0.151G | | net_t.layer1.0.conv1.weight | (64, 64, 3, 3) | | | net_t.layer1.0.bn1 | 0.128K | 0.524M | | net_t.layer1.0.bn1.weight | (64,) | | | net_t.layer1.0.bn1.bias | (64,) | | | net_t.layer1.0.conv2 | 36.864K | 0.151G | | net_t.layer1.0.conv2.weight | (64, 64, 3, 3) | | | net_t.layer1.0.bn2 | 0.128K | 0.524M | | net_t.layer1.0.bn2.weight | (64,) | | | net_t.layer1.0.bn2.bias | (64,) | | | net_t.layer1.1 | 73.984K | 0.303G | | net_t.layer1.1.conv1 | 36.864K | 0.151G | | net_t.layer1.1.conv1.weight | (64, 64, 3, 3) | | | net_t.layer1.1.bn1 | 0.128K | 0.524M | | net_t.layer1.1.bn1.weight | (64,) | | | net_t.layer1.1.bn1.bias | (64,) | | | net_t.layer1.1.conv2 | 36.864K | 0.151G | | net_t.layer1.1.conv2.weight | (64, 64, 3, 3) | | | net_t.layer1.1.bn2 | 0.128K | 0.524M | | net_t.layer1.1.bn2.weight | (64,) | | | net_t.layer1.1.bn2.bias | (64,) | | | net_t.layer1.2 | 73.984K | 0.303G | | net_t.layer1.2.conv1 | 36.864K | 0.151G | | net_t.layer1.2.conv1.weight | (64, 64, 3, 3) | | | net_t.layer1.2.bn1 | 0.128K | 0.524M | | net_t.layer1.2.bn1.weight | (64,) | | | net_t.layer1.2.bn1.bias | (64,) | | | net_t.layer1.2.conv2 | 36.864K | 0.151G | | net_t.layer1.2.conv2.weight | (64, 64, 3, 3) | | | net_t.layer1.2.bn2 | 0.128K | 0.524M | | net_t.layer1.2.bn2.weight | (64,) | | | net_t.layer1.2.bn2.bias | (64,) | | | net_t.layer2 | 1.116M | 1.143G | | net_t.layer2.0 | 0.23M | 0.236G | | net_t.layer2.0.conv1 | 73.728K | 75.497M | | net_t.layer2.0.conv1.weight | (128, 64, 3, 3) | | | net_t.layer2.0.bn1 | 0.256K | 0.262M | | net_t.layer2.0.bn1.weight | (128,) | | | net_t.layer2.0.bn1.bias | (128,) | | | net_t.layer2.0.conv2 | 0.147M | 0.151G | | net_t.layer2.0.conv2.weight | (128, 128, 3, 3) | | | net_t.layer2.0.bn2 | 0.256K | 0.262M | | net_t.layer2.0.bn2.weight | (128,) | | | net_t.layer2.0.bn2.bias | (128,) | | | net_t.layer2.0.downsample | 8.448K | 8.651M | | net_t.layer2.0.downsample.0 | 8.192K | 8.389M | | net_t.layer2.0.downsample.1 | 0.256K | 0.262M | | net_t.layer2.1 | 0.295M | 0.303G | | net_t.layer2.1.conv1 | 0.147M | 0.151G | | net_t.layer2.1.conv1.weight | (128, 128, 3, 3) | | | net_t.layer2.1.bn1 | 0.256K | 0.262M | | net_t.layer2.1.bn1.weight | (128,) | | | net_t.layer2.1.bn1.bias | (128,) | | | net_t.layer2.1.conv2 | 0.147M | 0.151G | | net_t.layer2.1.conv2.weight | (128, 128, 3, 3) | | | net_t.layer2.1.bn2 | 0.256K | 0.262M | | net_t.layer2.1.bn2.weight | (128,) | | | net_t.layer2.1.bn2.bias | (128,) | | | net_t.layer2.2 | 0.295M | 0.303G | | net_t.layer2.2.conv1 | 0.147M | 0.151G | | net_t.layer2.2.conv1.weight | (128, 128, 3, 3) | | | net_t.layer2.2.bn1 | 0.256K | 0.262M | | net_t.layer2.2.bn1.weight | (128,) | | | net_t.layer2.2.bn1.bias | (128,) | | | net_t.layer2.2.conv2 | 0.147M | 0.151G | | net_t.layer2.2.conv2.weight | (128, 128, 3, 3) | | | net_t.layer2.2.bn2 | 0.256K | 0.262M | | net_t.layer2.2.bn2.weight | (128,) | | | net_t.layer2.2.bn2.bias | (128,) | | | net_t.layer2.3 | 0.295M | 0.303G | | net_t.layer2.3.conv1 | 0.147M | 0.151G | | net_t.layer2.3.conv1.weight | (128, 128, 3, 3) | | | net_t.layer2.3.bn1 | 0.256K | 0.262M | | net_t.layer2.3.bn1.weight | (128,) | | | net_t.layer2.3.bn1.bias | (128,) | | | net_t.layer2.3.conv2 | 0.147M | 0.151G | | net_t.layer2.3.conv2.weight | (128, 128, 3, 3) | | | net_t.layer2.3.bn2 | 0.256K | 0.262M | | net_t.layer2.3.bn2.weight | (128,) | | | net_t.layer2.3.bn2.bias | (128,) | | | net_t.layer3 | 6.822M | 1.747G | | net_t.layer3.0 | 0.919M | 0.235G | | net_t.layer3.0.conv1 | 0.295M | 75.497M | | net_t.layer3.0.conv1.weight | (256, 128, 3, 3) | | | net_t.layer3.0.bn1 | 0.512K | 0.131M | | net_t.layer3.0.bn1.weight | (256,) | | | net_t.layer3.0.bn1.bias | (256,) | | | net_t.layer3.0.conv2 | 0.59M | 0.151G | | net_t.layer3.0.conv2.weight | (256, 256, 3, 3) | | | net_t.layer3.0.bn2 | 0.512K | 0.131M | | net_t.layer3.0.bn2.weight | (256,) | | | net_t.layer3.0.bn2.bias | (256,) | | | net_t.layer3.0.downsample | 33.28K | 8.52M | | net_t.layer3.0.downsample.0 | 32.768K | 8.389M | | net_t.layer3.0.downsample.1 | 0.512K | 0.131M | | net_t.layer3.1 | 1.181M | 0.302G | | net_t.layer3.1.conv1 | 0.59M | 0.151G | | net_t.layer3.1.conv1.weight | (256, 256, 3, 3) | | | net_t.layer3.1.bn1 | 0.512K | 0.131M | | net_t.layer3.1.bn1.weight | (256,) | | | net_t.layer3.1.bn1.bias | (256,) | | | net_t.layer3.1.conv2 | 0.59M | 0.151G | | net_t.layer3.1.conv2.weight | (256, 256, 3, 3) | | | net_t.layer3.1.bn2 | 0.512K | 0.131M | | net_t.layer3.1.bn2.weight | (256,) | | | net_t.layer3.1.bn2.bias | (256,) | | | net_t.layer3.2 | 1.181M | 0.302G | | net_t.layer3.2.conv1 | 0.59M | 0.151G | | net_t.layer3.2.conv1.weight | (256, 256, 3, 3) | | | net_t.layer3.2.bn1 | 0.512K | 0.131M | | net_t.layer3.2.bn1.weight | (256,) | | | net_t.layer3.2.bn1.bias | (256,) | | | net_t.layer3.2.conv2 | 0.59M | 0.151G | | net_t.layer3.2.conv2.weight | (256, 256, 3, 3) | | | net_t.layer3.2.bn2 | 0.512K | 0.131M | | net_t.layer3.2.bn2.weight | (256,) | | | net_t.layer3.2.bn2.bias | (256,) | | | net_t.layer3.3 | 1.181M | 0.302G | | net_t.layer3.3.conv1 | 0.59M | 0.151G | | net_t.layer3.3.conv1.weight | (256, 256, 3, 3) | | | net_t.layer3.3.bn1 | 0.512K | 0.131M | | net_t.layer3.3.bn1.weight | (256,) | | | net_t.layer3.3.bn1.bias | (256,) | | | net_t.layer3.3.conv2 | 0.59M | 0.151G | | net_t.layer3.3.conv2.weight | (256, 256, 3, 3) | | | net_t.layer3.3.bn2 | 0.512K | 0.131M | | net_t.layer3.3.bn2.weight | (256,) | | | net_t.layer3.3.bn2.bias | (256,) | | | net_t.layer3.4 | 1.181M | 0.302G | | net_t.layer3.4.conv1 | 0.59M | 0.151G | | net_t.layer3.4.conv1.weight | (256, 256, 3, 3) | | | net_t.layer3.4.bn1 | 0.512K | 0.131M | | net_t.layer3.4.bn1.weight | (256,) | | | net_t.layer3.4.bn1.bias | (256,) | | | net_t.layer3.4.conv2 | 0.59M | 0.151G | | net_t.layer3.4.conv2.weight | (256, 256, 3, 3) | | | net_t.layer3.4.bn2 | 0.512K | 0.131M | | net_t.layer3.4.bn2.weight | (256,) | | | net_t.layer3.4.bn2.bias | (256,) | | | net_t.layer3.5 | 1.181M | 0.302G | | net_t.layer3.5.conv1 | 0.59M | 0.151G | | net_t.layer3.5.conv1.weight | (256, 256, 3, 3) | | | net_t.layer3.5.bn1 | 0.512K | 0.131M | | net_t.layer3.5.bn1.weight | (256,) | | | net_t.layer3.5.bn1.bias | (256,) | | | net_t.layer3.5.conv2 | 0.59M | 0.151G | | net_t.layer3.5.conv2.weight | (256, 256, 3, 3) | | | net_t.layer3.5.bn2 | 0.512K | 0.131M | | net_t.layer3.5.bn2.weight | (256,) | | | net_t.layer3.5.bn2.bias | (256,) | | | mff_oce | 1.458M | 0.269G | | mff_oce.bn_layer | 0.94M | 66.47M | | mff_oce.bn_layer.0 | 0.379M | 30.622M | | mff_oce.bn_layer.0.conv1 | 32.768K | 8.389M | | mff_oce.bn_layer.0.conv1.weight | (128, 256, 1, 1) | | | mff_oce.bn_layer.0.bn1 | 0.256K | 65.536K | | mff_oce.bn_layer.0.bn1.weight | (128,) | | | mff_oce.bn_layer.0.bn1.bias | (128,) | | | mff_oce.bn_layer.0.conv2 | 0.147M | 9.437M | | mff_oce.bn_layer.0.conv2.weight | (128, 128, 3, 3) | | | mff_oce.bn_layer.0.bn2 | 0.256K | 16.384K | | mff_oce.bn_layer.0.bn2.weight | (128,) | | | mff_oce.bn_layer.0.bn2.bias | (128,) | | | mff_oce.bn_layer.0.conv3 | 65.536K | 4.194M | | mff_oce.bn_layer.0.conv3.weight | (512, 128, 1, 1) | | | mff_oce.bn_layer.0.bn3 | 1.024K | 65.536K | | mff_oce.bn_layer.0.bn3.weight | (512,) | | | mff_oce.bn_layer.0.bn3.bias | (512,) | | | mff_oce.bn_layer.0.downsample | 0.132M | 8.454M | | mff_oce.bn_layer.0.downsample.0 | 0.131M | 8.389M | | mff_oce.bn_layer.0.downsample.1 | 1.024K | 65.536K | | mff_oce.bn_layer.1 | 0.28M | 17.924M | | mff_oce.bn_layer.1.conv1 | 65.536K | 4.194M | | mff_oce.bn_layer.1.conv1.weight | (128, 512, 1, 1) | | | mff_oce.bn_layer.1.bn1 | 0.256K | 16.384K | | mff_oce.bn_layer.1.bn1.weight | (128,) | | | mff_oce.bn_layer.1.bn1.bias | (128,) | | | mff_oce.bn_layer.1.conv2 | 0.147M | 9.437M | | mff_oce.bn_layer.1.conv2.weight | (128, 128, 3, 3) | | | mff_oce.bn_layer.1.bn2 | 0.256K | 16.384K | | mff_oce.bn_layer.1.bn2.weight | (128,) | | | mff_oce.bn_layer.1.bn2.bias | (128,) | | | mff_oce.bn_layer.1.conv3 | 65.536K | 4.194M | | mff_oce.bn_layer.1.conv3.weight | (512, 128, 1, 1) | | | mff_oce.bn_layer.1.bn3 | 1.024K | 65.536K | | mff_oce.bn_layer.1.bn3.weight | (512,) | | | mff_oce.bn_layer.1.bn3.bias | (512,) | | | mff_oce.bn_layer.2 | 0.28M | 17.924M | | mff_oce.bn_layer.2.conv1 | 65.536K | 4.194M | | mff_oce.bn_layer.2.conv1.weight | (128, 512, 1, 1) | | | mff_oce.bn_layer.2.bn1 | 0.256K | 16.384K | | mff_oce.bn_layer.2.bn1.weight | (128,) | | | mff_oce.bn_layer.2.bn1.bias | (128,) | | | mff_oce.bn_layer.2.conv2 | 0.147M | 9.437M | | mff_oce.bn_layer.2.conv2.weight | (128, 128, 3, 3) | | | mff_oce.bn_layer.2.bn2 | 0.256K | 16.384K | | mff_oce.bn_layer.2.bn2.weight | (128,) | | | mff_oce.bn_layer.2.bn2.bias | (128,) | | | mff_oce.bn_layer.2.conv3 | 65.536K | 4.194M | | mff_oce.bn_layer.2.conv3.weight | (512, 128, 1, 1) | | | mff_oce.bn_layer.2.bn3 | 1.024K | 65.536K | | mff_oce.bn_layer.2.bn3.weight | (512,) | | | mff_oce.bn_layer.2.bn3.bias | (512,) | | | mff_oce.conv1 | 73.728K | 75.497M | | mff_oce.conv1.weight | (128, 64, 3, 3) | | | mff_oce.bn1 | 0.256K | 0.262M | | mff_oce.bn1.weight | (128,) | | | mff_oce.bn1.bias | (128,) | | | mff_oce.conv2 | 0.295M | 75.497M | | mff_oce.conv2.weight | (256, 128, 3, 3) | | | mff_oce.bn2 | 0.512K | 0.131M | | mff_oce.bn2.weight | (256,) | | | mff_oce.bn2.bias | (256,) | | | mff_oce.conv21 | 16.512K | 16.777M | | mff_oce.conv21.weight | (128, 128, 1, 1) | | | mff_oce.conv21.bias | (128,) | | | mff_oce.bn21 | 0.256K | 0.262M | | mff_oce.bn21.weight | (128,) | | | mff_oce.bn21.bias | (128,) | | | mff_oce.conv31 | 65.792K | 16.777M | | mff_oce.conv31.weight | (256, 256, 1, 1) | | | mff_oce.conv31.bias | (256,) | | | mff_oce.bn31 | 0.512K | 0.131M | | mff_oce.bn31.weight | (256,) | | | mff_oce.bn31.bias | (256,) | | | mff_oce.convf | 65.792K | 16.777M | | mff_oce.convf.weight | (256, 256, 1, 1) | | | mff_oce.convf.bias | (256,) | | | mff_oce.bnf | 0.512K | 0.131M | | mff_oce.bnf.weight | (256,) | | | mff_oce.bnf.bias | (256,) | | | net_s.layers_up | 13.471M | 3.313G | | net_s.layers_up.0.blocks.0 | 8.04M | 0.501G | | net_s.layers_up.0.blocks.0.smm_blocks | 6.16M | 0.381G | | net_s.layers_up.0.blocks.0.smm_blocks.0 | 2.053M | 0.127G | | net_s.layers_up.0.blocks.0.smm_blocks.0.ln_1 | 1.024K | 0.164M | | net_s.layers_up.0.blocks.0.smm_blocks.0.self_attention | 2.052M | 0.127G | | net_s.layers_up.0.blocks.0.smm_blocks.1 | 2.053M | 0.127G | | net_s.layers_up.0.blocks.0.smm_blocks.1.ln_1 | 1.024K | 0.164M | | net_s.layers_up.0.blocks.0.smm_blocks.1.self_attention | 2.052M | 0.127G | | net_s.layers_up.0.blocks.0.smm_blocks.2 | 2.053M | 0.127G | | net_s.layers_up.0.blocks.0.smm_blocks.2.ln_1 | 1.024K | 0.164M | | net_s.layers_up.0.blocks.0.smm_blocks.2.self_attention | 2.052M | 0.127G | | net_s.layers_up.0.blocks.0.conv1b3 | 0.263M | 16.908M | | net_s.layers_up.0.blocks.0.conv1b3.0 | 0.263M | 16.777M | | net_s.layers_up.0.blocks.0.conv1b3.0.weight | (512, 512, 1, 1) | | | net_s.layers_up.0.blocks.0.conv1b3.0.bias | (512,) | | | net_s.layers_up.0.blocks.0.conv1b3.1 | | 0.131M | | net_s.layers_up.0.blocks.0.conv1a3 | 0.263M | 16.908M | | net_s.layers_up.0.blocks.0.conv1a3.0 | 0.263M | 16.777M | | net_s.layers_up.0.blocks.0.conv1a3.0.weight | (512, 512, 1, 1) | | | net_s.layers_up.0.blocks.0.conv1a3.0.bias | (512,) | | | net_s.layers_up.0.blocks.0.conv1a3.1 | | 0.131M | | net_s.layers_up.0.blocks.0.conv1b5 | 0.263M | 16.908M | | net_s.layers_up.0.blocks.0.conv1b5.0 | 0.263M | 16.777M | | net_s.layers_up.0.blocks.0.conv1b5.0.weight | (512, 512, 1, 1) | | | net_s.layers_up.0.blocks.0.conv1b5.0.bias | (512,) | | | net_s.layers_up.0.blocks.0.conv1b5.1 | | 0.131M | | net_s.layers_up.0.blocks.0.conv1a5 | 0.263M | 16.908M | | net_s.layers_up.0.blocks.0.conv1a5.0 | 0.263M | 16.777M | | net_s.layers_up.0.blocks.0.conv1a5.0.weight | (512, 512, 1, 1) | | | net_s.layers_up.0.blocks.0.conv1a5.0.bias | (512,) | | | net_s.layers_up.0.blocks.0.conv1a5.1 | | 0.131M | | net_s.layers_up.0.blocks.0.conv33.0 | 4.608K | | | net_s.layers_up.0.blocks.0.conv33.0.weight | (512, 1, 3, 3) | | | net_s.layers_up.0.blocks.0.conv55 | 12.8K | 0.95M | | net_s.layers_up.0.blocks.0.conv55.0 | 12.8K | 0.819M | | net_s.layers_up.0.blocks.0.conv55.0.weight | (512, 1, 5, 5) | | | net_s.layers_up.0.blocks.0.conv55.1 | | 0.131M | | net_s.layers_up.0.blocks.0.conv77 | 25.088K | 1.737M | | net_s.layers_up.0.blocks.0.conv77.0 | 25.088K | 1.606M | | net_s.layers_up.0.blocks.0.conv77.0.weight | (512, 1, 7, 7) | | | net_s.layers_up.0.blocks.0.conv77.1 | | 0.131M | | net_s.layers_up.0.blocks.0.finalconv11 | 0.787M | 50.332M | | net_s.layers_up.0.blocks.0.finalconv11.weight | (512, 1536, 1, 1) | | | net_s.layers_up.0.blocks.0.finalconv11.bias | (512,) | | | net_s.layers_up.1 | 3.761M | 0.827G | | net_s.layers_up.1.blocks | 3.236M | 0.793G | | net_s.layers_up.1.blocks.0 | 1.618M | 0.397G | | net_s.layers_up.1.blocks.0.smm_blocks | 1.137M | 0.273G | | net_s.layers_up.1.blocks.0.conv1b3 | 65.792K | 17.039M | | net_s.layers_up.1.blocks.0.conv1a3 | 65.792K | 17.039M | | net_s.layers_up.1.blocks.0.conv1b5 | 65.792K | 17.039M | | net_s.layers_up.1.blocks.0.conv1a5 | 65.792K | 17.039M | | net_s.layers_up.1.blocks.0.conv33.0 | 2.304K | | | net_s.layers_up.1.blocks.0.conv55 | 6.4K | 1.901M | | net_s.layers_up.1.blocks.0.conv77 | 12.544K | 3.473M | | net_s.layers_up.1.blocks.0.finalconv11 | 0.197M | 50.332M | | net_s.layers_up.1.blocks.1 | 1.618M | 0.397G | | net_s.layers_up.1.blocks.1.smm_blocks | 1.137M | 0.273G | | net_s.layers_up.1.blocks.1.conv1b3 | 65.792K | 17.039M | | net_s.layers_up.1.blocks.1.conv1a3 | 65.792K | 17.039M | | net_s.layers_up.1.blocks.1.conv1b5 | 65.792K | 17.039M | | net_s.layers_up.1.blocks.1.conv1a5 | 65.792K | 17.039M | | net_s.layers_up.1.blocks.1.conv33.0 | 2.304K | | | net_s.layers_up.1.blocks.1.conv55 | 6.4K | 1.901M | | net_s.layers_up.1.blocks.1.conv77 | 12.544K | 3.473M | | net_s.layers_up.1.blocks.1.finalconv11 | 0.197M | 50.332M | | net_s.layers_up.1.upsample | 0.525M | 33.882M | | net_s.layers_up.1.upsample.expand | 0.524M | 33.554M | | net_s.layers_up.1.upsample.expand.weight | (1024, 512) | | | net_s.layers_up.1.upsample.norm | 0.512K | 0.328M | | net_s.layers_up.1.upsample.norm.weight | (256,) | | | net_s.layers_up.1.upsample.norm.bias | (256,) | | | net_s.layers_up.2 | 1.411M | 1.227G | | net_s.layers_up.2.blocks | 1.279M | 1.193G | | net_s.layers_up.2.blocks.0 | 0.64M | 0.596G | | net_s.layers_up.2.blocks.0.smm_blocks | 0.514M | 0.466G | | net_s.layers_up.2.blocks.0.conv1b3 | 16.512K | 17.302M | | net_s.layers_up.2.blocks.0.conv1a3 | 16.512K | 17.302M | | net_s.layers_up.2.blocks.0.conv1b5 | 16.512K | 17.302M | | net_s.layers_up.2.blocks.0.conv1a5 | 16.512K | 17.302M | | net_s.layers_up.2.blocks.0.conv33.0 | 1.152K | | | net_s.layers_up.2.blocks.0.conv55 | 3.2K | 3.801M | | net_s.layers_up.2.blocks.0.conv77 | 6.272K | 6.947M | | net_s.layers_up.2.blocks.0.finalconv11 | 49.28K | 50.332M | | net_s.layers_up.2.blocks.1 | 0.64M | 0.596G | | net_s.layers_up.2.blocks.1.smm_blocks | 0.514M | 0.466G | | net_s.layers_up.2.blocks.1.conv1b3 | 16.512K | 17.302M | | net_s.layers_up.2.blocks.1.conv1a3 | 16.512K | 17.302M | | net_s.layers_up.2.blocks.1.conv1b5 | 16.512K | 17.302M | | net_s.layers_up.2.blocks.1.conv1a5 | 16.512K | 17.302M | | net_s.layers_up.2.blocks.1.conv33.0 | 1.152K | | | net_s.layers_up.2.blocks.1.conv55 | 3.2K | 3.801M | | net_s.layers_up.2.blocks.1.conv77 | 6.272K | 6.947M | | net_s.layers_up.2.blocks.1.finalconv11 | 49.28K | 50.332M | | net_s.layers_up.2.upsample | 0.131M | 34.21M | | net_s.layers_up.2.upsample.expand | 0.131M | 33.554M | | net_s.layers_up.2.upsample.expand.weight | (512, 256) | | | net_s.layers_up.2.upsample.norm | 0.256K | 0.655M | | net_s.layers_up.2.upsample.norm.weight | (128,) | | | net_s.layers_up.2.upsample.norm.bias | (128,) | | | net_s.layers_up.3 | 0.26M | 0.758G | | net_s.layers_up.3.blocks.0 | 0.227M | 0.723G | | net_s.layers_up.3.blocks.0.smm_blocks | 0.192M | 0.58G | | net_s.layers_up.3.blocks.0.smm_blocks.0 | 64.128K | 0.193G | | net_s.layers_up.3.blocks.0.smm_blocks.1 | 64.128K | 0.193G | | net_s.layers_up.3.blocks.0.smm_blocks.2 | 64.128K | 0.193G | | net_s.layers_up.3.blocks.0.conv1b3 | 4.16K | 17.826M | | net_s.layers_up.3.blocks.0.conv1b3.0 | 4.16K | 16.777M | | net_s.layers_up.3.blocks.0.conv1b3.1 | | 1.049M | | net_s.layers_up.3.blocks.0.conv1a3 | 4.16K | 17.826M | | net_s.layers_up.3.blocks.0.conv1a3.0 | 4.16K | 16.777M | | net_s.layers_up.3.blocks.0.conv1a3.1 | | 1.049M | | net_s.layers_up.3.blocks.0.conv1b5 | 4.16K | 17.826M | | net_s.layers_up.3.blocks.0.conv1b5.0 | 4.16K | 16.777M | | net_s.layers_up.3.blocks.0.conv1b5.1 | | 1.049M | | net_s.layers_up.3.blocks.0.conv1a5 | 4.16K | 17.826M | | net_s.layers_up.3.blocks.0.conv1a5.0 | 4.16K | 16.777M | | net_s.layers_up.3.blocks.0.conv1a5.1 | | 1.049M | | net_s.layers_up.3.blocks.0.conv33.0 | 0.576K | | | net_s.layers_up.3.blocks.0.conv33.0.weight | (64, 1, 3, 3) | | | net_s.layers_up.3.blocks.0.conv55 | 1.6K | 7.602M | | net_s.layers_up.3.blocks.0.conv55.0 | 1.6K | 6.554M | | net_s.layers_up.3.blocks.0.conv55.1 | | 1.049M | | net_s.layers_up.3.blocks.0.conv77 | 3.136K | 13.894M | | net_s.layers_up.3.blocks.0.conv77.0 | 3.136K | 12.845M | | net_s.layers_up.3.blocks.0.conv77.1 | | 1.049M | | net_s.layers_up.3.blocks.0.finalconv11 | 12.352K | 50.332M | | net_s.layers_up.3.blocks.0.finalconv11.weight | (64, 192, 1, 1) | | | net_s.layers_up.3.blocks.0.finalconv11.bias | (64,) | | | net_s.layers_up.3.upsample | 32.896K | 34.865M | | net_s.layers_up.3.upsample.expand | 32.768K | 33.554M | | net_s.layers_up.3.upsample.expand.weight | (256, 128) | | | net_s.layers_up.3.upsample.norm | 0.128K | 1.311M | | net_s.layers_up.3.upsample.norm.weight | (64,) | | | net_s.layers_up.3.upsample.norm.bias | (64,) | | --------------------------------------------------------------------------------- 2024-04-02 17:57:39,709 - ==> Creating optimizer 2024-04-02 17:57:39,714 - ==> Loading dataset: DefaultAD 2024-04-02 17:57:40,054 - ==> ********** cfg ********** epoch_full : 1000 fvcore_b : 1 fvcore_c : 3 metrics : ['mAUROC_sp_max', 'mAP_sp_max', 'mF1_max_sp_max', 'mAUROC_px', 'mAP_px', 'mF1_max_px', 'mAUPRO_px', 'mF1_px_0.2_0.8_0.1', 'mAcc_px_0.2_0.8_0.1', 'mIoU_px_0.2_0.8_0.1', 'mIoU_max_px'] evaluator.kwargs : {'metrics': ['mAUROC_sp_max', 'mAP_sp_max', 'mF1_max_sp_max', 'mAUROC_px', 'mAP_px', 'mF1_max_px', 'mAUPRO_px', 'mF1_px_0.2_0.8_0.1', 'mAcc_px_0.2_0.8_0.1', 'mIoU_px_0.2_0.8_0.1', 'mIoU_max_px'], 'pooling_ks': None, 'max_step_aupro': 100} optim.lr : 0.005 optim.kwargs : {'name': 'adamw', 'betas': (0.9, 0.999), 'eps': 1e-08, 'weight_decay': 0.0001, 'amsgrad': False} trainer.name : MAMBAADTrainer trainer.checkpoint : runs trainer.logdir_sub : trainer.resume_dir : trainer.cuda_deterministic : False trainer.epoch_full : 1000 trainer.scheduler_kwargs : {'name': 'step', 'lr_noise': None, 'noise_pct': 0.67, 'noise_std': 1.0, 'noise_seed': 42, 'lr_min': 5e-05, 'warmup_lr': 5e-06, 'warmup_iters': -1, 'cooldown_iters': 0, 'warmup_epochs': 0, 'cooldown_epochs': 0, 'use_iters': True, 'patience_iters': 0, 'patience_epochs': 0, 'decay_iters': 0, 'decay_epochs': 800, 'cycle_decay': 0.1, 'decay_rate': 0.1} trainer.mixup_kwargs : {'mixup_alpha': 0.8, 'cutmix_alpha': 1.0, 'cutmix_minmax': None, 'prob': 0.0, 'switch_prob': 0.5, 'mode': 'batch', 'correct_lam': True, 'label_smoothing': 0.1} trainer.test_start_epoch : 1000 trainer.test_per_epoch : 50 trainer.find_unused_parameters : False trainer.sync_BN : apex trainer.dist_BN : trainer.scaler : none trainer.data.batch_size : 16 trainer.data.batch_size_per_gpu : 16 trainer.data.batch_size_test : 16 trainer.data.batch_size_per_gpu_test : 16 trainer.data.num_workers_per_gpu : 4 trainer.data.drop_last : True trainer.data.pin_memory : True trainer.data.persistent_workers : False trainer.data.num_workers : 4 trainer.iter : 0 trainer.epoch : 0 trainer.iter_full : 4942000 trainer.metric_recorder : {'mAUROC_sp_max_coco': [], 'mAP_sp_max_coco': [], 'mF1_max_sp_max_coco': [], 'mAUROC_px_coco': [], 'mAP_px_coco': [], 'mF1_max_px_coco': [], 'mAUPRO_px_coco': [], 'mF1_px_0.2_0.8_0.1_coco': [], 'mAcc_px_0.2_0.8_0.1_coco': [], 'mIoU_px_0.2_0.8_0.1_coco': [], 'mIoU_max_px_coco': []} loss.loss_terms : [{'type': 'L2Loss', 'name': 'pixel', 'lam': 5.0}] loss.clip_grad : 5.0 loss.create_graph : False loss.retain_graph : False adv : False logging.log_terms_train : [{'name': 'batch_t', 'fmt': ':>5.3f', 'add_name': 'avg'}, {'name': 'data_t', 'fmt': ':>5.3f'}, {'name': 'optim_t', 'fmt': ':>5.3f'}, {'name': 'lr', 'fmt': ':>7.6f'}, {'name': 'cos', 'suffixes': [''], 'fmt': ':>5.3f', 'add_name': 'avg'}] logging.log_terms_test : [{'name': 'batch_t', 'fmt': ':>5.3f', 'add_name': 'avg'}, {'name': 'cos', 'suffixes': [''], 'fmt': ':>5.3f', 'add_name': 'avg'}] logging.train_reset_log_per : 100 logging.train_log_per : 100 logging.test_log_per : 50 data.sampler : naive data.loader_type : pil data.loader_type_target : pil_L data.type : DefaultAD data.root : data/coco data.meta : meta_20_2.json data.cls_names : ['coco'] data.train_transforms : [{'type': 'Resize', 'size': (256, 256), 'interpolation': }, {'type': 'CenterCrop', 'size': (256, 256)}, {'type': 'ToTensor'}, {'type': 'Normalize', 'mean': (0.485, 0.456, 0.406), 'std': (0.229, 0.224, 0.225), 'inplace': True}] data.test_transforms : [{'type': 'Resize', 'size': (256, 256), 'interpolation': }, {'type': 'CenterCrop', 'size': (256, 256)}, {'type': 'ToTensor'}, {'type': 'Normalize', 'mean': (0.485, 0.456, 0.406), 'std': (0.229, 0.224, 0.225), 'inplace': True}] data.target_transforms : [{'type': 'Resize', 'size': (256, 256), 'interpolation': }, {'type': 'CenterCrop', 'size': (256, 256)}, {'type': 'ToTensor'}] data.train_size : 4942 data.test_size : 310 data.train_length : 79083 data.test_length : 4952 model_t.name : timm_resnet34 model_t.kwargs : {'pretrained': False, 'checkpoint_path': 'model/pretrain/resnet34-43635321.pth', 'strict': False, 'features_only': True, 'out_indices': [1, 2, 3]} model_s : {'depths_decoder': [3, 4, 6, 3], 'scan_type': 'hilbert', 'num_direction': 4} model.name : mambaadhcs57c1 model.kwargs : {'pretrained': False, 'checkpoint_path': '', 'strict': True, 'model_t': Namespace(name='timm_resnet34', kwargs={'pretrained': False, 'checkpoint_path': 'model/pretrain/resnet34-43635321.pth', 'strict': False, 'features_only': True, 'out_indices': [1, 2, 3]}), 'model_s': {'depths_decoder': [3, 4, 6, 3], 'scan_type': 'hilbert', 'num_direction': 4}} seed : 42 size : 256 warmup_epochs : 0 test_start_epoch : 1000 test_per_epoch : 50 batch_train : 16 batch_test_per : 16 lr : 0.005 weight_decay : 0.0001 cfg_path : configs.mambaad.mambaad_coco2_nhcs57c1d4 mode : train sleep : -1 memory : -1 dist_url : env:// logger_rank : 0 opts : [] command : python3 -m torch.distributed.launch --nproc_per_node=$nproc_per_node --nnodes=$nnodes --node_rank=$node_rank --master_addr=$master_addr --master_port=$master_port --use_env run.py -c configs.mambaad.mambaad_coco2_nhcs57c1d4 -m train --sleep -1 --memory -1 --dist_url env:// --logger_rank 0 task_start_time : 10045732.720708951 dist : False world_size : 1 rank : 0 local_rank : 0 ngpus_per_node : 1 nnodes : 1 master : True logdir : runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720 logger.filters : [] logger.name : root logger.level : 20 logger.parent : None logger.propagate : True logger.disabled : False logdir_train : runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720/show_train logdir_test : runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720/show_test 2024-04-02 17:57:40,055 - ==> Starting training with 1 nodes x 1 GPUs 2024-04-02 17:59:05,584 - Train: 0.00% [100/4942000] [0.0/1000.0] [batch_t 0.762 (0.852)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-02 18:00:22,558 - Train: 0.00% [200/4942000] [0.0/1000.0] [batch_t 0.771 (0.770)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-02 18:01:39,505 - Train: 0.01% [300/4942000] [0.1/1000.0] [batch_t 0.762 (0.769)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-02 18:02:56,506 - Train: 0.01% [400/4942000] [0.1/1000.0] [batch_t 0.766 (0.770)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-02 18:04:13,523 - Train: 0.01% [500/4942000] [0.1/1000.0] [batch_t 0.772 (0.770)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-02 18:05:30,439 - Train: 0.01% [600/4942000] [0.1/1000.0] [batch_t 0.771 (0.769)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-02 18:06:47,486 - Train: 0.01% [700/4942000] [0.1/1000.0] [batch_t 0.761 (0.770)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-02 18:08:04,445 - Train: 0.02% [800/4942000] [0.2/1000.0] [batch_t 0.771 (0.769)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-02 18:09:25,425 - Train: 0.02% [900/4942000] [0.2/1000.0] [batch_t 0.772 (0.790)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-02 18:10:42,305 - Train: 0.02% [1000/4942000] [0.2/1000.0] [batch_t 0.768 (0.769)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-02 18:11:59,251 - Train: 0.02% [1100/4942000] [0.2/1000.0] [batch_t 0.766 (0.769)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-02 18:13:18,564 - Train: 0.02% [1200/4942000] [0.2/1000.0] [batch_t 0.770 (0.793)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-02 18:14:35,388 - Train: 0.03% [1300/4942000] [0.3/1000.0] [batch_t 0.766 (0.768)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-02 18:15:54,832 - Train: 0.03% [1400/4942000] [0.3/1000.0] [batch_t 0.765 (0.794)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-02 18:17:11,810 - Train: 0.03% [1500/4942000] [0.3/1000.0] [batch_t 0.777 (0.770)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-02 18:18:28,756 - Train: 0.03% [1600/4942000] [0.3/1000.0] [batch_t 0.767 (0.769)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-02 18:19:46,438 - Train: 0.03% [1700/4942000] [0.3/1000.0] [batch_t 0.774 (0.777)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-02 18:21:03,421 - Train: 0.04% [1800/4942000] [0.4/1000.0] [batch_t 0.771 (0.770)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-02 18:22:20,298 - Train: 0.04% [1900/4942000] [0.4/1000.0] [batch_t 0.776 (0.769)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-02 18:23:37,290 - Train: 0.04% [2000/4942000] [0.4/1000.0] [batch_t 0.767 (0.770)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-02 18:24:54,192 - Train: 0.04% [2100/4942000] [0.4/1000.0] [batch_t 0.763 (0.769)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-02 18:26:10,993 - Train: 0.04% [2200/4942000] [0.4/1000.0] [batch_t 0.772 (0.768)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-02 18:27:27,946 - Train: 0.05% [2300/4942000] [0.5/1000.0] [batch_t 0.768 (0.769)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-02 18:28:44,884 - Train: 0.05% [2400/4942000] [0.5/1000.0] [batch_t 0.761 (0.769)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-02 18:30:01,846 - Train: 0.05% [2500/4942000] [0.5/1000.0] [batch_t 0.767 (0.770)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-02 18:31:18,816 - Train: 0.05% [2600/4942000] [0.5/1000.0] [batch_t 0.766 (0.770)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-02 18:32:35,819 - Train: 0.05% [2700/4942000] [0.5/1000.0] [batch_t 0.771 (0.770)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-02 18:33:52,821 - Train: 0.06% [2800/4942000] [0.6/1000.0] [batch_t 0.772 (0.770)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-02 18:35:09,899 - Train: 0.06% [2900/4942000] [0.6/1000.0] [batch_t 0.767 (0.771)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-02 18:36:26,715 - Train: 0.06% [3000/4942000] [0.6/1000.0] [batch_t 0.766 (0.768)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-02 18:37:43,710 - Train: 0.06% [3100/4942000] [0.6/1000.0] [batch_t 0.768 (0.770)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-02 18:39:10,534 - Train: 0.06% [3200/4942000] [0.6/1000.0] [batch_t 0.766 (0.868)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-02 18:40:27,346 - Train: 0.07% [3300/4942000] [0.7/1000.0] [batch_t 0.768 (0.768)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-02 18:41:44,285 - Train: 0.07% [3400/4942000] [0.7/1000.0] [batch_t 0.770 (0.769)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-02 18:43:01,236 - Train: 0.07% [3500/4942000] [0.7/1000.0] [batch_t 0.763 (0.769)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-02 18:44:26,095 - Train: 0.07% [3600/4942000] [0.7/1000.0] [batch_t 0.762 (0.849)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-02 18:45:43,103 - Train: 0.07% [3700/4942000] [0.7/1000.0] [batch_t 0.773 (0.770)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-02 18:47:00,157 - Train: 0.08% [3800/4942000] [0.8/1000.0] [batch_t 0.763 (0.770)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-02 18:48:17,133 - Train: 0.08% [3900/4942000] [0.8/1000.0] [batch_t 0.763 (0.770)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-02 18:49:34,049 - Train: 0.08% [4000/4942000] [0.8/1000.0] [batch_t 0.777 (0.769)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-02 18:50:50,986 - Train: 0.08% [4100/4942000] [0.8/1000.0] [batch_t 0.771 (0.769)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-02 18:52:08,036 - Train: 0.08% [4200/4942000] [0.8/1000.0] [batch_t 0.770 (0.770)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-02 18:53:24,945 - Train: 0.09% [4300/4942000] [0.9/1000.0] [batch_t 0.768 (0.769)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-02 18:54:41,894 - Train: 0.09% [4400/4942000] [0.9/1000.0] [batch_t 0.767 (0.769)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-02 18:55:58,843 - Train: 0.09% [4500/4942000] [0.9/1000.0] [batch_t 0.766 (0.769)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-02 19:00:17,951 - Train: 0.09% [4600/4942000] [0.9/1000.0] [batch_t 1.102 (2.591)] [data_t 0.335] [optim_t 0.767] [lr 0.005000] 2024-04-02 19:04:58,038 - Train: 0.10% [4700/4942000] [1.0/1000.0] [batch_t 3.598 (2.801)] [data_t 2.826] [optim_t 0.772] [lr 0.005000] 2024-04-02 19:09:30,114 - Train: 0.10% [4800/4942000] [1.0/1000.0] [batch_t 3.152 (2.721)] [data_t 2.374] [optim_t 0.777] [lr 0.005000] 2024-04-02 19:14:40,043 - Train: 0.10% [4900/4942000] [1.0/1000.0] [batch_t 0.772 (3.099)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-02 19:15:30,541 - ==> Total time: 1:18:09 Eta: 54 days, 5:23:57 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-02 19:16:26,469 - Train: 0.10% [5000/4942000] [1.0/1000.0] [batch_t 0.765 (0.926)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-02 19:17:55,835 - Train: 0.10% [5100/4942000] [1.0/1000.0] [batch_t 0.774 (0.894)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-02 19:19:12,641 - Train: 0.11% [5200/4942000] [1.1/1000.0] [batch_t 0.764 (0.768)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-02 19:20:29,688 - Train: 0.11% [5300/4942000] [1.1/1000.0] [batch_t 0.777 (0.770)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-02 19:21:52,735 - Train: 0.11% [5400/4942000] [1.1/1000.0] [batch_t 2.494 (0.830)] [data_t 1.731] [optim_t 0.763] [lr 0.005000] 2024-04-02 19:23:57,025 - Train: 0.11% [5500/4942000] [1.1/1000.0] [batch_t 0.770 (1.243)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-02 19:25:24,377 - Train: 0.11% [5600/4942000] [1.1/1000.0] [batch_t 0.766 (0.873)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-02 19:26:45,076 - Train: 0.12% [5700/4942000] [1.2/1000.0] [batch_t 0.759 (0.807)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-02 19:28:02,060 - Train: 0.12% [5800/4942000] [1.2/1000.0] [batch_t 0.776 (0.770)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-02 19:29:26,937 - Train: 0.12% [5900/4942000] [1.2/1000.0] [batch_t 0.766 (0.849)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-02 19:31:07,996 - Train: 0.12% [6000/4942000] [1.2/1000.0] [batch_t 0.776 (1.010)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-02 19:32:30,045 - Train: 0.12% [6100/4942000] [1.2/1000.0] [batch_t 0.771 (0.820)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-02 19:33:47,081 - Train: 0.13% [6200/4942000] [1.3/1000.0] [batch_t 0.768 (0.770)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-02 19:35:04,041 - Train: 0.13% [6300/4942000] [1.3/1000.0] [batch_t 0.766 (0.769)] [data_t 0.002] [optim_t 0.763] [lr 0.005000] 2024-04-02 19:36:20,930 - Train: 0.13% [6400/4942000] [1.3/1000.0] [batch_t 0.772 (0.769)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-02 19:37:42,289 - Train: 0.13% [6500/4942000] [1.3/1000.0] [batch_t 0.775 (0.813)] [data_t 0.003] [optim_t 0.772] [lr 0.005000] 2024-04-02 19:38:59,199 - Train: 0.13% [6600/4942000] [1.3/1000.0] [batch_t 0.773 (0.769)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-02 19:40:16,040 - Train: 0.14% [6700/4942000] [1.4/1000.0] [batch_t 0.770 (0.768)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-02 19:41:32,966 - Train: 0.14% [6800/4942000] [1.4/1000.0] [batch_t 0.769 (0.769)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-02 19:42:49,980 - Train: 0.14% [6900/4942000] [1.4/1000.0] [batch_t 0.768 (0.770)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-02 19:44:06,931 - Train: 0.14% [7000/4942000] [1.4/1000.0] [batch_t 0.771 (0.769)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-02 19:45:23,891 - Train: 0.14% [7100/4942000] [1.4/1000.0] [batch_t 0.771 (0.770)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-02 19:46:40,867 - Train: 0.15% [7200/4942000] [1.5/1000.0] [batch_t 0.768 (0.770)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-02 19:47:57,906 - Train: 0.15% [7300/4942000] [1.5/1000.0] [batch_t 0.767 (0.770)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-02 19:49:14,909 - Train: 0.15% [7400/4942000] [1.5/1000.0] [batch_t 0.766 (0.770)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-02 19:50:31,926 - Train: 0.15% [7500/4942000] [1.5/1000.0] [batch_t 0.768 (0.770)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-02 19:51:48,860 - Train: 0.15% [7600/4942000] [1.5/1000.0] [batch_t 0.767 (0.769)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-02 19:53:05,884 - Train: 0.16% [7700/4942000] [1.6/1000.0] [batch_t 0.777 (0.770)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-02 19:54:24,969 - Train: 0.16% [7800/4942000] [1.6/1000.0] [batch_t 0.771 (0.791)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-02 19:55:42,014 - Train: 0.16% [7900/4942000] [1.6/1000.0] [batch_t 0.769 (0.770)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-02 19:56:59,082 - Train: 0.16% [8000/4942000] [1.6/1000.0] [batch_t 0.783 (0.771)] [data_t 0.003] [optim_t 0.780] [lr 0.005000] 2024-04-02 19:58:16,143 - Train: 0.16% [8100/4942000] [1.6/1000.0] [batch_t 0.773 (0.771)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-02 19:59:33,093 - Train: 0.17% [8200/4942000] [1.7/1000.0] [batch_t 0.767 (0.769)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-02 20:00:50,112 - Train: 0.17% [8300/4942000] [1.7/1000.0] [batch_t 0.776 (0.770)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-02 20:02:30,403 - Train: 0.17% [8400/4942000] [1.7/1000.0] [batch_t 0.770 (1.003)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-02 20:04:18,796 - Train: 0.17% [8500/4942000] [1.7/1000.0] [batch_t 0.824 (1.084)] [data_t 0.002] [optim_t 0.822] [lr 0.005000] 2024-04-02 20:05:48,420 - Train: 0.17% [8600/4942000] [1.7/1000.0] [batch_t 0.771 (0.896)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-02 20:07:12,968 - Train: 0.18% [8700/4942000] [1.8/1000.0] [batch_t 0.777 (0.845)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-02 20:08:29,927 - Train: 0.18% [8800/4942000] [1.8/1000.0] [batch_t 0.763 (0.770)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-02 20:09:56,451 - Train: 0.18% [8900/4942000] [1.8/1000.0] [batch_t 1.106 (0.865)] [data_t 0.337] [optim_t 0.769] [lr 0.005000] 2024-04-02 20:11:37,096 - Train: 0.18% [9000/4942000] [1.8/1000.0] [batch_t 0.771 (1.006)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-02 20:12:58,196 - Train: 0.18% [9100/4942000] [1.8/1000.0] [batch_t 0.758 (0.811)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-02 20:14:18,959 - Train: 0.19% [9200/4942000] [1.9/1000.0] [batch_t 0.752 (0.808)] [data_t 0.002] [optim_t 0.749] [lr 0.005000] 2024-04-02 20:15:44,873 - Train: 0.19% [9300/4942000] [1.9/1000.0] [batch_t 0.767 (0.859)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-02 20:17:14,508 - Train: 0.19% [9400/4942000] [1.9/1000.0] [batch_t 0.763 (0.896)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-02 20:18:31,613 - Train: 0.19% [9500/4942000] [1.9/1000.0] [batch_t 0.764 (0.771)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-02 20:19:48,627 - Train: 0.19% [9600/4942000] [1.9/1000.0] [batch_t 0.763 (0.770)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-02 20:21:05,649 - Train: 0.20% [9700/4942000] [2.0/1000.0] [batch_t 0.776 (0.770)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-02 20:22:22,530 - Train: 0.20% [9800/4942000] [2.0/1000.0] [batch_t 0.775 (0.769)] [data_t 0.002] [optim_t 0.773] [lr 0.005000] 2024-04-02 20:23:31,640 - ==> Total time: 2:26:10 Eta: 50 days, 15:44:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-02 20:23:51,596 - Train: 0.20% [9900/4942000] [2.0/1000.0] [batch_t 0.767 (1.037)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-02 20:25:40,426 - Train: 0.20% [10000/4942000] [2.0/1000.0] [batch_t 0.766 (1.088)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-02 20:27:39,694 - Train: 0.20% [10100/4942000] [2.0/1000.0] [batch_t 1.725 (1.193)] [data_t 0.949] [optim_t 0.777] [lr 0.005000] 2024-04-02 20:29:29,554 - Train: 0.21% [10200/4942000] [2.1/1000.0] [batch_t 0.768 (1.098)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-02 20:30:46,505 - Train: 0.21% [10300/4942000] [2.1/1000.0] [batch_t 0.766 (0.769)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-02 20:32:03,420 - Train: 0.21% [10400/4942000] [2.1/1000.0] [batch_t 0.778 (0.769)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-02 20:33:20,401 - Train: 0.21% [10500/4942000] [2.1/1000.0] [batch_t 0.768 (0.770)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-02 20:34:37,459 - Train: 0.21% [10600/4942000] [2.1/1000.0] [batch_t 0.762 (0.770)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-02 20:35:54,405 - Train: 0.22% [10700/4942000] [2.2/1000.0] [batch_t 0.782 (0.769)] [data_t 0.003] [optim_t 0.780] [lr 0.005000] 2024-04-02 20:37:11,326 - Train: 0.22% [10800/4942000] [2.2/1000.0] [batch_t 0.771 (0.769)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-02 20:38:28,188 - Train: 0.22% [10900/4942000] [2.2/1000.0] [batch_t 0.773 (0.769)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-02 20:39:45,031 - Train: 0.22% [11000/4942000] [2.2/1000.0] [batch_t 0.768 (0.768)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-02 20:41:02,034 - Train: 0.22% [11100/4942000] [2.2/1000.0] [batch_t 0.772 (0.770)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-02 20:42:19,591 - Train: 0.23% [11200/4942000] [2.3/1000.0] [batch_t 0.762 (0.775)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-02 20:43:36,503 - Train: 0.23% [11300/4942000] [2.3/1000.0] [batch_t 0.781 (0.769)] [data_t 0.003] [optim_t 0.778] [lr 0.005000] 2024-04-02 20:44:53,423 - Train: 0.23% [11400/4942000] [2.3/1000.0] [batch_t 0.777 (0.769)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-02 20:46:11,545 - Train: 0.23% [11500/4942000] [2.3/1000.0] [batch_t 0.772 (0.781)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-02 20:47:37,700 - Train: 0.23% [11600/4942000] [2.3/1000.0] [batch_t 8.722 (0.861)] [data_t 7.951] [optim_t 0.771] [lr 0.005000] 2024-04-02 20:48:56,436 - Train: 0.24% [11700/4942000] [2.4/1000.0] [batch_t 0.766 (0.785)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-02 20:50:22,539 - Train: 0.24% [11800/4942000] [2.4/1000.0] [batch_t 0.767 (0.861)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-02 20:51:39,685 - Train: 0.24% [11900/4942000] [2.4/1000.0] [batch_t 0.770 (0.771)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-02 20:53:06,288 - Train: 0.24% [12000/4942000] [2.4/1000.0] [batch_t 0.772 (0.866)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-02 20:54:24,079 - Train: 0.24% [12100/4942000] [2.4/1000.0] [batch_t 0.762 (0.778)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-02 20:55:56,480 - Train: 0.25% [12200/4942000] [2.5/1000.0] [batch_t 0.766 (0.924)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-02 20:57:32,685 - Train: 0.25% [12300/4942000] [2.5/1000.0] [batch_t 0.772 (0.962)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-02 20:59:17,154 - Train: 0.25% [12400/4942000] [2.5/1000.0] [batch_t 0.768 (1.045)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-02 21:00:51,882 - Train: 0.25% [12500/4942000] [2.5/1000.0] [batch_t 0.766 (0.947)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-02 21:02:09,466 - Train: 0.25% [12600/4942000] [2.5/1000.0] [batch_t 0.775 (0.776)] [data_t 0.002] [optim_t 0.773] [lr 0.005000] 2024-04-02 21:03:27,768 - Train: 0.26% [12700/4942000] [2.6/1000.0] [batch_t 0.771 (0.783)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-02 21:04:44,712 - Train: 0.26% [12800/4942000] [2.6/1000.0] [batch_t 0.767 (0.769)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-02 21:06:02,784 - Train: 0.26% [12900/4942000] [2.6/1000.0] [batch_t 0.772 (0.781)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-02 21:07:28,458 - Train: 0.26% [13000/4942000] [2.6/1000.0] [batch_t 0.765 (0.857)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-02 21:08:46,148 - Train: 0.27% [13100/4942000] [2.7/1000.0] [batch_t 0.758 (0.777)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-02 21:10:04,383 - Train: 0.27% [13200/4942000] [2.7/1000.0] [batch_t 0.767 (0.782)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-02 21:11:30,698 - Train: 0.27% [13300/4942000] [2.7/1000.0] [batch_t 1.464 (0.863)] [data_t 0.689] [optim_t 0.775] [lr 0.005000] 2024-04-02 21:13:44,294 - Train: 0.27% [13400/4942000] [2.7/1000.0] [batch_t 0.772 (1.336)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-02 21:16:37,297 - Train: 0.27% [13500/4942000] [2.7/1000.0] [batch_t 0.787 (1.730)] [data_t 0.003] [optim_t 0.784] [lr 0.005000] 2024-04-02 21:18:52,466 - Train: 0.28% [13600/4942000] [2.8/1000.0] [batch_t 0.772 (1.352)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-02 21:20:09,808 - Train: 0.28% [13700/4942000] [2.8/1000.0] [batch_t 0.772 (0.773)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-02 21:21:43,919 - Train: 0.28% [13800/4942000] [2.8/1000.0] [batch_t 0.765 (0.941)] [data_t 0.004] [optim_t 0.761] [lr 0.005000] 2024-04-02 21:23:22,104 - Train: 0.28% [13900/4942000] [2.8/1000.0] [batch_t 0.772 (0.981)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-02 21:24:45,035 - Train: 0.28% [14000/4942000] [2.8/1000.0] [batch_t 0.767 (0.829)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-02 21:26:20,861 - Train: 0.29% [14100/4942000] [2.9/1000.0] [batch_t 0.775 (0.958)] [data_t 0.003] [optim_t 0.772] [lr 0.005000] 2024-04-02 21:27:37,777 - Train: 0.29% [14200/4942000] [2.9/1000.0] [batch_t 0.772 (0.769)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-02 21:28:54,735 - Train: 0.29% [14300/4942000] [2.9/1000.0] [batch_t 0.757 (0.769)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-02 21:30:12,874 - Train: 0.29% [14400/4942000] [2.9/1000.0] [batch_t 0.757 (0.781)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-02 21:31:37,236 - Train: 0.29% [14500/4942000] [2.9/1000.0] [batch_t 0.768 (0.844)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-02 21:32:54,075 - Train: 0.30% [14600/4942000] [3.0/1000.0] [batch_t 0.765 (0.768)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-02 21:34:52,640 - Train: 0.30% [14700/4942000] [3.0/1000.0] [batch_t 0.766 (1.186)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-02 21:39:17,112 - Train: 0.30% [14800/4942000] [3.0/1000.0] [batch_t 13.437 (2.645)] [data_t 12.658] [optim_t 0.780] [lr 0.005000] 2024-04-02 21:40:15,389 - ==> Total time: 3:42:54 Eta: 51 days, 10:40:16 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-02 21:42:18,335 - Train: 0.30% [14900/4942000] [3.0/1000.0] [batch_t 0.771 (1.624)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-02 21:43:41,158 - Train: 0.30% [15000/4942000] [3.0/1000.0] [batch_t 0.764 (0.828)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-02 21:45:24,121 - Train: 0.31% [15100/4942000] [3.1/1000.0] [batch_t 0.850 (1.030)] [data_t 0.088] [optim_t 0.762] [lr 0.005000] 2024-04-02 21:46:47,061 - Train: 0.31% [15200/4942000] [3.1/1000.0] [batch_t 0.771 (0.829)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-02 21:48:04,649 - Train: 0.31% [15300/4942000] [3.1/1000.0] [batch_t 0.758 (0.776)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-02 21:49:24,046 - Train: 0.31% [15400/4942000] [3.1/1000.0] [batch_t 0.772 (0.794)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-02 21:50:42,452 - Train: 0.31% [15500/4942000] [3.1/1000.0] [batch_t 0.771 (0.784)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-02 21:52:25,295 - Train: 0.32% [15600/4942000] [3.2/1000.0] [batch_t 0.782 (1.028)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-02 21:54:52,212 - Train: 0.32% [15700/4942000] [3.2/1000.0] [batch_t 0.773 (1.469)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-02 21:56:12,161 - Train: 0.32% [15800/4942000] [3.2/1000.0] [batch_t 0.751 (0.799)] [data_t 0.002] [optim_t 0.749] [lr 0.005000] 2024-04-02 21:57:29,159 - Train: 0.32% [15900/4942000] [3.2/1000.0] [batch_t 0.767 (0.770)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-02 21:58:46,453 - Train: 0.32% [16000/4942000] [3.2/1000.0] [batch_t 0.778 (0.773)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-02 22:00:03,492 - Train: 0.33% [16100/4942000] [3.3/1000.0] [batch_t 0.776 (0.770)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-02 22:01:20,563 - Train: 0.33% [16200/4942000] [3.3/1000.0] [batch_t 0.759 (0.771)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-02 22:02:43,240 - Train: 0.33% [16300/4942000] [3.3/1000.0] [batch_t 0.767 (0.827)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-02 22:04:07,382 - Train: 0.33% [16400/4942000] [3.3/1000.0] [batch_t 0.770 (0.841)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-02 22:05:36,543 - Train: 0.33% [16500/4942000] [3.3/1000.0] [batch_t 0.778 (0.892)] [data_t 0.004] [optim_t 0.774] [lr 0.005000] 2024-04-02 22:06:54,553 - Train: 0.34% [16600/4942000] [3.4/1000.0] [batch_t 0.768 (0.780)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-02 22:08:11,924 - Train: 0.34% [16700/4942000] [3.4/1000.0] [batch_t 0.756 (0.774)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-02 22:09:28,861 - Train: 0.34% [16800/4942000] [3.4/1000.0] [batch_t 0.761 (0.769)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-02 22:10:49,813 - Train: 0.34% [16900/4942000] [3.4/1000.0] [batch_t 0.768 (0.809)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-02 22:12:06,831 - Train: 0.34% [17000/4942000] [3.4/1000.0] [batch_t 0.770 (0.770)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-02 22:13:24,212 - Train: 0.35% [17100/4942000] [3.5/1000.0] [batch_t 0.779 (0.774)] [data_t 0.002] [optim_t 0.777] [lr 0.005000] 2024-04-02 22:14:41,101 - Train: 0.35% [17200/4942000] [3.5/1000.0] [batch_t 0.759 (0.769)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-02 22:15:58,041 - Train: 0.35% [17300/4942000] [3.5/1000.0] [batch_t 0.766 (0.769)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-02 22:17:14,964 - Train: 0.35% [17400/4942000] [3.5/1000.0] [batch_t 0.768 (0.769)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-02 22:18:32,054 - Train: 0.35% [17500/4942000] [3.5/1000.0] [batch_t 0.769 (0.771)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-02 22:19:49,017 - Train: 0.36% [17600/4942000] [3.6/1000.0] [batch_t 0.772 (0.770)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-02 22:21:05,943 - Train: 0.36% [17700/4942000] [3.6/1000.0] [batch_t 0.767 (0.769)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-02 22:22:22,715 - Train: 0.36% [17800/4942000] [3.6/1000.0] [batch_t 0.771 (0.768)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-02 22:23:39,672 - Train: 0.36% [17900/4942000] [3.6/1000.0] [batch_t 0.766 (0.769)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-02 22:25:12,170 - Train: 0.36% [18000/4942000] [3.6/1000.0] [batch_t 0.771 (0.925)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-02 22:26:29,047 - Train: 0.37% [18100/4942000] [3.7/1000.0] [batch_t 0.777 (0.769)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-02 22:27:45,934 - Train: 0.37% [18200/4942000] [3.7/1000.0] [batch_t 0.773 (0.769)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-02 22:29:11,665 - Train: 0.37% [18300/4942000] [3.7/1000.0] [batch_t 0.779 (0.857)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-02 22:30:28,654 - Train: 0.37% [18400/4942000] [3.7/1000.0] [batch_t 0.763 (0.770)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-02 22:31:45,716 - Train: 0.37% [18500/4942000] [3.7/1000.0] [batch_t 0.767 (0.771)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-02 22:33:24,437 - Train: 0.38% [18600/4942000] [3.8/1000.0] [batch_t 0.749 (0.987)] [data_t 0.003] [optim_t 0.746] [lr 0.005000] 2024-04-02 22:34:41,418 - Train: 0.38% [18700/4942000] [3.8/1000.0] [batch_t 0.766 (0.770)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-02 22:35:59,119 - Train: 0.38% [18800/4942000] [3.8/1000.0] [batch_t 0.782 (0.777)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-02 22:37:16,088 - Train: 0.38% [18900/4942000] [3.8/1000.0] [batch_t 0.762 (0.770)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-02 22:38:33,011 - Train: 0.38% [19000/4942000] [3.8/1000.0] [batch_t 0.761 (0.769)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-02 22:39:49,946 - Train: 0.39% [19100/4942000] [3.9/1000.0] [batch_t 0.776 (0.769)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-02 22:41:07,935 - Train: 0.39% [19200/4942000] [3.9/1000.0] [batch_t 0.768 (0.780)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-02 22:42:24,727 - Train: 0.39% [19300/4942000] [3.9/1000.0] [batch_t 0.768 (0.768)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-02 22:43:41,743 - Train: 0.39% [19400/4942000] [3.9/1000.0] [batch_t 0.771 (0.770)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-02 22:44:58,645 - Train: 0.39% [19500/4942000] [3.9/1000.0] [batch_t 0.771 (0.769)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-02 22:46:15,546 - Train: 0.40% [19600/4942000] [4.0/1000.0] [batch_t 0.745 (0.769)] [data_t 0.003] [optim_t 0.742] [lr 0.005000] 2024-04-02 22:47:32,318 - Train: 0.40% [19700/4942000] [4.0/1000.0] [batch_t 0.753 (0.768)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-02 22:48:25,578 - ==> Total time: 4:51:04 Eta: 50 days, 7:58:46 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-02 22:48:52,787 - Train: 0.40% [19800/4942000] [4.0/1000.0] [batch_t 0.771 (0.799)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-02 22:50:16,489 - Train: 0.40% [19900/4942000] [4.0/1000.0] [batch_t 7.531 (0.837)] [data_t 6.764] [optim_t 0.767] [lr 0.005000] 2024-04-02 22:51:33,476 - Train: 0.40% [20000/4942000] [4.0/1000.0] [batch_t 0.767 (0.770)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-02 22:52:51,811 - Train: 0.41% [20100/4942000] [4.1/1000.0] [batch_t 0.752 (0.780)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-02 22:54:09,258 - Train: 0.41% [20200/4942000] [4.1/1000.0] [batch_t 1.328 (0.774)] [data_t 0.559] [optim_t 0.769] [lr 0.005000] 2024-04-02 22:55:26,879 - Train: 0.41% [20300/4942000] [4.1/1000.0] [batch_t 0.780 (0.776)] [data_t 0.003] [optim_t 0.777] [lr 0.005000] 2024-04-02 22:56:43,903 - Train: 0.41% [20400/4942000] [4.1/1000.0] [batch_t 0.768 (0.770)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-02 22:58:00,906 - Train: 0.41% [20500/4942000] [4.1/1000.0] [batch_t 0.763 (0.770)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-02 22:59:19,510 - Train: 0.42% [20600/4942000] [4.2/1000.0] [batch_t 0.767 (0.786)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-02 23:00:36,433 - Train: 0.42% [20700/4942000] [4.2/1000.0] [batch_t 0.758 (0.769)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-02 23:01:54,590 - Train: 0.42% [20800/4942000] [4.2/1000.0] [batch_t 0.766 (0.781)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-02 23:03:11,446 - Train: 0.42% [20900/4942000] [4.2/1000.0] [batch_t 0.767 (0.768)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-02 23:04:28,458 - Train: 0.42% [21000/4942000] [4.2/1000.0] [batch_t 0.763 (0.770)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-02 23:05:45,400 - Train: 0.43% [21100/4942000] [4.3/1000.0] [batch_t 0.767 (0.769)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-02 23:07:02,359 - Train: 0.43% [21200/4942000] [4.3/1000.0] [batch_t 0.776 (0.769)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-02 23:08:19,234 - Train: 0.43% [21300/4942000] [4.3/1000.0] [batch_t 0.765 (0.769)] [data_t 0.002] [optim_t 0.763] [lr 0.005000] 2024-04-02 23:09:36,477 - Train: 0.43% [21400/4942000] [4.3/1000.0] [batch_t 0.776 (0.772)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-02 23:10:53,433 - Train: 0.44% [21500/4942000] [4.4/1000.0] [batch_t 0.782 (0.769)] [data_t 0.002] [optim_t 0.779] [lr 0.005000] 2024-04-02 23:12:20,266 - Train: 0.44% [21600/4942000] [4.4/1000.0] [batch_t 2.079 (0.868)] [data_t 1.318] [optim_t 0.762] [lr 0.005000] 2024-04-02 23:13:40,864 - Train: 0.44% [21700/4942000] [4.4/1000.0] [batch_t 0.769 (0.806)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-02 23:14:57,657 - Train: 0.44% [21800/4942000] [4.4/1000.0] [batch_t 0.768 (0.768)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-02 23:16:27,920 - Train: 0.44% [21900/4942000] [4.4/1000.0] [batch_t 0.781 (0.903)] [data_t 0.002] [optim_t 0.779] [lr 0.005000] 2024-04-02 23:17:44,753 - Train: 0.45% [22000/4942000] [4.5/1000.0] [batch_t 0.767 (0.768)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-02 23:19:01,646 - Train: 0.45% [22100/4942000] [4.5/1000.0] [batch_t 0.758 (0.769)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-02 23:20:19,894 - Train: 0.45% [22200/4942000] [4.5/1000.0] [batch_t 0.760 (0.782)] [data_t 0.002] [optim_t 0.758] [lr 0.005000] 2024-04-02 23:21:37,307 - Train: 0.45% [22300/4942000] [4.5/1000.0] [batch_t 0.776 (0.774)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-02 23:22:54,138 - Train: 0.45% [22400/4942000] [4.5/1000.0] [batch_t 0.762 (0.768)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-02 23:24:16,202 - Train: 0.46% [22500/4942000] [4.6/1000.0] [batch_t 0.758 (0.821)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-02 23:25:34,573 - Train: 0.46% [22600/4942000] [4.6/1000.0] [batch_t 0.766 (0.784)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-02 23:26:52,708 - Train: 0.46% [22700/4942000] [4.6/1000.0] [batch_t 0.772 (0.781)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-02 23:28:09,875 - Train: 0.46% [22800/4942000] [4.6/1000.0] [batch_t 0.767 (0.772)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-02 23:29:26,756 - Train: 0.46% [22900/4942000] [4.6/1000.0] [batch_t 0.767 (0.769)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-02 23:30:44,015 - Train: 0.47% [23000/4942000] [4.7/1000.0] [batch_t 0.771 (0.772)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-02 23:32:00,801 - Train: 0.47% [23100/4942000] [4.7/1000.0] [batch_t 0.762 (0.768)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-02 23:33:20,683 - Train: 0.47% [23200/4942000] [4.7/1000.0] [batch_t 0.776 (0.799)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-02 23:34:37,944 - Train: 0.47% [23300/4942000] [4.7/1000.0] [batch_t 0.775 (0.772)] [data_t 0.003] [optim_t 0.772] [lr 0.005000] 2024-04-02 23:35:54,794 - Train: 0.47% [23400/4942000] [4.7/1000.0] [batch_t 0.772 (0.768)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-02 23:37:11,671 - Train: 0.48% [23500/4942000] [4.8/1000.0] [batch_t 0.775 (0.769)] [data_t 0.003] [optim_t 0.772] [lr 0.005000] 2024-04-02 23:38:28,502 - Train: 0.48% [23600/4942000] [4.8/1000.0] [batch_t 0.767 (0.768)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-02 23:39:48,961 - Train: 0.48% [23700/4942000] [4.8/1000.0] [batch_t 0.771 (0.805)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-02 23:41:06,276 - Train: 0.48% [23800/4942000] [4.8/1000.0] [batch_t 0.761 (0.773)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-02 23:42:25,402 - Train: 0.48% [23900/4942000] [4.8/1000.0] [batch_t 0.763 (0.790)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-02 23:43:44,825 - Train: 0.49% [24000/4942000] [4.9/1000.0] [batch_t 0.771 (0.794)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-02 23:45:04,041 - Train: 0.49% [24100/4942000] [4.9/1000.0] [batch_t 0.769 (0.792)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-02 23:46:20,998 - Train: 0.49% [24200/4942000] [4.9/1000.0] [batch_t 0.761 (0.769)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-02 23:47:38,818 - Train: 0.49% [24300/4942000] [4.9/1000.0] [batch_t 0.766 (0.778)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-02 23:48:55,775 - Train: 0.49% [24400/4942000] [4.9/1000.0] [batch_t 0.760 (0.769)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-02 23:50:12,749 - Train: 0.50% [24500/4942000] [5.0/1000.0] [batch_t 0.773 (0.770)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-02 23:51:29,552 - Train: 0.50% [24600/4942000] [5.0/1000.0] [batch_t 0.756 (0.768)] [data_t 0.002] [optim_t 0.754] [lr 0.005000] 2024-04-02 23:52:46,483 - Train: 0.50% [24700/4942000] [5.0/1000.0] [batch_t 0.753 (0.769)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-02 23:52:54,200 - ==> Total time: 5:55:33 Eta: 49 days, 3:15:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-02 23:54:05,153 - Train: 0.50% [24800/4942000] [5.0/1000.0] [batch_t 0.762 (0.772)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-02 23:55:22,067 - Train: 0.50% [24900/4942000] [5.0/1000.0] [batch_t 0.757 (0.769)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-02 23:56:38,912 - Train: 0.51% [25000/4942000] [5.1/1000.0] [batch_t 0.770 (0.768)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-02 23:57:55,764 - Train: 0.51% [25100/4942000] [5.1/1000.0] [batch_t 0.769 (0.768)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-02 23:59:12,716 - Train: 0.51% [25200/4942000] [5.1/1000.0] [batch_t 0.772 (0.769)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-03 00:00:29,644 - Train: 0.51% [25300/4942000] [5.1/1000.0] [batch_t 0.776 (0.769)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-03 00:01:46,523 - Train: 0.51% [25400/4942000] [5.1/1000.0] [batch_t 0.777 (0.769)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-03 00:03:03,456 - Train: 0.52% [25500/4942000] [5.2/1000.0] [batch_t 0.766 (0.769)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-03 00:04:20,406 - Train: 0.52% [25600/4942000] [5.2/1000.0] [batch_t 0.772 (0.769)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-03 00:05:37,213 - Train: 0.52% [25700/4942000] [5.2/1000.0] [batch_t 0.776 (0.768)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-03 00:06:54,177 - Train: 0.52% [25800/4942000] [5.2/1000.0] [batch_t 0.768 (0.770)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-03 00:08:11,149 - Train: 0.52% [25900/4942000] [5.2/1000.0] [batch_t 0.775 (0.770)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-03 00:09:28,042 - Train: 0.53% [26000/4942000] [5.3/1000.0] [batch_t 0.764 (0.769)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-03 00:10:44,894 - Train: 0.53% [26100/4942000] [5.3/1000.0] [batch_t 0.772 (0.768)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-03 00:12:01,783 - Train: 0.53% [26200/4942000] [5.3/1000.0] [batch_t 0.773 (0.769)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-03 00:13:18,790 - Train: 0.53% [26300/4942000] [5.3/1000.0] [batch_t 0.767 (0.770)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-03 00:14:35,700 - Train: 0.53% [26400/4942000] [5.3/1000.0] [batch_t 0.770 (0.769)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-03 00:15:52,531 - Train: 0.54% [26500/4942000] [5.4/1000.0] [batch_t 0.777 (0.768)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-03 00:17:09,468 - Train: 0.54% [26600/4942000] [5.4/1000.0] [batch_t 0.758 (0.769)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-03 00:18:26,416 - Train: 0.54% [26700/4942000] [5.4/1000.0] [batch_t 0.771 (0.769)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-03 00:19:43,348 - Train: 0.54% [26800/4942000] [5.4/1000.0] [batch_t 0.768 (0.769)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-03 00:21:00,227 - Train: 0.54% [26900/4942000] [5.4/1000.0] [batch_t 0.772 (0.769)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-03 00:22:17,193 - Train: 0.55% [27000/4942000] [5.5/1000.0] [batch_t 0.764 (0.770)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-03 00:23:34,079 - Train: 0.55% [27100/4942000] [5.5/1000.0] [batch_t 0.773 (0.769)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-03 00:24:50,946 - Train: 0.55% [27200/4942000] [5.5/1000.0] [batch_t 0.770 (0.769)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-03 00:26:07,949 - Train: 0.55% [27300/4942000] [5.5/1000.0] [batch_t 0.768 (0.770)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-03 00:27:24,981 - Train: 0.55% [27400/4942000] [5.5/1000.0] [batch_t 0.754 (0.770)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-03 00:28:41,982 - Train: 0.56% [27500/4942000] [5.6/1000.0] [batch_t 0.778 (0.770)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-03 00:29:59,014 - Train: 0.56% [27600/4942000] [5.6/1000.0] [batch_t 0.772 (0.770)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-03 00:31:15,942 - Train: 0.56% [27700/4942000] [5.6/1000.0] [batch_t 0.764 (0.769)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-03 00:32:33,061 - Train: 0.56% [27800/4942000] [5.6/1000.0] [batch_t 0.778 (0.771)] [data_t 0.002] [optim_t 0.776] [lr 0.005000] 2024-04-03 00:33:50,025 - Train: 0.56% [27900/4942000] [5.6/1000.0] [batch_t 0.773 (0.770)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-03 00:35:07,081 - Train: 0.57% [28000/4942000] [5.7/1000.0] [batch_t 0.771 (0.770)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-03 00:36:24,152 - Train: 0.57% [28100/4942000] [5.7/1000.0] [batch_t 0.764 (0.770)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-03 00:37:41,133 - Train: 0.57% [28200/4942000] [5.7/1000.0] [batch_t 0.775 (0.770)] [data_t 0.003] [optim_t 0.772] [lr 0.005000] 2024-04-03 00:38:58,192 - Train: 0.57% [28300/4942000] [5.7/1000.0] [batch_t 0.772 (0.771)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-03 00:40:15,226 - Train: 0.57% [28400/4942000] [5.7/1000.0] [batch_t 0.773 (0.770)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-03 00:41:32,244 - Train: 0.58% [28500/4942000] [5.8/1000.0] [batch_t 0.768 (0.770)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-03 00:42:49,279 - Train: 0.58% [28600/4942000] [5.8/1000.0] [batch_t 0.767 (0.770)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 00:44:06,247 - Train: 0.58% [28700/4942000] [5.8/1000.0] [batch_t 0.768 (0.770)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-03 00:45:23,447 - Train: 0.58% [28800/4942000] [5.8/1000.0] [batch_t 0.759 (0.770)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-03 00:46:40,346 - Train: 0.58% [28900/4942000] [5.8/1000.0] [batch_t 0.757 (0.769)] [data_t 0.003] [optim_t 0.753] [lr 0.005000] 2024-04-03 00:47:57,356 - Train: 0.59% [29000/4942000] [5.9/1000.0] [batch_t 0.771 (0.770)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-03 00:49:14,328 - Train: 0.59% [29100/4942000] [5.9/1000.0] [batch_t 0.772 (0.770)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-03 00:50:31,286 - Train: 0.59% [29200/4942000] [5.9/1000.0] [batch_t 0.764 (0.769)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-03 00:51:48,272 - Train: 0.59% [29300/4942000] [5.9/1000.0] [batch_t 0.774 (0.770)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-03 00:53:05,251 - Train: 0.59% [29400/4942000] [5.9/1000.0] [batch_t 0.772 (0.770)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-03 00:54:22,218 - Train: 0.60% [29500/4942000] [6.0/1000.0] [batch_t 0.764 (0.770)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-03 00:55:39,221 - Train: 0.60% [29600/4942000] [6.0/1000.0] [batch_t 0.762 (0.770)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-03 00:56:19,174 - ==> Total time: 6:58:58 Eta: 48 days, 4:49:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 00:56:57,977 - Train: 0.60% [29700/4942000] [6.0/1000.0] [batch_t 0.768 (0.776)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-03 00:58:15,005 - Train: 0.60% [29800/4942000] [6.0/1000.0] [batch_t 0.773 (0.770)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-03 00:59:32,001 - Train: 0.61% [29900/4942000] [6.1/1000.0] [batch_t 0.771 (0.770)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-03 01:00:49,091 - Train: 0.61% [30000/4942000] [6.1/1000.0] [batch_t 0.772 (0.771)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-03 01:02:06,093 - Train: 0.61% [30100/4942000] [6.1/1000.0] [batch_t 0.764 (0.770)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-03 01:03:23,177 - Train: 0.61% [30200/4942000] [6.1/1000.0] [batch_t 0.773 (0.771)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-03 01:04:40,075 - Train: 0.61% [30300/4942000] [6.1/1000.0] [batch_t 0.755 (0.769)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-03 01:05:57,031 - Train: 0.62% [30400/4942000] [6.2/1000.0] [batch_t 0.770 (0.769)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-03 01:07:14,025 - Train: 0.62% [30500/4942000] [6.2/1000.0] [batch_t 0.769 (0.770)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-03 01:08:31,078 - Train: 0.62% [30600/4942000] [6.2/1000.0] [batch_t 0.776 (0.770)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-03 01:09:48,057 - Train: 0.62% [30700/4942000] [6.2/1000.0] [batch_t 0.774 (0.770)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-03 01:11:04,944 - Train: 0.62% [30800/4942000] [6.2/1000.0] [batch_t 0.776 (0.769)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-03 01:12:21,963 - Train: 0.63% [30900/4942000] [6.3/1000.0] [batch_t 0.782 (0.770)] [data_t 0.002] [optim_t 0.780] [lr 0.005000] 2024-04-03 01:13:39,318 - Train: 0.63% [31000/4942000] [6.3/1000.0] [batch_t 0.763 (0.771)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-03 01:14:56,280 - Train: 0.63% [31100/4942000] [6.3/1000.0] [batch_t 0.767 (0.770)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-03 01:16:13,661 - Train: 0.63% [31200/4942000] [6.3/1000.0] [batch_t 0.758 (0.774)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-03 01:17:30,709 - Train: 0.63% [31300/4942000] [6.3/1000.0] [batch_t 0.763 (0.770)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-03 01:18:47,743 - Train: 0.64% [31400/4942000] [6.4/1000.0] [batch_t 0.765 (0.770)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-03 01:20:04,802 - Train: 0.64% [31500/4942000] [6.4/1000.0] [batch_t 0.768 (0.770)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-03 01:21:21,778 - Train: 0.64% [31600/4942000] [6.4/1000.0] [batch_t 0.772 (0.770)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-03 01:22:38,857 - Train: 0.64% [31700/4942000] [6.4/1000.0] [batch_t 0.762 (0.771)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-03 01:23:55,845 - Train: 0.64% [31800/4942000] [6.4/1000.0] [batch_t 0.772 (0.770)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-03 01:25:12,804 - Train: 0.65% [31900/4942000] [6.5/1000.0] [batch_t 0.769 (0.770)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-03 01:26:29,853 - Train: 0.65% [32000/4942000] [6.5/1000.0] [batch_t 0.772 (0.770)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-03 01:27:46,872 - Train: 0.65% [32100/4942000] [6.5/1000.0] [batch_t 0.778 (0.770)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-03 01:29:03,911 - Train: 0.65% [32200/4942000] [6.5/1000.0] [batch_t 0.760 (0.770)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-03 01:30:20,933 - Train: 0.65% [32300/4942000] [6.5/1000.0] [batch_t 0.769 (0.770)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-03 01:31:37,961 - Train: 0.66% [32400/4942000] [6.6/1000.0] [batch_t 0.773 (0.770)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-03 01:32:54,980 - Train: 0.66% [32500/4942000] [6.6/1000.0] [batch_t 0.773 (0.770)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-03 01:34:11,995 - Train: 0.66% [32600/4942000] [6.6/1000.0] [batch_t 0.779 (0.770)] [data_t 0.003] [optim_t 0.777] [lr 0.005000] 2024-04-03 01:35:28,932 - Train: 0.66% [32700/4942000] [6.6/1000.0] [batch_t 0.777 (0.769)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-03 01:36:45,969 - Train: 0.66% [32800/4942000] [6.6/1000.0] [batch_t 0.767 (0.770)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 01:38:02,956 - Train: 0.67% [32900/4942000] [6.7/1000.0] [batch_t 0.765 (0.770)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-03 01:39:19,995 - Train: 0.67% [33000/4942000] [6.7/1000.0] [batch_t 0.768 (0.770)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-03 01:40:37,050 - Train: 0.67% [33100/4942000] [6.7/1000.0] [batch_t 0.767 (0.770)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-03 01:41:54,121 - Train: 0.67% [33200/4942000] [6.7/1000.0] [batch_t 0.779 (0.771)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-03 01:43:11,175 - Train: 0.67% [33300/4942000] [6.7/1000.0] [batch_t 0.767 (0.770)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 01:44:28,104 - Train: 0.68% [33400/4942000] [6.8/1000.0] [batch_t 0.777 (0.769)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-03 01:45:45,130 - Train: 0.68% [33500/4942000] [6.8/1000.0] [batch_t 0.768 (0.770)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-03 01:47:02,104 - Train: 0.68% [33600/4942000] [6.8/1000.0] [batch_t 0.764 (0.770)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-03 01:48:19,167 - Train: 0.68% [33700/4942000] [6.8/1000.0] [batch_t 0.783 (0.771)] [data_t 0.003] [optim_t 0.781] [lr 0.005000] 2024-04-03 01:49:36,181 - Train: 0.68% [33800/4942000] [6.8/1000.0] [batch_t 0.772 (0.770)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-03 01:50:53,169 - Train: 0.69% [33900/4942000] [6.9/1000.0] [batch_t 0.779 (0.770)] [data_t 0.002] [optim_t 0.776] [lr 0.005000] 2024-04-03 01:52:10,103 - Train: 0.69% [34000/4942000] [6.9/1000.0] [batch_t 0.769 (0.769)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-03 01:53:27,118 - Train: 0.69% [34100/4942000] [6.9/1000.0] [batch_t 0.767 (0.770)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-03 01:54:44,068 - Train: 0.69% [34200/4942000] [6.9/1000.0] [batch_t 0.766 (0.769)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-03 01:56:01,238 - Train: 0.69% [34300/4942000] [6.9/1000.0] [batch_t 0.769 (0.772)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-03 01:57:18,219 - Train: 0.70% [34400/4942000] [7.0/1000.0] [batch_t 0.772 (0.770)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-03 01:58:35,077 - Train: 0.70% [34500/4942000] [7.0/1000.0] [batch_t 0.767 (0.769)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-03 01:59:47,411 - ==> Total time: 8:02:26 Eta: 47 days, 12:38:01 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 01:59:53,931 - Train: 0.70% [34600/4942000] [7.0/1000.0] [batch_t 0.766 (0.819)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-03 02:01:10,942 - Train: 0.70% [34700/4942000] [7.0/1000.0] [batch_t 0.768 (0.770)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-03 02:02:27,910 - Train: 0.70% [34800/4942000] [7.0/1000.0] [batch_t 0.767 (0.770)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-03 02:03:44,879 - Train: 0.71% [34900/4942000] [7.1/1000.0] [batch_t 0.772 (0.770)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-03 02:05:01,804 - Train: 0.71% [35000/4942000] [7.1/1000.0] [batch_t 0.771 (0.769)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-03 02:06:18,826 - Train: 0.71% [35100/4942000] [7.1/1000.0] [batch_t 0.771 (0.770)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-03 02:07:35,779 - Train: 0.71% [35200/4942000] [7.1/1000.0] [batch_t 0.767 (0.769)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-03 02:08:52,754 - Train: 0.71% [35300/4942000] [7.1/1000.0] [batch_t 0.771 (0.770)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-03 02:10:09,722 - Train: 0.72% [35400/4942000] [7.2/1000.0] [batch_t 0.773 (0.770)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-03 02:11:26,626 - Train: 0.72% [35500/4942000] [7.2/1000.0] [batch_t 0.767 (0.769)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-03 02:12:43,577 - Train: 0.72% [35600/4942000] [7.2/1000.0] [batch_t 0.771 (0.769)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-03 02:14:00,531 - Train: 0.72% [35700/4942000] [7.2/1000.0] [batch_t 0.783 (0.769)] [data_t 0.003] [optim_t 0.780] [lr 0.005000] 2024-04-03 02:15:17,478 - Train: 0.72% [35800/4942000] [7.2/1000.0] [batch_t 0.776 (0.769)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-03 02:16:34,592 - Train: 0.73% [35900/4942000] [7.3/1000.0] [batch_t 0.777 (0.771)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-03 02:17:51,640 - Train: 0.73% [36000/4942000] [7.3/1000.0] [batch_t 0.772 (0.770)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-03 02:19:08,614 - Train: 0.73% [36100/4942000] [7.3/1000.0] [batch_t 0.765 (0.770)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-03 02:20:25,563 - Train: 0.73% [36200/4942000] [7.3/1000.0] [batch_t 0.761 (0.769)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-03 02:21:42,501 - Train: 0.73% [36300/4942000] [7.3/1000.0] [batch_t 0.767 (0.769)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 02:22:59,443 - Train: 0.74% [36400/4942000] [7.4/1000.0] [batch_t 0.770 (0.769)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-03 02:24:16,401 - Train: 0.74% [36500/4942000] [7.4/1000.0] [batch_t 0.767 (0.769)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 02:25:33,278 - Train: 0.74% [36600/4942000] [7.4/1000.0] [batch_t 0.777 (0.769)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-03 02:26:50,236 - Train: 0.74% [36700/4942000] [7.4/1000.0] [batch_t 0.768 (0.770)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-03 02:28:07,270 - Train: 0.74% [36800/4942000] [7.4/1000.0] [batch_t 0.771 (0.770)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-03 02:29:24,312 - Train: 0.75% [36900/4942000] [7.5/1000.0] [batch_t 0.771 (0.770)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-03 02:30:41,267 - Train: 0.75% [37000/4942000] [7.5/1000.0] [batch_t 0.756 (0.769)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-03 02:31:58,253 - Train: 0.75% [37100/4942000] [7.5/1000.0] [batch_t 0.772 (0.770)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-03 02:33:15,285 - Train: 0.75% [37200/4942000] [7.5/1000.0] [batch_t 0.767 (0.770)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-03 02:34:32,290 - Train: 0.75% [37300/4942000] [7.5/1000.0] [batch_t 0.770 (0.770)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-03 02:35:49,316 - Train: 0.76% [37400/4942000] [7.6/1000.0] [batch_t 0.767 (0.770)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 02:37:06,322 - Train: 0.76% [37500/4942000] [7.6/1000.0] [batch_t 0.759 (0.770)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-03 02:38:23,320 - Train: 0.76% [37600/4942000] [7.6/1000.0] [batch_t 0.768 (0.770)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-03 02:39:40,363 - Train: 0.76% [37700/4942000] [7.6/1000.0] [batch_t 0.769 (0.770)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-03 02:40:57,397 - Train: 0.76% [37800/4942000] [7.6/1000.0] [batch_t 0.763 (0.770)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-03 02:42:14,463 - Train: 0.77% [37900/4942000] [7.7/1000.0] [batch_t 0.773 (0.771)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-03 02:43:31,394 - Train: 0.77% [38000/4942000] [7.7/1000.0] [batch_t 0.756 (0.769)] [data_t 0.003] [optim_t 0.753] [lr 0.005000] 2024-04-03 02:44:48,404 - Train: 0.77% [38100/4942000] [7.7/1000.0] [batch_t 0.767 (0.770)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 02:46:05,413 - Train: 0.77% [38200/4942000] [7.7/1000.0] [batch_t 0.772 (0.770)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-03 02:47:22,458 - Train: 0.77% [38300/4942000] [7.7/1000.0] [batch_t 0.777 (0.770)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-03 02:48:39,467 - Train: 0.78% [38400/4942000] [7.8/1000.0] [batch_t 0.766 (0.770)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-03 02:49:56,354 - Train: 0.78% [38500/4942000] [7.8/1000.0] [batch_t 0.776 (0.769)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-03 02:51:13,362 - Train: 0.78% [38600/4942000] [7.8/1000.0] [batch_t 0.771 (0.770)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-03 02:52:30,345 - Train: 0.78% [38700/4942000] [7.8/1000.0] [batch_t 0.767 (0.770)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 02:53:47,329 - Train: 0.79% [38800/4942000] [7.9/1000.0] [batch_t 0.773 (0.770)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-03 02:55:04,331 - Train: 0.79% [38900/4942000] [7.9/1000.0] [batch_t 0.773 (0.770)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-03 02:56:21,345 - Train: 0.79% [39000/4942000] [7.9/1000.0] [batch_t 0.767 (0.770)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-03 02:57:38,321 - Train: 0.79% [39100/4942000] [7.9/1000.0] [batch_t 0.771 (0.770)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-03 02:58:55,210 - Train: 0.79% [39200/4942000] [7.9/1000.0] [batch_t 0.777 (0.769)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-03 03:00:12,168 - Train: 0.80% [39300/4942000] [8.0/1000.0] [batch_t 0.763 (0.769)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-03 03:01:29,300 - Train: 0.80% [39400/4942000] [8.0/1000.0] [batch_t 0.766 (0.771)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-03 03:02:46,262 - Train: 0.80% [39500/4942000] [8.0/1000.0] [batch_t 0.770 (0.770)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-03 03:03:13,970 - ==> Total time: 9:05:53 Eta: 47 days, 0:09:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 03:04:05,005 - Train: 0.80% [39600/4942000] [8.0/1000.0] [batch_t 0.772 (0.774)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-03 03:05:22,012 - Train: 0.80% [39700/4942000] [8.0/1000.0] [batch_t 0.772 (0.770)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-03 03:06:39,077 - Train: 0.81% [39800/4942000] [8.1/1000.0] [batch_t 0.782 (0.771)] [data_t 0.002] [optim_t 0.780] [lr 0.005000] 2024-04-03 03:07:55,987 - Train: 0.81% [39900/4942000] [8.1/1000.0] [batch_t 0.775 (0.769)] [data_t 0.003] [optim_t 0.772] [lr 0.005000] 2024-04-03 03:09:12,989 - Train: 0.81% [40000/4942000] [8.1/1000.0] [batch_t 0.761 (0.770)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-03 03:10:29,983 - Train: 0.81% [40100/4942000] [8.1/1000.0] [batch_t 0.778 (0.770)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-03 03:11:47,002 - Train: 0.81% [40200/4942000] [8.1/1000.0] [batch_t 0.776 (0.770)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-03 03:13:03,990 - Train: 0.82% [40300/4942000] [8.2/1000.0] [batch_t 0.757 (0.770)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-03 03:14:21,099 - Train: 0.82% [40400/4942000] [8.2/1000.0] [batch_t 0.781 (0.771)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-03 03:15:38,096 - Train: 0.82% [40500/4942000] [8.2/1000.0] [batch_t 0.782 (0.770)] [data_t 0.003] [optim_t 0.780] [lr 0.005000] 2024-04-03 03:16:55,135 - Train: 0.82% [40600/4942000] [8.2/1000.0] [batch_t 0.766 (0.770)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-03 03:18:12,086 - Train: 0.82% [40700/4942000] [8.2/1000.0] [batch_t 0.766 (0.769)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-03 03:19:29,060 - Train: 0.83% [40800/4942000] [8.3/1000.0] [batch_t 0.767 (0.770)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 03:20:46,114 - Train: 0.83% [40900/4942000] [8.3/1000.0] [batch_t 0.776 (0.770)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-03 03:22:03,074 - Train: 0.83% [41000/4942000] [8.3/1000.0] [batch_t 0.772 (0.769)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-03 03:23:20,106 - Train: 0.83% [41100/4942000] [8.3/1000.0] [batch_t 0.762 (0.770)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-03 03:24:37,090 - Train: 0.83% [41200/4942000] [8.3/1000.0] [batch_t 0.773 (0.770)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-03 03:25:54,006 - Train: 0.84% [41300/4942000] [8.4/1000.0] [batch_t 0.776 (0.769)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-03 03:27:10,890 - Train: 0.84% [41400/4942000] [8.4/1000.0] [batch_t 0.773 (0.769)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-03 03:28:27,855 - Train: 0.84% [41500/4942000] [8.4/1000.0] [batch_t 0.772 (0.770)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-03 03:29:44,790 - Train: 0.84% [41600/4942000] [8.4/1000.0] [batch_t 0.773 (0.769)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-03 03:31:01,670 - Train: 0.84% [41700/4942000] [8.4/1000.0] [batch_t 0.771 (0.769)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-03 03:32:18,698 - Train: 0.85% [41800/4942000] [8.5/1000.0] [batch_t 0.777 (0.770)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-03 03:33:35,746 - Train: 0.85% [41900/4942000] [8.5/1000.0] [batch_t 0.778 (0.770)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-03 03:34:52,747 - Train: 0.85% [42000/4942000] [8.5/1000.0] [batch_t 0.768 (0.770)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-03 03:36:09,841 - Train: 0.85% [42100/4942000] [8.5/1000.0] [batch_t 0.767 (0.771)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-03 03:37:26,747 - Train: 0.85% [42200/4942000] [8.5/1000.0] [batch_t 0.772 (0.769)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-03 03:38:43,729 - Train: 0.86% [42300/4942000] [8.6/1000.0] [batch_t 0.771 (0.770)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-03 03:40:00,857 - Train: 0.86% [42400/4942000] [8.6/1000.0] [batch_t 0.782 (0.771)] [data_t 0.002] [optim_t 0.780] [lr 0.005000] 2024-04-03 03:41:17,884 - Train: 0.86% [42500/4942000] [8.6/1000.0] [batch_t 0.774 (0.770)] [data_t 0.002] [optim_t 0.772] [lr 0.005000] 2024-04-03 03:42:34,876 - Train: 0.86% [42600/4942000] [8.6/1000.0] [batch_t 0.777 (0.770)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-03 03:43:51,878 - Train: 0.86% [42700/4942000] [8.6/1000.0] [batch_t 0.772 (0.770)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-03 03:45:08,838 - Train: 0.87% [42800/4942000] [8.7/1000.0] [batch_t 0.763 (0.770)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-03 03:46:25,841 - Train: 0.87% [42900/4942000] [8.7/1000.0] [batch_t 0.773 (0.770)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-03 03:47:42,941 - Train: 0.87% [43000/4942000] [8.7/1000.0] [batch_t 0.759 (0.771)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-03 03:49:00,088 - Train: 0.87% [43100/4942000] [8.7/1000.0] [batch_t 0.783 (0.771)] [data_t 0.003] [optim_t 0.781] [lr 0.005000] 2024-04-03 03:50:17,088 - Train: 0.87% [43200/4942000] [8.7/1000.0] [batch_t 0.774 (0.770)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-03 03:51:34,076 - Train: 0.88% [43300/4942000] [8.8/1000.0] [batch_t 0.767 (0.770)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 03:52:51,147 - Train: 0.88% [43400/4942000] [8.8/1000.0] [batch_t 0.779 (0.771)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-03 03:54:08,197 - Train: 0.88% [43500/4942000] [8.8/1000.0] [batch_t 0.763 (0.770)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-03 03:55:25,120 - Train: 0.88% [43600/4942000] [8.8/1000.0] [batch_t 0.762 (0.769)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-03 03:56:42,077 - Train: 0.88% [43700/4942000] [8.8/1000.0] [batch_t 0.772 (0.769)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-03 03:57:59,098 - Train: 0.89% [43800/4942000] [8.9/1000.0] [batch_t 0.772 (0.770)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-03 03:59:16,152 - Train: 0.89% [43900/4942000] [8.9/1000.0] [batch_t 0.765 (0.770)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-03 04:00:33,135 - Train: 0.89% [44000/4942000] [8.9/1000.0] [batch_t 0.770 (0.770)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-03 04:01:50,070 - Train: 0.89% [44100/4942000] [8.9/1000.0] [batch_t 0.778 (0.769)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-03 04:03:07,098 - Train: 0.89% [44200/4942000] [8.9/1000.0] [batch_t 0.774 (0.770)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-03 04:04:24,038 - Train: 0.90% [44300/4942000] [9.0/1000.0] [batch_t 0.772 (0.769)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-03 04:05:41,131 - Train: 0.90% [44400/4942000] [9.0/1000.0] [batch_t 0.772 (0.771)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-03 04:06:41,140 - ==> Total time: 10:09:20 Eta: 46 days, 14:14:58 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 04:06:59,710 - Train: 0.90% [44500/4942000] [9.0/1000.0] [batch_t 0.758 (0.780)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-03 04:08:16,936 - Train: 0.90% [44600/4942000] [9.0/1000.0] [batch_t 0.772 (0.772)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-03 04:09:33,923 - Train: 0.90% [44700/4942000] [9.0/1000.0] [batch_t 0.772 (0.770)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-03 04:10:50,911 - Train: 0.91% [44800/4942000] [9.1/1000.0] [batch_t 0.776 (0.770)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-03 04:12:07,766 - Train: 0.91% [44900/4942000] [9.1/1000.0] [batch_t 0.748 (0.768)] [data_t 0.002] [optim_t 0.746] [lr 0.005000] 2024-04-03 04:13:24,779 - Train: 0.91% [45000/4942000] [9.1/1000.0] [batch_t 0.766 (0.770)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-03 04:14:41,711 - Train: 0.91% [45100/4942000] [9.1/1000.0] [batch_t 0.768 (0.769)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-03 04:15:58,689 - Train: 0.91% [45200/4942000] [9.1/1000.0] [batch_t 0.768 (0.770)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-03 04:17:15,621 - Train: 0.92% [45300/4942000] [9.2/1000.0] [batch_t 0.763 (0.769)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-03 04:18:32,573 - Train: 0.92% [45400/4942000] [9.2/1000.0] [batch_t 0.768 (0.769)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-03 04:19:49,760 - Train: 0.92% [45500/4942000] [9.2/1000.0] [batch_t 0.765 (0.772)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-03 04:21:06,800 - Train: 0.92% [45600/4942000] [9.2/1000.0] [batch_t 0.772 (0.770)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-03 04:22:23,725 - Train: 0.92% [45700/4942000] [9.2/1000.0] [batch_t 0.778 (0.769)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-03 04:23:40,699 - Train: 0.93% [45800/4942000] [9.3/1000.0] [batch_t 0.766 (0.770)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-03 04:24:57,529 - Train: 0.93% [45900/4942000] [9.3/1000.0] [batch_t 0.768 (0.768)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-03 04:26:14,618 - Train: 0.93% [46000/4942000] [9.3/1000.0] [batch_t 0.772 (0.771)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-03 04:27:31,639 - Train: 0.93% [46100/4942000] [9.3/1000.0] [batch_t 0.773 (0.770)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-03 04:28:48,465 - Train: 0.93% [46200/4942000] [9.3/1000.0] [batch_t 0.772 (0.768)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-03 04:30:05,362 - Train: 0.94% [46300/4942000] [9.4/1000.0] [batch_t 0.761 (0.769)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-03 04:31:23,440 - Train: 0.94% [46400/4942000] [9.4/1000.0] [batch_t 0.766 (0.781)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-03 04:32:40,443 - Train: 0.94% [46500/4942000] [9.4/1000.0] [batch_t 0.771 (0.770)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-03 04:33:57,424 - Train: 0.94% [46600/4942000] [9.4/1000.0] [batch_t 0.763 (0.770)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-03 04:35:14,487 - Train: 0.94% [46700/4942000] [9.4/1000.0] [batch_t 0.766 (0.771)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-03 04:36:31,610 - Train: 0.95% [46800/4942000] [9.5/1000.0] [batch_t 0.768 (0.771)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-03 04:37:48,680 - Train: 0.95% [46900/4942000] [9.5/1000.0] [batch_t 0.775 (0.771)] [data_t 0.002] [optim_t 0.773] [lr 0.005000] 2024-04-03 04:39:05,731 - Train: 0.95% [47000/4942000] [9.5/1000.0] [batch_t 0.762 (0.770)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-03 04:40:22,727 - Train: 0.95% [47100/4942000] [9.5/1000.0] [batch_t 0.775 (0.770)] [data_t 0.002] [optim_t 0.773] [lr 0.005000] 2024-04-03 04:41:39,703 - Train: 0.96% [47200/4942000] [9.6/1000.0] [batch_t 0.757 (0.770)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-03 04:42:56,813 - Train: 0.96% [47300/4942000] [9.6/1000.0] [batch_t 0.771 (0.771)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-03 04:44:13,793 - Train: 0.96% [47400/4942000] [9.6/1000.0] [batch_t 0.775 (0.770)] [data_t 0.003] [optim_t 0.772] [lr 0.005000] 2024-04-03 04:45:30,731 - Train: 0.96% [47500/4942000] [9.6/1000.0] [batch_t 0.762 (0.769)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-03 04:46:47,648 - Train: 0.96% [47600/4942000] [9.6/1000.0] [batch_t 0.772 (0.769)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-03 04:48:04,665 - Train: 0.97% [47700/4942000] [9.7/1000.0] [batch_t 0.770 (0.770)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-03 04:49:21,638 - Train: 0.97% [47800/4942000] [9.7/1000.0] [batch_t 0.763 (0.770)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-03 04:50:38,669 - Train: 0.97% [47900/4942000] [9.7/1000.0] [batch_t 0.776 (0.770)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-03 04:51:55,685 - Train: 0.97% [48000/4942000] [9.7/1000.0] [batch_t 0.763 (0.770)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-03 04:53:12,689 - Train: 0.97% [48100/4942000] [9.7/1000.0] [batch_t 0.763 (0.770)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-03 04:54:29,568 - Train: 0.98% [48200/4942000] [9.8/1000.0] [batch_t 0.759 (0.769)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-03 04:55:46,497 - Train: 0.98% [48300/4942000] [9.8/1000.0] [batch_t 0.769 (0.769)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-03 04:57:03,413 - Train: 0.98% [48400/4942000] [9.8/1000.0] [batch_t 0.772 (0.769)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-03 04:58:20,442 - Train: 0.98% [48500/4942000] [9.8/1000.0] [batch_t 0.768 (0.770)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-03 04:59:37,525 - Train: 0.98% [48600/4942000] [9.8/1000.0] [batch_t 0.773 (0.771)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-03 05:00:54,514 - Train: 0.99% [48700/4942000] [9.9/1000.0] [batch_t 0.764 (0.770)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-03 05:02:11,493 - Train: 0.99% [48800/4942000] [9.9/1000.0] [batch_t 0.768 (0.770)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-03 05:03:28,565 - Train: 0.99% [48900/4942000] [9.9/1000.0] [batch_t 0.777 (0.771)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-03 05:04:45,476 - Train: 0.99% [49000/4942000] [9.9/1000.0] [batch_t 0.765 (0.769)] [data_t 0.002] [optim_t 0.763] [lr 0.005000] 2024-04-03 05:06:02,281 - Train: 0.99% [49100/4942000] [9.9/1000.0] [batch_t 0.762 (0.768)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-03 05:07:19,389 - Train: 1.00% [49200/4942000] [10.0/1000.0] [batch_t 0.776 (0.771)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-03 05:08:36,518 - Train: 1.00% [49300/4942000] [10.0/1000.0] [batch_t 0.762 (0.771)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-03 05:09:53,429 - Train: 1.00% [49400/4942000] [10.0/1000.0] [batch_t 0.776 (0.769)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-03 05:10:08,839 - ==> Total time: 11:12:48 Eta: 46 days, 6:07:14 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 05:11:12,064 - Train: 1.00% [49500/4942000] [10.0/1000.0] [batch_t 0.759 (0.770)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-03 05:12:29,022 - Train: 1.00% [49600/4942000] [10.0/1000.0] [batch_t 0.759 (0.769)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-03 05:13:46,016 - Train: 1.01% [49700/4942000] [10.1/1000.0] [batch_t 0.776 (0.770)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-03 05:15:03,010 - Train: 1.01% [49800/4942000] [10.1/1000.0] [batch_t 0.767 (0.770)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 05:16:19,920 - Train: 1.01% [49900/4942000] [10.1/1000.0] [batch_t 0.768 (0.769)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-03 05:17:36,834 - Train: 1.01% [50000/4942000] [10.1/1000.0] [batch_t 0.755 (0.769)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-03 05:18:53,812 - Train: 1.01% [50100/4942000] [10.1/1000.0] [batch_t 0.762 (0.770)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-03 05:20:10,877 - Train: 1.02% [50200/4942000] [10.2/1000.0] [batch_t 0.777 (0.771)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-03 05:21:27,842 - Train: 1.02% [50300/4942000] [10.2/1000.0] [batch_t 0.772 (0.770)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-03 05:22:44,775 - Train: 1.02% [50400/4942000] [10.2/1000.0] [batch_t 0.771 (0.769)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-03 05:24:01,730 - Train: 1.02% [50500/4942000] [10.2/1000.0] [batch_t 0.767 (0.769)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 05:25:18,652 - Train: 1.02% [50600/4942000] [10.2/1000.0] [batch_t 0.777 (0.769)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-03 05:26:35,746 - Train: 1.03% [50700/4942000] [10.3/1000.0] [batch_t 0.772 (0.771)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-03 05:27:52,759 - Train: 1.03% [50800/4942000] [10.3/1000.0] [batch_t 0.767 (0.770)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-03 05:29:09,814 - Train: 1.03% [50900/4942000] [10.3/1000.0] [batch_t 0.766 (0.770)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-03 05:30:26,883 - Train: 1.03% [51000/4942000] [10.3/1000.0] [batch_t 0.773 (0.771)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-03 05:31:43,833 - Train: 1.03% [51100/4942000] [10.3/1000.0] [batch_t 0.781 (0.769)] [data_t 0.002] [optim_t 0.779] [lr 0.005000] 2024-04-03 05:33:00,763 - Train: 1.04% [51200/4942000] [10.4/1000.0] [batch_t 0.768 (0.769)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-03 05:34:17,724 - Train: 1.04% [51300/4942000] [10.4/1000.0] [batch_t 0.769 (0.770)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-03 05:35:34,618 - Train: 1.04% [51400/4942000] [10.4/1000.0] [batch_t 0.762 (0.769)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-03 05:36:51,608 - Train: 1.04% [51500/4942000] [10.4/1000.0] [batch_t 0.753 (0.770)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-03 05:38:08,606 - Train: 1.04% [51600/4942000] [10.4/1000.0] [batch_t 0.772 (0.770)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-03 05:39:25,484 - Train: 1.05% [51700/4942000] [10.5/1000.0] [batch_t 0.770 (0.769)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-03 05:40:42,368 - Train: 1.05% [51800/4942000] [10.5/1000.0] [batch_t 0.763 (0.769)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-03 05:41:59,417 - Train: 1.05% [51900/4942000] [10.5/1000.0] [batch_t 0.766 (0.770)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 05:43:16,458 - Train: 1.05% [52000/4942000] [10.5/1000.0] [batch_t 0.758 (0.770)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-03 05:44:33,412 - Train: 1.05% [52100/4942000] [10.5/1000.0] [batch_t 0.777 (0.769)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-03 05:45:50,429 - Train: 1.06% [52200/4942000] [10.6/1000.0] [batch_t 0.767 (0.770)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 05:47:07,313 - Train: 1.06% [52300/4942000] [10.6/1000.0] [batch_t 0.770 (0.769)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-03 05:48:24,237 - Train: 1.06% [52400/4942000] [10.6/1000.0] [batch_t 0.762 (0.769)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-03 05:49:41,340 - Train: 1.06% [52500/4942000] [10.6/1000.0] [batch_t 0.769 (0.771)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-03 05:50:58,305 - Train: 1.06% [52600/4942000] [10.6/1000.0] [batch_t 0.772 (0.770)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-03 05:52:15,273 - Train: 1.07% [52700/4942000] [10.7/1000.0] [batch_t 0.770 (0.770)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-03 05:53:32,150 - Train: 1.07% [52800/4942000] [10.7/1000.0] [batch_t 0.774 (0.769)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-03 05:54:49,008 - Train: 1.07% [52900/4942000] [10.7/1000.0] [batch_t 0.777 (0.768)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-03 05:56:05,983 - Train: 1.07% [53000/4942000] [10.7/1000.0] [batch_t 0.771 (0.770)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-03 05:57:23,210 - Train: 1.07% [53100/4942000] [10.7/1000.0] [batch_t 0.769 (0.770)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-03 05:58:40,195 - Train: 1.08% [53200/4942000] [10.8/1000.0] [batch_t 0.772 (0.770)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-03 05:59:57,093 - Train: 1.08% [53300/4942000] [10.8/1000.0] [batch_t 0.757 (0.769)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-03 06:01:14,174 - Train: 1.08% [53400/4942000] [10.8/1000.0] [batch_t 0.763 (0.771)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-03 06:02:31,168 - Train: 1.08% [53500/4942000] [10.8/1000.0] [batch_t 0.761 (0.770)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-03 06:03:48,211 - Train: 1.08% [53600/4942000] [10.8/1000.0] [batch_t 0.766 (0.770)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-03 06:05:05,006 - Train: 1.09% [53700/4942000] [10.9/1000.0] [batch_t 0.778 (0.768)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-03 06:06:22,089 - Train: 1.09% [53800/4942000] [10.9/1000.0] [batch_t 0.766 (0.771)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-03 06:07:39,028 - Train: 1.09% [53900/4942000] [10.9/1000.0] [batch_t 0.770 (0.769)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-03 06:08:56,158 - Train: 1.09% [54000/4942000] [10.9/1000.0] [batch_t 0.777 (0.771)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-03 06:10:13,257 - Train: 1.09% [54100/4942000] [10.9/1000.0] [batch_t 0.771 (0.771)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-03 06:11:30,170 - Train: 1.10% [54200/4942000] [11.0/1000.0] [batch_t 0.767 (0.769)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-03 06:12:47,264 - Train: 1.10% [54300/4942000] [11.0/1000.0] [batch_t 0.770 (0.771)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-03 06:13:34,892 - ==> Total time: 12:16:14 Eta: 45 days, 23:14:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 06:14:06,109 - Train: 1.10% [54400/4942000] [11.0/1000.0] [batch_t 0.765 (0.777)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-03 06:15:23,063 - Train: 1.10% [54500/4942000] [11.0/1000.0] [batch_t 0.765 (0.769)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-03 06:16:40,066 - Train: 1.10% [54600/4942000] [11.0/1000.0] [batch_t 0.771 (0.770)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-03 06:17:57,076 - Train: 1.11% [54700/4942000] [11.1/1000.0] [batch_t 0.774 (0.770)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-03 06:19:13,935 - Train: 1.11% [54800/4942000] [11.1/1000.0] [batch_t 0.771 (0.768)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-03 06:20:30,876 - Train: 1.11% [54900/4942000] [11.1/1000.0] [batch_t 0.781 (0.769)] [data_t 0.002] [optim_t 0.778] [lr 0.005000] 2024-04-03 06:21:47,852 - Train: 1.11% [55000/4942000] [11.1/1000.0] [batch_t 0.769 (0.770)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-03 06:23:04,883 - Train: 1.11% [55100/4942000] [11.1/1000.0] [batch_t 0.768 (0.770)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-03 06:24:21,938 - Train: 1.12% [55200/4942000] [11.2/1000.0] [batch_t 0.757 (0.770)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-03 06:25:38,831 - Train: 1.12% [55300/4942000] [11.2/1000.0] [batch_t 0.762 (0.769)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-03 06:26:55,784 - Train: 1.12% [55400/4942000] [11.2/1000.0] [batch_t 0.759 (0.769)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-03 06:28:12,914 - Train: 1.12% [55500/4942000] [11.2/1000.0] [batch_t 0.773 (0.771)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-03 06:29:29,796 - Train: 1.13% [55600/4942000] [11.3/1000.0] [batch_t 0.762 (0.769)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-03 06:30:46,821 - Train: 1.13% [55700/4942000] [11.3/1000.0] [batch_t 0.762 (0.770)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-03 06:32:03,738 - Train: 1.13% [55800/4942000] [11.3/1000.0] [batch_t 0.773 (0.769)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-03 06:33:20,712 - Train: 1.13% [55900/4942000] [11.3/1000.0] [batch_t 0.768 (0.770)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-03 06:34:37,590 - Train: 1.13% [56000/4942000] [11.3/1000.0] [batch_t 0.759 (0.769)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-03 06:35:54,614 - Train: 1.14% [56100/4942000] [11.4/1000.0] [batch_t 0.779 (0.770)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-03 06:37:11,660 - Train: 1.14% [56200/4942000] [11.4/1000.0] [batch_t 0.772 (0.770)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-03 06:38:28,626 - Train: 1.14% [56300/4942000] [11.4/1000.0] [batch_t 0.764 (0.770)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-03 06:39:45,606 - Train: 1.14% [56400/4942000] [11.4/1000.0] [batch_t 0.768 (0.770)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-03 06:41:02,575 - Train: 1.14% [56500/4942000] [11.4/1000.0] [batch_t 0.763 (0.770)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-03 06:42:19,652 - Train: 1.15% [56600/4942000] [11.5/1000.0] [batch_t 0.777 (0.771)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-03 06:43:36,719 - Train: 1.15% [56700/4942000] [11.5/1000.0] [batch_t 0.783 (0.771)] [data_t 0.002] [optim_t 0.781] [lr 0.005000] 2024-04-03 06:44:53,699 - Train: 1.15% [56800/4942000] [11.5/1000.0] [batch_t 0.777 (0.770)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-03 06:46:10,780 - Train: 1.15% [56900/4942000] [11.5/1000.0] [batch_t 0.754 (0.771)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-03 06:47:27,934 - Train: 1.15% [57000/4942000] [11.5/1000.0] [batch_t 0.778 (0.771)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-03 06:48:44,948 - Train: 1.16% [57100/4942000] [11.6/1000.0] [batch_t 0.772 (0.770)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-03 06:50:01,893 - Train: 1.16% [57200/4942000] [11.6/1000.0] [batch_t 0.768 (0.769)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-03 06:51:18,902 - Train: 1.16% [57300/4942000] [11.6/1000.0] [batch_t 0.767 (0.770)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-03 06:52:35,852 - Train: 1.16% [57400/4942000] [11.6/1000.0] [batch_t 0.770 (0.769)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-03 06:53:52,872 - Train: 1.16% [57500/4942000] [11.6/1000.0] [batch_t 0.769 (0.770)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-03 06:55:09,933 - Train: 1.17% [57600/4942000] [11.7/1000.0] [batch_t 0.772 (0.771)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-03 06:56:26,913 - Train: 1.17% [57700/4942000] [11.7/1000.0] [batch_t 0.761 (0.770)] [data_t 0.002] [optim_t 0.758] [lr 0.005000] 2024-04-03 06:57:43,865 - Train: 1.17% [57800/4942000] [11.7/1000.0] [batch_t 0.768 (0.769)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-03 06:59:00,891 - Train: 1.17% [57900/4942000] [11.7/1000.0] [batch_t 0.766 (0.770)] [data_t 0.002] [optim_t 0.763] [lr 0.005000] 2024-04-03 07:00:17,783 - Train: 1.17% [58000/4942000] [11.7/1000.0] [batch_t 0.764 (0.769)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-03 07:01:34,722 - Train: 1.18% [58100/4942000] [11.8/1000.0] [batch_t 0.763 (0.769)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-03 07:02:51,725 - Train: 1.18% [58200/4942000] [11.8/1000.0] [batch_t 0.777 (0.770)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-03 07:04:08,598 - Train: 1.18% [58300/4942000] [11.8/1000.0] [batch_t 0.778 (0.769)] [data_t 0.002] [optim_t 0.776] [lr 0.005000] 2024-04-03 07:05:25,643 - Train: 1.18% [58400/4942000] [11.8/1000.0] [batch_t 0.762 (0.770)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-03 07:06:42,573 - Train: 1.18% [58500/4942000] [11.8/1000.0] [batch_t 0.771 (0.769)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-03 07:07:59,352 - Train: 1.19% [58600/4942000] [11.9/1000.0] [batch_t 0.764 (0.768)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-03 07:09:16,355 - Train: 1.19% [58700/4942000] [11.9/1000.0] [batch_t 0.769 (0.770)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-03 07:10:33,438 - Train: 1.19% [58800/4942000] [11.9/1000.0] [batch_t 0.773 (0.771)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-03 07:11:50,397 - Train: 1.19% [58900/4942000] [11.9/1000.0] [batch_t 0.768 (0.770)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-03 07:13:07,454 - Train: 1.19% [59000/4942000] [11.9/1000.0] [batch_t 0.779 (0.770)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-03 07:14:24,429 - Train: 1.20% [59100/4942000] [12.0/1000.0] [batch_t 0.777 (0.770)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-03 07:15:41,495 - Train: 1.20% [59200/4942000] [12.0/1000.0] [batch_t 0.773 (0.771)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-03 07:16:58,378 - Train: 1.20% [59300/4942000] [12.0/1000.0] [batch_t 0.748 (0.769)] [data_t 0.003] [optim_t 0.745] [lr 0.005000] 2024-04-03 07:17:01,473 - ==> Total time: 13:19:40 Eta: 45 days, 17:20:07 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 07:18:19,433 - Train: 1.20% [59400/4942000] [12.0/1000.0] [batch_t 0.768 (0.780)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-03 07:19:36,496 - Train: 1.20% [59500/4942000] [12.0/1000.0] [batch_t 0.770 (0.771)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-03 07:20:53,477 - Train: 1.21% [59600/4942000] [12.1/1000.0] [batch_t 0.775 (0.770)] [data_t 0.002] [optim_t 0.773] [lr 0.005000] 2024-04-03 07:22:10,450 - Train: 1.21% [59700/4942000] [12.1/1000.0] [batch_t 0.763 (0.770)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-03 07:23:27,347 - Train: 1.21% [59800/4942000] [12.1/1000.0] [batch_t 0.766 (0.769)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 07:24:44,413 - Train: 1.21% [59900/4942000] [12.1/1000.0] [batch_t 0.776 (0.771)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-03 07:26:01,461 - Train: 1.21% [60000/4942000] [12.1/1000.0] [batch_t 0.773 (0.770)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-03 07:27:18,474 - Train: 1.22% [60100/4942000] [12.2/1000.0] [batch_t 0.749 (0.770)] [data_t 0.002] [optim_t 0.746] [lr 0.005000] 2024-04-03 07:28:35,446 - Train: 1.22% [60200/4942000] [12.2/1000.0] [batch_t 0.765 (0.770)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-03 07:29:52,407 - Train: 1.22% [60300/4942000] [12.2/1000.0] [batch_t 0.772 (0.769)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-03 07:31:09,487 - Train: 1.22% [60400/4942000] [12.2/1000.0] [batch_t 0.767 (0.771)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 07:32:26,372 - Train: 1.22% [60500/4942000] [12.2/1000.0] [batch_t 0.774 (0.769)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-03 07:33:43,453 - Train: 1.23% [60600/4942000] [12.3/1000.0] [batch_t 0.773 (0.771)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-03 07:35:00,484 - Train: 1.23% [60700/4942000] [12.3/1000.0] [batch_t 0.774 (0.770)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-03 07:36:17,565 - Train: 1.23% [60800/4942000] [12.3/1000.0] [batch_t 0.768 (0.771)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-03 07:37:34,644 - Train: 1.23% [60900/4942000] [12.3/1000.0] [batch_t 0.769 (0.771)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-03 07:38:51,590 - Train: 1.23% [61000/4942000] [12.3/1000.0] [batch_t 0.765 (0.769)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-03 07:40:08,613 - Train: 1.24% [61100/4942000] [12.4/1000.0] [batch_t 0.763 (0.770)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-03 07:41:25,542 - Train: 1.24% [61200/4942000] [12.4/1000.0] [batch_t 0.771 (0.769)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-03 07:42:42,544 - Train: 1.24% [61300/4942000] [12.4/1000.0] [batch_t 0.782 (0.770)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-03 07:43:59,541 - Train: 1.24% [61400/4942000] [12.4/1000.0] [batch_t 0.762 (0.770)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-03 07:45:16,467 - Train: 1.24% [61500/4942000] [12.4/1000.0] [batch_t 0.769 (0.769)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-03 07:46:33,409 - Train: 1.25% [61600/4942000] [12.5/1000.0] [batch_t 0.767 (0.769)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 07:47:50,468 - Train: 1.25% [61700/4942000] [12.5/1000.0] [batch_t 0.773 (0.770)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-03 07:49:07,457 - Train: 1.25% [61800/4942000] [12.5/1000.0] [batch_t 0.767 (0.770)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-03 07:50:24,583 - Train: 1.25% [61900/4942000] [12.5/1000.0] [batch_t 0.763 (0.771)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-03 07:51:41,564 - Train: 1.25% [62000/4942000] [12.5/1000.0] [batch_t 0.772 (0.770)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-03 07:52:58,610 - Train: 1.26% [62100/4942000] [12.6/1000.0] [batch_t 0.757 (0.770)] [data_t 0.002] [optim_t 0.754] [lr 0.005000] 2024-04-03 07:54:15,557 - Train: 1.26% [62200/4942000] [12.6/1000.0] [batch_t 0.764 (0.769)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-03 07:55:32,570 - Train: 1.26% [62300/4942000] [12.6/1000.0] [batch_t 0.777 (0.770)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-03 07:56:49,556 - Train: 1.26% [62400/4942000] [12.6/1000.0] [batch_t 0.767 (0.770)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-03 07:58:06,580 - Train: 1.26% [62500/4942000] [12.6/1000.0] [batch_t 0.771 (0.770)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-03 07:59:23,632 - Train: 1.27% [62600/4942000] [12.7/1000.0] [batch_t 0.763 (0.770)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-03 08:00:41,088 - Train: 1.27% [62700/4942000] [12.7/1000.0] [batch_t 0.773 (0.774)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-03 08:01:58,085 - Train: 1.27% [62800/4942000] [12.7/1000.0] [batch_t 0.767 (0.770)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 08:03:15,024 - Train: 1.27% [62900/4942000] [12.7/1000.0] [batch_t 0.767 (0.769)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 08:04:31,953 - Train: 1.27% [63000/4942000] [12.7/1000.0] [batch_t 0.767 (0.769)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 08:05:48,997 - Train: 1.28% [63100/4942000] [12.8/1000.0] [batch_t 0.772 (0.770)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-03 08:07:05,946 - Train: 1.28% [63200/4942000] [12.8/1000.0] [batch_t 0.773 (0.769)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-03 08:08:23,011 - Train: 1.28% [63300/4942000] [12.8/1000.0] [batch_t 0.777 (0.771)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-03 08:09:40,012 - Train: 1.28% [63400/4942000] [12.8/1000.0] [batch_t 0.767 (0.770)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 08:10:56,944 - Train: 1.28% [63500/4942000] [12.8/1000.0] [batch_t 0.771 (0.769)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-03 08:12:13,808 - Train: 1.29% [63600/4942000] [12.9/1000.0] [batch_t 0.767 (0.769)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 08:13:30,768 - Train: 1.29% [63700/4942000] [12.9/1000.0] [batch_t 0.773 (0.770)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-03 08:14:47,673 - Train: 1.29% [63800/4942000] [12.9/1000.0] [batch_t 0.771 (0.769)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-03 08:16:04,704 - Train: 1.29% [63900/4942000] [12.9/1000.0] [batch_t 0.761 (0.770)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-03 08:17:21,703 - Train: 1.30% [64000/4942000] [13.0/1000.0] [batch_t 0.776 (0.770)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-03 08:18:38,551 - Train: 1.30% [64100/4942000] [13.0/1000.0] [batch_t 0.766 (0.768)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-03 08:19:55,519 - Train: 1.30% [64200/4942000] [13.0/1000.0] [batch_t 0.763 (0.770)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-03 08:20:31,049 - ==> Total time: 14:23:10 Eta: 45 days, 12:14:33 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 08:21:14,295 - Train: 1.30% [64300/4942000] [13.0/1000.0] [batch_t 0.754 (0.774)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-03 08:22:31,246 - Train: 1.30% [64400/4942000] [13.0/1000.0] [batch_t 0.766 (0.769)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 08:23:48,225 - Train: 1.31% [64500/4942000] [13.1/1000.0] [batch_t 0.779 (0.770)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-03 08:25:05,186 - Train: 1.31% [64600/4942000] [13.1/1000.0] [batch_t 0.771 (0.770)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-03 08:26:22,112 - Train: 1.31% [64700/4942000] [13.1/1000.0] [batch_t 0.768 (0.769)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-03 08:27:39,029 - Train: 1.31% [64800/4942000] [13.1/1000.0] [batch_t 0.746 (0.769)] [data_t 0.002] [optim_t 0.744] [lr 0.005000] 2024-04-03 08:28:55,985 - Train: 1.31% [64900/4942000] [13.1/1000.0] [batch_t 0.776 (0.769)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-03 08:30:12,935 - Train: 1.32% [65000/4942000] [13.2/1000.0] [batch_t 0.762 (0.769)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-03 08:31:29,933 - Train: 1.32% [65100/4942000] [13.2/1000.0] [batch_t 0.767 (0.770)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 08:32:46,867 - Train: 1.32% [65200/4942000] [13.2/1000.0] [batch_t 0.767 (0.769)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 08:34:03,852 - Train: 1.32% [65300/4942000] [13.2/1000.0] [batch_t 0.765 (0.770)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-03 08:35:20,815 - Train: 1.32% [65400/4942000] [13.2/1000.0] [batch_t 0.770 (0.770)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-03 08:36:37,834 - Train: 1.33% [65500/4942000] [13.3/1000.0] [batch_t 0.763 (0.770)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-03 08:37:54,815 - Train: 1.33% [65600/4942000] [13.3/1000.0] [batch_t 0.775 (0.770)] [data_t 0.003] [optim_t 0.772] [lr 0.005000] 2024-04-03 08:39:11,693 - Train: 1.33% [65700/4942000] [13.3/1000.0] [batch_t 0.767 (0.769)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 08:40:28,566 - Train: 1.33% [65800/4942000] [13.3/1000.0] [batch_t 0.772 (0.769)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-03 08:41:45,614 - Train: 1.33% [65900/4942000] [13.3/1000.0] [batch_t 0.759 (0.770)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-03 08:43:02,459 - Train: 1.34% [66000/4942000] [13.4/1000.0] [batch_t 0.773 (0.768)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-03 08:44:19,382 - Train: 1.34% [66100/4942000] [13.4/1000.0] [batch_t 0.770 (0.769)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-03 08:45:36,307 - Train: 1.34% [66200/4942000] [13.4/1000.0] [batch_t 0.772 (0.769)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-03 08:46:53,315 - Train: 1.34% [66300/4942000] [13.4/1000.0] [batch_t 0.773 (0.770)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-03 08:48:10,284 - Train: 1.34% [66400/4942000] [13.4/1000.0] [batch_t 0.758 (0.770)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-03 08:49:27,174 - Train: 1.35% [66500/4942000] [13.5/1000.0] [batch_t 0.766 (0.769)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-03 08:50:44,162 - Train: 1.35% [66600/4942000] [13.5/1000.0] [batch_t 0.773 (0.770)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-03 08:52:01,109 - Train: 1.35% [66700/4942000] [13.5/1000.0] [batch_t 0.761 (0.769)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-03 08:53:17,996 - Train: 1.35% [66800/4942000] [13.5/1000.0] [batch_t 0.777 (0.769)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-03 08:54:34,948 - Train: 1.35% [66900/4942000] [13.5/1000.0] [batch_t 0.766 (0.769)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-03 08:55:51,899 - Train: 1.36% [67000/4942000] [13.6/1000.0] [batch_t 0.772 (0.769)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-03 08:57:08,849 - Train: 1.36% [67100/4942000] [13.6/1000.0] [batch_t 0.771 (0.769)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-03 08:58:25,718 - Train: 1.36% [67200/4942000] [13.6/1000.0] [batch_t 0.768 (0.769)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-03 08:59:42,608 - Train: 1.36% [67300/4942000] [13.6/1000.0] [batch_t 0.767 (0.769)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-03 09:00:59,638 - Train: 1.36% [67400/4942000] [13.6/1000.0] [batch_t 0.767 (0.770)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 09:02:16,468 - Train: 1.37% [67500/4942000] [13.7/1000.0] [batch_t 0.768 (0.768)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-03 09:03:33,431 - Train: 1.37% [67600/4942000] [13.7/1000.0] [batch_t 0.772 (0.770)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-03 09:04:50,392 - Train: 1.37% [67700/4942000] [13.7/1000.0] [batch_t 0.774 (0.770)] [data_t 0.003] [optim_t 0.772] [lr 0.005000] 2024-04-03 09:06:07,150 - Train: 1.37% [67800/4942000] [13.7/1000.0] [batch_t 0.767 (0.767)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 09:07:24,103 - Train: 1.37% [67900/4942000] [13.7/1000.0] [batch_t 0.772 (0.769)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-03 09:08:41,035 - Train: 1.38% [68000/4942000] [13.8/1000.0] [batch_t 0.760 (0.769)] [data_t 0.002] [optim_t 0.758] [lr 0.005000] 2024-04-03 09:09:57,944 - Train: 1.38% [68100/4942000] [13.8/1000.0] [batch_t 0.763 (0.769)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-03 09:11:14,945 - Train: 1.38% [68200/4942000] [13.8/1000.0] [batch_t 0.773 (0.770)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-03 09:12:31,798 - Train: 1.38% [68300/4942000] [13.8/1000.0] [batch_t 0.766 (0.768)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-03 09:13:48,698 - Train: 1.38% [68400/4942000] [13.8/1000.0] [batch_t 0.766 (0.769)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-03 09:15:05,610 - Train: 1.39% [68500/4942000] [13.9/1000.0] [batch_t 0.772 (0.769)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-03 09:16:22,471 - Train: 1.39% [68600/4942000] [13.9/1000.0] [batch_t 0.772 (0.769)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-03 09:17:39,469 - Train: 1.39% [68700/4942000] [13.9/1000.0] [batch_t 0.763 (0.770)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-03 09:18:56,396 - Train: 1.39% [68800/4942000] [13.9/1000.0] [batch_t 0.777 (0.769)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-03 09:20:13,223 - Train: 1.39% [68900/4942000] [13.9/1000.0] [batch_t 0.757 (0.768)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-03 09:21:30,139 - Train: 1.40% [69000/4942000] [14.0/1000.0] [batch_t 0.767 (0.769)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 09:22:46,976 - Train: 1.40% [69100/4942000] [14.0/1000.0] [batch_t 0.752 (0.768)] [data_t 0.003] [optim_t 0.749] [lr 0.005000] 2024-04-03 09:23:54,655 - ==> Total time: 15:26:33 Eta: 45 days, 7:36:34 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 09:24:05,675 - Train: 1.40% [69200/4942000] [14.0/1000.0] [batch_t 0.769 (0.787)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-03 09:25:22,627 - Train: 1.40% [69300/4942000] [14.0/1000.0] [batch_t 0.753 (0.769)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-03 09:26:39,430 - Train: 1.40% [69400/4942000] [14.0/1000.0] [batch_t 0.761 (0.768)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-03 09:27:56,352 - Train: 1.41% [69500/4942000] [14.1/1000.0] [batch_t 0.764 (0.769)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-03 09:28:42,306 - Train: 1.41% [69600/4942000] [14.1/1000.0] [batch_t 0.330 (0.459)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:29:17,486 - Train: 1.41% [69700/4942000] [14.1/1000.0] [batch_t 0.327 (0.352)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 09:29:50,401 - Train: 1.41% [69800/4942000] [14.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 09:30:23,317 - Train: 1.41% [69900/4942000] [14.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:30:56,245 - Train: 1.42% [70000/4942000] [14.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:31:30,176 - Train: 1.42% [70100/4942000] [14.2/1000.0] [batch_t 0.325 (0.339)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 09:32:03,102 - Train: 1.42% [70200/4942000] [14.2/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 09:32:36,750 - Train: 1.42% [70300/4942000] [14.2/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:33:10,140 - Train: 1.42% [70400/4942000] [14.2/1000.0] [batch_t 0.330 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:33:43,063 - Train: 1.43% [70500/4942000] [14.3/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 09:34:16,661 - Train: 1.43% [70600/4942000] [14.3/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:34:49,639 - Train: 1.43% [70700/4942000] [14.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 09:35:24,773 - Train: 1.43% [70800/4942000] [14.3/1000.0] [batch_t 0.331 (0.351)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 09:35:57,681 - Train: 1.43% [70900/4942000] [14.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:36:30,542 - Train: 1.44% [71000/4942000] [14.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:37:03,454 - Train: 1.44% [71100/4942000] [14.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 09:37:37,769 - Train: 1.44% [71200/4942000] [14.4/1000.0] [batch_t 0.330 (0.343)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:38:12,486 - Train: 1.44% [71300/4942000] [14.4/1000.0] [batch_t 0.327 (0.347)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 09:38:45,411 - Train: 1.44% [71400/4942000] [14.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 09:39:19,393 - Train: 1.45% [71500/4942000] [14.5/1000.0] [batch_t 0.331 (0.340)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 09:39:52,362 - Train: 1.45% [71600/4942000] [14.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 09:40:25,669 - Train: 1.45% [71700/4942000] [14.5/1000.0] [batch_t 0.331 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:40:58,619 - Train: 1.45% [71800/4942000] [14.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:41:32,705 - Train: 1.45% [71900/4942000] [14.5/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 09:42:08,225 - Train: 1.46% [72000/4942000] [14.6/1000.0] [batch_t 0.325 (0.355)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 09:42:41,173 - Train: 1.46% [72100/4942000] [14.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 09:43:15,146 - Train: 1.46% [72200/4942000] [14.6/1000.0] [batch_t 0.330 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 09:43:48,078 - Train: 1.46% [72300/4942000] [14.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 09:44:22,173 - Train: 1.46% [72400/4942000] [14.6/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:44:55,100 - Train: 1.47% [72500/4942000] [14.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 09:45:30,047 - Train: 1.47% [72600/4942000] [14.7/1000.0] [batch_t 0.331 (0.349)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 09:46:02,993 - Train: 1.47% [72700/4942000] [14.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 09:46:38,754 - Train: 1.47% [72800/4942000] [14.7/1000.0] [batch_t 0.329 (0.358)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 09:47:12,647 - Train: 1.48% [72900/4942000] [14.8/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:47:45,568 - Train: 1.48% [73000/4942000] [14.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 09:48:19,110 - Train: 1.48% [73100/4942000] [14.8/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:48:52,056 - Train: 1.48% [73200/4942000] [14.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:49:26,018 - Train: 1.48% [73300/4942000] [14.8/1000.0] [batch_t 0.321 (0.340)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-03 09:49:58,949 - Train: 1.49% [73400/4942000] [14.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 09:50:35,049 - Train: 1.49% [73500/4942000] [14.9/1000.0] [batch_t 0.327 (0.361)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 09:51:08,592 - Train: 1.49% [73600/4942000] [14.9/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:51:41,630 - Train: 1.49% [73700/4942000] [14.9/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 09:52:14,523 - Train: 1.49% [73800/4942000] [14.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 09:52:47,385 - Train: 1.50% [73900/4942000] [15.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:53:20,367 - Train: 1.50% [74000/4942000] [15.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:53:53,372 - Train: 1.50% [74100/4942000] [15.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 09:54:03,253 - ==> Total time: 15:56:42 Eta: 43 days, 15:03:46 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 09:54:28,655 - Train: 1.50% [74200/4942000] [15.0/1000.0] [batch_t 0.325 (0.336)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 09:55:01,568 - Train: 1.50% [74300/4942000] [15.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 09:55:35,891 - Train: 1.51% [74400/4942000] [15.1/1000.0] [batch_t 0.332 (0.343)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 09:56:12,009 - Train: 1.51% [74500/4942000] [15.1/1000.0] [batch_t 0.330 (0.361)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:56:45,055 - Train: 1.51% [74600/4942000] [15.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 09:57:19,254 - Train: 1.51% [74700/4942000] [15.1/1000.0] [batch_t 0.327 (0.342)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 09:57:52,197 - Train: 1.51% [74800/4942000] [15.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 09:58:25,449 - Train: 1.52% [74900/4942000] [15.2/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 09:58:58,379 - Train: 1.52% [75000/4942000] [15.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 09:59:32,849 - Train: 1.52% [75100/4942000] [15.2/1000.0] [batch_t 0.326 (0.345)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 10:00:07,136 - Train: 1.52% [75200/4942000] [15.2/1000.0] [batch_t 0.330 (0.343)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 10:00:40,041 - Train: 1.52% [75300/4942000] [15.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:01:12,941 - Train: 1.53% [75400/4942000] [15.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 10:01:45,817 - Train: 1.53% [75500/4942000] [15.3/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 10:02:19,643 - Train: 1.53% [75600/4942000] [15.3/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:02:52,528 - Train: 1.53% [75700/4942000] [15.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 10:03:26,930 - Train: 1.53% [75800/4942000] [15.3/1000.0] [batch_t 0.332 (0.344)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 10:03:59,858 - Train: 1.54% [75900/4942000] [15.4/1000.0] [batch_t 0.339 (0.329)] [data_t 0.003] [optim_t 0.336] [lr 0.005000] 2024-04-03 10:04:35,994 - Train: 1.54% [76000/4942000] [15.4/1000.0] [batch_t 0.332 (0.361)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 10:05:11,484 - Train: 1.54% [76100/4942000] [15.4/1000.0] [batch_t 0.329 (0.355)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 10:05:44,360 - Train: 1.54% [76200/4942000] [15.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 10:06:20,183 - Train: 1.54% [76300/4942000] [15.4/1000.0] [batch_t 0.330 (0.358)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 10:06:53,138 - Train: 1.55% [76400/4942000] [15.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 10:07:29,470 - Train: 1.55% [76500/4942000] [15.5/1000.0] [batch_t 0.329 (0.363)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:08:02,492 - Train: 1.55% [76600/4942000] [15.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:08:42,441 - Train: 1.55% [76700/4942000] [15.5/1000.0] [batch_t 0.330 (0.399)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 10:09:17,641 - Train: 1.55% [76800/4942000] [15.5/1000.0] [batch_t 0.328 (0.352)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 10:09:50,605 - Train: 1.56% [76900/4942000] [15.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 10:10:24,653 - Train: 1.56% [77000/4942000] [15.6/1000.0] [batch_t 0.333 (0.340)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 10:10:57,637 - Train: 1.56% [77100/4942000] [15.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 10:11:33,133 - Train: 1.56% [77200/4942000] [15.6/1000.0] [batch_t 0.327 (0.355)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 10:12:10,034 - Train: 1.56% [77300/4942000] [15.6/1000.0] [batch_t 0.331 (0.369)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 10:12:45,631 - Train: 1.57% [77400/4942000] [15.7/1000.0] [batch_t 0.329 (0.356)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:13:23,676 - Train: 1.57% [77500/4942000] [15.7/1000.0] [batch_t 0.328 (0.380)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 10:13:57,019 - Train: 1.57% [77600/4942000] [15.7/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 10:14:34,971 - Train: 1.57% [77700/4942000] [15.7/1000.0] [batch_t 0.329 (0.379)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:15:40,350 - Train: 1.57% [77800/4942000] [15.7/1000.0] [batch_t 0.373 (0.654)] [data_t 0.045] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:16:25,300 - Train: 1.58% [77900/4942000] [15.8/1000.0] [batch_t 0.328 (0.449)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 10:17:12,159 - Train: 1.58% [78000/4942000] [15.8/1000.0] [batch_t 1.027 (0.468)] [data_t 0.700] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:17:56,548 - Train: 1.58% [78100/4942000] [15.8/1000.0] [batch_t 0.330 (0.444)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 10:18:45,345 - Train: 1.58% [78200/4942000] [15.8/1000.0] [batch_t 0.561 (0.488)] [data_t 0.235] [optim_t 0.325] [lr 0.005000] 2024-04-03 10:19:21,790 - Train: 1.58% [78300/4942000] [15.8/1000.0] [batch_t 0.329 (0.364)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 10:20:45,954 - Train: 1.59% [78400/4942000] [15.9/1000.0] [batch_t 0.325 (0.842)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 10:21:36,349 - Train: 1.59% [78500/4942000] [15.9/1000.0] [batch_t 0.326 (0.504)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 10:22:49,068 - Train: 1.59% [78600/4942000] [15.9/1000.0] [batch_t 0.673 (0.727)] [data_t 0.349] [optim_t 0.324] [lr 0.005000] 2024-04-03 10:23:50,614 - Train: 1.59% [78700/4942000] [15.9/1000.0] [batch_t 0.329 (0.615)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:25:16,416 - Train: 1.59% [78800/4942000] [15.9/1000.0] [batch_t 1.292 (0.858)] [data_t 0.966] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:27:56,838 - Train: 1.60% [78900/4942000] [16.0/1000.0] [batch_t 3.018 (1.604)] [data_t 2.686] [optim_t 0.332] [lr 0.005000] 2024-04-03 10:29:52,561 - Train: 1.60% [79000/4942000] [16.0/1000.0] [batch_t 0.329 (1.157)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 10:30:46,525 - ==> Total time: 16:33:25 Eta: 42 days, 10:15:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 10:30:59,505 - Train: 1.60% [79100/4942000] [16.0/1000.0] [batch_t 1.201 (0.411)] [data_t 0.875] [optim_t 0.326] [lr 0.005000] 2024-04-03 10:32:19,913 - Train: 1.60% [79200/4942000] [16.0/1000.0] [batch_t 1.066 (0.804)] [data_t 0.745] [optim_t 0.321] [lr 0.005000] 2024-04-03 10:33:32,598 - Train: 1.60% [79300/4942000] [16.0/1000.0] [batch_t 0.331 (0.727)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 10:35:09,679 - Train: 1.61% [79400/4942000] [16.1/1000.0] [batch_t 0.327 (0.971)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 10:36:19,681 - Train: 1.61% [79500/4942000] [16.1/1000.0] [batch_t 0.751 (0.700)] [data_t 0.427] [optim_t 0.324] [lr 0.005000] 2024-04-03 10:36:55,907 - Train: 1.61% [79600/4942000] [16.1/1000.0] [batch_t 0.327 (0.362)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 10:37:33,164 - Train: 1.61% [79700/4942000] [16.1/1000.0] [batch_t 0.328 (0.372)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 10:38:09,062 - Train: 1.61% [79800/4942000] [16.1/1000.0] [batch_t 0.329 (0.359)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:38:41,993 - Train: 1.62% [79900/4942000] [16.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 10:39:16,840 - Train: 1.62% [80000/4942000] [16.2/1000.0] [batch_t 0.328 (0.348)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 10:39:49,759 - Train: 1.62% [80100/4942000] [16.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:40:35,019 - Train: 1.62% [80200/4942000] [16.2/1000.0] [batch_t 0.325 (0.453)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 10:41:12,321 - Train: 1.62% [80300/4942000] [16.2/1000.0] [batch_t 0.328 (0.373)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 10:41:48,562 - Train: 1.63% [80400/4942000] [16.3/1000.0] [batch_t 0.329 (0.362)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:42:25,409 - Train: 1.63% [80500/4942000] [16.3/1000.0] [batch_t 0.329 (0.368)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:43:01,045 - Train: 1.63% [80600/4942000] [16.3/1000.0] [batch_t 0.329 (0.356)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:43:54,696 - Train: 1.63% [80700/4942000] [16.3/1000.0] [batch_t 0.335 (0.536)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 10:44:32,628 - Train: 1.63% [80800/4942000] [16.3/1000.0] [batch_t 0.333 (0.379)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 10:45:09,190 - Train: 1.64% [80900/4942000] [16.4/1000.0] [batch_t 0.328 (0.366)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 10:45:42,136 - Train: 1.64% [81000/4942000] [16.4/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 10:46:26,121 - Train: 1.64% [81100/4942000] [16.4/1000.0] [batch_t 0.324 (0.440)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 10:47:44,184 - Train: 1.64% [81200/4942000] [16.4/1000.0] [batch_t 0.842 (0.781)] [data_t 0.518] [optim_t 0.323] [lr 0.005000] 2024-04-03 10:49:10,564 - Train: 1.65% [81300/4942000] [16.5/1000.0] [batch_t 1.071 (0.864)] [data_t 0.744] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:49:58,400 - Train: 1.65% [81400/4942000] [16.5/1000.0] [batch_t 0.328 (0.478)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 10:50:37,814 - Train: 1.65% [81500/4942000] [16.5/1000.0] [batch_t 0.325 (0.394)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 10:51:13,601 - Train: 1.65% [81600/4942000] [16.5/1000.0] [batch_t 0.331 (0.358)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 10:51:46,546 - Train: 1.65% [81700/4942000] [16.5/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 10:52:22,198 - Train: 1.66% [81800/4942000] [16.6/1000.0] [batch_t 0.324 (0.356)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 10:52:58,671 - Train: 1.66% [81900/4942000] [16.6/1000.0] [batch_t 0.325 (0.365)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 10:53:41,816 - Train: 1.66% [82000/4942000] [16.6/1000.0] [batch_t 0.329 (0.431)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:54:18,955 - Train: 1.66% [82100/4942000] [16.6/1000.0] [batch_t 0.331 (0.371)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-03 10:54:51,894 - Train: 1.66% [82200/4942000] [16.6/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 10:55:27,490 - Train: 1.67% [82300/4942000] [16.7/1000.0] [batch_t 0.326 (0.356)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-03 10:56:00,421 - Train: 1.67% [82400/4942000] [16.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 10:56:42,379 - Train: 1.67% [82500/4942000] [16.7/1000.0] [batch_t 0.672 (0.420)] [data_t 0.346] [optim_t 0.326] [lr 0.005000] 2024-04-03 10:57:24,131 - Train: 1.67% [82600/4942000] [16.7/1000.0] [batch_t 0.330 (0.417)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 10:57:57,135 - Train: 1.67% [82700/4942000] [16.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 10:58:33,386 - Train: 1.68% [82800/4942000] [16.8/1000.0] [batch_t 0.335 (0.362)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 10:59:10,288 - Train: 1.68% [82900/4942000] [16.8/1000.0] [batch_t 0.332 (0.369)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 10:59:43,195 - Train: 1.68% [83000/4942000] [16.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 11:00:20,159 - Train: 1.68% [83100/4942000] [16.8/1000.0] [batch_t 0.330 (0.370)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 11:00:53,115 - Train: 1.68% [83200/4942000] [16.8/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 11:01:28,923 - Train: 1.69% [83300/4942000] [16.9/1000.0] [batch_t 0.329 (0.358)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:02:01,904 - Train: 1.69% [83400/4942000] [16.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 11:02:36,817 - Train: 1.69% [83500/4942000] [16.9/1000.0] [batch_t 0.325 (0.349)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 11:03:12,467 - Train: 1.69% [83600/4942000] [16.9/1000.0] [batch_t 0.328 (0.356)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 11:03:45,371 - Train: 1.69% [83700/4942000] [16.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 11:04:19,983 - Train: 1.70% [83800/4942000] [17.0/1000.0] [batch_t 0.327 (0.346)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 11:04:52,942 - Train: 1.70% [83900/4942000] [17.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 11:05:27,756 - Train: 1.70% [84000/4942000] [17.0/1000.0] [batch_t 0.330 (0.348)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 11:05:32,376 - ==> Total time: 17:08:11 Eta: 41 days, 6:53:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 11:06:06,682 - Train: 1.70% [84100/4942000] [17.0/1000.0] [batch_t 0.326 (0.378)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 11:06:45,063 - Train: 1.70% [84200/4942000] [17.0/1000.0] [batch_t 0.331 (0.384)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 11:07:23,094 - Train: 1.71% [84300/4942000] [17.1/1000.0] [batch_t 0.330 (0.380)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 11:07:56,074 - Train: 1.71% [84400/4942000] [17.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:08:31,437 - Train: 1.71% [84500/4942000] [17.1/1000.0] [batch_t 0.330 (0.354)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 11:09:06,551 - Train: 1.71% [84600/4942000] [17.1/1000.0] [batch_t 2.322 (0.351)] [data_t 1.996] [optim_t 0.326] [lr 0.005000] 2024-04-03 11:09:40,173 - Train: 1.71% [84700/4942000] [17.1/1000.0] [batch_t 0.332 (0.336)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 11:10:14,804 - Train: 1.72% [84800/4942000] [17.2/1000.0] [batch_t 0.329 (0.346)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:10:48,963 - Train: 1.72% [84900/4942000] [17.2/1000.0] [batch_t 0.330 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:11:24,602 - Train: 1.72% [85000/4942000] [17.2/1000.0] [batch_t 0.329 (0.356)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:11:57,594 - Train: 1.72% [85100/4942000] [17.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:12:37,279 - Train: 1.72% [85200/4942000] [17.2/1000.0] [batch_t 0.329 (0.397)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 11:13:13,327 - Train: 1.73% [85300/4942000] [17.3/1000.0] [batch_t 0.330 (0.360)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 11:13:46,259 - Train: 1.73% [85400/4942000] [17.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 11:14:22,074 - Train: 1.73% [85500/4942000] [17.3/1000.0] [batch_t 0.329 (0.358)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:14:54,972 - Train: 1.73% [85600/4942000] [17.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 11:15:31,421 - Train: 1.73% [85700/4942000] [17.3/1000.0] [batch_t 0.325 (0.364)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 11:16:04,361 - Train: 1.74% [85800/4942000] [17.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 11:16:40,059 - Train: 1.74% [85900/4942000] [17.4/1000.0] [batch_t 0.330 (0.357)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 11:17:16,524 - Train: 1.74% [86000/4942000] [17.4/1000.0] [batch_t 0.330 (0.365)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 11:17:49,478 - Train: 1.74% [86100/4942000] [17.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 11:18:23,979 - Train: 1.74% [86200/4942000] [17.4/1000.0] [batch_t 0.330 (0.345)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 11:18:56,889 - Train: 1.75% [86300/4942000] [17.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 11:19:31,360 - Train: 1.75% [86400/4942000] [17.5/1000.0] [batch_t 0.332 (0.345)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 11:20:06,502 - Train: 1.75% [86500/4942000] [17.5/1000.0] [batch_t 2.552 (0.351)] [data_t 2.225] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:20:40,555 - Train: 1.75% [86600/4942000] [17.5/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 11:21:18,888 - Train: 1.75% [86700/4942000] [17.5/1000.0] [batch_t 0.331 (0.383)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 11:21:51,892 - Train: 1.76% [86800/4942000] [17.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 11:22:28,766 - Train: 1.76% [86900/4942000] [17.6/1000.0] [batch_t 0.337 (0.369)] [data_t 0.003] [optim_t 0.335] [lr 0.005000] 2024-04-03 11:23:01,693 - Train: 1.76% [87000/4942000] [17.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 11:23:37,244 - Train: 1.76% [87100/4942000] [17.6/1000.0] [batch_t 0.330 (0.355)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 11:24:13,395 - Train: 1.76% [87200/4942000] [17.6/1000.0] [batch_t 0.328 (0.361)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 11:24:46,317 - Train: 1.77% [87300/4942000] [17.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 11:25:20,949 - Train: 1.77% [87400/4942000] [17.7/1000.0] [batch_t 0.327 (0.346)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 11:25:53,988 - Train: 1.77% [87500/4942000] [17.7/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 11:26:28,878 - Train: 1.77% [87600/4942000] [17.7/1000.0] [batch_t 0.327 (0.349)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 11:27:01,764 - Train: 1.77% [87700/4942000] [17.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 11:27:37,119 - Train: 1.78% [87800/4942000] [17.8/1000.0] [batch_t 0.328 (0.353)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 11:28:12,399 - Train: 1.78% [87900/4942000] [17.8/1000.0] [batch_t 0.329 (0.353)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:28:45,316 - Train: 1.78% [88000/4942000] [17.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 11:29:21,735 - Train: 1.78% [88100/4942000] [17.8/1000.0] [batch_t 0.330 (0.364)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 11:29:54,649 - Train: 1.78% [88200/4942000] [17.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:30:29,933 - Train: 1.79% [88300/4942000] [17.9/1000.0] [batch_t 0.332 (0.353)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 11:31:02,844 - Train: 1.79% [88400/4942000] [17.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 11:31:38,141 - Train: 1.79% [88500/4942000] [17.9/1000.0] [batch_t 0.327 (0.353)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 11:32:13,383 - Train: 1.79% [88600/4942000] [17.9/1000.0] [batch_t 0.330 (0.352)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 11:32:46,322 - Train: 1.79% [88700/4942000] [17.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:33:22,583 - Train: 1.80% [88800/4942000] [18.0/1000.0] [batch_t 0.327 (0.363)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 11:33:55,596 - Train: 1.80% [88900/4942000] [18.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 11:34:18,877 - ==> Total time: 17:36:58 Eta: 40 days, 1:03:27 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 11:34:36,100 - Train: 1.80% [89000/4942000] [18.0/1000.0] [batch_t 0.329 (0.347)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 11:35:12,444 - Train: 1.80% [89100/4942000] [18.0/1000.0] [batch_t 0.325 (0.363)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 11:35:45,364 - Train: 1.80% [89200/4942000] [18.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:36:19,100 - Train: 1.81% [89300/4942000] [18.1/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 11:36:56,625 - Train: 1.81% [89400/4942000] [18.1/1000.0] [batch_t 0.329 (0.375)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:37:34,194 - Train: 1.81% [89500/4942000] [18.1/1000.0] [batch_t 0.330 (0.376)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:38:11,169 - Train: 1.81% [89600/4942000] [18.1/1000.0] [batch_t 0.328 (0.370)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 11:38:53,482 - Train: 1.82% [89700/4942000] [18.2/1000.0] [batch_t 0.327 (0.423)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 11:39:59,450 - Train: 1.82% [89800/4942000] [18.2/1000.0] [batch_t 0.637 (0.660)] [data_t 0.310] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:40:55,453 - Train: 1.82% [89900/4942000] [18.2/1000.0] [batch_t 0.329 (0.560)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:55:54,625 - Train: 1.82% [90000/4942000] [18.2/1000.0] [batch_t 0.773 (8.992)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-03 11:58:27,064 - Train: 1.82% [90100/4942000] [18.2/1000.0] [batch_t 0.771 (1.524)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-03 11:59:51,883 - Train: 1.83% [90200/4942000] [18.3/1000.0] [batch_t 0.759 (0.848)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-03 12:01:16,456 - Train: 1.83% [90300/4942000] [18.3/1000.0] [batch_t 0.766 (0.846)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-03 12:02:33,435 - Train: 1.83% [90400/4942000] [18.3/1000.0] [batch_t 0.768 (0.770)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-03 12:03:50,384 - Train: 1.83% [90500/4942000] [18.3/1000.0] [batch_t 0.764 (0.769)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-03 12:05:07,362 - Train: 1.83% [90600/4942000] [18.3/1000.0] [batch_t 0.764 (0.770)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-03 12:06:28,794 - Train: 1.84% [90700/4942000] [18.4/1000.0] [batch_t 0.766 (0.814)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-03 12:07:45,642 - Train: 1.84% [90800/4942000] [18.4/1000.0] [batch_t 0.768 (0.768)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-03 12:09:02,791 - Train: 1.84% [90900/4942000] [18.4/1000.0] [batch_t 0.766 (0.771)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-03 12:10:20,857 - Train: 1.84% [91000/4942000] [18.4/1000.0] [batch_t 0.768 (0.781)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-03 12:11:37,814 - Train: 1.84% [91100/4942000] [18.4/1000.0] [batch_t 0.775 (0.769)] [data_t 0.003] [optim_t 0.772] [lr 0.005000] 2024-04-03 12:12:54,753 - Train: 1.85% [91200/4942000] [18.5/1000.0] [batch_t 0.767 (0.769)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 12:14:11,728 - Train: 1.85% [91300/4942000] [18.5/1000.0] [batch_t 0.777 (0.770)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-03 12:15:28,640 - Train: 1.85% [91400/4942000] [18.5/1000.0] [batch_t 0.772 (0.769)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-03 12:16:45,565 - Train: 1.85% [91500/4942000] [18.5/1000.0] [batch_t 0.768 (0.769)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-03 12:18:02,380 - Train: 1.85% [91600/4942000] [18.5/1000.0] [batch_t 0.767 (0.768)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-03 12:19:19,418 - Train: 1.86% [91700/4942000] [18.6/1000.0] [batch_t 0.776 (0.770)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-03 12:20:36,286 - Train: 1.86% [91800/4942000] [18.6/1000.0] [batch_t 0.766 (0.769)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-03 12:21:53,335 - Train: 1.86% [91900/4942000] [18.6/1000.0] [batch_t 0.759 (0.770)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-03 12:23:10,337 - Train: 1.86% [92000/4942000] [18.6/1000.0] [batch_t 0.777 (0.770)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-03 12:24:27,353 - Train: 1.86% [92100/4942000] [18.6/1000.0] [batch_t 0.770 (0.770)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-03 12:25:44,266 - Train: 1.87% [92200/4942000] [18.7/1000.0] [batch_t 0.757 (0.769)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-03 12:27:01,160 - Train: 1.87% [92300/4942000] [18.7/1000.0] [batch_t 0.772 (0.769)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-03 12:28:18,105 - Train: 1.87% [92400/4942000] [18.7/1000.0] [batch_t 0.769 (0.769)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-03 12:28:59,409 - Train: 1.87% [92500/4942000] [18.7/1000.0] [batch_t 0.330 (0.413)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:29:32,382 - Train: 1.87% [92600/4942000] [18.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 12:30:05,306 - Train: 1.88% [92700/4942000] [18.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 12:30:38,266 - Train: 1.88% [92800/4942000] [18.8/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-03 12:31:14,423 - Train: 1.88% [92900/4942000] [18.8/1000.0] [batch_t 0.329 (0.361)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:31:47,412 - Train: 1.88% [93000/4942000] [18.8/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 12:32:20,653 - Train: 1.88% [93100/4942000] [18.8/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 12:32:53,815 - Train: 1.89% [93200/4942000] [18.9/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 12:33:26,762 - Train: 1.89% [93300/4942000] [18.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 12:33:59,722 - Train: 1.89% [93400/4942000] [18.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-03 12:34:32,638 - Train: 1.89% [93500/4942000] [18.9/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 12:35:08,173 - Train: 1.89% [93600/4942000] [18.9/1000.0] [batch_t 0.334 (0.355)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 12:35:41,247 - Train: 1.90% [93700/4942000] [19.0/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:36:15,600 - Train: 1.90% [93800/4942000] [19.0/1000.0] [batch_t 0.330 (0.343)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 12:36:47,881 - ==> Total time: 18:39:27 Eta: 40 days, 3:19:01 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 12:36:50,038 - Train: 1.90% [93900/4942000] [19.0/1000.0] [batch_t 0.327 (0.421)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 12:37:22,969 - Train: 1.90% [94000/4942000] [19.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 12:37:55,881 - Train: 1.90% [94100/4942000] [19.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 12:38:28,832 - Train: 1.91% [94200/4942000] [19.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 12:39:01,763 - Train: 1.91% [94300/4942000] [19.1/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 12:39:34,765 - Train: 1.91% [94400/4942000] [19.1/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 12:40:07,799 - Train: 1.91% [94500/4942000] [19.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:40:40,753 - Train: 1.91% [94600/4942000] [19.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 12:41:15,875 - Train: 1.92% [94700/4942000] [19.2/1000.0] [batch_t 0.327 (0.351)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 12:41:48,807 - Train: 1.92% [94800/4942000] [19.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 12:42:21,830 - Train: 1.92% [94900/4942000] [19.2/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 12:42:54,714 - Train: 1.92% [95000/4942000] [19.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:43:27,752 - Train: 1.92% [95100/4942000] [19.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:44:00,701 - Train: 1.93% [95200/4942000] [19.3/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 12:44:34,768 - Train: 1.93% [95300/4942000] [19.3/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:45:08,558 - Train: 1.93% [95400/4942000] [19.3/1000.0] [batch_t 0.333 (0.338)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 12:45:41,474 - Train: 1.93% [95500/4942000] [19.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 12:46:14,461 - Train: 1.93% [95600/4942000] [19.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-03 12:46:47,375 - Train: 1.94% [95700/4942000] [19.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 12:47:20,918 - Train: 1.94% [95800/4942000] [19.4/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 12:47:53,934 - Train: 1.94% [95900/4942000] [19.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:48:27,520 - Train: 1.94% [96000/4942000] [19.4/1000.0] [batch_t 0.331 (0.336)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 12:49:00,455 - Train: 1.94% [96100/4942000] [19.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 12:49:34,640 - Train: 1.95% [96200/4942000] [19.5/1000.0] [batch_t 0.328 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 12:50:08,635 - Train: 1.95% [96300/4942000] [19.5/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:50:41,536 - Train: 1.95% [96400/4942000] [19.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:51:15,740 - Train: 1.95% [96500/4942000] [19.5/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:51:48,643 - Train: 1.95% [96600/4942000] [19.5/1000.0] [batch_t 0.326 (0.329)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-03 12:52:21,595 - Train: 1.96% [96700/4942000] [19.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 12:52:54,611 - Train: 1.96% [96800/4942000] [19.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 12:53:27,556 - Train: 1.96% [96900/4942000] [19.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:54:00,501 - Train: 1.96% [97000/4942000] [19.6/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 12:54:34,894 - Train: 1.96% [97100/4942000] [19.6/1000.0] [batch_t 0.329 (0.344)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:55:08,841 - Train: 1.97% [97200/4942000] [19.7/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:55:41,827 - Train: 1.97% [97300/4942000] [19.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 12:56:14,944 - Train: 1.97% [97400/4942000] [19.7/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:56:47,882 - Train: 1.97% [97500/4942000] [19.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:57:20,829 - Train: 1.97% [97600/4942000] [19.7/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 12:57:53,741 - Train: 1.98% [97700/4942000] [19.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 12:58:26,993 - Train: 1.98% [97800/4942000] [19.8/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 12:58:59,953 - Train: 1.98% [97900/4942000] [19.8/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 12:59:33,710 - Train: 1.98% [98000/4942000] [19.8/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:00:09,535 - Train: 1.99% [98100/4942000] [19.9/1000.0] [batch_t 0.325 (0.358)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 13:00:42,466 - Train: 1.99% [98200/4942000] [19.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:01:15,843 - Train: 1.99% [98300/4942000] [19.9/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:01:48,754 - Train: 1.99% [98400/4942000] [19.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 13:02:22,782 - Train: 1.99% [98500/4942000] [19.9/1000.0] [batch_t 0.331 (0.340)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 13:02:55,742 - Train: 2.00% [98600/4942000] [20.0/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 13:03:29,136 - Train: 2.00% [98700/4942000] [20.0/1000.0] [batch_t 0.330 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 13:04:02,240 - Train: 2.00% [98800/4942000] [20.0/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 13:04:15,932 - ==> Total time: 19:06:55 Eta: 39 days, 0:39:00 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 13:04:37,110 - Train: 2.00% [98900/4942000] [20.0/1000.0] [batch_t 0.333 (0.332)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 13:05:11,684 - Train: 2.00% [99000/4942000] [20.0/1000.0] [batch_t 0.327 (0.346)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 13:05:44,627 - Train: 2.01% [99100/4942000] [20.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 13:06:17,602 - Train: 2.01% [99200/4942000] [20.1/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 13:06:50,557 - Train: 2.01% [99300/4942000] [20.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:07:26,079 - Train: 2.01% [99400/4942000] [20.1/1000.0] [batch_t 0.334 (0.355)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 13:07:59,003 - Train: 2.01% [99500/4942000] [20.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:08:32,286 - Train: 2.02% [99600/4942000] [20.2/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:09:07,838 - Train: 2.02% [99700/4942000] [20.2/1000.0] [batch_t 0.330 (0.355)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:09:40,762 - Train: 2.02% [99800/4942000] [20.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 13:10:14,618 - Train: 2.02% [99900/4942000] [20.2/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:10:47,547 - Train: 2.02% [100000/4942000] [20.2/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 13:11:21,351 - Train: 2.03% [100100/4942000] [20.3/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:11:54,378 - Train: 2.03% [100200/4942000] [20.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:12:28,092 - Train: 2.03% [100300/4942000] [20.3/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:13:01,064 - Train: 2.03% [100400/4942000] [20.3/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 13:13:35,096 - Train: 2.03% [100500/4942000] [20.3/1000.0] [batch_t 0.329 (0.340)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:14:08,095 - Train: 2.04% [100600/4942000] [20.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 13:14:41,062 - Train: 2.04% [100700/4942000] [20.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:15:14,599 - Train: 2.04% [100800/4942000] [20.4/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:15:47,576 - Train: 2.04% [100900/4942000] [20.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:16:20,528 - Train: 2.04% [101000/4942000] [20.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 13:16:53,513 - Train: 2.05% [101100/4942000] [20.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:17:26,504 - Train: 2.05% [101200/4942000] [20.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:17:59,416 - Train: 2.05% [101300/4942000] [20.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:18:32,344 - Train: 2.05% [101400/4942000] [20.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 13:19:05,268 - Train: 2.05% [101500/4942000] [20.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 13:19:38,409 - Train: 2.06% [101600/4942000] [20.6/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 13:20:11,377 - Train: 2.06% [101700/4942000] [20.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 13:20:44,305 - Train: 2.06% [101800/4942000] [20.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:21:17,216 - Train: 2.06% [101900/4942000] [20.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 13:21:50,138 - Train: 2.06% [102000/4942000] [20.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 13:22:23,038 - Train: 2.07% [102100/4942000] [20.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 13:22:55,961 - Train: 2.07% [102200/4942000] [20.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 13:23:28,933 - Train: 2.07% [102300/4942000] [20.7/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 13:24:01,852 - Train: 2.07% [102400/4942000] [20.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 13:24:34,760 - Train: 2.07% [102500/4942000] [20.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 13:25:07,686 - Train: 2.08% [102600/4942000] [20.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 13:25:40,606 - Train: 2.08% [102700/4942000] [20.8/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 13:26:13,526 - Train: 2.08% [102800/4942000] [20.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 13:26:46,441 - Train: 2.08% [102900/4942000] [20.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:27:19,477 - Train: 2.08% [103000/4942000] [20.8/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 13:27:52,360 - Train: 2.09% [103100/4942000] [20.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 13:28:25,800 - Train: 2.09% [103200/4942000] [20.9/1000.0] [batch_t 0.330 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 13:28:58,706 - Train: 2.09% [103300/4942000] [20.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 13:29:31,771 - Train: 2.09% [103400/4942000] [20.9/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 13:30:04,707 - Train: 2.09% [103500/4942000] [20.9/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 13:30:38,059 - Train: 2.10% [103600/4942000] [21.0/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 13:31:10,966 - Train: 2.10% [103700/4942000] [21.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:31:37,969 - ==> Total time: 19:34:17 Eta: 38 days, 0:24:05 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 13:31:45,487 - Train: 2.10% [103800/4942000] [21.0/1000.0] [batch_t 0.329 (0.346)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:32:18,437 - Train: 2.10% [103900/4942000] [21.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:32:51,394 - Train: 2.10% [104000/4942000] [21.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 13:33:24,351 - Train: 2.11% [104100/4942000] [21.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:33:57,258 - Train: 2.11% [104200/4942000] [21.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 13:34:30,184 - Train: 2.11% [104300/4942000] [21.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 13:35:03,213 - Train: 2.11% [104400/4942000] [21.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:35:37,337 - Train: 2.11% [104500/4942000] [21.1/1000.0] [batch_t 0.333 (0.341)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 13:36:10,224 - Train: 2.12% [104600/4942000] [21.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 13:36:43,168 - Train: 2.12% [104700/4942000] [21.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:37:16,706 - Train: 2.12% [104800/4942000] [21.2/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 13:37:49,620 - Train: 2.12% [104900/4942000] [21.2/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 13:38:22,523 - Train: 2.12% [105000/4942000] [21.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 13:38:55,436 - Train: 2.13% [105100/4942000] [21.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:39:29,038 - Train: 2.13% [105200/4942000] [21.3/1000.0] [batch_t 0.331 (0.336)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 13:40:01,987 - Train: 2.13% [105300/4942000] [21.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:40:35,326 - Train: 2.13% [105400/4942000] [21.3/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 13:41:08,259 - Train: 2.13% [105500/4942000] [21.3/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 13:41:41,121 - Train: 2.14% [105600/4942000] [21.4/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 13:42:13,982 - Train: 2.14% [105700/4942000] [21.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 13:42:46,967 - Train: 2.14% [105800/4942000] [21.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:43:19,913 - Train: 2.14% [105900/4942000] [21.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 13:43:52,881 - Train: 2.14% [106000/4942000] [21.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:44:25,848 - Train: 2.15% [106100/4942000] [21.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:44:58,776 - Train: 2.15% [106200/4942000] [21.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 13:45:31,709 - Train: 2.15% [106300/4942000] [21.5/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-03 13:46:04,639 - Train: 2.15% [106400/4942000] [21.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 13:46:37,583 - Train: 2.15% [106500/4942000] [21.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 13:47:10,562 - Train: 2.16% [106600/4942000] [21.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 13:47:43,484 - Train: 2.16% [106700/4942000] [21.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:48:16,425 - Train: 2.16% [106800/4942000] [21.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:48:49,411 - Train: 2.16% [106900/4942000] [21.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 13:49:23,055 - Train: 2.17% [107000/4942000] [21.7/1000.0] [batch_t 0.333 (0.336)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 13:49:56,165 - Train: 2.17% [107100/4942000] [21.7/1000.0] [batch_t 0.334 (0.331)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 13:50:30,397 - Train: 2.17% [107200/4942000] [21.7/1000.0] [batch_t 0.328 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:51:03,317 - Train: 2.17% [107300/4942000] [21.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:51:38,969 - Train: 2.17% [107400/4942000] [21.7/1000.0] [batch_t 0.325 (0.356)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 13:52:13,464 - Train: 2.18% [107500/4942000] [21.8/1000.0] [batch_t 0.331 (0.345)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 13:52:46,363 - Train: 2.18% [107600/4942000] [21.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:53:19,363 - Train: 2.18% [107700/4942000] [21.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-03 13:53:52,482 - Train: 2.18% [107800/4942000] [21.8/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:54:26,425 - Train: 2.18% [107900/4942000] [21.8/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:54:59,343 - Train: 2.19% [108000/4942000] [21.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:55:32,254 - Train: 2.19% [108100/4942000] [21.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 13:56:06,015 - Train: 2.19% [108200/4942000] [21.9/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 13:56:39,236 - Train: 2.19% [108300/4942000] [21.9/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 13:57:12,174 - Train: 2.19% [108400/4942000] [21.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:57:45,169 - Train: 2.20% [108500/4942000] [22.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:58:18,070 - Train: 2.20% [108600/4942000] [22.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:58:51,132 - Train: 2.20% [108700/4942000] [22.0/1000.0] [batch_t 0.325 (0.331)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 13:58:59,064 - ==> Total time: 20:01:38 Eta: 37 days, 2:18:14 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 13:59:26,100 - Train: 2.20% [108800/4942000] [22.0/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 13:59:58,991 - Train: 2.20% [108900/4942000] [22.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 14:00:32,463 - Train: 2.21% [109000/4942000] [22.1/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 14:01:05,812 - Train: 2.21% [109100/4942000] [22.1/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 14:01:38,732 - Train: 2.21% [109200/4942000] [22.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 14:02:12,250 - Train: 2.21% [109300/4942000] [22.1/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 14:02:45,138 - Train: 2.21% [109400/4942000] [22.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 14:03:18,115 - Train: 2.22% [109500/4942000] [22.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 14:03:51,050 - Train: 2.22% [109600/4942000] [22.2/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 14:04:23,960 - Train: 2.22% [109700/4942000] [22.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 14:04:56,877 - Train: 2.22% [109800/4942000] [22.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 14:05:29,771 - Train: 2.22% [109900/4942000] [22.2/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 14:06:02,805 - Train: 2.23% [110000/4942000] [22.3/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-03 14:06:36,599 - Train: 2.23% [110100/4942000] [22.3/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 14:07:10,085 - Train: 2.23% [110200/4942000] [22.3/1000.0] [batch_t 0.331 (0.335)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 14:07:43,050 - Train: 2.23% [110300/4942000] [22.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 14:08:16,364 - Train: 2.23% [110400/4942000] [22.3/1000.0] [batch_t 0.325 (0.333)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 14:08:49,275 - Train: 2.24% [110500/4942000] [22.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 14:09:22,237 - Train: 2.24% [110600/4942000] [22.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 14:09:55,223 - Train: 2.24% [110700/4942000] [22.4/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 14:10:28,164 - Train: 2.24% [110800/4942000] [22.4/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 14:11:01,053 - Train: 2.24% [110900/4942000] [22.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 14:11:34,664 - Train: 2.25% [111000/4942000] [22.5/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 14:12:08,382 - Train: 2.25% [111100/4942000] [22.5/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 14:12:41,379 - Train: 2.25% [111200/4942000] [22.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 14:13:14,415 - Train: 2.25% [111300/4942000] [22.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 14:13:47,468 - Train: 2.25% [111400/4942000] [22.5/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 14:14:24,190 - Train: 2.26% [111500/4942000] [22.6/1000.0] [batch_t 0.330 (0.367)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-03 14:14:57,107 - Train: 2.26% [111600/4942000] [22.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 14:15:31,105 - Train: 2.26% [111700/4942000] [22.6/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 14:20:32,873 - Train: 2.26% [111800/4942000] [22.6/1000.0] [batch_t 0.323 (3.018)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-03 14:21:06,910 - Train: 2.26% [111900/4942000] [22.6/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 14:21:39,800 - Train: 2.27% [112000/4942000] [22.7/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 14:22:14,703 - Train: 2.27% [112100/4942000] [22.7/1000.0] [batch_t 0.327 (0.349)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 14:22:47,638 - Train: 2.27% [112200/4942000] [22.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-03 14:23:22,325 - Train: 2.27% [112300/4942000] [22.7/1000.0] [batch_t 0.329 (0.347)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 14:23:55,311 - Train: 2.27% [112400/4942000] [22.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 14:24:29,007 - Train: 2.28% [112500/4942000] [22.8/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 14:25:01,899 - Train: 2.28% [112600/4942000] [22.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 14:25:38,953 - Train: 2.28% [112700/4942000] [22.8/1000.0] [batch_t 0.328 (0.370)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 14:26:15,414 - Train: 2.28% [112800/4942000] [22.8/1000.0] [batch_t 0.328 (0.365)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 14:26:48,502 - Train: 2.28% [112900/4942000] [22.8/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 14:27:22,564 - Train: 2.29% [113000/4942000] [22.9/1000.0] [batch_t 0.334 (0.341)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 14:27:55,638 - Train: 2.29% [113100/4942000] [22.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 14:28:31,101 - Train: 2.29% [113200/4942000] [22.9/1000.0] [batch_t 0.327 (0.355)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 14:29:04,015 - Train: 2.29% [113300/4942000] [22.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 14:29:38,008 - Train: 2.29% [113400/4942000] [22.9/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 14:30:14,111 - Train: 2.30% [113500/4942000] [23.0/1000.0] [batch_t 0.334 (0.361)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-03 14:31:13,582 - Train: 2.30% [113600/4942000] [23.0/1000.0] [batch_t 0.804 (0.595)] [data_t 0.479] [optim_t 0.324] [lr 0.005000] 2024-04-03 14:40:56,452 - ==> Total time: 20:43:35 Eta: 36 days, 16:25:42 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 14:41:31,666 - Train: 2.30% [113700/4942000] [23.0/1000.0] [batch_t 0.761 (0.986)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-03 14:42:49,585 - Train: 2.30% [113800/4942000] [23.0/1000.0] [batch_t 0.767 (0.779)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 14:44:06,556 - Train: 2.30% [113900/4942000] [23.0/1000.0] [batch_t 0.768 (0.770)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-03 14:45:23,376 - Train: 2.31% [114000/4942000] [23.1/1000.0] [batch_t 0.762 (0.768)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-03 14:46:40,315 - Train: 2.31% [114100/4942000] [23.1/1000.0] [batch_t 0.782 (0.769)] [data_t 0.003] [optim_t 0.780] [lr 0.005000] 2024-04-03 14:47:57,267 - Train: 2.31% [114200/4942000] [23.1/1000.0] [batch_t 0.772 (0.769)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-03 14:49:14,315 - Train: 2.31% [114300/4942000] [23.1/1000.0] [batch_t 0.767 (0.770)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-03 14:50:31,463 - Train: 2.31% [114400/4942000] [23.1/1000.0] [batch_t 0.776 (0.771)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-03 14:51:48,380 - Train: 2.32% [114500/4942000] [23.2/1000.0] [batch_t 0.771 (0.769)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-03 14:53:05,389 - Train: 2.32% [114600/4942000] [23.2/1000.0] [batch_t 0.771 (0.770)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-03 14:54:22,381 - Train: 2.32% [114700/4942000] [23.2/1000.0] [batch_t 0.767 (0.770)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 14:55:39,292 - Train: 2.32% [114800/4942000] [23.2/1000.0] [batch_t 0.754 (0.769)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-03 14:56:56,130 - Train: 2.32% [114900/4942000] [23.2/1000.0] [batch_t 0.768 (0.768)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-03 14:58:12,974 - Train: 2.33% [115000/4942000] [23.3/1000.0] [batch_t 0.762 (0.768)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-03 14:59:29,828 - Train: 2.33% [115100/4942000] [23.3/1000.0] [batch_t 0.764 (0.768)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-03 15:00:46,807 - Train: 2.33% [115200/4942000] [23.3/1000.0] [batch_t 0.779 (0.770)] [data_t 0.002] [optim_t 0.776] [lr 0.005000] 2024-04-03 15:02:03,724 - Train: 2.33% [115300/4942000] [23.3/1000.0] [batch_t 0.775 (0.769)] [data_t 0.003] [optim_t 0.772] [lr 0.005000] 2024-04-03 15:03:20,672 - Train: 2.34% [115400/4942000] [23.4/1000.0] [batch_t 0.762 (0.769)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-03 15:04:37,650 - Train: 2.34% [115500/4942000] [23.4/1000.0] [batch_t 0.761 (0.770)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-03 15:05:54,513 - Train: 2.34% [115600/4942000] [23.4/1000.0] [batch_t 0.770 (0.769)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-03 15:07:11,499 - Train: 2.34% [115700/4942000] [23.4/1000.0] [batch_t 0.767 (0.770)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-03 15:08:28,443 - Train: 2.34% [115800/4942000] [23.4/1000.0] [batch_t 0.773 (0.769)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-03 15:09:45,286 - Train: 2.35% [115900/4942000] [23.5/1000.0] [batch_t 0.768 (0.768)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-03 15:11:02,114 - Train: 2.35% [116000/4942000] [23.5/1000.0] [batch_t 0.774 (0.768)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-03 15:12:19,049 - Train: 2.35% [116100/4942000] [23.5/1000.0] [batch_t 0.754 (0.769)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-03 15:13:35,904 - Train: 2.35% [116200/4942000] [23.5/1000.0] [batch_t 0.758 (0.768)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-03 15:14:53,952 - Train: 2.35% [116300/4942000] [23.5/1000.0] [batch_t 0.768 (0.780)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-03 15:16:10,957 - Train: 2.36% [116400/4942000] [23.6/1000.0] [batch_t 0.767 (0.770)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 15:17:27,869 - Train: 2.36% [116500/4942000] [23.6/1000.0] [batch_t 0.767 (0.769)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-03 15:18:44,720 - Train: 2.36% [116600/4942000] [23.6/1000.0] [batch_t 0.769 (0.768)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-03 15:20:01,627 - Train: 2.36% [116700/4942000] [23.6/1000.0] [batch_t 0.772 (0.769)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-03 15:21:18,564 - Train: 2.36% [116800/4942000] [23.6/1000.0] [batch_t 0.757 (0.769)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-03 15:22:35,484 - Train: 2.37% [116900/4942000] [23.7/1000.0] [batch_t 0.754 (0.769)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-03 15:23:52,393 - Train: 2.37% [117000/4942000] [23.7/1000.0] [batch_t 0.757 (0.769)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-03 15:25:09,346 - Train: 2.37% [117100/4942000] [23.7/1000.0] [batch_t 0.765 (0.769)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-03 15:26:26,329 - Train: 2.37% [117200/4942000] [23.7/1000.0] [batch_t 0.766 (0.770)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-03 15:27:43,274 - Train: 2.37% [117300/4942000] [23.7/1000.0] [batch_t 0.762 (0.769)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-03 15:28:47,904 - Train: 2.38% [117400/4942000] [23.8/1000.0] [batch_t 0.330 (0.646)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 15:29:22,192 - Train: 2.38% [117500/4942000] [23.8/1000.0] [batch_t 0.329 (0.343)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 15:29:55,129 - Train: 2.38% [117600/4942000] [23.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 15:30:29,150 - Train: 2.38% [117700/4942000] [23.8/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 15:31:02,052 - Train: 2.38% [117800/4942000] [23.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 15:31:35,381 - Train: 2.39% [117900/4942000] [23.9/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 15:32:09,218 - Train: 2.39% [118000/4942000] [23.9/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 15:32:42,203 - Train: 2.39% [118100/4942000] [23.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 15:33:16,314 - Train: 2.39% [118200/4942000] [23.9/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 15:33:49,231 - Train: 2.39% [118300/4942000] [23.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 15:34:23,053 - Train: 2.40% [118400/4942000] [24.0/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 15:34:56,129 - Train: 2.40% [118500/4942000] [24.0/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 15:35:29,602 - Train: 2.40% [118600/4942000] [24.0/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 15:35:32,242 - ==> Total time: 21:38:11 Eta: 36 days, 15:53:04 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 15:36:04,053 - Train: 2.40% [118700/4942000] [24.0/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 15:36:38,415 - Train: 2.40% [118800/4942000] [24.0/1000.0] [batch_t 0.329 (0.344)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 15:37:11,338 - Train: 2.41% [118900/4942000] [24.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 15:37:44,257 - Train: 2.41% [119000/4942000] [24.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 15:38:17,266 - Train: 2.41% [119100/4942000] [24.1/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 15:38:50,175 - Train: 2.41% [119200/4942000] [24.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 15:39:23,108 - Train: 2.41% [119300/4942000] [24.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 15:39:55,980 - Train: 2.42% [119400/4942000] [24.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 15:40:28,875 - Train: 2.42% [119500/4942000] [24.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 15:41:01,776 - Train: 2.42% [119600/4942000] [24.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 15:41:34,741 - Train: 2.42% [119700/4942000] [24.2/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 15:42:07,726 - Train: 2.42% [119800/4942000] [24.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 15:42:40,637 - Train: 2.43% [119900/4942000] [24.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 15:43:13,552 - Train: 2.43% [120000/4942000] [24.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 15:43:46,500 - Train: 2.43% [120100/4942000] [24.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 15:44:19,428 - Train: 2.43% [120200/4942000] [24.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 15:44:52,343 - Train: 2.43% [120300/4942000] [24.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 15:45:25,286 - Train: 2.44% [120400/4942000] [24.4/1000.0] [batch_t 0.336 (0.329)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-03 15:45:58,223 - Train: 2.44% [120500/4942000] [24.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 15:46:31,111 - Train: 2.44% [120600/4942000] [24.4/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 15:47:04,038 - Train: 2.44% [120700/4942000] [24.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 15:47:36,918 - Train: 2.44% [120800/4942000] [24.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 15:48:10,307 - Train: 2.45% [120900/4942000] [24.5/1000.0] [batch_t 0.330 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 15:48:43,247 - Train: 2.45% [121000/4942000] [24.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 15:49:17,869 - Train: 2.45% [121100/4942000] [24.5/1000.0] [batch_t 0.330 (0.346)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 15:49:50,813 - Train: 2.45% [121200/4942000] [24.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 15:50:25,497 - Train: 2.45% [121300/4942000] [24.5/1000.0] [batch_t 0.327 (0.347)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 15:50:58,456 - Train: 2.46% [121400/4942000] [24.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 15:51:33,068 - Train: 2.46% [121500/4942000] [24.6/1000.0] [batch_t 0.326 (0.346)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 15:52:06,540 - Train: 2.46% [121600/4942000] [24.6/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 15:52:39,453 - Train: 2.46% [121700/4942000] [24.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 15:53:14,870 - Train: 2.46% [121800/4942000] [24.6/1000.0] [batch_t 0.328 (0.354)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 15:53:51,949 - Train: 2.47% [121900/4942000] [24.7/1000.0] [batch_t 0.563 (0.371)] [data_t 0.237] [optim_t 0.326] [lr 0.005000] 2024-04-03 15:54:31,613 - Train: 2.47% [122000/4942000] [24.7/1000.0] [batch_t 0.332 (0.397)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 15:55:06,238 - Train: 2.47% [122100/4942000] [24.7/1000.0] [batch_t 2.020 (0.346)] [data_t 1.692] [optim_t 0.327] [lr 0.005000] 2024-04-03 15:55:40,785 - Train: 2.47% [122200/4942000] [24.7/1000.0] [batch_t 0.328 (0.345)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 15:56:13,696 - Train: 2.47% [122300/4942000] [24.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 15:56:46,614 - Train: 2.48% [122400/4942000] [24.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 15:57:19,534 - Train: 2.48% [122500/4942000] [24.8/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 15:57:52,413 - Train: 2.48% [122600/4942000] [24.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 15:58:27,631 - Train: 2.48% [122700/4942000] [24.8/1000.0] [batch_t 0.332 (0.352)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 15:59:00,602 - Train: 2.48% [122800/4942000] [24.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 15:59:34,070 - Train: 2.49% [122900/4942000] [24.9/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:00:07,240 - Train: 2.49% [123000/4942000] [24.9/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:00:40,164 - Train: 2.49% [123100/4942000] [24.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:01:14,549 - Train: 2.49% [123200/4942000] [24.9/1000.0] [batch_t 0.325 (0.344)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 16:01:47,500 - Train: 2.49% [123300/4942000] [24.9/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 16:02:21,059 - Train: 2.50% [123400/4942000] [25.0/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:02:53,948 - Train: 2.50% [123500/4942000] [25.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 16:03:10,431 - ==> Total time: 22:05:49 Eta: 35 days, 21:47:15 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 16:03:28,638 - Train: 2.50% [123600/4942000] [25.0/1000.0] [batch_t 0.340 (0.333)] [data_t 0.002] [optim_t 0.338] [lr 0.005000] 2024-04-03 16:04:01,597 - Train: 2.50% [123700/4942000] [25.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:04:35,513 - Train: 2.51% [123800/4942000] [25.1/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:05:10,831 - Train: 2.51% [123900/4942000] [25.1/1000.0] [batch_t 0.328 (0.353)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:05:43,760 - Train: 2.51% [124000/4942000] [25.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:06:17,000 - Train: 2.51% [124100/4942000] [25.1/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:06:49,913 - Train: 2.51% [124200/4942000] [25.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:07:24,276 - Train: 2.52% [124300/4942000] [25.2/1000.0] [batch_t 0.328 (0.344)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:07:57,253 - Train: 2.52% [124400/4942000] [25.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:08:30,995 - Train: 2.52% [124500/4942000] [25.2/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:09:05,632 - Train: 2.52% [124600/4942000] [25.2/1000.0] [batch_t 1.054 (0.346)] [data_t 0.729] [optim_t 0.325] [lr 0.005000] 2024-04-03 16:09:38,640 - Train: 2.52% [124700/4942000] [25.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:10:13,396 - Train: 2.53% [124800/4942000] [25.3/1000.0] [batch_t 0.326 (0.347)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 16:10:46,296 - Train: 2.53% [124900/4942000] [25.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:11:20,358 - Train: 2.53% [125000/4942000] [25.3/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:11:53,358 - Train: 2.53% [125100/4942000] [25.3/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 16:12:26,961 - Train: 2.53% [125200/4942000] [25.3/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:12:59,898 - Train: 2.54% [125300/4942000] [25.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 16:13:34,037 - Train: 2.54% [125400/4942000] [25.4/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:14:07,570 - Train: 2.54% [125500/4942000] [25.4/1000.0] [batch_t 0.326 (0.335)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 16:14:40,499 - Train: 2.54% [125600/4942000] [25.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:15:14,571 - Train: 2.54% [125700/4942000] [25.4/1000.0] [batch_t 0.331 (0.341)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 16:15:47,511 - Train: 2.55% [125800/4942000] [25.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:16:21,532 - Train: 2.55% [125900/4942000] [25.5/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:16:54,446 - Train: 2.55% [126000/4942000] [25.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:17:32,977 - Train: 2.55% [126100/4942000] [25.5/1000.0] [batch_t 0.329 (0.385)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:18:07,434 - Train: 2.55% [126200/4942000] [25.5/1000.0] [batch_t 0.327 (0.344)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 16:18:40,336 - Train: 2.56% [126300/4942000] [25.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 16:19:14,700 - Train: 2.56% [126400/4942000] [25.6/1000.0] [batch_t 0.329 (0.344)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:19:47,615 - Train: 2.56% [126500/4942000] [25.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 16:20:21,121 - Train: 2.56% [126600/4942000] [25.6/1000.0] [batch_t 0.325 (0.335)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 16:20:54,023 - Train: 2.56% [126700/4942000] [25.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:21:26,993 - Train: 2.57% [126800/4942000] [25.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:21:59,913 - Train: 2.57% [126900/4942000] [25.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:22:33,584 - Train: 2.57% [127000/4942000] [25.7/1000.0] [batch_t 0.331 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:23:07,223 - Train: 2.57% [127100/4942000] [25.7/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 16:23:40,133 - Train: 2.57% [127200/4942000] [25.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 16:24:17,179 - Train: 2.58% [127300/4942000] [25.8/1000.0] [batch_t 0.328 (0.370)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 16:24:50,113 - Train: 2.58% [127400/4942000] [25.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 16:25:23,615 - Train: 2.58% [127500/4942000] [25.8/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:25:56,546 - Train: 2.58% [127600/4942000] [25.8/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 16:26:30,344 - Train: 2.58% [127700/4942000] [25.8/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 16:27:03,229 - Train: 2.59% [127800/4942000] [25.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:27:37,921 - Train: 2.59% [127900/4942000] [25.9/1000.0] [batch_t 0.328 (0.347)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:28:14,287 - Train: 2.59% [128000/4942000] [25.9/1000.0] [batch_t 0.325 (0.364)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 16:28:47,215 - Train: 2.59% [128100/4942000] [25.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 16:29:22,396 - Train: 2.59% [128200/4942000] [25.9/1000.0] [batch_t 0.329 (0.352)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:29:55,264 - Train: 2.60% [128300/4942000] [26.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:30:31,738 - Train: 2.60% [128400/4942000] [26.0/1000.0] [batch_t 0.330 (0.365)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 16:31:02,046 - ==> Total time: 22:33:41 Eta: 35 days, 5:11:12 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 16:31:09,400 - Train: 2.60% [128500/4942000] [26.0/1000.0] [batch_t 0.328 (0.446)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:31:42,353 - Train: 2.60% [128600/4942000] [26.0/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 16:32:18,541 - Train: 2.60% [128700/4942000] [26.0/1000.0] [batch_t 0.326 (0.362)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 16:32:51,445 - Train: 2.61% [128800/4942000] [26.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:33:27,279 - Train: 2.61% [128900/4942000] [26.1/1000.0] [batch_t 0.332 (0.358)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 16:34:00,149 - Train: 2.61% [129000/4942000] [26.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:34:35,523 - Train: 2.61% [129100/4942000] [26.1/1000.0] [batch_t 0.330 (0.354)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:35:11,268 - Train: 2.61% [129200/4942000] [26.1/1000.0] [batch_t 0.327 (0.357)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 16:35:44,181 - Train: 2.62% [129300/4942000] [26.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:36:19,886 - Train: 2.62% [129400/4942000] [26.2/1000.0] [batch_t 0.330 (0.357)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:36:52,777 - Train: 2.62% [129500/4942000] [26.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:37:28,003 - Train: 2.62% [129600/4942000] [26.2/1000.0] [batch_t 0.329 (0.352)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:38:01,827 - Train: 2.62% [129700/4942000] [26.2/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 16:38:37,741 - Train: 2.63% [129800/4942000] [26.3/1000.0] [batch_t 0.331 (0.359)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 16:39:12,854 - Train: 2.63% [129900/4942000] [26.3/1000.0] [batch_t 0.331 (0.351)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 16:39:45,730 - Train: 2.63% [130000/4942000] [26.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 16:40:20,241 - Train: 2.63% [130100/4942000] [26.3/1000.0] [batch_t 0.328 (0.345)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:40:53,170 - Train: 2.63% [130200/4942000] [26.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:41:27,645 - Train: 2.64% [130300/4942000] [26.4/1000.0] [batch_t 0.331 (0.345)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:42:00,697 - Train: 2.64% [130400/4942000] [26.4/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-03 16:42:37,695 - Train: 2.64% [130500/4942000] [26.4/1000.0] [batch_t 0.328 (0.370)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 16:43:11,970 - Train: 2.64% [130600/4942000] [26.4/1000.0] [batch_t 0.328 (0.343)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:43:44,914 - Train: 2.64% [130700/4942000] [26.4/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 16:44:20,463 - Train: 2.65% [130800/4942000] [26.5/1000.0] [batch_t 0.330 (0.355)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:44:53,373 - Train: 2.65% [130900/4942000] [26.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:45:27,647 - Train: 2.65% [131000/4942000] [26.5/1000.0] [batch_t 0.324 (0.343)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 16:46:00,587 - Train: 2.65% [131100/4942000] [26.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 16:46:35,310 - Train: 2.65% [131200/4942000] [26.5/1000.0] [batch_t 0.331 (0.347)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 16:47:09,324 - Train: 2.66% [131300/4942000] [26.6/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 16:47:42,278 - Train: 2.66% [131400/4942000] [26.6/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 16:48:18,117 - Train: 2.66% [131500/4942000] [26.6/1000.0] [batch_t 0.331 (0.358)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 16:48:51,059 - Train: 2.66% [131600/4942000] [26.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:49:28,879 - Train: 2.66% [131700/4942000] [26.6/1000.0] [batch_t 0.329 (0.378)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:50:01,806 - Train: 2.67% [131800/4942000] [26.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 16:50:37,557 - Train: 2.67% [131900/4942000] [26.7/1000.0] [batch_t 0.329 (0.357)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:51:13,152 - Train: 2.67% [132000/4942000] [26.7/1000.0] [batch_t 0.330 (0.356)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:51:46,075 - Train: 2.67% [132100/4942000] [26.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:52:23,193 - Train: 2.68% [132200/4942000] [26.8/1000.0] [batch_t 0.330 (0.371)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:52:56,120 - Train: 2.68% [132300/4942000] [26.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:53:31,346 - Train: 2.68% [132400/4942000] [26.8/1000.0] [batch_t 0.330 (0.352)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:54:04,297 - Train: 2.68% [132500/4942000] [26.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:54:41,106 - Train: 2.68% [132600/4942000] [26.8/1000.0] [batch_t 0.332 (0.368)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 16:55:18,590 - Train: 2.69% [132700/4942000] [26.9/1000.0] [batch_t 0.330 (0.375)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:55:51,825 - Train: 2.69% [132800/4942000] [26.9/1000.0] [batch_t 0.325 (0.332)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 16:56:27,754 - Train: 2.69% [132900/4942000] [26.9/1000.0] [batch_t 0.331 (0.359)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 16:57:00,691 - Train: 2.69% [133000/4942000] [26.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:57:37,154 - Train: 2.69% [133100/4942000] [26.9/1000.0] [batch_t 0.329 (0.365)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:58:13,181 - Train: 2.70% [133200/4942000] [27.0/1000.0] [batch_t 0.327 (0.360)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 16:58:46,131 - Train: 2.70% [133300/4942000] [27.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:59:21,676 - Train: 2.70% [133400/4942000] [27.0/1000.0] [batch_t 0.330 (0.355)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:59:32,864 - ==> Total time: 23:02:12 Eta: 34 days, 14:10:25 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 16:59:56,775 - Train: 2.70% [133500/4942000] [27.0/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:00:31,921 - Train: 2.70% [133600/4942000] [27.0/1000.0] [batch_t 0.322 (0.351)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-03 17:01:05,216 - Train: 2.71% [133700/4942000] [27.1/1000.0] [batch_t 0.683 (0.333)] [data_t 0.359] [optim_t 0.325] [lr 0.005000] 2024-04-03 17:01:40,401 - Train: 2.71% [133800/4942000] [27.1/1000.0] [batch_t 0.328 (0.352)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:02:15,415 - Train: 2.71% [133900/4942000] [27.1/1000.0] [batch_t 0.329 (0.350)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:02:48,504 - Train: 2.71% [134000/4942000] [27.1/1000.0] [batch_t 0.333 (0.331)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-03 17:03:24,445 - Train: 2.71% [134100/4942000] [27.1/1000.0] [batch_t 0.330 (0.359)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:03:57,369 - Train: 2.72% [134200/4942000] [27.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:04:33,335 - Train: 2.72% [134300/4942000] [27.2/1000.0] [batch_t 0.331 (0.360)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 17:05:08,320 - Train: 2.72% [134400/4942000] [27.2/1000.0] [batch_t 0.326 (0.350)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 17:05:41,256 - Train: 2.72% [134500/4942000] [27.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:06:20,317 - Train: 2.72% [134600/4942000] [27.2/1000.0] [batch_t 0.328 (0.391)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:06:53,213 - Train: 2.73% [134700/4942000] [27.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:07:26,581 - Train: 2.73% [134800/4942000] [27.3/1000.0] [batch_t 0.332 (0.334)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 17:07:59,549 - Train: 2.73% [134900/4942000] [27.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:08:35,025 - Train: 2.73% [135000/4942000] [27.3/1000.0] [batch_t 0.330 (0.355)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:09:10,134 - Train: 2.73% [135100/4942000] [27.3/1000.0] [batch_t 0.329 (0.351)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:09:43,040 - Train: 2.74% [135200/4942000] [27.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 17:10:20,004 - Train: 2.74% [135300/4942000] [27.4/1000.0] [batch_t 0.327 (0.370)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 17:10:53,066 - Train: 2.74% [135400/4942000] [27.4/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 17:11:27,950 - Train: 2.74% [135500/4942000] [27.4/1000.0] [batch_t 0.325 (0.349)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 17:12:00,932 - Train: 2.74% [135600/4942000] [27.4/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 17:12:37,957 - Train: 2.75% [135700/4942000] [27.5/1000.0] [batch_t 0.332 (0.370)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 17:13:12,669 - Train: 2.75% [135800/4942000] [27.5/1000.0] [batch_t 0.328 (0.347)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:13:45,606 - Train: 2.75% [135900/4942000] [27.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:14:21,117 - Train: 2.75% [136000/4942000] [27.5/1000.0] [batch_t 0.329 (0.355)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:14:54,158 - Train: 2.75% [136100/4942000] [27.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:15:31,619 - Train: 2.76% [136200/4942000] [27.6/1000.0] [batch_t 0.326 (0.375)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 17:16:04,573 - Train: 2.76% [136300/4942000] [27.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 17:16:39,815 - Train: 2.76% [136400/4942000] [27.6/1000.0] [batch_t 0.328 (0.352)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:17:15,690 - Train: 2.76% [136500/4942000] [27.6/1000.0] [batch_t 0.330 (0.359)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:17:54,467 - Train: 2.76% [136600/4942000] [27.6/1000.0] [batch_t 0.332 (0.388)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 17:18:31,086 - Train: 2.77% [136700/4942000] [27.7/1000.0] [batch_t 0.330 (0.366)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:19:04,044 - Train: 2.77% [136800/4942000] [27.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 17:19:39,912 - Train: 2.77% [136900/4942000] [27.7/1000.0] [batch_t 0.330 (0.359)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:20:17,496 - Train: 2.77% [137000/4942000] [27.7/1000.0] [batch_t 0.327 (0.376)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 17:20:50,441 - Train: 2.77% [137100/4942000] [27.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-03 17:21:26,478 - Train: 2.78% [137200/4942000] [27.8/1000.0] [batch_t 0.327 (0.360)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 17:21:59,425 - Train: 2.78% [137300/4942000] [27.8/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 17:22:35,620 - Train: 2.78% [137400/4942000] [27.8/1000.0] [batch_t 0.330 (0.362)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:23:12,223 - Train: 2.78% [137500/4942000] [27.8/1000.0] [batch_t 0.329 (0.366)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:23:45,135 - Train: 2.78% [137600/4942000] [27.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 17:24:36,420 - Train: 2.79% [137700/4942000] [27.9/1000.0] [batch_t 0.326 (0.513)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 17:25:10,378 - Train: 2.79% [137800/4942000] [27.9/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:25:43,280 - Train: 2.79% [137900/4942000] [27.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 17:26:17,950 - Train: 2.79% [138000/4942000] [27.9/1000.0] [batch_t 0.325 (0.347)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 17:26:50,844 - Train: 2.79% [138100/4942000] [27.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:27:28,222 - Train: 2.80% [138200/4942000] [28.0/1000.0] [batch_t 0.330 (0.374)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:28:01,227 - Train: 2.80% [138300/4942000] [28.0/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 17:28:27,174 - ==> Total time: 23:31:06 Eta: 34 days, 0:25:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 17:28:37,700 - Train: 2.80% [138400/4942000] [28.0/1000.0] [batch_t 0.329 (0.366)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:29:12,088 - Train: 2.80% [138500/4942000] [28.0/1000.0] [batch_t 0.331 (0.344)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 17:29:45,007 - Train: 2.80% [138600/4942000] [28.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:30:19,152 - Train: 2.81% [138700/4942000] [28.1/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:30:52,087 - Train: 2.81% [138800/4942000] [28.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 17:31:25,304 - Train: 2.81% [138900/4942000] [28.1/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:31:58,270 - Train: 2.81% [139000/4942000] [28.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:32:32,095 - Train: 2.81% [139100/4942000] [28.1/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:33:06,176 - Train: 2.82% [139200/4942000] [28.2/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 17:33:39,138 - Train: 2.82% [139300/4942000] [28.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:34:12,727 - Train: 2.82% [139400/4942000] [28.2/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:34:45,661 - Train: 2.82% [139500/4942000] [28.2/1000.0] [batch_t 0.326 (0.329)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-03 17:35:18,756 - Train: 2.82% [139600/4942000] [28.2/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:35:51,857 - Train: 2.83% [139700/4942000] [28.3/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:36:25,836 - Train: 2.83% [139800/4942000] [28.3/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:36:58,755 - Train: 2.83% [139900/4942000] [28.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:37:33,886 - Train: 2.83% [140000/4942000] [28.3/1000.0] [batch_t 0.326 (0.351)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 17:38:07,473 - Train: 2.83% [140100/4942000] [28.3/1000.0] [batch_t 0.326 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 17:38:40,517 - Train: 2.84% [140200/4942000] [28.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:39:15,200 - Train: 2.84% [140300/4942000] [28.4/1000.0] [batch_t 0.331 (0.347)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 17:39:48,132 - Train: 2.84% [140400/4942000] [28.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:40:22,248 - Train: 2.84% [140500/4942000] [28.4/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:40:55,172 - Train: 2.85% [140600/4942000] [28.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:41:28,602 - Train: 2.85% [140700/4942000] [28.5/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:42:28,935 - Train: 2.85% [140800/4942000] [28.5/1000.0] [batch_t 0.331 (0.603)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 17:43:22,640 - Train: 2.85% [140900/4942000] [28.5/1000.0] [batch_t 0.325 (0.537)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 17:44:04,634 - Train: 2.85% [141000/4942000] [28.5/1000.0] [batch_t 0.331 (0.420)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 17:44:42,085 - Train: 2.86% [141100/4942000] [28.6/1000.0] [batch_t 0.325 (0.374)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 17:45:16,372 - Train: 2.86% [141200/4942000] [28.6/1000.0] [batch_t 0.329 (0.343)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:45:49,365 - Train: 2.86% [141300/4942000] [28.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:46:27,937 - Train: 2.86% [141400/4942000] [28.6/1000.0] [batch_t 0.330 (0.386)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:47:23,428 - Train: 2.86% [141500/4942000] [28.6/1000.0] [batch_t 0.366 (0.555)] [data_t 0.039] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:48:14,067 - Train: 2.87% [141600/4942000] [28.7/1000.0] [batch_t 0.328 (0.506)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:48:48,734 - Train: 2.87% [141700/4942000] [28.7/1000.0] [batch_t 1.793 (0.347)] [data_t 1.466] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:49:43,147 - Train: 2.87% [141800/4942000] [28.7/1000.0] [batch_t 0.330 (0.541)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:50:25,941 - Train: 2.87% [141900/4942000] [28.7/1000.0] [batch_t 0.329 (0.428)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:51:56,886 - Train: 2.87% [142000/4942000] [28.7/1000.0] [batch_t 9.002 (0.909)] [data_t 8.675] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:52:50,904 - Train: 2.88% [142100/4942000] [28.8/1000.0] [batch_t 0.804 (0.540)] [data_t 0.475] [optim_t 0.329] [lr 0.005000] 2024-04-03 17:53:32,744 - Train: 2.88% [142200/4942000] [28.8/1000.0] [batch_t 0.331 (0.418)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 17:54:09,712 - Train: 2.88% [142300/4942000] [28.8/1000.0] [batch_t 0.329 (0.370)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:54:48,660 - Train: 2.88% [142400/4942000] [28.8/1000.0] [batch_t 0.325 (0.389)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 17:55:42,950 - Train: 2.88% [142500/4942000] [28.8/1000.0] [batch_t 0.321 (0.543)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-03 17:56:18,334 - Train: 2.89% [142600/4942000] [28.9/1000.0] [batch_t 0.329 (0.354)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:56:51,218 - Train: 2.89% [142700/4942000] [28.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 17:57:28,304 - Train: 2.89% [142800/4942000] [28.9/1000.0] [batch_t 0.330 (0.371)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:58:03,104 - Train: 2.89% [142900/4942000] [28.9/1000.0] [batch_t 0.329 (0.348)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:58:48,022 - Train: 2.89% [143000/4942000] [28.9/1000.0] [batch_t 0.331 (0.449)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 17:59:42,923 - Train: 2.90% [143100/4942000] [29.0/1000.0] [batch_t 0.327 (0.549)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 18:00:44,067 - Train: 2.90% [143200/4942000] [29.0/1000.0] [batch_t 0.332 (0.611)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 18:01:46,750 - Train: 2.90% [143300/4942000] [29.0/1000.0] [batch_t 0.329 (0.625)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:01:53,312 - ==> Total time: 1 day, 0:04:32 Eta: 33 days, 14:07:14 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 18:02:21,801 - Train: 2.90% [143400/4942000] [29.0/1000.0] [batch_t 0.332 (0.332)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 18:02:54,770 - Train: 2.90% [143500/4942000] [29.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 18:03:30,429 - Train: 2.91% [143600/4942000] [29.1/1000.0] [batch_t 0.330 (0.357)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 18:04:03,353 - Train: 2.91% [143700/4942000] [29.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 18:04:37,690 - Train: 2.91% [143800/4942000] [29.1/1000.0] [batch_t 0.328 (0.343)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:05:12,155 - Train: 2.91% [143900/4942000] [29.1/1000.0] [batch_t 0.329 (0.345)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:05:45,072 - Train: 2.91% [144000/4942000] [29.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 18:06:19,001 - Train: 2.92% [144100/4942000] [29.2/1000.0] [batch_t 0.331 (0.339)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 18:06:51,835 - Train: 2.92% [144200/4942000] [29.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 18:07:24,714 - Train: 2.92% [144300/4942000] [29.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:07:57,612 - Train: 2.92% [144400/4942000] [29.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:08:30,531 - Train: 2.92% [144500/4942000] [29.2/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 18:09:03,571 - Train: 2.93% [144600/4942000] [29.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:09:37,884 - Train: 2.93% [144700/4942000] [29.3/1000.0] [batch_t 0.336 (0.343)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-03 18:10:11,945 - Train: 2.93% [144800/4942000] [29.3/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:10:44,887 - Train: 2.93% [144900/4942000] [29.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 18:11:18,414 - Train: 2.93% [145000/4942000] [29.3/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:11:51,408 - Train: 2.94% [145100/4942000] [29.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 18:12:24,471 - Train: 2.94% [145200/4942000] [29.4/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:12:57,375 - Train: 2.94% [145300/4942000] [29.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 18:13:30,467 - Train: 2.94% [145400/4942000] [29.4/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:14:03,402 - Train: 2.94% [145500/4942000] [29.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 18:14:37,431 - Train: 2.95% [145600/4942000] [29.5/1000.0] [batch_t 0.330 (0.340)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 18:15:11,428 - Train: 2.95% [145700/4942000] [29.5/1000.0] [batch_t 0.326 (0.340)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 18:15:44,438 - Train: 2.95% [145800/4942000] [29.5/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 18:16:18,662 - Train: 2.95% [145900/4942000] [29.5/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:16:51,579 - Train: 2.95% [146000/4942000] [29.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 18:17:25,716 - Train: 2.96% [146100/4942000] [29.6/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:17:58,600 - Train: 2.96% [146200/4942000] [29.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 18:18:31,542 - Train: 2.96% [146300/4942000] [29.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:19:04,451 - Train: 2.96% [146400/4942000] [29.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:19:38,366 - Train: 2.96% [146500/4942000] [29.6/1000.0] [batch_t 0.325 (0.339)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 18:20:11,296 - Train: 2.97% [146600/4942000] [29.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:20:44,172 - Train: 2.97% [146700/4942000] [29.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 18:21:17,171 - Train: 2.97% [146800/4942000] [29.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:21:50,096 - Train: 2.97% [146900/4942000] [29.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 18:22:23,053 - Train: 2.97% [147000/4942000] [29.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:22:55,997 - Train: 2.98% [147100/4942000] [29.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 18:23:28,861 - Train: 2.98% [147200/4942000] [29.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 18:24:01,724 - Train: 2.98% [147300/4942000] [29.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 18:24:34,609 - Train: 2.98% [147400/4942000] [29.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:25:07,567 - Train: 2.98% [147500/4942000] [29.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 18:25:40,483 - Train: 2.99% [147600/4942000] [29.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 18:26:13,422 - Train: 2.99% [147700/4942000] [29.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 18:26:46,386 - Train: 2.99% [147800/4942000] [29.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 18:27:20,023 - Train: 2.99% [147900/4942000] [29.9/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:27:52,914 - Train: 2.99% [148000/4942000] [29.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 18:28:25,881 - Train: 3.00% [148100/4942000] [30.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 18:28:59,001 - Train: 3.00% [148200/4942000] [30.0/1000.0] [batch_t 0.325 (0.331)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 18:29:18,815 - ==> Total time: 1 day, 0:31:58 Eta: 33 days, 1:13:35 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 18:29:33,328 - Train: 3.00% [148300/4942000] [30.0/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:30:06,404 - Train: 3.00% [148400/4942000] [30.0/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:30:39,331 - Train: 3.00% [148500/4942000] [30.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-03 18:31:12,258 - Train: 3.01% [148600/4942000] [30.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:31:45,239 - Train: 3.01% [148700/4942000] [30.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 18:32:21,924 - Train: 3.01% [148800/4942000] [30.1/1000.0] [batch_t 0.331 (0.367)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-03 18:32:54,837 - Train: 3.01% [148900/4942000] [30.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 18:33:27,771 - Train: 3.01% [149000/4942000] [30.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:34:00,712 - Train: 3.02% [149100/4942000] [30.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:34:34,239 - Train: 3.02% [149200/4942000] [30.2/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:35:09,547 - Train: 3.02% [149300/4942000] [30.2/1000.0] [batch_t 0.328 (0.353)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 18:35:42,447 - Train: 3.02% [149400/4942000] [30.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 18:36:16,855 - Train: 3.03% [149500/4942000] [30.3/1000.0] [batch_t 0.329 (0.344)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:36:49,945 - Train: 3.03% [149600/4942000] [30.3/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:37:25,202 - Train: 3.03% [149700/4942000] [30.3/1000.0] [batch_t 0.328 (0.352)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:37:58,119 - Train: 3.03% [149800/4942000] [30.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:38:33,914 - Train: 3.03% [149900/4942000] [30.3/1000.0] [batch_t 0.330 (0.358)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 18:39:06,889 - Train: 3.04% [150000/4942000] [30.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:39:39,872 - Train: 3.04% [150100/4942000] [30.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 18:40:12,810 - Train: 3.04% [150200/4942000] [30.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:40:45,717 - Train: 3.04% [150300/4942000] [30.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 18:41:18,625 - Train: 3.04% [150400/4942000] [30.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 18:41:51,520 - Train: 3.05% [150500/4942000] [30.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:42:24,506 - Train: 3.05% [150600/4942000] [30.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:42:57,535 - Train: 3.05% [150700/4942000] [30.5/1000.0] [batch_t 0.339 (0.330)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-03 18:43:30,508 - Train: 3.05% [150800/4942000] [30.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:44:03,405 - Train: 3.05% [150900/4942000] [30.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 18:44:38,291 - Train: 3.06% [151000/4942000] [30.6/1000.0] [batch_t 0.336 (0.349)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-03 18:45:13,398 - Train: 3.06% [151100/4942000] [30.6/1000.0] [batch_t 0.329 (0.351)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:45:46,349 - Train: 3.06% [151200/4942000] [30.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 18:46:20,370 - Train: 3.06% [151300/4942000] [30.6/1000.0] [batch_t 0.332 (0.340)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 18:46:53,334 - Train: 3.06% [151400/4942000] [30.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:47:26,909 - Train: 3.07% [151500/4942000] [30.7/1000.0] [batch_t 0.331 (0.336)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 18:47:59,858 - Train: 3.07% [151600/4942000] [30.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:48:32,788 - Train: 3.07% [151700/4942000] [30.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 18:49:05,754 - Train: 3.07% [151800/4942000] [30.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 18:49:38,694 - Train: 3.07% [151900/4942000] [30.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 18:50:11,650 - Train: 3.08% [152000/4942000] [30.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 18:50:44,610 - Train: 3.08% [152100/4942000] [30.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:51:17,516 - Train: 3.08% [152200/4942000] [30.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 18:51:50,409 - Train: 3.08% [152300/4942000] [30.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:52:23,480 - Train: 3.08% [152400/4942000] [30.8/1000.0] [batch_t 0.334 (0.331)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 18:52:56,460 - Train: 3.09% [152500/4942000] [30.9/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 18:53:29,374 - Train: 3.09% [152600/4942000] [30.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 18:54:02,286 - Train: 3.09% [152700/4942000] [30.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:54:35,211 - Train: 3.09% [152800/4942000] [30.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 18:55:08,141 - Train: 3.09% [152900/4942000] [30.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:55:41,064 - Train: 3.10% [153000/4942000] [31.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:56:14,041 - Train: 3.10% [153100/4942000] [31.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 18:56:47,007 - Train: 3.10% [153200/4942000] [31.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:56:47,668 - ==> Total time: 1 day, 0:59:26 Eta: 32 days, 13:09:49 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 18:57:21,512 - Train: 3.10% [153300/4942000] [31.0/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 18:57:54,458 - Train: 3.10% [153400/4942000] [31.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:58:27,415 - Train: 3.11% [153500/4942000] [31.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 18:59:00,365 - Train: 3.11% [153600/4942000] [31.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:59:33,370 - Train: 3.11% [153700/4942000] [31.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:00:06,622 - Train: 3.11% [153800/4942000] [31.1/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:00:39,520 - Train: 3.11% [153900/4942000] [31.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:01:12,440 - Train: 3.12% [154000/4942000] [31.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:01:45,334 - Train: 3.12% [154100/4942000] [31.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:02:18,273 - Train: 3.12% [154200/4942000] [31.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:02:51,208 - Train: 3.12% [154300/4942000] [31.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:03:24,180 - Train: 3.12% [154400/4942000] [31.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:03:57,111 - Train: 3.13% [154500/4942000] [31.3/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 19:04:30,003 - Train: 3.13% [154600/4942000] [31.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:05:02,958 - Train: 3.13% [154700/4942000] [31.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:05:35,870 - Train: 3.13% [154800/4942000] [31.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:06:08,833 - Train: 3.13% [154900/4942000] [31.3/1000.0] [batch_t 0.335 (0.330)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-03 19:06:41,828 - Train: 3.14% [155000/4942000] [31.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:07:14,720 - Train: 3.14% [155100/4942000] [31.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:07:47,709 - Train: 3.14% [155200/4942000] [31.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:08:20,633 - Train: 3.14% [155300/4942000] [31.4/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-03 19:08:53,604 - Train: 3.14% [155400/4942000] [31.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:09:26,557 - Train: 3.15% [155500/4942000] [31.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:09:59,478 - Train: 3.15% [155600/4942000] [31.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:10:32,404 - Train: 3.15% [155700/4942000] [31.5/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 19:11:06,078 - Train: 3.15% [155800/4942000] [31.5/1000.0] [batch_t 1.125 (0.337)] [data_t 0.798] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:11:39,037 - Train: 3.15% [155900/4942000] [31.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:12:12,672 - Train: 3.16% [156000/4942000] [31.6/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 19:12:45,574 - Train: 3.16% [156100/4942000] [31.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:13:18,493 - Train: 3.16% [156200/4942000] [31.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:14:09,828 - Train: 3.16% [156300/4942000] [31.6/1000.0] [batch_t 0.328 (0.513)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:14:42,762 - Train: 3.16% [156400/4942000] [31.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:15:18,325 - Train: 3.17% [156500/4942000] [31.7/1000.0] [batch_t 0.329 (0.356)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:15:51,343 - Train: 3.17% [156600/4942000] [31.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:16:27,217 - Train: 3.17% [156700/4942000] [31.7/1000.0] [batch_t 0.331 (0.359)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:17:00,201 - Train: 3.17% [156800/4942000] [31.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:17:33,877 - Train: 3.17% [156900/4942000] [31.7/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:18:06,760 - Train: 3.18% [157000/4942000] [31.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:18:39,679 - Train: 3.18% [157100/4942000] [31.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 19:19:12,643 - Train: 3.18% [157200/4942000] [31.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:19:45,592 - Train: 3.18% [157300/4942000] [31.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:20:18,505 - Train: 3.18% [157400/4942000] [31.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 19:20:51,410 - Train: 3.19% [157500/4942000] [31.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:21:24,341 - Train: 3.19% [157600/4942000] [31.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:21:57,317 - Train: 3.19% [157700/4942000] [31.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:22:30,321 - Train: 3.19% [157800/4942000] [31.9/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 19:23:03,260 - Train: 3.20% [157900/4942000] [32.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 19:23:36,113 - Train: 3.20% [158000/4942000] [32.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:24:09,179 - Train: 3.20% [158100/4942000] [32.0/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:24:23,667 - ==> Total time: 1 day, 1:27:02 Eta: 32 days, 1:53:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 19:24:43,386 - Train: 3.20% [158200/4942000] [32.0/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:25:16,320 - Train: 3.20% [158300/4942000] [32.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:25:49,221 - Train: 3.21% [158400/4942000] [32.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:26:22,126 - Train: 3.21% [158500/4942000] [32.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 19:26:55,070 - Train: 3.21% [158600/4942000] [32.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 19:27:27,990 - Train: 3.21% [158700/4942000] [32.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:28:00,919 - Train: 3.21% [158800/4942000] [32.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:28:33,841 - Train: 3.22% [158900/4942000] [32.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 19:29:06,713 - Train: 3.22% [159000/4942000] [32.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 19:29:39,655 - Train: 3.22% [159100/4942000] [32.2/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 19:30:12,603 - Train: 3.22% [159200/4942000] [32.2/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 19:30:45,556 - Train: 3.22% [159300/4942000] [32.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:31:18,616 - Train: 3.23% [159400/4942000] [32.3/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:31:51,553 - Train: 3.23% [159500/4942000] [32.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 19:32:25,341 - Train: 3.23% [159600/4942000] [32.3/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 19:32:58,215 - Train: 3.23% [159700/4942000] [32.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 19:33:32,422 - Train: 3.23% [159800/4942000] [32.3/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:34:06,057 - Train: 3.24% [159900/4942000] [32.4/1000.0] [batch_t 0.907 (0.336)] [data_t 0.577] [optim_t 0.330] [lr 0.005000] 2024-04-03 19:34:38,973 - Train: 3.24% [160000/4942000] [32.4/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 19:35:11,880 - Train: 3.24% [160100/4942000] [32.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:35:44,787 - Train: 3.24% [160200/4942000] [32.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 19:36:17,844 - Train: 3.24% [160300/4942000] [32.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:36:50,761 - Train: 3.25% [160400/4942000] [32.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:37:24,605 - Train: 3.25% [160500/4942000] [32.5/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:37:57,542 - Train: 3.25% [160600/4942000] [32.5/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 19:38:30,869 - Train: 3.25% [160700/4942000] [32.5/1000.0] [batch_t 0.330 (0.333)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:39:03,876 - Train: 3.25% [160800/4942000] [32.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 19:39:36,801 - Train: 3.26% [160900/4942000] [32.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:40:09,689 - Train: 3.26% [161000/4942000] [32.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:40:42,586 - Train: 3.26% [161100/4942000] [32.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:41:15,517 - Train: 3.26% [161200/4942000] [32.6/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 19:41:48,473 - Train: 3.26% [161300/4942000] [32.6/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 19:42:21,410 - Train: 3.27% [161400/4942000] [32.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 19:42:54,343 - Train: 3.27% [161500/4942000] [32.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:43:27,245 - Train: 3.27% [161600/4942000] [32.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:44:00,175 - Train: 3.27% [161700/4942000] [32.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:44:33,051 - Train: 3.27% [161800/4942000] [32.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 19:45:05,953 - Train: 3.28% [161900/4942000] [32.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:45:38,906 - Train: 3.28% [162000/4942000] [32.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:46:11,876 - Train: 3.28% [162100/4942000] [32.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:46:44,819 - Train: 3.28% [162200/4942000] [32.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:47:17,827 - Train: 3.28% [162300/4942000] [32.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:47:50,746 - Train: 3.29% [162400/4942000] [32.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:48:23,664 - Train: 3.29% [162500/4942000] [32.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:48:56,601 - Train: 3.29% [162600/4942000] [32.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 19:49:29,551 - Train: 3.29% [162700/4942000] [32.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:50:02,515 - Train: 3.29% [162800/4942000] [32.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:50:35,440 - Train: 3.30% [162900/4942000] [33.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:51:08,366 - Train: 3.30% [163000/4942000] [33.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:51:36,679 - ==> Total time: 1 day, 1:54:15 Eta: 31 days, 15:04:39 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 19:51:42,639 - Train: 3.30% [163100/4942000] [33.0/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:52:15,522 - Train: 3.30% [163200/4942000] [33.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 19:52:48,463 - Train: 3.30% [163300/4942000] [33.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:53:21,399 - Train: 3.31% [163400/4942000] [33.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:53:54,295 - Train: 3.31% [163500/4942000] [33.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:54:27,351 - Train: 3.31% [163600/4942000] [33.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:55:00,253 - Train: 3.31% [163700/4942000] [33.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 19:55:33,168 - Train: 3.31% [163800/4942000] [33.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 19:56:07,285 - Train: 3.32% [163900/4942000] [33.2/1000.0] [batch_t 0.335 (0.341)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 19:56:40,173 - Train: 3.32% [164000/4942000] [33.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:57:13,107 - Train: 3.32% [164100/4942000] [33.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:57:46,051 - Train: 3.32% [164200/4942000] [33.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:58:18,953 - Train: 3.32% [164300/4942000] [33.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:58:51,864 - Train: 3.33% [164400/4942000] [33.3/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 19:59:26,524 - Train: 3.33% [164500/4942000] [33.3/1000.0] [batch_t 0.328 (0.347)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:59:59,429 - Train: 3.33% [164600/4942000] [33.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:00:34,148 - Train: 3.33% [164700/4942000] [33.3/1000.0] [batch_t 0.326 (0.347)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 20:01:07,071 - Train: 3.33% [164800/4942000] [33.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 20:01:40,020 - Train: 3.34% [164900/4942000] [33.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:02:13,056 - Train: 3.34% [165000/4942000] [33.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:02:45,950 - Train: 3.34% [165100/4942000] [33.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:03:18,873 - Train: 3.34% [165200/4942000] [33.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:03:51,761 - Train: 3.34% [165300/4942000] [33.4/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 20:04:24,660 - Train: 3.35% [165400/4942000] [33.5/1000.0] [batch_t 0.337 (0.329)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-03 20:04:57,613 - Train: 3.35% [165500/4942000] [33.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:05:30,552 - Train: 3.35% [165600/4942000] [33.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:06:03,448 - Train: 3.35% [165700/4942000] [33.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:06:36,356 - Train: 3.35% [165800/4942000] [33.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:07:09,276 - Train: 3.36% [165900/4942000] [33.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:07:42,192 - Train: 3.36% [166000/4942000] [33.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:08:15,102 - Train: 3.36% [166100/4942000] [33.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:08:48,028 - Train: 3.36% [166200/4942000] [33.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:09:20,950 - Train: 3.37% [166300/4942000] [33.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:09:53,968 - Train: 3.37% [166400/4942000] [33.7/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:10:26,879 - Train: 3.37% [166500/4942000] [33.7/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 20:10:59,772 - Train: 3.37% [166600/4942000] [33.7/1000.0] [batch_t 0.335 (0.329)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-03 20:11:32,757 - Train: 3.37% [166700/4942000] [33.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:12:05,715 - Train: 3.38% [166800/4942000] [33.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:12:38,632 - Train: 3.38% [166900/4942000] [33.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:13:11,542 - Train: 3.38% [167000/4942000] [33.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:13:44,442 - Train: 3.38% [167100/4942000] [33.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:14:18,128 - Train: 3.38% [167200/4942000] [33.8/1000.0] [batch_t 0.326 (0.337)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 20:14:51,026 - Train: 3.39% [167300/4942000] [33.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 20:15:23,932 - Train: 3.39% [167400/4942000] [33.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:15:56,797 - Train: 3.39% [167500/4942000] [33.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:16:29,684 - Train: 3.39% [167600/4942000] [33.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 20:17:02,654 - Train: 3.39% [167700/4942000] [33.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:17:35,549 - Train: 3.40% [167800/4942000] [34.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:18:08,584 - Train: 3.40% [167900/4942000] [34.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:18:41,516 - Train: 3.40% [168000/4942000] [34.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:18:50,738 - ==> Total time: 1 day, 2:21:29 Eta: 31 days, 4:53:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 20:19:15,755 - Train: 3.40% [168100/4942000] [34.0/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:19:48,668 - Train: 3.40% [168200/4942000] [34.0/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 20:20:21,557 - Train: 3.41% [168300/4942000] [34.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 20:20:54,470 - Train: 3.41% [168400/4942000] [34.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:21:27,354 - Train: 3.41% [168500/4942000] [34.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:22:00,267 - Train: 3.41% [168600/4942000] [34.1/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 20:22:33,187 - Train: 3.41% [168700/4942000] [34.1/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 20:23:06,101 - Train: 3.42% [168800/4942000] [34.2/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 20:23:39,000 - Train: 3.42% [168900/4942000] [34.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:24:11,931 - Train: 3.42% [169000/4942000] [34.2/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 20:24:44,796 - Train: 3.42% [169100/4942000] [34.2/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 20:25:17,830 - Train: 3.42% [169200/4942000] [34.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:25:50,753 - Train: 3.43% [169300/4942000] [34.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:26:23,659 - Train: 3.43% [169400/4942000] [34.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:26:56,563 - Train: 3.43% [169500/4942000] [34.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:27:29,436 - Train: 3.43% [169600/4942000] [34.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:28:02,347 - Train: 3.43% [169700/4942000] [34.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:28:36,940 - Train: 3.44% [169800/4942000] [34.4/1000.0] [batch_t 0.331 (0.346)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:29:09,840 - Train: 3.44% [169900/4942000] [34.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:29:42,741 - Train: 3.44% [170000/4942000] [34.4/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 20:30:15,656 - Train: 3.44% [170100/4942000] [34.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 20:30:48,565 - Train: 3.44% [170200/4942000] [34.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:31:22,960 - Train: 3.45% [170300/4942000] [34.5/1000.0] [batch_t 0.330 (0.344)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:31:55,899 - Train: 3.45% [170400/4942000] [34.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:32:29,591 - Train: 3.45% [170500/4942000] [34.5/1000.0] [batch_t 0.331 (0.337)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 20:33:02,524 - Train: 3.45% [170600/4942000] [34.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:33:35,590 - Train: 3.45% [170700/4942000] [34.5/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:34:08,500 - Train: 3.46% [170800/4942000] [34.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:34:41,448 - Train: 3.46% [170900/4942000] [34.6/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 20:35:14,380 - Train: 3.46% [171000/4942000] [34.6/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 20:35:47,326 - Train: 3.46% [171100/4942000] [34.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:36:20,246 - Train: 3.46% [171200/4942000] [34.6/1000.0] [batch_t 0.337 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-03 20:36:53,160 - Train: 3.47% [171300/4942000] [34.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:37:26,105 - Train: 3.47% [171400/4942000] [34.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:37:59,044 - Train: 3.47% [171500/4942000] [34.7/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 20:38:33,641 - Train: 3.47% [171600/4942000] [34.7/1000.0] [batch_t 0.332 (0.346)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 20:39:07,972 - Train: 3.47% [171700/4942000] [34.7/1000.0] [batch_t 0.328 (0.343)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:39:40,892 - Train: 3.48% [171800/4942000] [34.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:40:15,029 - Train: 3.48% [171900/4942000] [34.8/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:40:47,971 - Train: 3.48% [172000/4942000] [34.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:41:21,483 - Train: 3.48% [172100/4942000] [34.8/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:41:54,465 - Train: 3.48% [172200/4942000] [34.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:42:27,390 - Train: 3.49% [172300/4942000] [34.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:43:00,293 - Train: 3.49% [172400/4942000] [34.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:43:33,226 - Train: 3.49% [172500/4942000] [34.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:44:06,147 - Train: 3.49% [172600/4942000] [34.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 20:44:39,058 - Train: 3.49% [172700/4942000] [34.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:45:11,961 - Train: 3.50% [172800/4942000] [35.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:45:44,865 - Train: 3.50% [172900/4942000] [35.0/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 20:46:07,869 - ==> Total time: 1 day, 2:48:47 Eta: 30 days, 19:16:28 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 20:46:19,061 - Train: 3.50% [173000/4942000] [35.0/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:46:51,925 - Train: 3.50% [173100/4942000] [35.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:47:25,750 - Train: 3.50% [173200/4942000] [35.0/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:47:58,678 - Train: 3.51% [173300/4942000] [35.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:48:31,716 - Train: 3.51% [173400/4942000] [35.1/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:49:04,578 - Train: 3.51% [173500/4942000] [35.1/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 20:49:38,074 - Train: 3.51% [173600/4942000] [35.1/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:50:11,112 - Train: 3.51% [173700/4942000] [35.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:50:44,000 - Train: 3.52% [173800/4942000] [35.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 20:51:18,184 - Train: 3.52% [173900/4942000] [35.2/1000.0] [batch_t 0.327 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:51:51,073 - Train: 3.52% [174000/4942000] [35.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:52:24,153 - Train: 3.52% [174100/4942000] [35.2/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:52:57,028 - Train: 3.52% [174200/4942000] [35.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:53:29,948 - Train: 3.53% [174300/4942000] [35.3/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 20:54:02,836 - Train: 3.53% [174400/4942000] [35.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:54:35,796 - Train: 3.53% [174500/4942000] [35.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:55:09,699 - Train: 3.53% [174600/4942000] [35.3/1000.0] [batch_t 0.323 (0.339)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 20:55:42,579 - Train: 3.54% [174700/4942000] [35.4/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 20:56:15,524 - Train: 3.54% [174800/4942000] [35.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:56:48,488 - Train: 3.54% [174900/4942000] [35.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:57:21,365 - Train: 3.54% [175000/4942000] [35.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:57:54,224 - Train: 3.54% [175100/4942000] [35.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:58:27,252 - Train: 3.55% [175200/4942000] [35.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:59:00,189 - Train: 3.55% [175300/4942000] [35.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:59:33,070 - Train: 3.55% [175400/4942000] [35.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:00:06,012 - Train: 3.55% [175500/4942000] [35.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:00:38,940 - Train: 3.55% [175600/4942000] [35.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 21:01:11,908 - Train: 3.56% [175700/4942000] [35.6/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 21:01:44,874 - Train: 3.56% [175800/4942000] [35.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:02:17,825 - Train: 3.56% [175900/4942000] [35.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 21:02:50,775 - Train: 3.56% [176000/4942000] [35.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:03:23,741 - Train: 3.56% [176100/4942000] [35.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:03:56,676 - Train: 3.57% [176200/4942000] [35.7/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 21:04:29,756 - Train: 3.57% [176300/4942000] [35.7/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:05:02,716 - Train: 3.57% [176400/4942000] [35.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 21:05:35,631 - Train: 3.57% [176500/4942000] [35.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:06:10,116 - Train: 3.57% [176600/4942000] [35.7/1000.0] [batch_t 0.332 (0.345)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 21:06:43,078 - Train: 3.58% [176700/4942000] [35.8/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 21:07:17,315 - Train: 3.58% [176800/4942000] [35.8/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:07:50,231 - Train: 3.58% [176900/4942000] [35.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:08:23,127 - Train: 3.58% [177000/4942000] [35.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:08:56,066 - Train: 3.58% [177100/4942000] [35.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:09:28,991 - Train: 3.59% [177200/4942000] [35.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:10:01,890 - Train: 3.59% [177300/4942000] [35.9/1000.0] [batch_t 0.321 (0.329)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-03 21:10:34,790 - Train: 3.59% [177400/4942000] [35.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:11:08,826 - Train: 3.59% [177500/4942000] [35.9/1000.0] [batch_t 0.321 (0.340)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-03 21:11:41,710 - Train: 3.59% [177600/4942000] [35.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:12:14,798 - Train: 3.60% [177700/4942000] [36.0/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:12:47,883 - Train: 3.60% [177800/4942000] [36.0/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:13:21,780 - Train: 3.60% [177900/4942000] [36.0/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 21:13:25,736 - ==> Total time: 1 day, 3:16:04 Eta: 30 days, 10:10:38 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 21:13:56,068 - Train: 3.60% [178000/4942000] [36.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:14:28,968 - Train: 3.60% [178100/4942000] [36.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:15:01,849 - Train: 3.61% [178200/4942000] [36.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 21:15:35,961 - Train: 3.61% [178300/4942000] [36.1/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:16:09,760 - Train: 3.61% [178400/4942000] [36.1/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:16:42,696 - Train: 3.61% [178500/4942000] [36.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:17:15,633 - Train: 3.61% [178600/4942000] [36.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:17:48,562 - Train: 3.62% [178700/4942000] [36.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:18:21,901 - Train: 3.62% [178800/4942000] [36.2/1000.0] [batch_t 0.326 (0.333)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 21:18:54,815 - Train: 3.62% [178900/4942000] [36.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:19:28,502 - Train: 3.62% [179000/4942000] [36.2/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:20:01,546 - Train: 3.62% [179100/4942000] [36.2/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:20:34,497 - Train: 3.63% [179200/4942000] [36.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:21:07,422 - Train: 3.63% [179300/4942000] [36.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:21:40,328 - Train: 3.63% [179400/4942000] [36.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:22:13,250 - Train: 3.63% [179500/4942000] [36.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:22:46,119 - Train: 3.63% [179600/4942000] [36.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:23:19,191 - Train: 3.64% [179700/4942000] [36.4/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:23:52,096 - Train: 3.64% [179800/4942000] [36.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 21:24:25,282 - Train: 3.64% [179900/4942000] [36.4/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:24:58,174 - Train: 3.64% [180000/4942000] [36.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:25:32,357 - Train: 3.64% [180100/4942000] [36.4/1000.0] [batch_t 0.328 (0.342)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 21:26:06,068 - Train: 3.65% [180200/4942000] [36.5/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 21:26:39,031 - Train: 3.65% [180300/4942000] [36.5/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 21:27:11,937 - Train: 3.65% [180400/4942000] [36.5/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 21:27:45,003 - Train: 3.65% [180500/4942000] [36.5/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:28:18,772 - Train: 3.65% [180600/4942000] [36.5/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 21:28:51,713 - Train: 3.66% [180700/4942000] [36.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:29:25,343 - Train: 3.66% [180800/4942000] [36.6/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:29:58,240 - Train: 3.66% [180900/4942000] [36.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:30:31,299 - Train: 3.66% [181000/4942000] [36.6/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:31:04,207 - Train: 3.66% [181100/4942000] [36.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 21:31:37,818 - Train: 3.67% [181200/4942000] [36.7/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:32:11,943 - Train: 3.67% [181300/4942000] [36.7/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:32:44,868 - Train: 3.67% [181400/4942000] [36.7/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 21:33:17,751 - Train: 3.67% [181500/4942000] [36.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:33:50,658 - Train: 3.67% [181600/4942000] [36.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:34:23,553 - Train: 3.68% [181700/4942000] [36.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:34:56,440 - Train: 3.68% [181800/4942000] [36.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:35:29,737 - Train: 3.68% [181900/4942000] [36.8/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:36:02,638 - Train: 3.68% [182000/4942000] [36.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:36:36,322 - Train: 3.68% [182100/4942000] [36.8/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:37:09,208 - Train: 3.69% [182200/4942000] [36.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:37:42,173 - Train: 3.69% [182300/4942000] [36.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 21:38:16,588 - Train: 3.69% [182400/4942000] [36.9/1000.0] [batch_t 0.331 (0.344)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 21:38:49,500 - Train: 3.69% [182500/4942000] [36.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:39:22,609 - Train: 3.69% [182600/4942000] [36.9/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:39:55,547 - Train: 3.70% [182700/4942000] [37.0/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 21:40:30,096 - Train: 3.70% [182800/4942000] [37.0/1000.0] [batch_t 0.329 (0.345)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:40:47,896 - ==> Total time: 1 day, 3:43:27 Eta: 30 days, 1:34:41 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 21:41:04,370 - Train: 3.70% [182900/4942000] [37.0/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:41:38,198 - Train: 3.70% [183000/4942000] [37.0/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:42:11,891 - Train: 3.70% [183100/4942000] [37.0/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 21:42:44,835 - Train: 3.71% [183200/4942000] [37.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 21:43:17,850 - Train: 3.71% [183300/4942000] [37.1/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 21:43:50,725 - Train: 3.71% [183400/4942000] [37.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:44:23,904 - Train: 3.71% [183500/4942000] [37.1/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 21:44:56,799 - Train: 3.72% [183600/4942000] [37.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 21:45:31,319 - Train: 3.72% [183700/4942000] [37.2/1000.0] [batch_t 0.328 (0.345)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:46:04,214 - Train: 3.72% [183800/4942000] [37.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 21:46:41,389 - Train: 3.72% [183900/4942000] [37.2/1000.0] [batch_t 0.326 (0.372)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 21:47:18,632 - Train: 3.72% [184000/4942000] [37.2/1000.0] [batch_t 0.330 (0.372)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 21:47:51,507 - Train: 3.73% [184100/4942000] [37.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:48:30,344 - Train: 3.73% [184200/4942000] [37.3/1000.0] [batch_t 0.326 (0.388)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 21:49:03,191 - Train: 3.73% [184300/4942000] [37.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:49:37,321 - Train: 3.73% [184400/4942000] [37.3/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:50:11,784 - Train: 3.73% [184500/4942000] [37.3/1000.0] [batch_t 0.328 (0.345)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:50:44,649 - Train: 3.74% [184600/4942000] [37.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:51:17,621 - Train: 3.74% [184700/4942000] [37.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 21:51:50,504 - Train: 3.74% [184800/4942000] [37.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:52:23,404 - Train: 3.74% [184900/4942000] [37.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:53:00,041 - Train: 3.74% [185000/4942000] [37.4/1000.0] [batch_t 0.488 (0.366)] [data_t 0.162] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:53:57,298 - Train: 3.75% [185100/4942000] [37.5/1000.0] [batch_t 1.016 (0.572)] [data_t 0.688] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:54:36,253 - Train: 3.75% [185200/4942000] [37.5/1000.0] [batch_t 0.328 (0.389)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:55:09,899 - Train: 3.75% [185300/4942000] [37.5/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:55:42,803 - Train: 3.75% [185400/4942000] [37.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:56:15,745 - Train: 3.75% [185500/4942000] [37.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:56:48,614 - Train: 3.76% [185600/4942000] [37.6/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 21:57:21,528 - Train: 3.76% [185700/4942000] [37.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 21:57:54,439 - Train: 3.76% [185800/4942000] [37.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:58:27,388 - Train: 3.76% [185900/4942000] [37.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:59:00,274 - Train: 3.76% [186000/4942000] [37.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:59:33,236 - Train: 3.77% [186100/4942000] [37.7/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 22:00:06,264 - Train: 3.77% [186200/4942000] [37.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:00:39,171 - Train: 3.77% [186300/4942000] [37.7/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 22:01:12,100 - Train: 3.77% [186400/4942000] [37.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:01:44,997 - Train: 3.77% [186500/4942000] [37.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:02:17,941 - Train: 3.78% [186600/4942000] [37.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:02:50,846 - Train: 3.78% [186700/4942000] [37.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 22:03:23,793 - Train: 3.78% [186800/4942000] [37.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:03:56,691 - Train: 3.78% [186900/4942000] [37.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:04:29,621 - Train: 3.78% [187000/4942000] [37.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:05:02,524 - Train: 3.79% [187100/4942000] [37.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:05:35,996 - Train: 3.79% [187200/4942000] [37.9/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:06:09,623 - Train: 3.79% [187300/4942000] [37.9/1000.0] [batch_t 0.325 (0.336)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 22:06:42,503 - Train: 3.79% [187400/4942000] [37.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:07:15,414 - Train: 3.79% [187500/4942000] [37.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:07:48,401 - Train: 3.80% [187600/4942000] [38.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:08:21,301 - Train: 3.80% [187700/4942000] [38.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:08:52,889 - ==> Total time: 1 day, 4:11:32 Eta: 29 days, 17:42:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 22:08:55,761 - Train: 3.80% [187800/4942000] [38.0/1000.0] [batch_t 0.329 (0.403)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:09:28,869 - Train: 3.80% [187900/4942000] [38.0/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:10:01,748 - Train: 3.80% [188000/4942000] [38.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:10:35,356 - Train: 3.81% [188100/4942000] [38.1/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:11:08,281 - Train: 3.81% [188200/4942000] [38.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 22:11:41,179 - Train: 3.81% [188300/4942000] [38.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:12:14,499 - Train: 3.81% [188400/4942000] [38.1/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:12:47,352 - Train: 3.81% [188500/4942000] [38.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:13:20,209 - Train: 3.82% [188600/4942000] [38.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:13:53,861 - Train: 3.82% [188700/4942000] [38.2/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:14:26,770 - Train: 3.82% [188800/4942000] [38.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:14:59,861 - Train: 3.82% [188900/4942000] [38.2/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:15:32,777 - Train: 3.82% [189000/4942000] [38.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:16:05,678 - Train: 3.83% [189100/4942000] [38.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:16:38,546 - Train: 3.83% [189200/4942000] [38.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:17:12,305 - Train: 3.83% [189300/4942000] [38.3/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:17:45,196 - Train: 3.83% [189400/4942000] [38.3/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 22:18:18,055 - Train: 3.83% [189500/4942000] [38.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:18:50,951 - Train: 3.84% [189600/4942000] [38.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:19:23,824 - Train: 3.84% [189700/4942000] [38.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:19:56,710 - Train: 3.84% [189800/4942000] [38.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:20:29,597 - Train: 3.84% [189900/4942000] [38.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:21:02,502 - Train: 3.84% [190000/4942000] [38.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:21:36,718 - Train: 3.85% [190100/4942000] [38.5/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:22:09,626 - Train: 3.85% [190200/4942000] [38.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:22:42,510 - Train: 3.85% [190300/4942000] [38.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:23:15,451 - Train: 3.85% [190400/4942000] [38.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:23:48,354 - Train: 3.85% [190500/4942000] [38.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:24:21,305 - Train: 3.86% [190600/4942000] [38.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:24:54,204 - Train: 3.86% [190700/4942000] [38.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:25:27,089 - Train: 3.86% [190800/4942000] [38.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:26:00,011 - Train: 3.86% [190900/4942000] [38.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:26:32,953 - Train: 3.86% [191000/4942000] [38.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:27:06,427 - Train: 3.87% [191100/4942000] [38.7/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:27:39,376 - Train: 3.87% [191200/4942000] [38.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:28:12,559 - Train: 3.87% [191300/4942000] [38.7/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:28:45,437 - Train: 3.87% [191400/4942000] [38.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:29:18,325 - Train: 3.87% [191500/4942000] [38.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:29:51,221 - Train: 3.88% [191600/4942000] [38.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:30:24,086 - Train: 3.88% [191700/4942000] [38.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:30:57,071 - Train: 3.88% [191800/4942000] [38.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:31:29,953 - Train: 3.88% [191900/4942000] [38.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:32:02,844 - Train: 3.89% [192000/4942000] [38.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:32:35,745 - Train: 3.89% [192100/4942000] [38.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:33:08,636 - Train: 3.89% [192200/4942000] [38.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 22:33:41,591 - Train: 3.89% [192300/4942000] [38.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:34:14,490 - Train: 3.89% [192400/4942000] [38.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:34:47,405 - Train: 3.90% [192500/4942000] [39.0/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 22:35:20,346 - Train: 3.90% [192600/4942000] [39.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:35:53,292 - Train: 3.90% [192700/4942000] [39.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:36:05,819 - ==> Total time: 1 day, 4:38:45 Eta: 29 days, 9:51:45 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 22:36:27,999 - Train: 3.90% [192800/4942000] [39.0/1000.0] [batch_t 0.326 (0.332)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 22:37:00,911 - Train: 3.90% [192900/4942000] [39.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:37:33,883 - Train: 3.91% [193000/4942000] [39.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 22:38:06,923 - Train: 3.91% [193100/4942000] [39.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:38:39,869 - Train: 3.91% [193200/4942000] [39.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 22:39:12,806 - Train: 3.91% [193300/4942000] [39.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:39:45,757 - Train: 3.91% [193400/4942000] [39.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:40:18,699 - Train: 3.92% [193500/4942000] [39.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:40:51,641 - Train: 3.92% [193600/4942000] [39.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:41:24,602 - Train: 3.92% [193700/4942000] [39.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:41:57,551 - Train: 3.92% [193800/4942000] [39.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:42:30,485 - Train: 3.92% [193900/4942000] [39.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:43:03,399 - Train: 3.93% [194000/4942000] [39.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:43:36,373 - Train: 3.93% [194100/4942000] [39.3/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 22:44:09,308 - Train: 3.93% [194200/4942000] [39.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:44:42,244 - Train: 3.93% [194300/4942000] [39.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:45:15,201 - Train: 3.93% [194400/4942000] [39.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:45:48,246 - Train: 3.94% [194500/4942000] [39.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 22:46:22,187 - Train: 3.94% [194600/4942000] [39.4/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:46:55,152 - Train: 3.94% [194700/4942000] [39.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:47:28,096 - Train: 3.94% [194800/4942000] [39.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 22:48:01,046 - Train: 3.94% [194900/4942000] [39.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:48:34,863 - Train: 3.95% [195000/4942000] [39.5/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 22:49:07,790 - Train: 3.95% [195100/4942000] [39.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:49:40,752 - Train: 3.95% [195200/4942000] [39.5/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:50:13,705 - Train: 3.95% [195300/4942000] [39.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:50:46,649 - Train: 3.95% [195400/4942000] [39.5/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 22:51:19,661 - Train: 3.96% [195500/4942000] [39.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:51:52,595 - Train: 3.96% [195600/4942000] [39.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:52:26,439 - Train: 3.96% [195700/4942000] [39.6/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:52:59,360 - Train: 3.96% [195800/4942000] [39.6/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 22:53:32,316 - Train: 3.96% [195900/4942000] [39.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 22:54:05,339 - Train: 3.97% [196000/4942000] [39.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:54:38,494 - Train: 3.97% [196100/4942000] [39.7/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 22:55:12,875 - Train: 3.97% [196200/4942000] [39.7/1000.0] [batch_t 0.329 (0.344)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:55:45,795 - Train: 3.97% [196300/4942000] [39.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:56:19,660 - Train: 3.97% [196400/4942000] [39.7/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:56:52,581 - Train: 3.98% [196500/4942000] [39.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:57:26,594 - Train: 3.98% [196600/4942000] [39.8/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:57:59,556 - Train: 3.98% [196700/4942000] [39.8/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 22:58:32,499 - Train: 3.98% [196800/4942000] [39.8/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 22:59:05,481 - Train: 3.98% [196900/4942000] [39.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:59:38,382 - Train: 3.99% [197000/4942000] [39.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 23:00:11,364 - Train: 3.99% [197100/4942000] [39.9/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:00:44,352 - Train: 3.99% [197200/4942000] [39.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:01:17,267 - Train: 3.99% [197300/4942000] [39.9/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 23:01:50,305 - Train: 3.99% [197400/4942000] [39.9/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 23:02:23,289 - Train: 4.00% [197500/4942000] [40.0/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:02:56,249 - Train: 4.00% [197600/4942000] [40.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:03:22,594 - ==> Total time: 1 day, 5:06:01 Eta: 29 days, 2:24:42 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 23:03:30,541 - Train: 4.00% [197700/4942000] [40.0/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:04:03,476 - Train: 4.00% [197800/4942000] [40.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:04:36,387 - Train: 4.00% [197900/4942000] [40.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:05:09,264 - Train: 4.01% [198000/4942000] [40.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 23:05:42,118 - Train: 4.01% [198100/4942000] [40.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 23:06:15,025 - Train: 4.01% [198200/4942000] [40.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:06:47,941 - Train: 4.01% [198300/4942000] [40.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 23:07:20,882 - Train: 4.01% [198400/4942000] [40.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 23:07:53,805 - Train: 4.02% [198500/4942000] [40.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:08:26,734 - Train: 4.02% [198600/4942000] [40.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:08:59,801 - Train: 4.02% [198700/4942000] [40.2/1000.0] [batch_t 0.325 (0.331)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 23:09:32,680 - Train: 4.02% [198800/4942000] [40.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:10:05,593 - Train: 4.02% [198900/4942000] [40.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 23:10:38,533 - Train: 4.03% [199000/4942000] [40.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:11:11,387 - Train: 4.03% [199100/4942000] [40.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 23:11:44,297 - Train: 4.03% [199200/4942000] [40.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:12:17,201 - Train: 4.03% [199300/4942000] [40.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:12:50,108 - Train: 4.03% [199400/4942000] [40.3/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 23:13:23,017 - Train: 4.04% [199500/4942000] [40.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:13:55,894 - Train: 4.04% [199600/4942000] [40.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 23:14:28,763 - Train: 4.04% [199700/4942000] [40.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:15:01,631 - Train: 4.04% [199800/4942000] [40.4/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 23:15:34,506 - Train: 4.04% [199900/4942000] [40.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 23:16:07,430 - Train: 4.05% [200000/4942000] [40.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 23:16:40,446 - Train: 4.05% [200100/4942000] [40.5/1000.0] [batch_t 0.422 (0.330)] [data_t 0.002] [optim_t 0.419] [lr 0.005000] 2024-04-03 23:17:13,333 - Train: 4.05% [200200/4942000] [40.5/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 23:17:46,224 - Train: 4.05% [200300/4942000] [40.5/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 23:18:19,169 - Train: 4.06% [200400/4942000] [40.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:18:52,094 - Train: 4.06% [200500/4942000] [40.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:19:25,947 - Train: 4.06% [200600/4942000] [40.6/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:19:58,873 - Train: 4.06% [200700/4942000] [40.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 23:20:31,827 - Train: 4.06% [200800/4942000] [40.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:21:04,769 - Train: 4.07% [200900/4942000] [40.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:21:37,663 - Train: 4.07% [201000/4942000] [40.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-03 23:22:10,587 - Train: 4.07% [201100/4942000] [40.7/1000.0] [batch_t 0.337 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-03 23:22:43,537 - Train: 4.07% [201200/4942000] [40.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:23:16,500 - Train: 4.07% [201300/4942000] [40.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:23:49,453 - Train: 4.08% [201400/4942000] [40.8/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 23:24:22,428 - Train: 4.08% [201500/4942000] [40.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:24:55,457 - Train: 4.08% [201600/4942000] [40.8/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 23:25:28,420 - Train: 4.08% [201700/4942000] [40.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:26:01,363 - Train: 4.08% [201800/4942000] [40.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-03 23:26:34,332 - Train: 4.09% [201900/4942000] [40.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:27:07,305 - Train: 4.09% [202000/4942000] [40.9/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:27:40,223 - Train: 4.09% [202100/4942000] [40.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:28:13,174 - Train: 4.09% [202200/4942000] [40.9/1000.0] [batch_t 0.333 (0.329)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-03 23:28:46,127 - Train: 4.09% [202300/4942000] [40.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:29:19,057 - Train: 4.10% [202400/4942000] [41.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 23:29:51,997 - Train: 4.10% [202500/4942000] [41.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 23:30:24,943 - Train: 4.10% [202600/4942000] [41.0/1000.0] [batch_t 0.334 (0.329)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-03 23:30:32,195 - ==> Total time: 1 day, 5:33:11 Eta: 28 days, 19:15:20 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 23:30:59,475 - Train: 4.10% [202700/4942000] [41.0/1000.0] [batch_t 0.325 (0.333)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 23:31:32,425 - Train: 4.10% [202800/4942000] [41.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:32:05,565 - Train: 4.11% [202900/4942000] [41.1/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 23:32:38,479 - Train: 4.11% [203000/4942000] [41.1/1000.0] [batch_t 0.333 (0.329)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-03 23:33:12,159 - Train: 4.11% [203100/4942000] [41.1/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 23:33:45,081 - Train: 4.11% [203200/4942000] [41.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:34:19,350 - Train: 4.11% [203300/4942000] [41.1/1000.0] [batch_t 0.326 (0.343)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 23:34:52,251 - Train: 4.12% [203400/4942000] [41.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:35:25,175 - Train: 4.12% [203500/4942000] [41.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:35:58,134 - Train: 4.12% [203600/4942000] [41.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:36:32,652 - Train: 4.12% [203700/4942000] [41.2/1000.0] [batch_t 0.329 (0.345)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:37:06,380 - Train: 4.12% [203800/4942000] [41.2/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:37:39,335 - Train: 4.13% [203900/4942000] [41.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:38:12,283 - Train: 4.13% [204000/4942000] [41.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 23:38:45,231 - Train: 4.13% [204100/4942000] [41.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:39:19,178 - Train: 4.13% [204200/4942000] [41.3/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:39:52,066 - Train: 4.13% [204300/4942000] [41.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:40:25,225 - Train: 4.14% [204400/4942000] [41.4/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:40:58,154 - Train: 4.14% [204500/4942000] [41.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:41:31,330 - Train: 4.14% [204600/4942000] [41.4/1000.0] [batch_t 0.324 (0.332)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 23:42:04,288 - Train: 4.14% [204700/4942000] [41.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:42:38,078 - Train: 4.14% [204800/4942000] [41.4/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 23:43:12,469 - Train: 4.15% [204900/4942000] [41.5/1000.0] [batch_t 0.327 (0.344)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:43:45,392 - Train: 4.15% [205000/4942000] [41.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:44:18,311 - Train: 4.15% [205100/4942000] [41.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 23:44:51,234 - Train: 4.15% [205200/4942000] [41.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:45:24,313 - Train: 4.15% [205300/4942000] [41.5/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:45:57,199 - Train: 4.16% [205400/4942000] [41.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:46:30,062 - Train: 4.16% [205500/4942000] [41.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:47:02,936 - Train: 4.16% [205600/4942000] [41.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:47:35,826 - Train: 4.16% [205700/4942000] [41.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 23:48:09,061 - Train: 4.16% [205800/4942000] [41.6/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:48:41,962 - Train: 4.17% [205900/4942000] [41.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:49:14,937 - Train: 4.17% [206000/4942000] [41.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 23:49:47,862 - Train: 4.17% [206100/4942000] [41.7/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 23:50:22,130 - Train: 4.17% [206200/4942000] [41.7/1000.0] [batch_t 0.327 (0.343)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:50:55,046 - Train: 4.17% [206300/4942000] [41.7/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 23:51:28,786 - Train: 4.18% [206400/4942000] [41.8/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:52:01,641 - Train: 4.18% [206500/4942000] [41.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:52:34,629 - Train: 4.18% [206600/4942000] [41.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:53:08,629 - Train: 4.18% [206700/4942000] [41.8/1000.0] [batch_t 0.330 (0.340)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 23:53:41,548 - Train: 4.18% [206800/4942000] [41.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:54:14,635 - Train: 4.19% [206900/4942000] [41.9/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 23:54:47,535 - Train: 4.19% [207000/4942000] [41.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 23:55:20,554 - Train: 4.19% [207100/4942000] [41.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:55:53,475 - Train: 4.19% [207200/4942000] [41.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:56:26,452 - Train: 4.19% [207300/4942000] [41.9/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 23:56:59,341 - Train: 4.20% [207400/4942000] [42.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:57:32,231 - Train: 4.20% [207500/4942000] [42.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:57:53,280 - ==> Total time: 1 day, 6:00:32 Eta: 28 days, 12:29:29 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-03 23:58:06,531 - Train: 4.20% [207600/4942000] [42.0/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:58:39,427 - Train: 4.20% [207700/4942000] [42.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:59:13,461 - Train: 4.20% [207800/4942000] [42.0/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:59:46,325 - Train: 4.21% [207900/4942000] [42.1/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 00:00:19,656 - Train: 4.21% [208000/4942000] [42.1/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 00:00:52,570 - Train: 4.21% [208100/4942000] [42.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 00:01:25,460 - Train: 4.21% [208200/4942000] [42.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 00:01:58,368 - Train: 4.21% [208300/4942000] [42.1/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 00:02:31,256 - Train: 4.22% [208400/4942000] [42.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:03:04,177 - Train: 4.22% [208500/4942000] [42.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 00:03:37,208 - Train: 4.22% [208600/4942000] [42.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 00:04:10,167 - Train: 4.22% [208700/4942000] [42.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 00:04:43,051 - Train: 4.23% [208800/4942000] [42.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 00:05:15,949 - Train: 4.23% [208900/4942000] [42.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:05:48,938 - Train: 4.23% [209000/4942000] [42.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 00:06:21,855 - Train: 4.23% [209100/4942000] [42.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 00:06:54,801 - Train: 4.23% [209200/4942000] [42.3/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 00:07:27,676 - Train: 4.24% [209300/4942000] [42.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 00:08:00,608 - Train: 4.24% [209400/4942000] [42.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:08:33,525 - Train: 4.24% [209500/4942000] [42.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 00:09:06,452 - Train: 4.24% [209600/4942000] [42.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 00:09:39,339 - Train: 4.24% [209700/4942000] [42.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 00:10:12,260 - Train: 4.25% [209800/4942000] [42.5/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 00:10:45,175 - Train: 4.25% [209900/4942000] [42.5/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 00:11:18,156 - Train: 4.25% [210000/4942000] [42.5/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 00:11:51,049 - Train: 4.25% [210100/4942000] [42.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 00:12:24,025 - Train: 4.25% [210200/4942000] [42.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:12:57,008 - Train: 4.26% [210300/4942000] [42.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 00:13:29,964 - Train: 4.26% [210400/4942000] [42.6/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 00:14:02,912 - Train: 4.26% [210500/4942000] [42.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 00:14:35,878 - Train: 4.26% [210600/4942000] [42.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:15:09,743 - Train: 4.26% [210700/4942000] [42.6/1000.0] [batch_t 0.332 (0.339)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 00:15:42,635 - Train: 4.27% [210800/4942000] [42.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 00:16:16,105 - Train: 4.27% [210900/4942000] [42.7/1000.0] [batch_t 0.331 (0.335)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-04 00:16:49,074 - Train: 4.27% [211000/4942000] [42.7/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 00:17:22,549 - Train: 4.27% [211100/4942000] [42.7/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:17:55,412 - Train: 4.27% [211200/4942000] [42.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 00:18:31,546 - Train: 4.28% [211300/4942000] [42.8/1000.0] [batch_t 0.330 (0.361)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:19:08,407 - Train: 4.28% [211400/4942000] [42.8/1000.0] [batch_t 1.734 (0.369)] [data_t 1.407] [optim_t 0.328] [lr 0.005000] 2024-04-04 00:19:43,054 - Train: 4.28% [211500/4942000] [42.8/1000.0] [batch_t 0.331 (0.346)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-04 00:20:17,490 - Train: 4.28% [211600/4942000] [42.8/1000.0] [batch_t 0.330 (0.344)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 00:20:50,373 - Train: 4.28% [211700/4942000] [42.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 00:21:24,916 - Train: 4.29% [211800/4942000] [42.9/1000.0] [batch_t 0.330 (0.345)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 00:21:57,821 - Train: 4.29% [211900/4942000] [42.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 00:22:31,587 - Train: 4.29% [212000/4942000] [42.9/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 00:23:04,528 - Train: 4.29% [212100/4942000] [42.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:23:38,038 - Train: 4.29% [212200/4942000] [42.9/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 00:24:10,954 - Train: 4.30% [212300/4942000] [43.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 00:24:43,868 - Train: 4.30% [212400/4942000] [43.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 00:25:16,785 - Train: 4.30% [212500/4942000] [43.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 00:25:18,760 - ==> Total time: 1 day, 6:27:57 Eta: 28 days, 6:02:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 00:25:51,169 - Train: 4.30% [212600/4942000] [43.0/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 00:26:24,104 - Train: 4.30% [212700/4942000] [43.0/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 00:26:57,200 - Train: 4.31% [212800/4942000] [43.1/1000.0] [batch_t 0.324 (0.331)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 00:27:30,144 - Train: 4.31% [212900/4942000] [43.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 00:28:03,091 - Train: 4.31% [213000/4942000] [43.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 00:28:36,034 - Train: 4.31% [213100/4942000] [43.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 00:29:08,982 - Train: 4.31% [213200/4942000] [43.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 00:29:41,923 - Train: 4.32% [213300/4942000] [43.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 00:30:14,880 - Train: 4.32% [213400/4942000] [43.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 00:30:47,850 - Train: 4.32% [213500/4942000] [43.2/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 00:31:20,813 - Train: 4.32% [213600/4942000] [43.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 00:31:53,781 - Train: 4.32% [213700/4942000] [43.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:32:26,743 - Train: 4.33% [213800/4942000] [43.3/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 00:32:59,728 - Train: 4.33% [213900/4942000] [43.3/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 00:33:32,705 - Train: 4.33% [214000/4942000] [43.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 00:34:05,646 - Train: 4.33% [214100/4942000] [43.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 00:34:38,718 - Train: 4.33% [214200/4942000] [43.3/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:35:11,644 - Train: 4.34% [214300/4942000] [43.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:35:44,593 - Train: 4.34% [214400/4942000] [43.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:36:17,574 - Train: 4.34% [214500/4942000] [43.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-04 00:36:50,573 - Train: 4.34% [214600/4942000] [43.4/1000.0] [batch_t 0.333 (0.330)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-04 00:37:23,541 - Train: 4.34% [214700/4942000] [43.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 00:37:56,440 - Train: 4.35% [214800/4942000] [43.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:38:29,398 - Train: 4.35% [214900/4942000] [43.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 00:39:02,318 - Train: 4.35% [215000/4942000] [43.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:39:35,212 - Train: 4.35% [215100/4942000] [43.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:40:08,123 - Train: 4.35% [215200/4942000] [43.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:40:41,069 - Train: 4.36% [215300/4942000] [43.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 00:41:14,067 - Train: 4.36% [215400/4942000] [43.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 00:41:47,010 - Train: 4.36% [215500/4942000] [43.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 00:42:20,062 - Train: 4.36% [215600/4942000] [43.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-04 00:42:53,020 - Train: 4.36% [215700/4942000] [43.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 00:43:25,986 - Train: 4.37% [215800/4942000] [43.7/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 00:43:58,940 - Train: 4.37% [215900/4942000] [43.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:44:31,916 - Train: 4.37% [216000/4942000] [43.7/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 00:45:04,894 - Train: 4.37% [216100/4942000] [43.7/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 00:45:37,861 - Train: 4.37% [216200/4942000] [43.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 00:46:10,837 - Train: 4.38% [216300/4942000] [43.8/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 00:46:43,818 - Train: 4.38% [216400/4942000] [43.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 00:47:16,790 - Train: 4.38% [216500/4942000] [43.8/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 00:47:49,709 - Train: 4.38% [216600/4942000] [43.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 00:48:22,723 - Train: 4.38% [216700/4942000] [43.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 00:48:55,662 - Train: 4.39% [216800/4942000] [43.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:49:28,594 - Train: 4.39% [216900/4942000] [43.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:50:01,645 - Train: 4.39% [217000/4942000] [43.9/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 00:50:34,597 - Train: 4.39% [217100/4942000] [43.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 00:51:08,948 - Train: 4.39% [217200/4942000] [43.9/1000.0] [batch_t 0.324 (0.343)] [data_t 0.003] [optim_t 0.321] [lr 0.005000] 2024-04-04 00:51:41,924 - Train: 4.40% [217300/4942000] [44.0/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 00:52:14,827 - Train: 4.40% [217400/4942000] [44.0/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 00:52:30,646 - ==> Total time: 1 day, 6:55:09 Eta: 27 days, 23:47:39 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 00:52:49,153 - Train: 4.40% [217500/4942000] [44.0/1000.0] [batch_t 0.326 (0.333)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 00:53:22,062 - Train: 4.40% [217600/4942000] [44.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 00:53:54,999 - Train: 4.41% [217700/4942000] [44.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:54:28,681 - Train: 4.41% [217800/4942000] [44.1/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:55:01,598 - Train: 4.41% [217900/4942000] [44.1/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 00:55:34,475 - Train: 4.41% [218000/4942000] [44.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:56:07,441 - Train: 4.41% [218100/4942000] [44.1/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 00:56:40,379 - Train: 4.42% [218200/4942000] [44.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 00:57:13,341 - Train: 4.42% [218300/4942000] [44.2/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 00:57:46,349 - Train: 4.42% [218400/4942000] [44.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 00:58:19,262 - Train: 4.42% [218500/4942000] [44.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 00:58:52,174 - Train: 4.42% [218600/4942000] [44.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 00:59:25,054 - Train: 4.43% [218700/4942000] [44.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 00:59:57,953 - Train: 4.43% [218800/4942000] [44.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 01:00:30,870 - Train: 4.43% [218900/4942000] [44.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 01:01:03,843 - Train: 4.43% [219000/4942000] [44.3/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:01:36,750 - Train: 4.43% [219100/4942000] [44.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 01:02:09,704 - Train: 4.44% [219200/4942000] [44.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 01:02:42,625 - Train: 4.44% [219300/4942000] [44.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:03:15,545 - Train: 4.44% [219400/4942000] [44.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 01:03:48,444 - Train: 4.44% [219500/4942000] [44.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:04:21,341 - Train: 4.44% [219600/4942000] [44.4/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 01:04:54,250 - Train: 4.45% [219700/4942000] [44.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 01:05:27,335 - Train: 4.45% [219800/4942000] [44.5/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:06:00,287 - Train: 4.45% [219900/4942000] [44.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:06:33,219 - Train: 4.45% [220000/4942000] [44.5/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 01:07:06,825 - Train: 4.45% [220100/4942000] [44.5/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 01:07:39,760 - Train: 4.46% [220200/4942000] [44.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:08:12,705 - Train: 4.46% [220300/4942000] [44.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 01:08:45,629 - Train: 4.46% [220400/4942000] [44.6/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 01:09:18,582 - Train: 4.46% [220500/4942000] [44.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 01:09:51,489 - Train: 4.46% [220600/4942000] [44.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:10:24,426 - Train: 4.47% [220700/4942000] [44.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 01:10:57,381 - Train: 4.47% [220800/4942000] [44.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:11:30,332 - Train: 4.47% [220900/4942000] [44.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-04 01:12:03,282 - Train: 4.47% [221000/4942000] [44.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 01:12:37,308 - Train: 4.47% [221100/4942000] [44.7/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:13:10,369 - Train: 4.48% [221200/4942000] [44.8/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:13:43,274 - Train: 4.48% [221300/4942000] [44.8/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 01:14:16,170 - Train: 4.48% [221400/4942000] [44.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:14:49,050 - Train: 4.48% [221500/4942000] [44.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:15:22,454 - Train: 4.48% [221600/4942000] [44.8/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 01:15:55,359 - Train: 4.49% [221700/4942000] [44.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 01:16:29,236 - Train: 4.49% [221800/4942000] [44.9/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:17:02,127 - Train: 4.49% [221900/4942000] [44.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:17:35,435 - Train: 4.49% [222000/4942000] [44.9/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 01:18:08,340 - Train: 4.49% [222100/4942000] [44.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:18:41,261 - Train: 4.50% [222200/4942000] [45.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:19:14,161 - Train: 4.50% [222300/4942000] [45.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 01:19:43,793 - ==> Total time: 1 day, 7:22:22 Eta: 27 days, 17:48:20 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 01:19:48,451 - Train: 4.50% [222400/4942000] [45.0/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:20:21,420 - Train: 4.50% [222500/4942000] [45.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:20:54,474 - Train: 4.50% [222600/4942000] [45.0/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 01:21:27,362 - Train: 4.51% [222700/4942000] [45.1/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 01:22:00,349 - Train: 4.51% [222800/4942000] [45.1/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 01:22:33,290 - Train: 4.51% [222900/4942000] [45.1/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:23:06,240 - Train: 4.51% [223000/4942000] [45.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:23:39,129 - Train: 4.51% [223100/4942000] [45.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 01:24:12,037 - Train: 4.52% [223200/4942000] [45.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 01:24:44,946 - Train: 4.52% [223300/4942000] [45.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:25:17,860 - Train: 4.52% [223400/4942000] [45.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:25:50,765 - Train: 4.52% [223500/4942000] [45.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:26:23,732 - Train: 4.52% [223600/4942000] [45.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:26:56,638 - Train: 4.53% [223700/4942000] [45.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:27:29,575 - Train: 4.53% [223800/4942000] [45.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:28:02,504 - Train: 4.53% [223900/4942000] [45.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:28:35,446 - Train: 4.53% [224000/4942000] [45.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 01:29:08,378 - Train: 4.53% [224100/4942000] [45.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:29:41,284 - Train: 4.54% [224200/4942000] [45.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:30:14,231 - Train: 4.54% [224300/4942000] [45.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 01:30:47,179 - Train: 4.54% [224400/4942000] [45.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:31:20,085 - Train: 4.54% [224500/4942000] [45.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 01:31:53,026 - Train: 4.54% [224600/4942000] [45.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 01:32:26,682 - Train: 4.55% [224700/4942000] [45.5/1000.0] [batch_t 0.329 (0.336)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:32:59,610 - Train: 4.55% [224800/4942000] [45.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 01:33:32,583 - Train: 4.55% [224900/4942000] [45.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:34:05,538 - Train: 4.55% [225000/4942000] [45.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 01:34:38,507 - Train: 4.55% [225100/4942000] [45.5/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 01:35:11,463 - Train: 4.56% [225200/4942000] [45.6/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 01:35:44,389 - Train: 4.56% [225300/4942000] [45.6/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 01:36:17,470 - Train: 4.56% [225400/4942000] [45.6/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 01:36:50,396 - Train: 4.56% [225500/4942000] [45.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:37:23,327 - Train: 4.56% [225600/4942000] [45.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 01:37:56,251 - Train: 4.57% [225700/4942000] [45.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 01:38:29,190 - Train: 4.57% [225800/4942000] [45.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 01:39:02,142 - Train: 4.57% [225900/4942000] [45.7/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 01:39:35,111 - Train: 4.57% [226000/4942000] [45.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:40:08,710 - Train: 4.58% [226100/4942000] [45.8/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:40:41,664 - Train: 4.58% [226200/4942000] [45.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:41:14,656 - Train: 4.58% [226300/4942000] [45.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:41:47,650 - Train: 4.58% [226400/4942000] [45.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:42:20,629 - Train: 4.58% [226500/4942000] [45.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:42:53,611 - Train: 4.59% [226600/4942000] [45.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:43:27,339 - Train: 4.59% [226700/4942000] [45.9/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:44:00,486 - Train: 4.59% [226800/4942000] [45.9/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 01:44:33,477 - Train: 4.59% [226900/4942000] [45.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 01:45:06,406 - Train: 4.59% [227000/4942000] [45.9/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 01:45:39,400 - Train: 4.60% [227100/4942000] [46.0/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 01:46:12,376 - Train: 4.60% [227200/4942000] [46.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:46:45,276 - Train: 4.60% [227300/4942000] [46.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 01:46:55,815 - ==> Total time: 1 day, 7:49:35 Eta: 27 days, 12:03:05 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 01:47:19,598 - Train: 4.60% [227400/4942000] [46.0/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:47:52,529 - Train: 4.60% [227500/4942000] [46.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:48:25,509 - Train: 4.61% [227600/4942000] [46.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:48:58,467 - Train: 4.61% [227700/4942000] [46.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:49:31,340 - Train: 4.61% [227800/4942000] [46.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 01:50:04,239 - Train: 4.61% [227900/4942000] [46.1/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 01:50:37,216 - Train: 4.61% [228000/4942000] [46.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:51:10,236 - Train: 4.62% [228100/4942000] [46.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:51:43,219 - Train: 4.62% [228200/4942000] [46.2/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 01:52:16,217 - Train: 4.62% [228300/4942000] [46.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 01:52:49,194 - Train: 4.62% [228400/4942000] [46.2/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 01:53:22,154 - Train: 4.62% [228500/4942000] [46.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:53:55,086 - Train: 4.63% [228600/4942000] [46.3/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 01:54:28,051 - Train: 4.63% [228700/4942000] [46.3/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 01:55:01,058 - Train: 4.63% [228800/4942000] [46.3/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:55:34,024 - Train: 4.63% [228900/4942000] [46.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:56:07,016 - Train: 4.63% [229000/4942000] [46.3/1000.0] [batch_t 0.339 (0.330)] [data_t 0.003] [optim_t 0.337] [lr 0.005000] 2024-04-04 01:56:40,012 - Train: 4.64% [229100/4942000] [46.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 01:57:13,053 - Train: 4.64% [229200/4942000] [46.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:57:46,090 - Train: 4.64% [229300/4942000] [46.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:58:19,084 - Train: 4.64% [229400/4942000] [46.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:58:52,042 - Train: 4.64% [229500/4942000] [46.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:59:25,815 - Train: 4.65% [229600/4942000] [46.5/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:59:58,748 - Train: 4.65% [229700/4942000] [46.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:00:31,668 - Train: 4.65% [229800/4942000] [46.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 02:01:04,542 - Train: 4.65% [229900/4942000] [46.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:01:37,444 - Train: 4.65% [230000/4942000] [46.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 02:02:10,326 - Train: 4.66% [230100/4942000] [46.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 02:02:43,243 - Train: 4.66% [230200/4942000] [46.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:03:16,201 - Train: 4.66% [230300/4942000] [46.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:03:49,145 - Train: 4.66% [230400/4942000] [46.6/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 02:04:22,118 - Train: 4.66% [230500/4942000] [46.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:04:55,096 - Train: 4.67% [230600/4942000] [46.7/1000.0] [batch_t 0.323 (0.330)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-04 02:05:28,060 - Train: 4.67% [230700/4942000] [46.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-04 02:06:01,011 - Train: 4.67% [230800/4942000] [46.7/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 02:06:34,030 - Train: 4.67% [230900/4942000] [46.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:07:07,105 - Train: 4.67% [231000/4942000] [46.7/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 02:07:40,063 - Train: 4.68% [231100/4942000] [46.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:08:12,979 - Train: 4.68% [231200/4942000] [46.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:08:45,854 - Train: 4.68% [231300/4942000] [46.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:09:18,751 - Train: 4.68% [231400/4942000] [46.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:09:51,628 - Train: 4.68% [231500/4942000] [46.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:10:24,524 - Train: 4.69% [231600/4942000] [46.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:10:57,447 - Train: 4.69% [231700/4942000] [46.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:11:30,387 - Train: 4.69% [231800/4942000] [46.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:12:03,275 - Train: 4.69% [231900/4942000] [46.9/1000.0] [batch_t 0.325 (0.329)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-04 02:12:36,174 - Train: 4.69% [232000/4942000] [46.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:13:09,056 - Train: 4.70% [232100/4942000] [47.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:13:41,978 - Train: 4.70% [232200/4942000] [47.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 02:14:06,304 - ==> Total time: 1 day, 8:16:45 Eta: 27 days, 6:30:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 02:14:16,488 - Train: 4.70% [232300/4942000] [47.0/1000.0] [batch_t 0.325 (0.336)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 02:14:49,388 - Train: 4.70% [232400/4942000] [47.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 02:15:22,319 - Train: 4.70% [232500/4942000] [47.0/1000.0] [batch_t 0.333 (0.329)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-04 02:15:55,304 - Train: 4.71% [232600/4942000] [47.1/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 02:16:28,282 - Train: 4.71% [232700/4942000] [47.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:17:01,227 - Train: 4.71% [232800/4942000] [47.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:17:34,176 - Train: 4.71% [232900/4942000] [47.1/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 02:18:07,144 - Train: 4.71% [233000/4942000] [47.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:18:40,068 - Train: 4.72% [233100/4942000] [47.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 02:19:12,986 - Train: 4.72% [233200/4942000] [47.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:19:45,886 - Train: 4.72% [233300/4942000] [47.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 02:20:18,831 - Train: 4.72% [233400/4942000] [47.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:20:51,825 - Train: 4.72% [233500/4942000] [47.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:21:24,746 - Train: 4.73% [233600/4942000] [47.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 02:21:57,673 - Train: 4.73% [233700/4942000] [47.3/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 02:22:30,676 - Train: 4.73% [233800/4942000] [47.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:23:03,612 - Train: 4.73% [233900/4942000] [47.3/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 02:23:36,506 - Train: 4.73% [234000/4942000] [47.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:24:09,456 - Train: 4.74% [234100/4942000] [47.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 02:24:42,339 - Train: 4.74% [234200/4942000] [47.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 02:25:15,303 - Train: 4.74% [234300/4942000] [47.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 02:25:48,245 - Train: 4.74% [234400/4942000] [47.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:26:22,290 - Train: 4.75% [234500/4942000] [47.5/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:26:55,231 - Train: 4.75% [234600/4942000] [47.5/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 02:27:28,189 - Train: 4.75% [234700/4942000] [47.5/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 02:28:01,121 - Train: 4.75% [234800/4942000] [47.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 02:28:34,008 - Train: 4.75% [234900/4942000] [47.5/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 02:29:07,471 - Train: 4.76% [235000/4942000] [47.6/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 02:29:40,369 - Train: 4.76% [235100/4942000] [47.6/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 02:30:13,395 - Train: 4.76% [235200/4942000] [47.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 02:30:46,328 - Train: 4.76% [235300/4942000] [47.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:31:19,692 - Train: 4.76% [235400/4942000] [47.6/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:31:52,578 - Train: 4.77% [235500/4942000] [47.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:32:25,469 - Train: 4.77% [235600/4942000] [47.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:32:58,385 - Train: 4.77% [235700/4942000] [47.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 02:33:31,257 - Train: 4.77% [235800/4942000] [47.7/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 02:34:04,138 - Train: 4.77% [235900/4942000] [47.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:34:37,011 - Train: 4.78% [236000/4942000] [47.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 02:35:10,003 - Train: 4.78% [236100/4942000] [47.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:35:42,888 - Train: 4.78% [236200/4942000] [47.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:36:15,764 - Train: 4.78% [236300/4942000] [47.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 02:36:48,679 - Train: 4.78% [236400/4942000] [47.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 02:37:21,610 - Train: 4.79% [236500/4942000] [47.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 02:37:54,582 - Train: 4.79% [236600/4942000] [47.9/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 02:38:27,474 - Train: 4.79% [236700/4942000] [47.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:39:00,420 - Train: 4.79% [236800/4942000] [47.9/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 02:39:33,326 - Train: 4.79% [236900/4942000] [47.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:40:06,243 - Train: 4.80% [237000/4942000] [48.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:40:39,150 - Train: 4.80% [237100/4942000] [48.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 02:41:12,064 - Train: 4.80% [237200/4942000] [48.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:41:17,330 - ==> Total time: 1 day, 8:43:56 Eta: 27 days, 1:11:30 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 02:41:46,374 - Train: 4.80% [237300/4942000] [48.0/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:42:19,338 - Train: 4.80% [237400/4942000] [48.0/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 02:42:52,289 - Train: 4.81% [237500/4942000] [48.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 02:43:25,203 - Train: 4.81% [237600/4942000] [48.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:43:58,105 - Train: 4.81% [237700/4942000] [48.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:44:31,043 - Train: 4.81% [237800/4942000] [48.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:45:04,084 - Train: 4.81% [237900/4942000] [48.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 02:45:37,014 - Train: 4.82% [238000/4942000] [48.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 02:46:09,928 - Train: 4.82% [238100/4942000] [48.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 02:46:42,803 - Train: 4.82% [238200/4942000] [48.2/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 02:47:15,774 - Train: 4.82% [238300/4942000] [48.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:47:48,674 - Train: 4.82% [238400/4942000] [48.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:48:21,569 - Train: 4.83% [238500/4942000] [48.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:48:54,531 - Train: 4.83% [238600/4942000] [48.3/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 02:49:27,441 - Train: 4.83% [238700/4942000] [48.3/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 02:50:00,367 - Train: 4.83% [238800/4942000] [48.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:50:33,296 - Train: 4.83% [238900/4942000] [48.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:51:06,244 - Train: 4.84% [239000/4942000] [48.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:51:39,131 - Train: 4.84% [239100/4942000] [48.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 02:52:12,043 - Train: 4.84% [239200/4942000] [48.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 02:52:45,067 - Train: 4.84% [239300/4942000] [48.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:53:17,966 - Train: 4.84% [239400/4942000] [48.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:53:50,895 - Train: 4.85% [239500/4942000] [48.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:54:24,259 - Train: 4.85% [239600/4942000] [48.5/1000.0] [batch_t 0.334 (0.334)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 02:54:57,208 - Train: 4.85% [239700/4942000] [48.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:55:30,136 - Train: 4.85% [239800/4942000] [48.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:56:03,052 - Train: 4.85% [239900/4942000] [48.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:56:35,991 - Train: 4.86% [240000/4942000] [48.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:57:08,984 - Train: 4.86% [240100/4942000] [48.6/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 02:57:41,910 - Train: 4.86% [240200/4942000] [48.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 02:58:14,835 - Train: 4.86% [240300/4942000] [48.6/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 02:58:47,804 - Train: 4.86% [240400/4942000] [48.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:59:20,797 - Train: 4.87% [240500/4942000] [48.7/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 02:59:53,770 - Train: 4.87% [240600/4942000] [48.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:00:26,738 - Train: 4.87% [240700/4942000] [48.7/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 03:00:59,777 - Train: 4.87% [240800/4942000] [48.7/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 03:01:32,705 - Train: 4.87% [240900/4942000] [48.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:02:05,632 - Train: 4.88% [241000/4942000] [48.8/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 03:02:38,549 - Train: 4.88% [241100/4942000] [48.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 03:03:11,423 - Train: 4.88% [241200/4942000] [48.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:03:44,325 - Train: 4.88% [241300/4942000] [48.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:04:17,242 - Train: 4.88% [241400/4942000] [48.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:04:50,156 - Train: 4.89% [241500/4942000] [48.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:05:23,060 - Train: 4.89% [241600/4942000] [48.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:05:56,009 - Train: 4.89% [241700/4942000] [48.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 03:06:28,927 - Train: 4.89% [241800/4942000] [48.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 03:07:01,843 - Train: 4.89% [241900/4942000] [48.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:07:34,761 - Train: 4.90% [242000/4942000] [49.0/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 03:08:07,758 - Train: 4.90% [242100/4942000] [49.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:08:27,037 - ==> Total time: 1 day, 9:11:06 Eta: 26 days, 20:03:39 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 03:08:42,245 - Train: 4.90% [242200/4942000] [49.0/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:09:15,145 - Train: 4.90% [242300/4942000] [49.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:09:48,124 - Train: 4.90% [242400/4942000] [49.0/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 03:10:21,113 - Train: 4.91% [242500/4942000] [49.1/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 03:10:54,067 - Train: 4.91% [242600/4942000] [49.1/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 03:11:27,024 - Train: 4.91% [242700/4942000] [49.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:11:59,968 - Train: 4.91% [242800/4942000] [49.1/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 03:12:32,943 - Train: 4.92% [242900/4942000] [49.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:13:05,923 - Train: 4.92% [243000/4942000] [49.2/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 03:13:38,846 - Train: 4.92% [243100/4942000] [49.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:14:11,755 - Train: 4.92% [243200/4942000] [49.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 03:14:44,751 - Train: 4.92% [243300/4942000] [49.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:15:17,698 - Train: 4.93% [243400/4942000] [49.3/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 03:15:50,837 - Train: 4.93% [243500/4942000] [49.3/1000.0] [batch_t 0.476 (0.331)] [data_t 0.002] [optim_t 0.474] [lr 0.005000] 2024-04-04 03:16:23,774 - Train: 4.93% [243600/4942000] [49.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 03:16:56,711 - Train: 4.93% [243700/4942000] [49.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:17:29,684 - Train: 4.93% [243800/4942000] [49.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:18:02,631 - Train: 4.94% [243900/4942000] [49.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:18:35,591 - Train: 4.94% [244000/4942000] [49.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 03:19:08,899 - Train: 4.94% [244100/4942000] [49.4/1000.0] [batch_t 0.332 (0.333)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:19:41,841 - Train: 4.94% [244200/4942000] [49.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:20:14,841 - Train: 4.94% [244300/4942000] [49.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:20:47,794 - Train: 4.95% [244400/4942000] [49.5/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:21:20,747 - Train: 4.95% [244500/4942000] [49.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:21:53,672 - Train: 4.95% [244600/4942000] [49.5/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 03:22:26,596 - Train: 4.95% [244700/4942000] [49.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 03:22:59,553 - Train: 4.95% [244800/4942000] [49.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:23:32,474 - Train: 4.96% [244900/4942000] [49.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:24:05,538 - Train: 4.96% [245000/4942000] [49.6/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:24:38,512 - Train: 4.96% [245100/4942000] [49.6/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 03:25:11,490 - Train: 4.96% [245200/4942000] [49.6/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 03:25:44,435 - Train: 4.96% [245300/4942000] [49.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:26:17,358 - Train: 4.97% [245400/4942000] [49.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 03:26:50,320 - Train: 4.97% [245500/4942000] [49.7/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:27:23,246 - Train: 4.97% [245600/4942000] [49.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 03:27:56,152 - Train: 4.97% [245700/4942000] [49.7/1000.0] [batch_t 0.321 (0.329)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-04 03:28:29,100 - Train: 4.97% [245800/4942000] [49.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:29:02,060 - Train: 4.98% [245900/4942000] [49.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:29:35,039 - Train: 4.98% [246000/4942000] [49.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:30:08,344 - Train: 4.98% [246100/4942000] [49.8/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:30:41,291 - Train: 4.98% [246200/4942000] [49.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:31:14,287 - Train: 4.98% [246300/4942000] [49.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:31:47,346 - Train: 4.99% [246400/4942000] [49.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:32:21,107 - Train: 4.99% [246500/4942000] [49.9/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:32:54,046 - Train: 4.99% [246600/4942000] [49.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:33:26,976 - Train: 4.99% [246700/4942000] [49.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:33:59,911 - Train: 4.99% [246800/4942000] [49.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:34:32,801 - Train: 5.00% [246900/4942000] [50.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:35:05,747 - Train: 5.00% [247000/4942000] [50.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:35:38,643 - Train: 5.00% [247100/4942000] [50.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:35:47,122 - Test: 16.13% [50/310] [batch_t 0.153 (0.163)] 2024-04-04 03:35:54,758 - Test: 32.26% [100/310] [batch_t 0.156 (0.158)] 2024-04-04 03:36:02,392 - Test: 48.39% [150/310] [batch_t 0.157 (0.156)] 2024-04-04 03:36:11,622 - Test: 64.52% [200/310] [batch_t 0.154 (0.163)] 2024-04-04 03:36:19,320 - Test: 80.65% [250/310] [batch_t 0.167 (0.161)] 2024-04-04 03:36:27,007 - Test: 96.77% [300/310] [batch_t 0.155 (0.160)] 2024-04-04 03:36:28,929 - Test: 100.00% [310/310] [batch_t 0.449 (0.161)] 2024-04-04 04:01:38,329 - ==> Metric Time for coco : 0.004 (mAUROC_sp_max) 0.001 (mAP_sp_max) 0.001 (mF1_max_sp_max) 335.813 (mAUROC_px) 291.220 (mAP_px) 34.106 (mF1_max_px) 774.652 (mAUPRO_px) 10.933 (mF1_px_0.2_0.8_0.1) 10.817 (mAcc_px_0.2_0.8_0.1) 10.955 (mIoU_px_0.2_0.8_0.1) 31.868 (mIoU_max_px) 2024-04-04 04:01:38,730 - | Name | mAUROC_sp_max | mAUROC_sp_max (Max) | mAP_sp_max | mAP_sp_max (Max) | mF1_max_sp_max | mF1_max_sp_max (Max) | mAUROC_px | mAUROC_px (Max) | mAP_px | mAP_px (Max) | mF1_max_px | mF1_max_px (Max) | mAUPRO_px | mAUPRO_px (Max) | mF1_px_0.2_0.8_0.1 | mF1_px_0.2_0.8_0.1 (Max) | mAcc_px_0.2_0.8_0.1 | mAcc_px_0.2_0.8_0.1 (Max) | mIoU_px_0.2_0.8_0.1 | mIoU_px_0.2_0.8_0.1 (Max) | mIoU_max_px | mIoU_max_px (Max) | |:------:|:---------------:|:---------------------:|:------------:|:------------------:|:----------------:|:----------------------:|:-----------:|:------------------:|:--------:|:------------------:|:------------:|:------------------:|:-----------:|:------------------:|:--------------------:|:--------------------------:|:---------------------:|:---------------------------:|:---------------------:|:---------------------------:|:-------------:|:-------------------:| | coco | 66.882 | 66.882 (50 epoch) | 46.433 | 46.433 (50 epoch) | 54.576 | 54.576 (50 epoch) | 71.218 | 71.218 (50 epoch) | 13.870 | 13.870 (50 epoch) | 21.554 | 21.554 (50 epoch) | 44.441 | 44.441 (50 epoch) | 11.792 | 11.792 (50 epoch) | 44.665 | 44.665 (50 epoch) | 6.425 | 6.425 (50 epoch) | 12.079 | 12.079 (50 epoch) | 2024-04-04 04:01:39,079 - ==> Total time: 1 day, 10:04:18 Eta: 26 days, 23:21:47 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 04:02:14,915 - Train: 5.00% [247200/4942000] [50.0/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 04:02:47,672 - Train: 5.00% [247300/4942000] [50.0/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 04:03:20,452 - Train: 5.01% [247400/4942000] [50.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 04:03:53,208 - Train: 5.01% [247500/4942000] [50.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 04:04:25,927 - Train: 5.01% [247600/4942000] [50.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 04:04:58,785 - Train: 5.01% [247700/4942000] [50.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 04:05:31,533 - Train: 5.01% [247800/4942000] [50.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:06:04,320 - Train: 5.02% [247900/4942000] [50.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 04:06:37,055 - Train: 5.02% [248000/4942000] [50.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 04:07:09,792 - Train: 5.02% [248100/4942000] [50.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 04:07:42,514 - Train: 5.02% [248200/4942000] [50.2/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 04:08:15,331 - Train: 5.02% [248300/4942000] [50.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:08:48,105 - Train: 5.03% [248400/4942000] [50.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 04:09:21,396 - Train: 5.03% [248500/4942000] [50.3/1000.0] [batch_t 0.325 (0.333)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 04:09:54,157 - Train: 5.03% [248600/4942000] [50.3/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:10:26,949 - Train: 5.03% [248700/4942000] [50.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 04:10:59,667 - Train: 5.03% [248800/4942000] [50.3/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-04 04:11:32,437 - Train: 5.04% [248900/4942000] [50.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 04:12:05,154 - Train: 5.04% [249000/4942000] [50.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 04:12:37,946 - Train: 5.04% [249100/4942000] [50.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 04:13:10,716 - Train: 5.04% [249200/4942000] [50.4/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-04 04:13:43,501 - Train: 5.04% [249300/4942000] [50.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 04:14:16,313 - Train: 5.05% [249400/4942000] [50.5/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:14:49,175 - Train: 5.05% [249500/4942000] [50.5/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 04:15:22,026 - Train: 5.05% [249600/4942000] [50.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 04:15:54,847 - Train: 5.05% [249700/4942000] [50.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 04:16:27,679 - Train: 5.05% [249800/4942000] [50.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 04:17:00,512 - Train: 5.06% [249900/4942000] [50.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 04:17:33,307 - Train: 5.06% [250000/4942000] [50.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 04:18:06,110 - Train: 5.06% [250100/4942000] [50.6/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 04:18:38,796 - Train: 5.06% [250200/4942000] [50.6/1000.0] [batch_t 0.318 (0.327)] [data_t 0.002] [optim_t 0.317] [lr 0.005000] 2024-04-04 04:19:11,514 - Train: 5.06% [250300/4942000] [50.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 04:19:44,267 - Train: 5.07% [250400/4942000] [50.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 04:20:16,979 - Train: 5.07% [250500/4942000] [50.7/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 04:20:49,782 - Train: 5.07% [250600/4942000] [50.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:21:22,557 - Train: 5.07% [250700/4942000] [50.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 04:21:55,281 - Train: 5.07% [250800/4942000] [50.7/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 04:22:28,011 - Train: 5.08% [250900/4942000] [50.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 04:23:00,668 - Train: 5.08% [251000/4942000] [50.8/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:23:33,390 - Train: 5.08% [251100/4942000] [50.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 04:24:06,116 - Train: 5.08% [251200/4942000] [50.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 04:24:38,849 - Train: 5.08% [251300/4942000] [50.8/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 04:25:11,626 - Train: 5.09% [251400/4942000] [50.9/1000.0] [batch_t 0.321 (0.328)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-04 04:25:44,308 - Train: 5.09% [251500/4942000] [50.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 04:26:17,031 - Train: 5.09% [251600/4942000] [50.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 04:26:49,756 - Train: 5.09% [251700/4942000] [50.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 04:27:22,487 - Train: 5.10% [251800/4942000] [51.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 04:27:55,219 - Train: 5.10% [251900/4942000] [51.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 04:28:28,035 - Train: 5.10% [252000/4942000] [51.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 04:28:41,822 - ==> Total time: 1 day, 10:31:21 Eta: 26 days, 18:23:21 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 04:29:02,567 - Train: 5.10% [252100/4942000] [51.0/1000.0] [batch_t 0.331 (0.328)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-04 04:29:35,319 - Train: 5.10% [252200/4942000] [51.0/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 04:30:08,093 - Train: 5.11% [252300/4942000] [51.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 04:30:40,827 - Train: 5.11% [252400/4942000] [51.1/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 04:31:13,539 - Train: 5.11% [252500/4942000] [51.1/1000.0] [batch_t 0.320 (0.327)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-04 04:31:46,230 - Train: 5.11% [252600/4942000] [51.1/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 04:32:18,972 - Train: 5.11% [252700/4942000] [51.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 04:32:51,686 - Train: 5.12% [252800/4942000] [51.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 04:33:24,422 - Train: 5.12% [252900/4942000] [51.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 04:33:57,127 - Train: 5.12% [253000/4942000] [51.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 04:34:29,853 - Train: 5.12% [253100/4942000] [51.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:35:02,591 - Train: 5.12% [253200/4942000] [51.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 04:35:35,460 - Train: 5.13% [253300/4942000] [51.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:36:08,219 - Train: 5.13% [253400/4942000] [51.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 04:36:40,939 - Train: 5.13% [253500/4942000] [51.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 04:37:13,710 - Train: 5.13% [253600/4942000] [51.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 04:37:46,465 - Train: 5.13% [253700/4942000] [51.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 04:38:19,212 - Train: 5.14% [253800/4942000] [51.4/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 04:38:51,937 - Train: 5.14% [253900/4942000] [51.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 04:39:24,698 - Train: 5.14% [254000/4942000] [51.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 04:39:57,433 - Train: 5.14% [254100/4942000] [51.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:40:30,215 - Train: 5.14% [254200/4942000] [51.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 04:41:02,968 - Train: 5.15% [254300/4942000] [51.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:41:35,692 - Train: 5.15% [254400/4942000] [51.5/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 04:42:08,411 - Train: 5.15% [254500/4942000] [51.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:42:41,140 - Train: 5.15% [254600/4942000] [51.5/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 04:43:13,993 - Train: 5.15% [254700/4942000] [51.5/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 04:43:46,712 - Train: 5.16% [254800/4942000] [51.6/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 04:44:19,439 - Train: 5.16% [254900/4942000] [51.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 04:44:52,246 - Train: 5.16% [255000/4942000] [51.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 04:45:25,008 - Train: 5.16% [255100/4942000] [51.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 04:45:57,745 - Train: 5.16% [255200/4942000] [51.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 04:46:30,475 - Train: 5.17% [255300/4942000] [51.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 04:47:03,286 - Train: 5.17% [255400/4942000] [51.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 04:47:36,054 - Train: 5.17% [255500/4942000] [51.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 04:48:08,774 - Train: 5.17% [255600/4942000] [51.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 04:48:41,499 - Train: 5.17% [255700/4942000] [51.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:49:14,251 - Train: 5.18% [255800/4942000] [51.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 04:49:47,034 - Train: 5.18% [255900/4942000] [51.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 04:50:19,813 - Train: 5.18% [256000/4942000] [51.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 04:50:52,554 - Train: 5.18% [256100/4942000] [51.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 04:51:25,395 - Train: 5.18% [256200/4942000] [51.8/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 04:51:58,124 - Train: 5.19% [256300/4942000] [51.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 04:52:30,891 - Train: 5.19% [256400/4942000] [51.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:53:06,326 - Train: 5.19% [256500/4942000] [51.9/1000.0] [batch_t 0.327 (0.354)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 04:53:39,055 - Train: 5.19% [256600/4942000] [51.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 04:54:11,786 - Train: 5.19% [256700/4942000] [51.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 04:54:44,493 - Train: 5.20% [256800/4942000] [52.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 04:55:18,827 - Train: 5.20% [256900/4942000] [52.0/1000.0] [batch_t 0.328 (0.343)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 04:55:46,339 - ==> Total time: 1 day, 10:58:25 Eta: 26 days, 13:35:54 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 04:55:53,666 - Train: 5.20% [257000/4942000] [52.0/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 04:56:26,444 - Train: 5.20% [257100/4942000] [52.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 04:56:59,158 - Train: 5.20% [257200/4942000] [52.0/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 04:57:32,957 - Train: 5.21% [257300/4942000] [52.1/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 04:58:06,926 - Train: 5.21% [257400/4942000] [52.1/1000.0] [batch_t 0.323 (0.340)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 04:58:39,790 - Train: 5.21% [257500/4942000] [52.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 04:59:12,525 - Train: 5.21% [257600/4942000] [52.1/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 04:59:45,325 - Train: 5.21% [257700/4942000] [52.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:00:18,092 - Train: 5.22% [257800/4942000] [52.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 05:00:50,845 - Train: 5.22% [257900/4942000] [52.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 05:01:23,607 - Train: 5.22% [258000/4942000] [52.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:01:56,297 - Train: 5.22% [258100/4942000] [52.2/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 05:02:29,041 - Train: 5.22% [258200/4942000] [52.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:03:01,828 - Train: 5.23% [258300/4942000] [52.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:03:34,597 - Train: 5.23% [258400/4942000] [52.3/1000.0] [batch_t 0.340 (0.328)] [data_t 0.003] [optim_t 0.338] [lr 0.005000] 2024-04-04 05:04:07,365 - Train: 5.23% [258500/4942000] [52.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:04:40,091 - Train: 5.23% [258600/4942000] [52.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:05:12,823 - Train: 5.23% [258700/4942000] [52.3/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 05:05:45,536 - Train: 5.24% [258800/4942000] [52.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 05:06:18,445 - Train: 5.24% [258900/4942000] [52.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 05:06:51,177 - Train: 5.24% [259000/4942000] [52.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 05:07:23,910 - Train: 5.24% [259100/4942000] [52.4/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 05:07:56,631 - Train: 5.24% [259200/4942000] [52.4/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-04 05:08:29,368 - Train: 5.25% [259300/4942000] [52.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:09:02,174 - Train: 5.25% [259400/4942000] [52.5/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 05:09:34,896 - Train: 5.25% [259500/4942000] [52.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 05:10:07,619 - Train: 5.25% [259600/4942000] [52.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:10:40,436 - Train: 5.25% [259700/4942000] [52.5/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 05:11:13,200 - Train: 5.26% [259800/4942000] [52.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:11:45,985 - Train: 5.26% [259900/4942000] [52.6/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 05:12:18,772 - Train: 5.26% [260000/4942000] [52.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 05:12:51,580 - Train: 5.26% [260100/4942000] [52.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:13:24,375 - Train: 5.27% [260200/4942000] [52.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 05:13:57,265 - Train: 5.27% [260300/4942000] [52.7/1000.0] [batch_t 0.322 (0.329)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-04 05:14:30,081 - Train: 5.27% [260400/4942000] [52.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:15:02,862 - Train: 5.27% [260500/4942000] [52.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 05:15:35,654 - Train: 5.27% [260600/4942000] [52.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:16:08,416 - Train: 5.28% [260700/4942000] [52.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:16:41,165 - Train: 5.28% [260800/4942000] [52.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 05:17:13,919 - Train: 5.28% [260900/4942000] [52.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:17:46,694 - Train: 5.28% [261000/4942000] [52.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:18:19,473 - Train: 5.28% [261100/4942000] [52.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:18:52,226 - Train: 5.29% [261200/4942000] [52.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:19:25,092 - Train: 5.29% [261300/4942000] [52.9/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 05:19:57,801 - Train: 5.29% [261400/4942000] [52.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:20:30,561 - Train: 5.29% [261500/4942000] [52.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:21:03,322 - Train: 5.29% [261600/4942000] [52.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:21:36,089 - Train: 5.30% [261700/4942000] [53.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 05:22:08,897 - Train: 5.30% [261800/4942000] [53.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 05:22:41,678 - Train: 5.30% [261900/4942000] [53.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:22:50,189 - ==> Total time: 1 day, 11:25:29 Eta: 26 days, 8:58:05 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 05:23:16,529 - Train: 5.30% [262000/4942000] [53.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:23:49,230 - Train: 5.30% [262100/4942000] [53.0/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 05:24:21,974 - Train: 5.31% [262200/4942000] [53.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:24:54,671 - Train: 5.31% [262300/4942000] [53.1/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 05:25:27,461 - Train: 5.31% [262400/4942000] [53.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 05:26:00,209 - Train: 5.31% [262500/4942000] [53.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 05:26:33,031 - Train: 5.31% [262600/4942000] [53.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:27:05,838 - Train: 5.32% [262700/4942000] [53.2/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 05:27:38,579 - Train: 5.32% [262800/4942000] [53.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:28:11,314 - Train: 5.32% [262900/4942000] [53.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 05:28:44,023 - Train: 5.32% [263000/4942000] [53.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:29:16,904 - Train: 5.32% [263100/4942000] [53.2/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 05:29:49,661 - Train: 5.33% [263200/4942000] [53.3/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 05:30:22,404 - Train: 5.33% [263300/4942000] [53.3/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:30:55,084 - Train: 5.33% [263400/4942000] [53.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:31:27,795 - Train: 5.33% [263500/4942000] [53.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 05:32:00,594 - Train: 5.33% [263600/4942000] [53.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 05:32:33,351 - Train: 5.34% [263700/4942000] [53.4/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 05:33:06,994 - Train: 5.34% [263800/4942000] [53.4/1000.0] [batch_t 0.326 (0.336)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 05:33:39,755 - Train: 5.34% [263900/4942000] [53.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 05:34:12,777 - Train: 5.34% [264000/4942000] [53.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:34:45,597 - Train: 5.34% [264100/4942000] [53.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 05:35:19,732 - Train: 5.35% [264200/4942000] [53.5/1000.0] [batch_t 0.332 (0.341)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-04 05:35:52,561 - Train: 5.35% [264300/4942000] [53.5/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 05:36:25,811 - Train: 5.35% [264400/4942000] [53.5/1000.0] [batch_t 0.325 (0.332)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 05:36:58,770 - Train: 5.35% [264500/4942000] [53.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:37:32,279 - Train: 5.35% [264600/4942000] [53.5/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 05:38:05,043 - Train: 5.36% [264700/4942000] [53.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 05:38:37,827 - Train: 5.36% [264800/4942000] [53.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:39:10,640 - Train: 5.36% [264900/4942000] [53.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:39:43,469 - Train: 5.36% [265000/4942000] [53.6/1000.0] [batch_t 0.321 (0.328)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-04 05:40:16,295 - Train: 5.36% [265100/4942000] [53.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:40:49,065 - Train: 5.37% [265200/4942000] [53.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 05:41:21,847 - Train: 5.37% [265300/4942000] [53.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:41:54,683 - Train: 5.37% [265400/4942000] [53.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 05:42:27,484 - Train: 5.37% [265500/4942000] [53.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:43:00,257 - Train: 5.37% [265600/4942000] [53.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 05:43:33,048 - Train: 5.38% [265700/4942000] [53.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 05:44:05,850 - Train: 5.38% [265800/4942000] [53.8/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 05:44:38,766 - Train: 5.38% [265900/4942000] [53.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:45:11,554 - Train: 5.38% [266000/4942000] [53.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 05:45:44,343 - Train: 5.38% [266100/4942000] [53.8/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 05:46:17,130 - Train: 5.39% [266200/4942000] [53.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:46:49,912 - Train: 5.39% [266300/4942000] [53.9/1000.0] [batch_t 0.321 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-04 05:47:22,756 - Train: 5.39% [266400/4942000] [53.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:47:55,602 - Train: 5.39% [266500/4942000] [53.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 05:48:28,425 - Train: 5.39% [266600/4942000] [53.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 05:49:01,249 - Train: 5.40% [266700/4942000] [54.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:49:34,045 - Train: 5.40% [266800/4942000] [54.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:49:56,348 - ==> Total time: 1 day, 11:52:35 Eta: 26 days, 4:30:13 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 05:50:08,906 - Train: 5.40% [266900/4942000] [54.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:50:41,670 - Train: 5.40% [267000/4942000] [54.0/1000.0] [batch_t 0.319 (0.328)] [data_t 0.002] [optim_t 0.317] [lr 0.005000] 2024-04-04 05:51:14,475 - Train: 5.40% [267100/4942000] [54.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:51:47,193 - Train: 5.41% [267200/4942000] [54.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:52:20,107 - Train: 5.41% [267300/4942000] [54.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 05:52:52,868 - Train: 5.41% [267400/4942000] [54.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 05:53:25,642 - Train: 5.41% [267500/4942000] [54.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 05:53:58,367 - Train: 5.41% [267600/4942000] [54.1/1000.0] [batch_t 0.320 (0.327)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-04 05:54:31,168 - Train: 5.42% [267700/4942000] [54.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:55:03,941 - Train: 5.42% [267800/4942000] [54.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:55:36,744 - Train: 5.42% [267900/4942000] [54.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 05:56:09,539 - Train: 5.42% [268000/4942000] [54.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 05:56:42,288 - Train: 5.42% [268100/4942000] [54.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:57:15,069 - Train: 5.43% [268200/4942000] [54.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:57:47,879 - Train: 5.43% [268300/4942000] [54.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:58:20,660 - Train: 5.43% [268400/4942000] [54.3/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:58:53,435 - Train: 5.43% [268500/4942000] [54.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:59:26,256 - Train: 5.44% [268600/4942000] [54.4/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 05:59:59,178 - Train: 5.44% [268700/4942000] [54.4/1000.0] [batch_t 0.321 (0.329)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-04 06:00:31,993 - Train: 5.44% [268800/4942000] [54.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 06:01:04,770 - Train: 5.44% [268900/4942000] [54.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:01:37,550 - Train: 5.44% [269000/4942000] [54.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 06:02:10,369 - Train: 5.45% [269100/4942000] [54.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 06:02:43,137 - Train: 5.45% [269200/4942000] [54.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:03:15,906 - Train: 5.45% [269300/4942000] [54.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:03:48,669 - Train: 5.45% [269400/4942000] [54.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:04:21,436 - Train: 5.45% [269500/4942000] [54.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:04:54,218 - Train: 5.46% [269600/4942000] [54.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 06:05:26,970 - Train: 5.46% [269700/4942000] [54.6/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 06:05:59,733 - Train: 5.46% [269800/4942000] [54.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:06:32,497 - Train: 5.46% [269900/4942000] [54.6/1000.0] [batch_t 0.337 (0.328)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-04 06:07:05,274 - Train: 5.46% [270000/4942000] [54.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:07:38,144 - Train: 5.47% [270100/4942000] [54.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 06:08:10,928 - Train: 5.47% [270200/4942000] [54.7/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 06:08:43,656 - Train: 5.47% [270300/4942000] [54.7/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 06:09:16,426 - Train: 5.47% [270400/4942000] [54.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:09:49,138 - Train: 5.47% [270500/4942000] [54.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 06:10:21,948 - Train: 5.48% [270600/4942000] [54.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 06:10:54,690 - Train: 5.48% [270700/4942000] [54.8/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 06:11:27,479 - Train: 5.48% [270800/4942000] [54.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:12:00,273 - Train: 5.48% [270900/4942000] [54.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 06:12:32,992 - Train: 5.48% [271000/4942000] [54.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 06:13:05,755 - Train: 5.49% [271100/4942000] [54.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:13:38,489 - Train: 5.49% [271200/4942000] [54.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:14:11,277 - Train: 5.49% [271300/4942000] [54.9/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 06:14:44,013 - Train: 5.49% [271400/4942000] [54.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:15:16,769 - Train: 5.49% [271500/4942000] [54.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 06:15:49,641 - Train: 5.50% [271600/4942000] [55.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 06:16:22,378 - Train: 5.50% [271700/4942000] [55.0/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 06:16:55,141 - Train: 5.50% [271800/4942000] [55.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 06:16:58,435 - ==> Total time: 1 day, 12:19:37 Eta: 26 days, 0:09:57 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 06:17:29,677 - Train: 5.50% [271900/4942000] [55.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 06:18:02,394 - Train: 5.50% [272000/4942000] [55.0/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 06:18:35,149 - Train: 5.51% [272100/4942000] [55.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 06:19:07,950 - Train: 5.51% [272200/4942000] [55.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:19:40,712 - Train: 5.51% [272300/4942000] [55.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:20:13,503 - Train: 5.51% [272400/4942000] [55.1/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 06:20:46,299 - Train: 5.51% [272500/4942000] [55.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 06:21:19,106 - Train: 5.52% [272600/4942000] [55.2/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 06:21:51,899 - Train: 5.52% [272700/4942000] [55.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:22:24,746 - Train: 5.52% [272800/4942000] [55.2/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:22:57,699 - Train: 5.52% [272900/4942000] [55.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:23:30,552 - Train: 5.52% [273000/4942000] [55.2/1000.0] [batch_t 0.331 (0.328)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:24:03,386 - Train: 5.53% [273100/4942000] [55.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:24:36,180 - Train: 5.53% [273200/4942000] [55.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:25:08,998 - Train: 5.53% [273300/4942000] [55.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 06:25:41,774 - Train: 5.53% [273400/4942000] [55.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 06:26:14,558 - Train: 5.53% [273500/4942000] [55.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:26:47,395 - Train: 5.54% [273600/4942000] [55.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:27:20,241 - Train: 5.54% [273700/4942000] [55.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 06:27:53,043 - Train: 5.54% [273800/4942000] [55.4/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 06:28:25,831 - Train: 5.54% [273900/4942000] [55.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:28:58,639 - Train: 5.54% [274000/4942000] [55.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 06:29:31,377 - Train: 5.55% [274100/4942000] [55.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:30:04,161 - Train: 5.55% [274200/4942000] [55.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:30:37,059 - Train: 5.55% [274300/4942000] [55.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 06:31:10,323 - Train: 5.55% [274400/4942000] [55.5/1000.0] [batch_t 0.326 (0.333)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 06:31:43,166 - Train: 5.55% [274500/4942000] [55.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:32:15,983 - Train: 5.56% [274600/4942000] [55.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 06:32:48,796 - Train: 5.56% [274700/4942000] [55.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:33:21,678 - Train: 5.56% [274800/4942000] [55.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:33:54,501 - Train: 5.56% [274900/4942000] [55.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 06:34:27,388 - Train: 5.56% [275000/4942000] [55.6/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 06:35:00,189 - Train: 5.57% [275100/4942000] [55.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:35:33,029 - Train: 5.57% [275200/4942000] [55.7/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 06:36:05,826 - Train: 5.57% [275300/4942000] [55.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:36:38,642 - Train: 5.57% [275400/4942000] [55.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 06:37:11,485 - Train: 5.57% [275500/4942000] [55.7/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 06:37:44,257 - Train: 5.58% [275600/4942000] [55.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:38:17,062 - Train: 5.58% [275700/4942000] [55.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:38:49,913 - Train: 5.58% [275800/4942000] [55.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:39:22,726 - Train: 5.58% [275900/4942000] [55.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:39:55,510 - Train: 5.58% [276000/4942000] [55.8/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 06:40:28,331 - Train: 5.59% [276100/4942000] [55.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:41:01,134 - Train: 5.59% [276200/4942000] [55.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:41:33,949 - Train: 5.59% [276300/4942000] [55.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:42:06,679 - Train: 5.59% [276400/4942000] [55.9/1000.0] [batch_t 0.330 (0.327)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:42:39,441 - Train: 5.59% [276500/4942000] [55.9/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-04 06:43:12,194 - Train: 5.60% [276600/4942000] [56.0/1000.0] [batch_t 0.325 (0.327)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-04 06:43:44,964 - Train: 5.60% [276700/4942000] [56.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 06:44:02,010 - ==> Total time: 1 day, 12:46:41 Eta: 25 days, 19:58:25 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 06:44:21,100 - Train: 5.60% [276800/4942000] [56.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:44:53,843 - Train: 5.60% [276900/4942000] [56.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:45:26,609 - Train: 5.61% [277000/4942000] [56.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:45:59,512 - Train: 5.61% [277100/4942000] [56.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:46:32,759 - Train: 5.61% [277200/4942000] [56.1/1000.0] [batch_t 0.323 (0.332)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 06:47:05,522 - Train: 5.61% [277300/4942000] [56.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:47:38,306 - Train: 5.61% [277400/4942000] [56.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:48:11,111 - Train: 5.62% [277500/4942000] [56.2/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 06:48:43,925 - Train: 5.62% [277600/4942000] [56.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:49:16,677 - Train: 5.62% [277700/4942000] [56.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 06:49:49,432 - Train: 5.62% [277800/4942000] [56.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 06:50:22,240 - Train: 5.62% [277900/4942000] [56.2/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 06:50:55,039 - Train: 5.63% [278000/4942000] [56.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:51:27,759 - Train: 5.63% [278100/4942000] [56.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:52:00,519 - Train: 5.63% [278200/4942000] [56.3/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:52:33,926 - Train: 5.63% [278300/4942000] [56.3/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 06:53:06,686 - Train: 5.63% [278400/4942000] [56.3/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 06:53:39,582 - Train: 5.64% [278500/4942000] [56.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 06:54:12,265 - Train: 5.64% [278600/4942000] [56.4/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 06:54:44,985 - Train: 5.64% [278700/4942000] [56.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:55:17,776 - Train: 5.64% [278800/4942000] [56.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 06:55:50,551 - Train: 5.64% [278900/4942000] [56.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:56:23,327 - Train: 5.65% [279000/4942000] [56.5/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 06:56:56,119 - Train: 5.65% [279100/4942000] [56.5/1000.0] [batch_t 0.331 (0.328)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-04 06:57:28,835 - Train: 5.65% [279200/4942000] [56.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 06:58:01,569 - Train: 5.65% [279300/4942000] [56.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:58:34,310 - Train: 5.65% [279400/4942000] [56.5/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 06:59:06,989 - Train: 5.66% [279500/4942000] [56.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:59:39,732 - Train: 5.66% [279600/4942000] [56.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 07:00:12,465 - Train: 5.66% [279700/4942000] [56.6/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 07:00:45,211 - Train: 5.66% [279800/4942000] [56.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 07:01:17,999 - Train: 5.66% [279900/4942000] [56.6/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 07:01:50,863 - Train: 5.67% [280000/4942000] [56.7/1000.0] [batch_t 0.325 (0.329)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-04 07:02:23,578 - Train: 5.67% [280100/4942000] [56.7/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 07:02:56,278 - Train: 5.67% [280200/4942000] [56.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 07:03:29,036 - Train: 5.67% [280300/4942000] [56.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 07:04:01,725 - Train: 5.67% [280400/4942000] [56.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 07:04:34,474 - Train: 5.68% [280500/4942000] [56.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 07:05:07,241 - Train: 5.68% [280600/4942000] [56.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 07:05:40,035 - Train: 5.68% [280700/4942000] [56.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 07:06:12,871 - Train: 5.68% [280800/4942000] [56.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:06:45,670 - Train: 5.68% [280900/4942000] [56.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 07:07:18,399 - Train: 5.69% [281000/4942000] [56.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 07:07:51,188 - Train: 5.69% [281100/4942000] [56.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 07:08:23,967 - Train: 5.69% [281200/4942000] [56.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 07:08:56,749 - Train: 5.69% [281300/4942000] [56.9/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 07:09:29,619 - Train: 5.69% [281400/4942000] [56.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 07:10:02,351 - Train: 5.70% [281500/4942000] [57.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 07:10:35,087 - Train: 5.70% [281600/4942000] [57.0/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 07:11:05,842 - ==> Total time: 1 day, 13:13:45 Eta: 25 days, 15:54:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 07:11:09,717 - Train: 5.70% [281700/4942000] [57.0/1000.0] [batch_t 0.326 (0.334)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 07:11:42,474 - Train: 5.70% [281800/4942000] [57.0/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 07:12:15,206 - Train: 5.70% [281900/4942000] [57.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 07:12:47,927 - Train: 5.71% [282000/4942000] [57.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 07:13:20,700 - Train: 5.71% [282100/4942000] [57.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 07:13:53,427 - Train: 5.71% [282200/4942000] [57.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 07:14:26,138 - Train: 5.71% [282300/4942000] [57.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 07:14:58,921 - Train: 5.71% [282400/4942000] [57.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 07:15:31,637 - Train: 5.72% [282500/4942000] [57.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 07:16:04,371 - Train: 5.72% [282600/4942000] [57.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 07:16:37,231 - Train: 5.72% [282700/4942000] [57.2/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 07:17:09,960 - Train: 5.72% [282800/4942000] [57.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 07:17:42,744 - Train: 5.72% [282900/4942000] [57.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:18:15,515 - Train: 5.73% [283000/4942000] [57.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 07:18:48,264 - Train: 5.73% [283100/4942000] [57.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 07:19:21,081 - Train: 5.73% [283200/4942000] [57.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 07:19:53,857 - Train: 5.73% [283300/4942000] [57.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 07:20:26,628 - Train: 5.73% [283400/4942000] [57.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 07:20:59,450 - Train: 5.74% [283500/4942000] [57.4/1000.0] [batch_t 0.336 (0.328)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-04 07:21:32,245 - Train: 5.74% [283600/4942000] [57.4/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:22:05,024 - Train: 5.74% [283700/4942000] [57.4/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-04 07:22:37,774 - Train: 5.74% [283800/4942000] [57.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 07:23:10,624 - Train: 5.74% [283900/4942000] [57.4/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-04 07:23:43,399 - Train: 5.75% [284000/4942000] [57.5/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 07:24:16,200 - Train: 5.75% [284100/4942000] [57.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 07:24:49,135 - Train: 5.75% [284200/4942000] [57.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 07:25:21,882 - Train: 5.75% [284300/4942000] [57.5/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 07:25:54,650 - Train: 5.75% [284400/4942000] [57.5/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-04 07:26:27,358 - Train: 5.76% [284500/4942000] [57.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 07:27:00,077 - Train: 5.76% [284600/4942000] [57.6/1000.0] [batch_t 0.321 (0.327)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-04 07:27:32,850 - Train: 5.76% [284700/4942000] [57.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 07:28:05,570 - Train: 5.76% [284800/4942000] [57.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 07:28:38,315 - Train: 5.76% [284900/4942000] [57.6/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 07:29:11,128 - Train: 5.77% [285000/4942000] [57.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 07:29:43,897 - Train: 5.77% [285100/4942000] [57.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 07:30:16,692 - Train: 5.77% [285200/4942000] [57.7/1000.0] [batch_t 0.319 (0.328)] [data_t 0.002] [optim_t 0.317] [lr 0.005000] 2024-04-04 07:30:49,420 - Train: 5.77% [285300/4942000] [57.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 07:31:22,217 - Train: 5.77% [285400/4942000] [57.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 07:31:54,996 - Train: 5.78% [285500/4942000] [57.8/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 07:32:27,941 - Train: 5.78% [285600/4942000] [57.8/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 07:33:00,707 - Train: 5.78% [285700/4942000] [57.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 07:33:33,505 - Train: 5.78% [285800/4942000] [57.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 07:34:06,326 - Train: 5.79% [285900/4942000] [57.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:34:39,117 - Train: 5.79% [286000/4942000] [57.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 07:35:11,927 - Train: 5.79% [286100/4942000] [57.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 07:35:44,729 - Train: 5.79% [286200/4942000] [57.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 07:36:17,583 - Train: 5.79% [286300/4942000] [57.9/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:36:50,413 - Train: 5.80% [286400/4942000] [58.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 07:37:23,259 - Train: 5.80% [286500/4942000] [58.0/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 07:37:56,039 - Train: 5.80% [286600/4942000] [58.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 07:38:07,836 - ==> Total time: 1 day, 13:40:47 Eta: 25 days, 11:58:14 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 07:38:30,830 - Train: 5.80% [286700/4942000] [58.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 07:39:03,623 - Train: 5.80% [286800/4942000] [58.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 07:39:36,510 - Train: 5.81% [286900/4942000] [58.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 07:40:09,302 - Train: 5.81% [287000/4942000] [58.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 07:40:42,029 - Train: 5.81% [287100/4942000] [58.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 07:41:15,606 - Train: 5.81% [287200/4942000] [58.1/1000.0] [batch_t 0.326 (0.336)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 07:41:48,330 - Train: 5.81% [287300/4942000] [58.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 07:42:21,055 - Train: 5.82% [287400/4942000] [58.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 07:42:53,801 - Train: 5.82% [287500/4942000] [58.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 07:43:26,493 - Train: 5.82% [287600/4942000] [58.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 07:43:59,191 - Train: 5.82% [287700/4942000] [58.2/1000.0] [batch_t 0.319 (0.327)] [data_t 0.002] [optim_t 0.317] [lr 0.005000] 2024-04-04 07:44:32,009 - Train: 5.82% [287800/4942000] [58.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 07:45:04,771 - Train: 5.83% [287900/4942000] [58.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 07:45:37,574 - Train: 5.83% [288000/4942000] [58.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 07:46:10,367 - Train: 5.83% [288100/4942000] [58.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 07:46:43,059 - Train: 5.83% [288200/4942000] [58.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 07:47:15,785 - Train: 5.83% [288300/4942000] [58.3/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:47:48,601 - Train: 5.84% [288400/4942000] [58.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 07:48:21,326 - Train: 5.84% [288500/4942000] [58.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 07:48:54,034 - Train: 5.84% [288600/4942000] [58.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 07:49:26,850 - Train: 5.84% [288700/4942000] [58.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 07:49:59,641 - Train: 5.84% [288800/4942000] [58.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 07:50:32,409 - Train: 5.85% [288900/4942000] [58.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 07:51:05,143 - Train: 5.85% [289000/4942000] [58.5/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 07:51:37,892 - Train: 5.85% [289100/4942000] [58.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 07:52:10,680 - Train: 5.85% [289200/4942000] [58.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 07:52:43,451 - Train: 5.85% [289300/4942000] [58.5/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 07:53:16,185 - Train: 5.86% [289400/4942000] [58.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 07:53:48,921 - Train: 5.86% [289500/4942000] [58.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 07:54:21,702 - Train: 5.86% [289600/4942000] [58.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 07:54:54,486 - Train: 5.86% [289700/4942000] [58.6/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 07:55:27,319 - Train: 5.86% [289800/4942000] [58.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 07:56:00,064 - Train: 5.87% [289900/4942000] [58.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 07:56:32,870 - Train: 5.87% [290000/4942000] [58.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:57:05,646 - Train: 5.87% [290100/4942000] [58.7/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 07:57:38,471 - Train: 5.87% [290200/4942000] [58.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 07:58:11,231 - Train: 5.87% [290300/4942000] [58.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 07:58:44,054 - Train: 5.88% [290400/4942000] [58.8/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 07:59:16,874 - Train: 5.88% [290500/4942000] [58.8/1000.0] [batch_t 0.323 (0.328)] [data_t 0.003] [optim_t 0.321] [lr 0.005000] 2024-04-04 07:59:49,609 - Train: 5.88% [290600/4942000] [58.8/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 08:00:22,323 - Train: 5.88% [290700/4942000] [58.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 08:00:55,065 - Train: 5.88% [290800/4942000] [58.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 08:01:27,832 - Train: 5.89% [290900/4942000] [58.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:02:00,549 - Train: 5.89% [291000/4942000] [58.9/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 08:02:33,307 - Train: 5.89% [291100/4942000] [58.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:03:06,019 - Train: 5.89% [291200/4942000] [58.9/1000.0] [batch_t 0.321 (0.327)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-04 08:03:38,843 - Train: 5.89% [291300/4942000] [58.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 08:04:11,617 - Train: 5.90% [291400/4942000] [59.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 08:04:44,385 - Train: 5.90% [291500/4942000] [59.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 08:05:09,928 - ==> Total time: 1 day, 14:07:49 Eta: 25 days, 8:08:46 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 08:05:18,989 - Train: 5.90% [291600/4942000] [59.0/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 08:05:51,751 - Train: 5.90% [291700/4942000] [59.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 08:06:24,517 - Train: 5.90% [291800/4942000] [59.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 08:06:57,221 - Train: 5.91% [291900/4942000] [59.1/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 08:07:29,914 - Train: 5.91% [292000/4942000] [59.1/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 08:08:02,649 - Train: 5.91% [292100/4942000] [59.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 08:08:35,391 - Train: 5.91% [292200/4942000] [59.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 08:09:08,154 - Train: 5.91% [292300/4942000] [59.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:09:40,925 - Train: 5.92% [292400/4942000] [59.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 08:10:13,626 - Train: 5.92% [292500/4942000] [59.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 08:10:46,471 - Train: 5.92% [292600/4942000] [59.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 08:11:19,211 - Train: 5.92% [292700/4942000] [59.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:11:51,961 - Train: 5.92% [292800/4942000] [59.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 08:12:24,731 - Train: 5.93% [292900/4942000] [59.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 08:12:57,526 - Train: 5.93% [293000/4942000] [59.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 08:13:30,266 - Train: 5.93% [293100/4942000] [59.3/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 08:14:03,014 - Train: 5.93% [293200/4942000] [59.3/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 08:14:35,847 - Train: 5.93% [293300/4942000] [59.3/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 08:15:08,669 - Train: 5.94% [293400/4942000] [59.4/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 08:15:41,485 - Train: 5.94% [293500/4942000] [59.4/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 08:16:14,325 - Train: 5.94% [293600/4942000] [59.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 08:16:47,131 - Train: 5.94% [293700/4942000] [59.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 08:17:19,945 - Train: 5.94% [293800/4942000] [59.4/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 08:17:52,801 - Train: 5.95% [293900/4942000] [59.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 08:18:25,770 - Train: 5.95% [294000/4942000] [59.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 08:18:58,594 - Train: 5.95% [294100/4942000] [59.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 08:19:31,419 - Train: 5.95% [294200/4942000] [59.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:20:04,234 - Train: 5.96% [294300/4942000] [59.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 08:20:37,060 - Train: 5.96% [294400/4942000] [59.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 08:21:09,864 - Train: 5.96% [294500/4942000] [59.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 08:21:42,706 - Train: 5.96% [294600/4942000] [59.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 08:22:16,571 - Train: 5.96% [294700/4942000] [59.6/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 08:22:49,396 - Train: 5.97% [294800/4942000] [59.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-04 08:23:22,230 - Train: 5.97% [294900/4942000] [59.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:23:55,062 - Train: 5.97% [295000/4942000] [59.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 08:24:27,895 - Train: 5.97% [295100/4942000] [59.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:25:00,692 - Train: 5.97% [295200/4942000] [59.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 08:25:33,472 - Train: 5.98% [295300/4942000] [59.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:26:06,322 - Train: 5.98% [295400/4942000] [59.8/1000.0] [batch_t 0.429 (0.328)] [data_t 0.002] [optim_t 0.427] [lr 0.005000] 2024-04-04 08:26:39,050 - Train: 5.98% [295500/4942000] [59.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 08:27:11,761 - Train: 5.98% [295600/4942000] [59.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 08:27:44,508 - Train: 5.98% [295700/4942000] [59.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:28:17,226 - Train: 5.99% [295800/4942000] [59.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 08:28:49,963 - Train: 5.99% [295900/4942000] [59.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 08:29:22,750 - Train: 5.99% [296000/4942000] [59.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 08:29:55,485 - Train: 5.99% [296100/4942000] [59.9/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 08:30:28,714 - Train: 5.99% [296200/4942000] [59.9/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 08:31:01,490 - Train: 6.00% [296300/4942000] [60.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 08:31:34,227 - Train: 6.00% [296400/4942000] [60.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 08:32:06,888 - Train: 6.00% [296500/4942000] [60.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:32:13,449 - ==> Total time: 1 day, 14:34:52 Eta: 25 days, 4:26:24 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 08:32:41,406 - Train: 6.00% [296600/4942000] [60.0/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 08:33:14,318 - Train: 6.00% [296700/4942000] [60.0/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 08:33:47,084 - Train: 6.01% [296800/4942000] [60.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 08:34:19,907 - Train: 6.01% [296900/4942000] [60.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:34:52,694 - Train: 6.01% [297000/4942000] [60.1/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-04 08:35:25,488 - Train: 6.01% [297100/4942000] [60.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 08:35:58,251 - Train: 6.01% [297200/4942000] [60.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 08:36:30,961 - Train: 6.02% [297300/4942000] [60.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 08:37:03,738 - Train: 6.02% [297400/4942000] [60.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 08:37:36,468 - Train: 6.02% [297500/4942000] [60.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 08:38:09,269 - Train: 6.02% [297600/4942000] [60.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:38:41,987 - Train: 6.02% [297700/4942000] [60.2/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:39:14,761 - Train: 6.03% [297800/4942000] [60.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 08:39:47,515 - Train: 6.03% [297900/4942000] [60.3/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:40:20,252 - Train: 6.03% [298000/4942000] [60.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:40:52,989 - Train: 6.03% [298100/4942000] [60.3/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:41:25,933 - Train: 6.03% [298200/4942000] [60.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 08:41:58,705 - Train: 6.04% [298300/4942000] [60.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 08:42:31,500 - Train: 6.04% [298400/4942000] [60.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 08:43:04,213 - Train: 6.04% [298500/4942000] [60.4/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 08:43:36,928 - Train: 6.04% [298600/4942000] [60.4/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 08:44:09,680 - Train: 6.04% [298700/4942000] [60.4/1000.0] [batch_t 0.323 (0.327)] [data_t 0.003] [optim_t 0.320] [lr 0.005000] 2024-04-04 08:44:42,351 - Train: 6.05% [298800/4942000] [60.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 08:45:15,091 - Train: 6.05% [298900/4942000] [60.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:45:47,849 - Train: 6.05% [299000/4942000] [60.5/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 08:46:20,563 - Train: 6.05% [299100/4942000] [60.5/1000.0] [batch_t 0.321 (0.327)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-04 08:46:53,298 - Train: 6.05% [299200/4942000] [60.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:47:26,060 - Train: 6.06% [299300/4942000] [60.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 08:47:58,799 - Train: 6.06% [299400/4942000] [60.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 08:48:31,637 - Train: 6.06% [299500/4942000] [60.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 08:49:04,491 - Train: 6.06% [299600/4942000] [60.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 08:49:37,235 - Train: 6.06% [299700/4942000] [60.6/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 08:50:09,955 - Train: 6.07% [299800/4942000] [60.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 08:50:42,659 - Train: 6.07% [299900/4942000] [60.7/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:51:15,401 - Train: 6.07% [300000/4942000] [60.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 08:51:48,106 - Train: 6.07% [300100/4942000] [60.7/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 08:52:20,857 - Train: 6.07% [300200/4942000] [60.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 08:52:53,617 - Train: 6.08% [300300/4942000] [60.8/1000.0] [batch_t 0.321 (0.328)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-04 08:53:26,369 - Train: 6.08% [300400/4942000] [60.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 08:53:59,125 - Train: 6.08% [300500/4942000] [60.8/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:54:31,836 - Train: 6.08% [300600/4942000] [60.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 08:55:04,587 - Train: 6.08% [300700/4942000] [60.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 08:55:37,291 - Train: 6.09% [300800/4942000] [60.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 08:56:09,998 - Train: 6.09% [300900/4942000] [60.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 08:56:42,733 - Train: 6.09% [301000/4942000] [60.9/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 08:57:15,593 - Train: 6.09% [301100/4942000] [60.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 08:57:48,324 - Train: 6.09% [301200/4942000] [60.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 08:58:21,887 - Train: 6.10% [301300/4942000] [61.0/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 08:58:54,679 - Train: 6.10% [301400/4942000] [61.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:59:14,995 - ==> Total time: 1 day, 15:01:54 Eta: 25 days, 0:49:56 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 08:59:29,442 - Train: 6.10% [301500/4942000] [61.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:00:02,240 - Train: 6.10% [301600/4942000] [61.0/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:00:35,018 - Train: 6.10% [301700/4942000] [61.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 09:01:07,831 - Train: 6.11% [301800/4942000] [61.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 09:01:40,553 - Train: 6.11% [301900/4942000] [61.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-04 09:02:13,307 - Train: 6.11% [302000/4942000] [61.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 09:02:46,108 - Train: 6.11% [302100/4942000] [61.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 09:03:18,955 - Train: 6.11% [302200/4942000] [61.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 09:03:51,759 - Train: 6.12% [302300/4942000] [61.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 09:04:24,705 - Train: 6.12% [302400/4942000] [61.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 09:04:57,516 - Train: 6.12% [302500/4942000] [61.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 09:05:30,281 - Train: 6.12% [302600/4942000] [61.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 09:06:03,079 - Train: 6.13% [302700/4942000] [61.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 09:06:36,392 - Train: 6.13% [302800/4942000] [61.3/1000.0] [batch_t 0.326 (0.333)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 09:07:09,624 - Train: 6.13% [302900/4942000] [61.3/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 09:07:42,362 - Train: 6.13% [303000/4942000] [61.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 09:08:15,817 - Train: 6.13% [303100/4942000] [61.3/1000.0] [batch_t 0.331 (0.334)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:08:48,554 - Train: 6.14% [303200/4942000] [61.4/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 09:09:21,357 - Train: 6.14% [303300/4942000] [61.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-04 09:09:54,174 - Train: 6.14% [303400/4942000] [61.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 09:10:26,979 - Train: 6.14% [303500/4942000] [61.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 09:10:59,702 - Train: 6.14% [303600/4942000] [61.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 09:11:33,466 - Train: 6.15% [303700/4942000] [61.5/1000.0] [batch_t 0.325 (0.338)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 09:12:06,910 - Train: 6.15% [303800/4942000] [61.5/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 09:12:39,748 - Train: 6.15% [303900/4942000] [61.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-04 09:13:12,471 - Train: 6.15% [304000/4942000] [61.5/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 09:13:45,234 - Train: 6.15% [304100/4942000] [61.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 09:14:18,040 - Train: 6.16% [304200/4942000] [61.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 09:14:50,827 - Train: 6.16% [304300/4942000] [61.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 09:15:23,620 - Train: 6.16% [304400/4942000] [61.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 09:15:56,389 - Train: 6.16% [304500/4942000] [61.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 09:16:29,139 - Train: 6.16% [304600/4942000] [61.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 09:17:01,869 - Train: 6.17% [304700/4942000] [61.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 09:17:34,615 - Train: 6.17% [304800/4942000] [61.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 09:18:07,418 - Train: 6.17% [304900/4942000] [61.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 09:18:40,198 - Train: 6.17% [305000/4942000] [61.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 09:19:13,022 - Train: 6.17% [305100/4942000] [61.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 09:19:45,840 - Train: 6.18% [305200/4942000] [61.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 09:20:18,760 - Train: 6.18% [305300/4942000] [61.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 09:20:51,544 - Train: 6.18% [305400/4942000] [61.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 09:21:24,317 - Train: 6.18% [305500/4942000] [61.8/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 09:21:57,082 - Train: 6.18% [305600/4942000] [61.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 09:22:29,852 - Train: 6.19% [305700/4942000] [61.9/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 09:23:02,588 - Train: 6.19% [305800/4942000] [61.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 09:23:35,324 - Train: 6.19% [305900/4942000] [61.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 09:24:08,076 - Train: 6.19% [306000/4942000] [61.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 09:24:40,867 - Train: 6.19% [306100/4942000] [61.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 09:25:13,612 - Train: 6.20% [306200/4942000] [62.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 09:25:46,348 - Train: 6.20% [306300/4942000] [62.0/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 09:26:19,038 - Train: 6.20% [306400/4942000] [62.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 09:26:20,351 - ==> Total time: 1 day, 15:28:59 Eta: 24 days, 21:20:33 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 09:26:53,647 - Train: 6.20% [306500/4942000] [62.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 09:27:26,507 - Train: 6.20% [306600/4942000] [62.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 09:27:59,263 - Train: 6.21% [306700/4942000] [62.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 09:28:31,994 - Train: 6.21% [306800/4942000] [62.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 09:29:04,723 - Train: 6.21% [306900/4942000] [62.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 09:29:37,423 - Train: 6.21% [307000/4942000] [62.1/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 09:30:10,211 - Train: 6.21% [307100/4942000] [62.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 09:30:42,952 - Train: 6.22% [307200/4942000] [62.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 09:31:15,685 - Train: 6.22% [307300/4942000] [62.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 09:31:48,446 - Train: 6.22% [307400/4942000] [62.2/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 09:32:21,261 - Train: 6.22% [307500/4942000] [62.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 09:32:54,029 - Train: 6.22% [307600/4942000] [62.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 09:33:26,820 - Train: 6.23% [307700/4942000] [62.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 09:33:59,629 - Train: 6.23% [307800/4942000] [62.3/1000.0] [batch_t 0.336 (0.328)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-04 09:34:32,379 - Train: 6.23% [307900/4942000] [62.3/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:35:05,152 - Train: 6.23% [308000/4942000] [62.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 09:35:38,035 - Train: 6.23% [308100/4942000] [62.3/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 09:36:10,804 - Train: 6.24% [308200/4942000] [62.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 09:36:43,586 - Train: 6.24% [308300/4942000] [62.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 09:37:16,351 - Train: 6.24% [308400/4942000] [62.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 09:37:49,202 - Train: 6.24% [308500/4942000] [62.4/1000.0] [batch_t 0.321 (0.328)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-04 09:38:22,681 - Train: 6.24% [308600/4942000] [62.4/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 09:38:55,492 - Train: 6.25% [308700/4942000] [62.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 09:39:28,298 - Train: 6.25% [308800/4942000] [62.5/1000.0] [batch_t 0.321 (0.328)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-04 09:40:01,115 - Train: 6.25% [308900/4942000] [62.5/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 09:40:33,957 - Train: 6.25% [309000/4942000] [62.5/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 09:41:06,728 - Train: 6.25% [309100/4942000] [62.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 09:41:39,445 - Train: 6.26% [309200/4942000] [62.6/1000.0] [batch_t 0.321 (0.327)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-04 09:42:12,250 - Train: 6.26% [309300/4942000] [62.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:42:45,087 - Train: 6.26% [309400/4942000] [62.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 09:43:18,027 - Train: 6.26% [309500/4942000] [62.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 09:43:50,806 - Train: 6.26% [309600/4942000] [62.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 09:44:23,594 - Train: 6.27% [309700/4942000] [62.7/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:44:56,363 - Train: 6.27% [309800/4942000] [62.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 09:45:29,129 - Train: 6.27% [309900/4942000] [62.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 09:46:01,918 - Train: 6.27% [310000/4942000] [62.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 09:46:34,724 - Train: 6.27% [310100/4942000] [62.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 09:47:07,538 - Train: 6.28% [310200/4942000] [62.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 09:47:40,304 - Train: 6.28% [310300/4942000] [62.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 09:48:13,134 - Train: 6.28% [310400/4942000] [62.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 09:48:45,941 - Train: 6.28% [310500/4942000] [62.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 09:49:18,707 - Train: 6.28% [310600/4942000] [62.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 09:49:51,480 - Train: 6.29% [310700/4942000] [62.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-04 09:50:24,313 - Train: 6.29% [310800/4942000] [62.9/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 09:50:57,326 - Train: 6.29% [310900/4942000] [62.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 09:51:30,065 - Train: 6.29% [311000/4942000] [62.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 09:52:02,830 - Train: 6.30% [311100/4942000] [63.0/1000.0] [batch_t 0.321 (0.328)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-04 09:52:35,567 - Train: 6.30% [311200/4942000] [63.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 09:53:08,371 - Train: 6.30% [311300/4942000] [63.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 09:53:23,459 - ==> Total time: 1 day, 15:56:02 Eta: 24 days, 17:56:24 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 09:53:43,103 - Train: 6.30% [311400/4942000] [63.0/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 09:54:15,869 - Train: 6.30% [311500/4942000] [63.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 09:54:48,611 - Train: 6.31% [311600/4942000] [63.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 09:55:21,365 - Train: 6.31% [311700/4942000] [63.1/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 09:55:54,098 - Train: 6.31% [311800/4942000] [63.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 09:56:26,845 - Train: 6.31% [311900/4942000] [63.1/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:56:59,550 - Train: 6.31% [312000/4942000] [63.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 09:57:32,326 - Train: 6.32% [312100/4942000] [63.2/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 09:58:05,225 - Train: 6.32% [312200/4942000] [63.2/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 09:58:37,946 - Train: 6.32% [312300/4942000] [63.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 09:59:10,705 - Train: 6.32% [312400/4942000] [63.2/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 09:59:43,427 - Train: 6.32% [312500/4942000] [63.2/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 10:00:16,172 - Train: 6.33% [312600/4942000] [63.3/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 10:00:48,945 - Train: 6.33% [312700/4942000] [63.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:01:21,702 - Train: 6.33% [312800/4942000] [63.3/1000.0] [batch_t 0.321 (0.327)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-04 10:01:54,469 - Train: 6.33% [312900/4942000] [63.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:02:27,299 - Train: 6.33% [313000/4942000] [63.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:03:00,081 - Train: 6.34% [313100/4942000] [63.4/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 10:03:32,832 - Train: 6.34% [313200/4942000] [63.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 10:04:05,601 - Train: 6.34% [313300/4942000] [63.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 10:04:38,361 - Train: 6.34% [313400/4942000] [63.4/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-04 10:05:11,108 - Train: 6.34% [313500/4942000] [63.4/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 10:05:44,046 - Train: 6.35% [313600/4942000] [63.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:06:16,821 - Train: 6.35% [313700/4942000] [63.5/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 10:06:49,564 - Train: 6.35% [313800/4942000] [63.5/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 10:07:22,394 - Train: 6.35% [313900/4942000] [63.5/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 10:07:55,199 - Train: 6.35% [314000/4942000] [63.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 10:08:27,950 - Train: 6.36% [314100/4942000] [63.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 10:09:00,786 - Train: 6.36% [314200/4942000] [63.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:09:33,560 - Train: 6.36% [314300/4942000] [63.6/1000.0] [batch_t 0.324 (0.328)] [data_t 0.003] [optim_t 0.321] [lr 0.005000] 2024-04-04 10:10:06,303 - Train: 6.36% [314400/4942000] [63.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 10:10:39,087 - Train: 6.36% [314500/4942000] [63.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 10:11:12,509 - Train: 6.37% [314600/4942000] [63.7/1000.0] [batch_t 0.323 (0.334)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 10:11:45,289 - Train: 6.37% [314700/4942000] [63.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:12:19,304 - Train: 6.37% [314800/4942000] [63.7/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:12:52,035 - Train: 6.37% [314900/4942000] [63.7/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 10:13:26,022 - Train: 6.37% [315000/4942000] [63.7/1000.0] [batch_t 0.326 (0.340)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 10:13:59,015 - Train: 6.38% [315100/4942000] [63.8/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 10:14:31,826 - Train: 6.38% [315200/4942000] [63.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:15:05,954 - Train: 6.38% [315300/4942000] [63.8/1000.0] [batch_t 0.333 (0.341)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-04 10:15:38,736 - Train: 6.38% [315400/4942000] [63.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 10:16:13,064 - Train: 6.38% [315500/4942000] [63.8/1000.0] [batch_t 0.326 (0.343)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 10:16:45,837 - Train: 6.39% [315600/4942000] [63.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 10:17:18,653 - Train: 6.39% [315700/4942000] [63.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 10:17:51,449 - Train: 6.39% [315800/4942000] [63.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 10:18:24,944 - Train: 6.39% [315900/4942000] [63.9/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 10:18:57,761 - Train: 6.39% [316000/4942000] [63.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:19:30,534 - Train: 6.40% [316100/4942000] [64.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 10:20:03,300 - Train: 6.40% [316200/4942000] [64.0/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 10:20:32,640 - ==> Total time: 1 day, 16:23:11 Eta: 24 days, 14:39:15 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 10:20:38,630 - Train: 6.40% [316300/4942000] [64.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:21:11,527 - Train: 6.40% [316400/4942000] [64.0/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 10:21:44,330 - Train: 6.40% [316500/4942000] [64.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 10:22:17,069 - Train: 6.41% [316600/4942000] [64.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 10:22:49,896 - Train: 6.41% [316700/4942000] [64.1/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:23:22,676 - Train: 6.41% [316800/4942000] [64.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:23:55,477 - Train: 6.41% [316900/4942000] [64.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:24:28,274 - Train: 6.41% [317000/4942000] [64.1/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 10:25:01,032 - Train: 6.42% [317100/4942000] [64.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:25:33,807 - Train: 6.42% [317200/4942000] [64.2/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 10:26:06,610 - Train: 6.42% [317300/4942000] [64.2/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:26:39,389 - Train: 6.42% [317400/4942000] [64.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:27:12,203 - Train: 6.42% [317500/4942000] [64.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 10:27:45,001 - Train: 6.43% [317600/4942000] [64.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:28:17,810 - Train: 6.43% [317700/4942000] [64.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:28:50,650 - Train: 6.43% [317800/4942000] [64.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:29:23,450 - Train: 6.43% [317900/4942000] [64.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 10:29:56,245 - Train: 6.43% [318000/4942000] [64.3/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:30:28,987 - Train: 6.44% [318100/4942000] [64.4/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 10:31:01,791 - Train: 6.44% [318200/4942000] [64.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 10:31:34,559 - Train: 6.44% [318300/4942000] [64.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:32:07,294 - Train: 6.44% [318400/4942000] [64.4/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 10:32:40,100 - Train: 6.44% [318500/4942000] [64.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 10:33:12,884 - Train: 6.45% [318600/4942000] [64.5/1000.0] [batch_t 0.324 (0.328)] [data_t 0.003] [optim_t 0.321] [lr 0.005000] 2024-04-04 10:33:45,670 - Train: 6.45% [318700/4942000] [64.5/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:34:18,444 - Train: 6.45% [318800/4942000] [64.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:34:51,227 - Train: 6.45% [318900/4942000] [64.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 10:35:24,022 - Train: 6.45% [319000/4942000] [64.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:35:56,799 - Train: 6.46% [319100/4942000] [64.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:36:29,591 - Train: 6.46% [319200/4942000] [64.6/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 10:37:02,464 - Train: 6.46% [319300/4942000] [64.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 10:37:35,274 - Train: 6.46% [319400/4942000] [64.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 10:38:08,028 - Train: 6.46% [319500/4942000] [64.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 10:38:40,839 - Train: 6.47% [319600/4942000] [64.7/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:39:13,599 - Train: 6.47% [319700/4942000] [64.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:39:46,388 - Train: 6.47% [319800/4942000] [64.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 10:40:19,153 - Train: 6.47% [319900/4942000] [64.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 10:40:51,933 - Train: 6.48% [320000/4942000] [64.8/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:41:24,679 - Train: 6.48% [320100/4942000] [64.8/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:41:57,431 - Train: 6.48% [320200/4942000] [64.8/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:42:30,192 - Train: 6.48% [320300/4942000] [64.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:43:02,925 - Train: 6.48% [320400/4942000] [64.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:43:35,697 - Train: 6.49% [320500/4942000] [64.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 10:44:08,453 - Train: 6.49% [320600/4942000] [64.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 10:44:41,383 - Train: 6.49% [320700/4942000] [64.9/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 10:45:14,200 - Train: 6.49% [320800/4942000] [64.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:45:46,929 - Train: 6.49% [320900/4942000] [64.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 10:46:19,705 - Train: 6.50% [321000/4942000] [65.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 10:46:52,438 - Train: 6.50% [321100/4942000] [65.0/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 10:47:25,190 - Train: 6.50% [321200/4942000] [65.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 10:47:35,012 - ==> Total time: 1 day, 16:50:14 Eta: 24 days, 11:25:42 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 10:47:59,868 - Train: 6.50% [321300/4942000] [65.0/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:48:32,570 - Train: 6.50% [321400/4942000] [65.0/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 10:49:05,276 - Train: 6.51% [321500/4942000] [65.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 10:49:38,032 - Train: 6.51% [321600/4942000] [65.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 10:50:10,787 - Train: 6.51% [321700/4942000] [65.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:50:43,523 - Train: 6.51% [321800/4942000] [65.1/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:51:16,279 - Train: 6.51% [321900/4942000] [65.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:51:49,193 - Train: 6.52% [322000/4942000] [65.2/1000.0] [batch_t 0.319 (0.329)] [data_t 0.002] [optim_t 0.317] [lr 0.005000] 2024-04-04 10:52:21,919 - Train: 6.52% [322100/4942000] [65.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 10:52:54,666 - Train: 6.52% [322200/4942000] [65.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:53:27,395 - Train: 6.52% [322300/4942000] [65.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 10:54:00,130 - Train: 6.52% [322400/4942000] [65.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 10:54:32,866 - Train: 6.53% [322500/4942000] [65.3/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 10:55:05,601 - Train: 6.53% [322600/4942000] [65.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:55:38,387 - Train: 6.53% [322700/4942000] [65.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:56:11,149 - Train: 6.53% [322800/4942000] [65.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 10:56:43,967 - Train: 6.53% [322900/4942000] [65.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 10:57:16,779 - Train: 6.54% [323000/4942000] [65.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 10:57:49,585 - Train: 6.54% [323100/4942000] [65.4/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 10:58:22,375 - Train: 6.54% [323200/4942000] [65.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 10:58:55,191 - Train: 6.54% [323300/4942000] [65.4/1000.0] [batch_t 0.336 (0.328)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-04 10:59:29,262 - Train: 6.54% [323400/4942000] [65.4/1000.0] [batch_t 0.326 (0.341)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 11:00:02,063 - Train: 6.55% [323500/4942000] [65.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 11:00:34,872 - Train: 6.55% [323600/4942000] [65.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:01:07,663 - Train: 6.55% [323700/4942000] [65.5/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 11:01:40,418 - Train: 6.55% [323800/4942000] [65.5/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 11:02:13,133 - Train: 6.55% [323900/4942000] [65.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 11:02:45,822 - Train: 6.56% [324000/4942000] [65.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 11:03:18,554 - Train: 6.56% [324100/4942000] [65.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 11:03:51,298 - Train: 6.56% [324200/4942000] [65.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 11:04:24,000 - Train: 6.56% [324300/4942000] [65.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 11:04:56,755 - Train: 6.56% [324400/4942000] [65.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 11:05:29,487 - Train: 6.57% [324500/4942000] [65.7/1000.0] [batch_t 0.325 (0.327)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-04 11:06:02,257 - Train: 6.57% [324600/4942000] [65.7/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 11:06:35,007 - Train: 6.57% [324700/4942000] [65.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 11:07:07,766 - Train: 6.57% [324800/4942000] [65.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:07:40,676 - Train: 6.57% [324900/4942000] [65.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 11:08:13,433 - Train: 6.58% [325000/4942000] [65.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 11:08:46,215 - Train: 6.58% [325100/4942000] [65.8/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 11:09:18,988 - Train: 6.58% [325200/4942000] [65.8/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 11:09:51,720 - Train: 6.58% [325300/4942000] [65.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 11:10:24,454 - Train: 6.58% [325400/4942000] [65.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 11:10:57,207 - Train: 6.59% [325500/4942000] [65.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 11:11:29,954 - Train: 6.59% [325600/4942000] [65.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 11:12:02,731 - Train: 6.59% [325700/4942000] [65.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 11:12:35,484 - Train: 6.59% [325800/4942000] [65.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 11:13:08,218 - Train: 6.59% [325900/4942000] [65.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 11:13:40,989 - Train: 6.60% [326000/4942000] [66.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:14:13,753 - Train: 6.60% [326100/4942000] [66.0/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 11:14:37,329 - ==> Total time: 1 day, 17:17:16 Eta: 24 days, 8:17:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 11:14:48,403 - Train: 6.60% [326200/4942000] [66.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:15:21,185 - Train: 6.60% [326300/4942000] [66.0/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 11:15:53,947 - Train: 6.60% [326400/4942000] [66.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 11:16:26,683 - Train: 6.61% [326500/4942000] [66.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:16:59,518 - Train: 6.61% [326600/4942000] [66.1/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 11:17:32,287 - Train: 6.61% [326700/4942000] [66.1/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 11:18:05,009 - Train: 6.61% [326800/4942000] [66.1/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:18:37,808 - Train: 6.61% [326900/4942000] [66.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 11:19:10,649 - Train: 6.62% [327000/4942000] [66.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 11:19:43,433 - Train: 6.62% [327100/4942000] [66.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 11:20:16,174 - Train: 6.62% [327200/4942000] [66.2/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:20:49,023 - Train: 6.62% [327300/4942000] [66.2/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 11:21:21,786 - Train: 6.62% [327400/4942000] [66.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 11:21:54,581 - Train: 6.63% [327500/4942000] [66.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:22:27,370 - Train: 6.63% [327600/4942000] [66.3/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 11:23:00,319 - Train: 6.63% [327700/4942000] [66.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 11:23:33,108 - Train: 6.63% [327800/4942000] [66.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 11:24:05,922 - Train: 6.63% [327900/4942000] [66.3/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 11:24:38,727 - Train: 6.64% [328000/4942000] [66.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 11:25:11,540 - Train: 6.64% [328100/4942000] [66.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 11:25:44,357 - Train: 6.64% [328200/4942000] [66.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 11:26:17,772 - Train: 6.64% [328300/4942000] [66.4/1000.0] [batch_t 0.328 (0.334)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-04 11:26:50,583 - Train: 6.65% [328400/4942000] [66.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:27:23,369 - Train: 6.65% [328500/4942000] [66.5/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 11:27:56,172 - Train: 6.65% [328600/4942000] [66.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 11:28:28,957 - Train: 6.65% [328700/4942000] [66.5/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 11:29:01,754 - Train: 6.65% [328800/4942000] [66.5/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 11:29:34,534 - Train: 6.66% [328900/4942000] [66.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 11:30:07,343 - Train: 6.66% [329000/4942000] [66.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 11:30:40,212 - Train: 6.66% [329100/4942000] [66.6/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 11:31:12,958 - Train: 6.66% [329200/4942000] [66.6/1000.0] [batch_t 0.321 (0.327)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-04 11:31:45,772 - Train: 6.66% [329300/4942000] [66.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 11:32:18,811 - Train: 6.67% [329400/4942000] [66.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:32:51,611 - Train: 6.67% [329500/4942000] [66.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 11:33:24,400 - Train: 6.67% [329600/4942000] [66.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:33:57,147 - Train: 6.67% [329700/4942000] [66.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:34:30,981 - Train: 6.67% [329800/4942000] [66.7/1000.0] [batch_t 0.325 (0.338)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 11:35:03,799 - Train: 6.68% [329900/4942000] [66.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:35:36,630 - Train: 6.68% [330000/4942000] [66.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:36:09,340 - Train: 6.68% [330100/4942000] [66.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 11:36:42,048 - Train: 6.68% [330200/4942000] [66.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 11:37:14,792 - Train: 6.68% [330300/4942000] [66.8/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:37:47,544 - Train: 6.69% [330400/4942000] [66.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 11:38:20,407 - Train: 6.69% [330500/4942000] [66.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 11:38:53,164 - Train: 6.69% [330600/4942000] [66.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 11:39:25,899 - Train: 6.69% [330700/4942000] [66.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 11:39:58,628 - Train: 6.69% [330800/4942000] [66.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 11:40:31,347 - Train: 6.70% [330900/4942000] [67.0/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 11:41:04,081 - Train: 6.70% [331000/4942000] [67.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:41:36,828 - Train: 6.70% [331100/4942000] [67.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 11:41:41,407 - ==> Total time: 1 day, 17:44:20 Eta: 24 days, 5:13:54 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 11:42:11,455 - Train: 6.70% [331200/4942000] [67.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 11:42:44,176 - Train: 6.70% [331300/4942000] [67.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 11:43:16,912 - Train: 6.71% [331400/4942000] [67.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 11:43:49,627 - Train: 6.71% [331500/4942000] [67.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 11:44:22,371 - Train: 6.71% [331600/4942000] [67.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 11:44:55,120 - Train: 6.71% [331700/4942000] [67.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 11:45:27,968 - Train: 6.71% [331800/4942000] [67.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 11:46:00,707 - Train: 6.72% [331900/4942000] [67.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 11:46:33,451 - Train: 6.72% [332000/4942000] [67.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 11:47:06,196 - Train: 6.72% [332100/4942000] [67.2/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 11:47:38,932 - Train: 6.72% [332200/4942000] [67.2/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 11:48:11,654 - Train: 6.72% [332300/4942000] [67.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 11:48:44,385 - Train: 6.73% [332400/4942000] [67.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 11:49:17,077 - Train: 6.73% [332500/4942000] [67.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:49:49,825 - Train: 6.73% [332600/4942000] [67.3/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 11:50:22,561 - Train: 6.73% [332700/4942000] [67.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 11:50:55,311 - Train: 6.73% [332800/4942000] [67.3/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 11:51:28,109 - Train: 6.74% [332900/4942000] [67.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 11:52:00,819 - Train: 6.74% [333000/4942000] [67.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 11:52:33,560 - Train: 6.74% [333100/4942000] [67.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 11:53:06,368 - Train: 6.74% [333200/4942000] [67.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 11:53:39,383 - Train: 6.74% [333300/4942000] [67.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 11:54:12,193 - Train: 6.75% [333400/4942000] [67.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:54:45,055 - Train: 6.75% [333500/4942000] [67.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:55:17,893 - Train: 6.75% [333600/4942000] [67.5/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 11:55:50,707 - Train: 6.75% [333700/4942000] [67.5/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:56:23,565 - Train: 6.75% [333800/4942000] [67.5/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 11:56:56,401 - Train: 6.76% [333900/4942000] [67.6/1000.0] [batch_t 0.325 (0.328)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-04 11:57:29,167 - Train: 6.76% [334000/4942000] [67.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 11:58:01,933 - Train: 6.76% [334100/4942000] [67.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:58:34,629 - Train: 6.76% [334200/4942000] [67.6/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 11:59:07,373 - Train: 6.76% [334300/4942000] [67.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 11:59:40,111 - Train: 6.77% [334400/4942000] [67.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 12:00:12,811 - Train: 6.77% [334500/4942000] [67.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 12:00:45,505 - Train: 6.77% [334600/4942000] [67.7/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 12:01:18,333 - Train: 6.77% [334700/4942000] [67.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 12:01:51,061 - Train: 6.77% [334800/4942000] [67.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 12:02:23,820 - Train: 6.78% [334900/4942000] [67.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 12:02:56,558 - Train: 6.78% [335000/4942000] [67.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 12:03:29,293 - Train: 6.78% [335100/4942000] [67.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 12:04:02,011 - Train: 6.78% [335200/4942000] [67.8/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-04 12:04:34,742 - Train: 6.78% [335300/4942000] [67.8/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 12:05:07,539 - Train: 6.79% [335400/4942000] [67.9/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 12:05:40,351 - Train: 6.79% [335500/4942000] [67.9/1000.0] [batch_t 0.325 (0.328)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-04 12:06:13,214 - Train: 6.79% [335600/4942000] [67.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 12:06:46,007 - Train: 6.79% [335700/4942000] [67.9/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 12:07:18,818 - Train: 6.79% [335800/4942000] [67.9/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-04 12:07:51,667 - Train: 6.80% [335900/4942000] [68.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 12:08:24,466 - Train: 6.80% [336000/4942000] [68.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 12:08:42,856 - ==> Total time: 1 day, 18:11:22 Eta: 24 days, 2:14:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 12:08:59,592 - Train: 6.80% [336100/4942000] [68.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 12:09:32,328 - Train: 6.80% [336200/4942000] [68.0/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 12:10:05,122 - Train: 6.80% [336300/4942000] [68.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 12:10:37,897 - Train: 6.81% [336400/4942000] [68.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-04 12:11:10,670 - Train: 6.81% [336500/4942000] [68.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 12:11:43,451 - Train: 6.81% [336600/4942000] [68.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 12:12:16,237 - Train: 6.81% [336700/4942000] [68.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 12:12:49,022 - Train: 6.82% [336800/4942000] [68.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 12:13:22,339 - Train: 6.82% [336900/4942000] [68.2/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 12:13:55,095 - Train: 6.82% [337000/4942000] [68.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 12:14:27,881 - Train: 6.82% [337100/4942000] [68.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 12:15:00,645 - Train: 6.82% [337200/4942000] [68.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 12:15:33,432 - Train: 6.83% [337300/4942000] [68.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 12:16:06,247 - Train: 6.83% [337400/4942000] [68.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 12:16:39,182 - Train: 6.83% [337500/4942000] [68.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 12:17:11,994 - Train: 6.83% [337600/4942000] [68.3/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 12:17:44,786 - Train: 6.83% [337700/4942000] [68.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-04 12:18:17,617 - Train: 6.84% [337800/4942000] [68.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 12:18:50,397 - Train: 6.84% [337900/4942000] [68.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 12:19:23,187 - Train: 6.84% [338000/4942000] [68.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 12:19:55,968 - Train: 6.84% [338100/4942000] [68.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 12:20:28,715 - Train: 6.84% [338200/4942000] [68.4/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 12:21:01,488 - Train: 6.85% [338300/4942000] [68.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 12:21:34,272 - Train: 6.85% [338400/4942000] [68.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 12:22:07,073 - Train: 6.85% [338500/4942000] [68.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-04 12:22:39,880 - Train: 6.85% [338600/4942000] [68.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 12:23:12,642 - Train: 6.85% [338700/4942000] [68.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 12:23:45,455 - Train: 6.86% [338800/4942000] [68.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 12:24:18,255 - Train: 6.86% [338900/4942000] [68.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 12:24:51,131 - Train: 6.86% [339000/4942000] [68.6/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 12:25:23,874 - Train: 6.86% [339100/4942000] [68.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 12:25:56,614 - Train: 6.86% [339200/4942000] [68.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 12:26:29,376 - Train: 6.87% [339300/4942000] [68.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 12:27:02,146 - Train: 6.87% [339400/4942000] [68.7/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 12:27:34,971 - Train: 6.87% [339500/4942000] [68.7/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:28:07,732 - Train: 6.87% [339600/4942000] [68.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 12:28:40,509 - Train: 6.87% [339700/4942000] [68.7/1000.0] [batch_t 0.324 (0.328)] [data_t 0.003] [optim_t 0.321] [lr 0.005000] 2024-04-04 12:29:13,351 - Train: 6.88% [339800/4942000] [68.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 12:29:46,128 - Train: 6.88% [339900/4942000] [68.8/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-04 12:30:18,855 - Train: 6.88% [340000/4942000] [68.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 12:30:51,545 - Train: 6.88% [340100/4942000] [68.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 12:31:24,322 - Train: 6.88% [340200/4942000] [68.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 12:31:57,030 - Train: 6.89% [340300/4942000] [68.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 12:32:30,007 - Train: 6.89% [340400/4942000] [68.9/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-04 12:33:02,732 - Train: 6.89% [340500/4942000] [68.9/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 12:33:35,512 - Train: 6.89% [340600/4942000] [68.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 12:34:08,391 - Train: 6.89% [340700/4942000] [68.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 12:34:41,188 - Train: 6.90% [340800/4942000] [69.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 12:35:13,996 - Train: 6.90% [340900/4942000] [69.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 12:35:46,118 - ==> Total time: 1 day, 18:38:25 Eta: 23 days, 23:20:09 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 12:35:48,880 - Train: 6.90% [341000/4942000] [69.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 12:36:21,598 - Train: 6.90% [341100/4942000] [69.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 12:36:54,317 - Train: 6.90% [341200/4942000] [69.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 12:37:27,097 - Train: 6.91% [341300/4942000] [69.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 12:37:59,896 - Train: 6.91% [341400/4942000] [69.1/1000.0] [batch_t 0.335 (0.328)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-04 12:38:32,753 - Train: 6.91% [341500/4942000] [69.1/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 12:39:05,585 - Train: 6.91% [341600/4942000] [69.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-04 12:39:38,554 - Train: 6.91% [341700/4942000] [69.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 12:40:11,356 - Train: 6.92% [341800/4942000] [69.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 12:40:44,189 - Train: 6.92% [341900/4942000] [69.2/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:41:16,990 - Train: 6.92% [342000/4942000] [69.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 12:41:49,808 - Train: 6.92% [342100/4942000] [69.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 12:42:22,582 - Train: 6.92% [342200/4942000] [69.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 12:42:55,403 - Train: 6.93% [342300/4942000] [69.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 12:43:28,244 - Train: 6.93% [342400/4942000] [69.3/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 12:44:01,024 - Train: 6.93% [342500/4942000] [69.3/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 12:44:33,848 - Train: 6.93% [342600/4942000] [69.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 12:45:06,962 - Train: 6.93% [342700/4942000] [69.3/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 12:45:39,736 - Train: 6.94% [342800/4942000] [69.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 12:46:12,566 - Train: 6.94% [342900/4942000] [69.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 12:46:45,397 - Train: 6.94% [343000/4942000] [69.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 12:47:18,639 - Train: 6.94% [343100/4942000] [69.4/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 12:47:51,491 - Train: 6.94% [343200/4942000] [69.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 12:48:24,301 - Train: 6.95% [343300/4942000] [69.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 12:48:57,062 - Train: 6.95% [343400/4942000] [69.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 12:49:29,827 - Train: 6.95% [343500/4942000] [69.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 12:50:02,610 - Train: 6.95% [343600/4942000] [69.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 12:50:35,337 - Train: 6.95% [343700/4942000] [69.5/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 12:51:08,099 - Train: 6.96% [343800/4942000] [69.6/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:51:40,846 - Train: 6.96% [343900/4942000] [69.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 12:52:13,621 - Train: 6.96% [344000/4942000] [69.6/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 12:52:46,437 - Train: 6.96% [344100/4942000] [69.6/1000.0] [batch_t 0.325 (0.328)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-04 12:53:19,211 - Train: 6.96% [344200/4942000] [69.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 12:53:52,016 - Train: 6.97% [344300/4942000] [69.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 12:54:24,809 - Train: 6.97% [344400/4942000] [69.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 12:54:57,556 - Train: 6.97% [344500/4942000] [69.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 12:55:30,417 - Train: 6.97% [344600/4942000] [69.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 12:56:03,203 - Train: 6.97% [344700/4942000] [69.7/1000.0] [batch_t 0.321 (0.328)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-04 12:56:35,950 - Train: 6.98% [344800/4942000] [69.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 12:57:08,709 - Train: 6.98% [344900/4942000] [69.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 12:57:41,444 - Train: 6.98% [345000/4942000] [69.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 12:58:14,471 - Train: 6.98% [345100/4942000] [69.8/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 12:58:47,221 - Train: 6.99% [345200/4942000] [69.9/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 12:59:19,998 - Train: 6.99% [345300/4942000] [69.9/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 12:59:52,755 - Train: 6.99% [345400/4942000] [69.9/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 13:00:25,533 - Train: 6.99% [345500/4942000] [69.9/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 13:00:58,292 - Train: 6.99% [345600/4942000] [69.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:01:31,057 - Train: 7.00% [345700/4942000] [70.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:02:03,850 - Train: 7.00% [345800/4942000] [70.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:02:36,676 - Train: 7.00% [345900/4942000] [70.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 13:02:49,785 - ==> Total time: 1 day, 19:05:28 Eta: 23 days, 20:29:59 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 13:03:11,649 - Train: 7.00% [346000/4942000] [70.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:03:44,430 - Train: 7.00% [346100/4942000] [70.0/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 13:04:17,166 - Train: 7.01% [346200/4942000] [70.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:04:49,930 - Train: 7.01% [346300/4942000] [70.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:05:22,714 - Train: 7.01% [346400/4942000] [70.1/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-04 13:05:55,471 - Train: 7.01% [346500/4942000] [70.1/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 13:06:28,251 - Train: 7.01% [346600/4942000] [70.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:07:00,986 - Train: 7.02% [346700/4942000] [70.2/1000.0] [batch_t 0.317 (0.327)] [data_t 0.002] [optim_t 0.315] [lr 0.005000] 2024-04-04 13:07:33,722 - Train: 7.02% [346800/4942000] [70.2/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-04 13:08:06,488 - Train: 7.02% [346900/4942000] [70.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 13:08:39,270 - Train: 7.02% [347000/4942000] [70.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:09:12,060 - Train: 7.02% [347100/4942000] [70.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 13:09:44,882 - Train: 7.03% [347200/4942000] [70.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:10:17,633 - Train: 7.03% [347300/4942000] [70.3/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 13:10:50,552 - Train: 7.03% [347400/4942000] [70.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 13:11:23,294 - Train: 7.03% [347500/4942000] [70.3/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 13:11:56,092 - Train: 7.03% [347600/4942000] [70.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:12:29,865 - Train: 7.04% [347700/4942000] [70.4/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:13:02,653 - Train: 7.04% [347800/4942000] [70.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 13:13:35,444 - Train: 7.04% [347900/4942000] [70.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:14:08,935 - Train: 7.04% [348000/4942000] [70.4/1000.0] [batch_t 0.324 (0.335)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 13:14:41,753 - Train: 7.04% [348100/4942000] [70.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 13:15:15,358 - Train: 7.05% [348200/4942000] [70.5/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:15:48,164 - Train: 7.05% [348300/4942000] [70.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 13:16:20,968 - Train: 7.05% [348400/4942000] [70.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 13:16:53,746 - Train: 7.05% [348500/4942000] [70.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:17:26,516 - Train: 7.05% [348600/4942000] [70.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 13:17:59,266 - Train: 7.06% [348700/4942000] [70.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 13:18:32,132 - Train: 7.06% [348800/4942000] [70.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:19:04,880 - Train: 7.06% [348900/4942000] [70.6/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 13:19:37,704 - Train: 7.06% [349000/4942000] [70.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:20:10,463 - Train: 7.06% [349100/4942000] [70.6/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 13:20:43,272 - Train: 7.07% [349200/4942000] [70.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:21:16,097 - Train: 7.07% [349300/4942000] [70.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 13:21:48,915 - Train: 7.07% [349400/4942000] [70.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 13:22:21,722 - Train: 7.07% [349500/4942000] [70.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 13:22:54,509 - Train: 7.07% [349600/4942000] [70.7/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 13:23:27,274 - Train: 7.08% [349700/4942000] [70.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 13:24:00,043 - Train: 7.08% [349800/4942000] [70.8/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 13:24:32,819 - Train: 7.08% [349900/4942000] [70.8/1000.0] [batch_t 0.321 (0.328)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-04 13:25:05,551 - Train: 7.08% [350000/4942000] [70.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 13:25:38,281 - Train: 7.08% [350100/4942000] [70.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 13:26:11,061 - Train: 7.09% [350200/4942000] [70.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:26:43,942 - Train: 7.09% [350300/4942000] [70.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:27:17,843 - Train: 7.09% [350400/4942000] [70.9/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 13:27:50,612 - Train: 7.09% [350500/4942000] [70.9/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 13:28:23,418 - Train: 7.09% [350600/4942000] [70.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:28:56,233 - Train: 7.10% [350700/4942000] [71.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:29:29,009 - Train: 7.10% [350800/4942000] [71.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 13:29:55,913 - ==> Total time: 1 day, 19:32:35 Eta: 23 days, 17:44:23 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 13:30:03,850 - Train: 7.10% [350900/4942000] [71.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 13:30:36,654 - Train: 7.10% [351000/4942000] [71.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:31:09,459 - Train: 7.10% [351100/4942000] [71.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 13:31:42,242 - Train: 7.11% [351200/4942000] [71.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 13:32:15,579 - Train: 7.11% [351300/4942000] [71.1/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:32:48,394 - Train: 7.11% [351400/4942000] [71.1/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 13:33:21,187 - Train: 7.11% [351500/4942000] [71.1/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 13:33:54,060 - Train: 7.11% [351600/4942000] [71.1/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 13:34:26,827 - Train: 7.12% [351700/4942000] [71.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:34:59,585 - Train: 7.12% [351800/4942000] [71.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:35:32,295 - Train: 7.12% [351900/4942000] [71.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:36:05,000 - Train: 7.12% [352000/4942000] [71.2/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 13:36:37,736 - Train: 7.12% [352100/4942000] [71.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:37:10,523 - Train: 7.13% [352200/4942000] [71.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:37:43,341 - Train: 7.13% [352300/4942000] [71.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 13:38:16,107 - Train: 7.13% [352400/4942000] [71.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:38:48,883 - Train: 7.13% [352500/4942000] [71.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:39:21,649 - Train: 7.13% [352600/4942000] [71.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 13:39:54,422 - Train: 7.14% [352700/4942000] [71.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 13:40:27,226 - Train: 7.14% [352800/4942000] [71.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:40:59,962 - Train: 7.14% [352900/4942000] [71.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:41:32,901 - Train: 7.14% [353000/4942000] [71.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:42:05,686 - Train: 7.14% [353100/4942000] [71.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:42:38,469 - Train: 7.15% [353200/4942000] [71.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:43:11,248 - Train: 7.15% [353300/4942000] [71.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 13:43:43,982 - Train: 7.15% [353400/4942000] [71.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 13:44:16,756 - Train: 7.15% [353500/4942000] [71.5/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 13:44:49,553 - Train: 7.15% [353600/4942000] [71.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:45:22,354 - Train: 7.16% [353700/4942000] [71.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:45:55,150 - Train: 7.16% [353800/4942000] [71.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 13:46:27,947 - Train: 7.16% [353900/4942000] [71.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 13:47:00,771 - Train: 7.16% [354000/4942000] [71.6/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 13:47:33,548 - Train: 7.17% [354100/4942000] [71.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:48:06,341 - Train: 7.17% [354200/4942000] [71.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 13:48:39,094 - Train: 7.17% [354300/4942000] [71.7/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 13:49:11,872 - Train: 7.17% [354400/4942000] [71.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 13:49:44,751 - Train: 7.17% [354500/4942000] [71.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:50:18,991 - Train: 7.18% [354600/4942000] [71.8/1000.0] [batch_t 0.330 (0.342)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:50:53,763 - Train: 7.18% [354700/4942000] [71.8/1000.0] [batch_t 0.328 (0.348)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:51:26,545 - Train: 7.18% [354800/4942000] [71.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 13:51:59,301 - Train: 7.18% [354900/4942000] [71.8/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:52:32,010 - Train: 7.18% [355000/4942000] [71.8/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 13:53:04,726 - Train: 7.19% [355100/4942000] [71.9/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 13:53:37,459 - Train: 7.19% [355200/4942000] [71.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 13:54:10,230 - Train: 7.19% [355300/4942000] [71.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 13:54:42,949 - Train: 7.19% [355400/4942000] [71.9/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:55:15,705 - Train: 7.19% [355500/4942000] [71.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:55:48,446 - Train: 7.20% [355600/4942000] [72.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:56:21,189 - Train: 7.20% [355700/4942000] [72.0/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 13:56:53,931 - Train: 7.20% [355800/4942000] [72.0/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 13:57:01,809 - ==> Total time: 1 day, 19:59:40 Eta: 23 days, 15:02:35 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 13:57:30,434 - Train: 7.20% [355900/4942000] [72.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 13:58:03,202 - Train: 7.20% [356000/4942000] [72.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:58:35,977 - Train: 7.21% [356100/4942000] [72.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:59:11,026 - Train: 7.21% [356200/4942000] [72.1/1000.0] [batch_t 0.329 (0.350)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:59:43,832 - Train: 7.21% [356300/4942000] [72.1/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 14:00:19,410 - Train: 7.21% [356400/4942000] [72.1/1000.0] [batch_t 0.329 (0.356)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 14:00:52,128 - Train: 7.21% [356500/4942000] [72.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 14:01:24,936 - Train: 7.22% [356600/4942000] [72.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 14:01:57,623 - Train: 7.22% [356700/4942000] [72.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 14:02:30,345 - Train: 7.22% [356800/4942000] [72.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 14:03:03,088 - Train: 7.22% [356900/4942000] [72.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:03:35,818 - Train: 7.22% [357000/4942000] [72.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 14:04:08,556 - Train: 7.23% [357100/4942000] [72.3/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 14:04:41,290 - Train: 7.23% [357200/4942000] [72.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 14:05:14,176 - Train: 7.23% [357300/4942000] [72.3/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 14:05:46,919 - Train: 7.23% [357400/4942000] [72.3/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 14:06:19,891 - Train: 7.23% [357500/4942000] [72.3/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 14:06:52,631 - Train: 7.24% [357600/4942000] [72.4/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 14:07:26,748 - Train: 7.24% [357700/4942000] [72.4/1000.0] [batch_t 0.326 (0.341)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 14:07:59,470 - Train: 7.24% [357800/4942000] [72.4/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 14:08:32,812 - Train: 7.24% [357900/4942000] [72.4/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 14:09:05,544 - Train: 7.24% [358000/4942000] [72.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:09:38,355 - Train: 7.25% [358100/4942000] [72.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:10:12,068 - Train: 7.25% [358200/4942000] [72.5/1000.0] [batch_t 0.325 (0.337)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 14:10:44,847 - Train: 7.25% [358300/4942000] [72.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 14:11:19,320 - Train: 7.25% [358400/4942000] [72.5/1000.0] [batch_t 0.328 (0.345)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:11:52,092 - Train: 7.25% [358500/4942000] [72.5/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 14:12:25,863 - Train: 7.26% [358600/4942000] [72.6/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:12:58,631 - Train: 7.26% [358700/4942000] [72.6/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 14:13:32,096 - Train: 7.26% [358800/4942000] [72.6/1000.0] [batch_t 0.325 (0.335)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 14:14:05,629 - Train: 7.26% [358900/4942000] [72.6/1000.0] [batch_t 1.102 (0.335)] [data_t 0.777] [optim_t 0.324] [lr 0.005000] 2024-04-04 14:14:38,402 - Train: 7.26% [359000/4942000] [72.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 14:15:12,625 - Train: 7.27% [359100/4942000] [72.7/1000.0] [batch_t 0.326 (0.342)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 14:15:45,433 - Train: 7.27% [359200/4942000] [72.7/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 14:16:19,307 - Train: 7.27% [359300/4942000] [72.7/1000.0] [batch_t 0.324 (0.339)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 14:16:52,020 - Train: 7.27% [359400/4942000] [72.7/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 14:17:25,350 - Train: 7.27% [359500/4942000] [72.7/1000.0] [batch_t 0.323 (0.333)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 14:17:58,101 - Train: 7.28% [359600/4942000] [72.8/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 14:18:31,341 - Train: 7.28% [359700/4942000] [72.8/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 14:19:04,070 - Train: 7.28% [359800/4942000] [72.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 14:19:36,823 - Train: 7.28% [359900/4942000] [72.8/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 14:20:10,600 - Train: 7.28% [360000/4942000] [72.8/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 14:20:43,325 - Train: 7.29% [360100/4942000] [72.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 14:21:16,238 - Train: 7.29% [360200/4942000] [72.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 14:21:48,989 - Train: 7.29% [360300/4942000] [72.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 14:22:21,757 - Train: 7.29% [360400/4942000] [72.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 14:22:54,490 - Train: 7.29% [360500/4942000] [72.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:23:27,237 - Train: 7.30% [360600/4942000] [73.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 14:23:59,982 - Train: 7.30% [360700/4942000] [73.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:24:22,356 - ==> Total time: 1 day, 20:27:01 Eta: 23 days, 12:27:34 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 14:24:35,206 - Train: 7.30% [360800/4942000] [73.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 14:25:07,957 - Train: 7.30% [360900/4942000] [73.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-04 14:25:40,753 - Train: 7.30% [361000/4942000] [73.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-04 14:26:13,533 - Train: 7.31% [361100/4942000] [73.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 14:26:46,312 - Train: 7.31% [361200/4942000] [73.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 14:27:19,124 - Train: 7.31% [361300/4942000] [73.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:27:51,956 - Train: 7.31% [361400/4942000] [73.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 14:28:24,940 - Train: 7.31% [361500/4942000] [73.1/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 14:28:57,711 - Train: 7.32% [361600/4942000] [73.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 14:29:30,520 - Train: 7.32% [361700/4942000] [73.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 14:30:03,346 - Train: 7.32% [361800/4942000] [73.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 14:30:36,177 - Train: 7.32% [361900/4942000] [73.2/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 14:31:08,952 - Train: 7.32% [362000/4942000] [73.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:31:41,711 - Train: 7.33% [362100/4942000] [73.3/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 14:32:14,525 - Train: 7.33% [362200/4942000] [73.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 14:32:47,299 - Train: 7.33% [362300/4942000] [73.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 14:33:20,151 - Train: 7.33% [362400/4942000] [73.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 14:33:52,983 - Train: 7.34% [362500/4942000] [73.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 14:34:25,810 - Train: 7.34% [362600/4942000] [73.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 14:34:58,540 - Train: 7.34% [362700/4942000] [73.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:35:31,333 - Train: 7.34% [362800/4942000] [73.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 14:36:04,131 - Train: 7.34% [362900/4942000] [73.4/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 14:36:37,048 - Train: 7.35% [363000/4942000] [73.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 14:37:09,856 - Train: 7.35% [363100/4942000] [73.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:37:42,713 - Train: 7.35% [363200/4942000] [73.5/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 14:38:15,527 - Train: 7.35% [363300/4942000] [73.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 14:38:48,299 - Train: 7.35% [363400/4942000] [73.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:39:21,075 - Train: 7.36% [363500/4942000] [73.6/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-04 14:39:53,899 - Train: 7.36% [363600/4942000] [73.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 14:40:26,855 - Train: 7.36% [363700/4942000] [73.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 14:40:59,599 - Train: 7.36% [363800/4942000] [73.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:41:32,424 - Train: 7.36% [363900/4942000] [73.6/1000.0] [batch_t 0.337 (0.328)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-04 14:42:05,226 - Train: 7.37% [364000/4942000] [73.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:42:37,967 - Train: 7.37% [364100/4942000] [73.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 14:43:10,779 - Train: 7.37% [364200/4942000] [73.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:43:43,585 - Train: 7.37% [364300/4942000] [73.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:44:16,456 - Train: 7.37% [364400/4942000] [73.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 14:44:49,282 - Train: 7.38% [364500/4942000] [73.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:45:22,047 - Train: 7.38% [364600/4942000] [73.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 14:45:54,842 - Train: 7.38% [364700/4942000] [73.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:46:27,600 - Train: 7.38% [364800/4942000] [73.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 14:47:00,408 - Train: 7.38% [364900/4942000] [73.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 14:47:33,196 - Train: 7.39% [365000/4942000] [73.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 14:48:05,972 - Train: 7.39% [365100/4942000] [73.9/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 14:48:38,766 - Train: 7.39% [365200/4942000] [73.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 14:49:11,607 - Train: 7.39% [365300/4942000] [73.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 14:49:44,369 - Train: 7.39% [365400/4942000] [73.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 14:50:17,897 - Train: 7.40% [365500/4942000] [74.0/1000.0] [batch_t 0.323 (0.335)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 14:50:50,708 - Train: 7.40% [365600/4942000] [74.0/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 14:51:23,436 - Train: 7.40% [365700/4942000] [74.0/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 14:51:26,069 - ==> Total time: 1 day, 20:54:05 Eta: 23 days, 9:52:30 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 14:51:58,453 - Train: 7.40% [365800/4942000] [74.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 14:52:31,205 - Train: 7.40% [365900/4942000] [74.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 14:53:03,951 - Train: 7.41% [366000/4942000] [74.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-04 14:53:36,768 - Train: 7.41% [366100/4942000] [74.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:54:09,571 - Train: 7.41% [366200/4942000] [74.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 14:54:42,398 - Train: 7.41% [366300/4942000] [74.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 14:55:15,342 - Train: 7.41% [366400/4942000] [74.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:55:48,161 - Train: 7.42% [366500/4942000] [74.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 14:56:22,837 - Train: 7.42% [366600/4942000] [74.2/1000.0] [batch_t 0.328 (0.347)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 14:56:55,644 - Train: 7.42% [366700/4942000] [74.2/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 14:57:28,432 - Train: 7.42% [366800/4942000] [74.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-04 14:58:01,259 - Train: 7.42% [366900/4942000] [74.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-04 14:58:34,136 - Train: 7.43% [367000/4942000] [74.3/1000.0] [batch_t 0.324 (0.329)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-04 14:59:06,963 - Train: 7.43% [367100/4942000] [74.3/1000.0] [batch_t 0.324 (0.328)] [data_t 0.003] [optim_t 0.321] [lr 0.005000] 2024-04-04 14:59:39,969 - Train: 7.43% [367200/4942000] [74.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:00:12,821 - Train: 7.43% [367300/4942000] [74.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-04 15:00:45,698 - Train: 7.43% [367400/4942000] [74.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 15:01:18,601 - Train: 7.44% [367500/4942000] [74.4/1000.0] [batch_t 0.325 (0.329)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-04 15:01:51,429 - Train: 7.44% [367600/4942000] [74.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-04 15:02:24,313 - Train: 7.44% [367700/4942000] [74.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:02:57,156 - Train: 7.44% [367800/4942000] [74.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:03:30,019 - Train: 7.44% [367900/4942000] [74.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-04 15:04:02,948 - Train: 7.45% [368000/4942000] [74.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:04:35,737 - Train: 7.45% [368100/4942000] [74.5/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 15:05:08,576 - Train: 7.45% [368200/4942000] [74.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-04 15:05:41,431 - Train: 7.45% [368300/4942000] [74.5/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 15:06:14,287 - Train: 7.45% [368400/4942000] [74.5/1000.0] [batch_t 0.324 (0.328)] [data_t 0.003] [optim_t 0.321] [lr 0.005000] 2024-04-04 15:06:47,138 - Train: 7.46% [368500/4942000] [74.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 15:07:20,168 - Train: 7.46% [368600/4942000] [74.6/1000.0] [batch_t 0.334 (0.330)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-04 15:07:53,031 - Train: 7.46% [368700/4942000] [74.6/1000.0] [batch_t 0.326 (0.329)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-04 15:08:25,792 - Train: 7.46% [368800/4942000] [74.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 15:08:58,600 - Train: 7.46% [368900/4942000] [74.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:09:31,457 - Train: 7.47% [369000/4942000] [74.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:10:04,314 - Train: 7.47% [369100/4942000] [74.7/1000.0] [batch_t 0.324 (0.328)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-04 15:10:37,437 - Train: 7.47% [369200/4942000] [74.7/1000.0] [batch_t 0.327 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 15:11:10,296 - Train: 7.47% [369300/4942000] [74.7/1000.0] [batch_t 0.326 (0.329)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-04 15:11:43,135 - Train: 7.47% [369400/4942000] [74.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:12:16,003 - Train: 7.48% [369500/4942000] [74.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:12:48,850 - Train: 7.48% [369600/4942000] [74.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-04 15:13:21,661 - Train: 7.48% [369700/4942000] [74.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:13:54,483 - Train: 7.48% [369800/4942000] [74.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 15:14:27,299 - Train: 7.48% [369900/4942000] [74.8/1000.0] [batch_t 0.324 (0.328)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-04 15:15:00,146 - Train: 7.49% [370000/4942000] [74.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-04 15:15:33,083 - Train: 7.49% [370100/4942000] [74.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 15:16:05,926 - Train: 7.49% [370200/4942000] [74.9/1000.0] [batch_t 0.331 (0.328)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-04 15:16:38,734 - Train: 7.49% [370300/4942000] [74.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-04 15:17:11,553 - Train: 7.49% [370400/4942000] [74.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 15:17:44,442 - Train: 7.50% [370500/4942000] [75.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 15:18:17,187 - Train: 7.50% [370600/4942000] [75.0/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 15:18:33,580 - ==> Total time: 1 day, 21:21:12 Eta: 23 days, 7:21:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 15:18:51,786 - Train: 7.50% [370700/4942000] [75.0/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 15:19:24,479 - Train: 7.50% [370800/4942000] [75.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 15:19:57,211 - Train: 7.51% [370900/4942000] [75.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 15:20:29,940 - Train: 7.51% [371000/4942000] [75.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 15:21:02,686 - Train: 7.51% [371100/4942000] [75.1/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 15:21:35,428 - Train: 7.51% [371200/4942000] [75.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 15:22:08,182 - Train: 7.51% [371300/4942000] [75.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:22:40,997 - Train: 7.52% [371400/4942000] [75.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 15:23:13,734 - Train: 7.52% [371500/4942000] [75.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 15:23:46,476 - Train: 7.52% [371600/4942000] [75.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 15:24:19,239 - Train: 7.52% [371700/4942000] [75.2/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 15:24:51,925 - Train: 7.52% [371800/4942000] [75.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 15:25:24,690 - Train: 7.53% [371900/4942000] [75.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 15:25:57,420 - Train: 7.53% [372000/4942000] [75.3/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 15:26:30,176 - Train: 7.53% [372100/4942000] [75.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:27:02,898 - Train: 7.53% [372200/4942000] [75.3/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 15:27:35,744 - Train: 7.53% [372300/4942000] [75.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-04 15:28:08,551 - Train: 7.54% [372400/4942000] [75.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 15:28:41,375 - Train: 7.54% [372500/4942000] [75.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 15:29:14,212 - Train: 7.54% [372600/4942000] [75.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 15:29:47,007 - Train: 7.54% [372700/4942000] [75.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 15:30:19,962 - Train: 7.54% [372800/4942000] [75.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 15:30:52,781 - Train: 7.55% [372900/4942000] [75.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 15:31:25,555 - Train: 7.55% [373000/4942000] [75.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 15:31:58,351 - Train: 7.55% [373100/4942000] [75.5/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 15:32:31,146 - Train: 7.55% [373200/4942000] [75.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 15:33:03,985 - Train: 7.55% [373300/4942000] [75.5/1000.0] [batch_t 0.320 (0.328)] [data_t 0.002] [optim_t 0.317] [lr 0.005000] 2024-04-04 15:33:36,804 - Train: 7.56% [373400/4942000] [75.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 15:34:09,561 - Train: 7.56% [373500/4942000] [75.6/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 15:34:42,355 - Train: 7.56% [373600/4942000] [75.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 15:35:15,117 - Train: 7.56% [373700/4942000] [75.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 15:35:47,926 - Train: 7.56% [373800/4942000] [75.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:36:20,757 - Train: 7.57% [373900/4942000] [75.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 15:36:53,487 - Train: 7.57% [374000/4942000] [75.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:37:26,258 - Train: 7.57% [374100/4942000] [75.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 15:37:59,124 - Train: 7.57% [374200/4942000] [75.7/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 15:38:31,846 - Train: 7.57% [374300/4942000] [75.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 15:39:04,550 - Train: 7.58% [374400/4942000] [75.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 15:39:37,339 - Train: 7.58% [374500/4942000] [75.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 15:40:10,093 - Train: 7.58% [374600/4942000] [75.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 15:40:42,875 - Train: 7.58% [374700/4942000] [75.8/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:41:15,631 - Train: 7.58% [374800/4942000] [75.8/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 15:41:48,443 - Train: 7.59% [374900/4942000] [75.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:42:21,163 - Train: 7.59% [375000/4942000] [75.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 15:42:53,945 - Train: 7.59% [375100/4942000] [75.9/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 15:43:26,675 - Train: 7.59% [375200/4942000] [75.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 15:43:59,459 - Train: 7.59% [375300/4942000] [75.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 15:44:32,248 - Train: 7.60% [375400/4942000] [76.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 15:45:05,013 - Train: 7.60% [375500/4942000] [76.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 15:45:35,233 - ==> Total time: 1 day, 21:48:14 Eta: 23 days, 4:52:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 15:45:39,947 - Train: 7.60% [375600/4942000] [76.0/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 15:46:12,648 - Train: 7.60% [375700/4942000] [76.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:46:45,400 - Train: 7.60% [375800/4942000] [76.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 15:47:18,194 - Train: 7.61% [375900/4942000] [76.1/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 15:47:50,951 - Train: 7.61% [376000/4942000] [76.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:48:23,756 - Train: 7.61% [376100/4942000] [76.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 15:48:56,527 - Train: 7.61% [376200/4942000] [76.1/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 15:49:29,306 - Train: 7.61% [376300/4942000] [76.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-04 15:50:02,054 - Train: 7.62% [376400/4942000] [76.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 15:50:34,832 - Train: 7.62% [376500/4942000] [76.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 15:51:07,628 - Train: 7.62% [376600/4942000] [76.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 15:51:40,395 - Train: 7.62% [376700/4942000] [76.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:52:13,135 - Train: 7.62% [376800/4942000] [76.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 15:52:46,036 - Train: 7.63% [376900/4942000] [76.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:53:18,800 - Train: 7.63% [377000/4942000] [76.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 15:53:51,579 - Train: 7.63% [377100/4942000] [76.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 15:54:24,329 - Train: 7.63% [377200/4942000] [76.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:54:57,076 - Train: 7.63% [377300/4942000] [76.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 15:55:30,524 - Train: 7.64% [377400/4942000] [76.4/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 15:56:03,251 - Train: 7.64% [377500/4942000] [76.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 15:56:36,094 - Train: 7.64% [377600/4942000] [76.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 15:57:08,853 - Train: 7.64% [377700/4942000] [76.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 15:57:41,588 - Train: 7.64% [377800/4942000] [76.4/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 15:58:14,485 - Train: 7.65% [377900/4942000] [76.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 15:58:47,240 - Train: 7.65% [378000/4942000] [76.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 15:59:20,008 - Train: 7.65% [378100/4942000] [76.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 15:59:52,780 - Train: 7.65% [378200/4942000] [76.5/1000.0] [batch_t 0.321 (0.328)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-04 16:00:25,558 - Train: 7.65% [378300/4942000] [76.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 16:00:58,548 - Train: 7.66% [378400/4942000] [76.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 16:01:31,351 - Train: 7.66% [378500/4942000] [76.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 16:02:04,119 - Train: 7.66% [378600/4942000] [76.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 16:02:36,924 - Train: 7.66% [378700/4942000] [76.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-04 16:03:09,700 - Train: 7.66% [378800/4942000] [76.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 16:03:42,440 - Train: 7.67% [378900/4942000] [76.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 16:04:15,148 - Train: 7.67% [379000/4942000] [76.7/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 16:04:47,878 - Train: 7.67% [379100/4942000] [76.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 16:05:20,628 - Train: 7.67% [379200/4942000] [76.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 16:05:53,355 - Train: 7.68% [379300/4942000] [76.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 16:06:26,089 - Train: 7.68% [379400/4942000] [76.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 16:06:58,846 - Train: 7.68% [379500/4942000] [76.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 16:07:31,578 - Train: 7.68% [379600/4942000] [76.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 16:08:04,300 - Train: 7.68% [379700/4942000] [76.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 16:08:38,500 - Train: 7.69% [379800/4942000] [76.9/1000.0] [batch_t 0.325 (0.342)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 16:09:11,225 - Train: 7.69% [379900/4942000] [76.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 16:09:43,999 - Train: 7.69% [380000/4942000] [76.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:10:16,754 - Train: 7.69% [380100/4942000] [76.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 16:10:49,526 - Train: 7.69% [380200/4942000] [76.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 16:11:22,298 - Train: 7.70% [380300/4942000] [77.0/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 16:11:55,018 - Train: 7.70% [380400/4942000] [77.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 16:12:27,799 - Train: 7.70% [380500/4942000] [77.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 16:12:38,938 - ==> Total time: 1 day, 22:15:18 Eta: 23 days, 2:27:34 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 16:13:02,620 - Train: 7.70% [380600/4942000] [77.0/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 16:13:35,372 - Train: 7.70% [380700/4942000] [77.0/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:14:08,976 - Train: 7.71% [380800/4942000] [77.1/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 16:14:41,752 - Train: 7.71% [380900/4942000] [77.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-04 16:15:14,995 - Train: 7.71% [381000/4942000] [77.1/1000.0] [batch_t 0.326 (0.332)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 16:15:47,976 - Train: 7.71% [381100/4942000] [77.1/1000.0] [batch_t 0.323 (0.330)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 16:16:20,771 - Train: 7.71% [381200/4942000] [77.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 16:16:54,490 - Train: 7.72% [381300/4942000] [77.2/1000.0] [batch_t 0.325 (0.337)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 16:17:39,133 - Train: 7.72% [381400/4942000] [77.2/1000.0] [batch_t 0.326 (0.446)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 16:18:12,319 - Train: 7.72% [381500/4942000] [77.2/1000.0] [batch_t 0.332 (0.332)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-04 16:18:45,110 - Train: 7.72% [381600/4942000] [77.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 16:19:17,890 - Train: 7.72% [381700/4942000] [77.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 16:19:50,655 - Train: 7.73% [381800/4942000] [77.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 16:20:23,473 - Train: 7.73% [381900/4942000] [77.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 16:20:56,221 - Train: 7.73% [382000/4942000] [77.3/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 16:21:28,936 - Train: 7.73% [382100/4942000] [77.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 16:22:01,672 - Train: 7.73% [382200/4942000] [77.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 16:22:34,388 - Train: 7.74% [382300/4942000] [77.4/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 16:23:07,875 - Train: 7.74% [382400/4942000] [77.4/1000.0] [batch_t 0.328 (0.335)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 16:23:40,759 - Train: 7.74% [382500/4942000] [77.4/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 16:24:14,085 - Train: 7.74% [382600/4942000] [77.4/1000.0] [batch_t 0.324 (0.333)] [data_t 0.003] [optim_t 0.321] [lr 0.005000] 2024-04-04 16:24:46,871 - Train: 7.74% [382700/4942000] [77.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 16:25:20,705 - Train: 7.75% [382800/4942000] [77.5/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 16:25:53,456 - Train: 7.75% [382900/4942000] [77.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 16:26:26,296 - Train: 7.75% [383000/4942000] [77.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 16:26:59,053 - Train: 7.75% [383100/4942000] [77.5/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 16:27:33,782 - Train: 7.75% [383200/4942000] [77.5/1000.0] [batch_t 0.330 (0.347)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:28:07,302 - Train: 7.76% [383300/4942000] [77.6/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 16:28:40,114 - Train: 7.76% [383400/4942000] [77.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 16:29:13,800 - Train: 7.76% [383500/4942000] [77.6/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 16:29:50,857 - Train: 7.76% [383600/4942000] [77.6/1000.0] [batch_t 0.325 (0.370)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 16:30:23,690 - Train: 7.76% [383700/4942000] [77.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 16:30:56,458 - Train: 7.77% [383800/4942000] [77.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 16:31:29,247 - Train: 7.77% [383900/4942000] [77.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 16:32:02,142 - Train: 7.77% [384000/4942000] [77.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 16:32:35,550 - Train: 7.77% [384100/4942000] [77.7/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 16:33:08,297 - Train: 7.77% [384200/4942000] [77.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 16:33:41,054 - Train: 7.78% [384300/4942000] [77.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 16:34:13,800 - Train: 7.78% [384400/4942000] [77.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 16:34:46,599 - Train: 7.78% [384500/4942000] [77.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:35:19,363 - Train: 7.78% [384600/4942000] [77.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-04 16:35:52,159 - Train: 7.78% [384700/4942000] [77.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-04 16:36:24,934 - Train: 7.79% [384800/4942000] [77.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 16:36:57,706 - Train: 7.79% [384900/4942000] [77.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 16:37:30,472 - Train: 7.79% [385000/4942000] [77.9/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:38:03,236 - Train: 7.79% [385100/4942000] [77.9/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 16:38:36,085 - Train: 7.79% [385200/4942000] [77.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 16:39:08,796 - Train: 7.80% [385300/4942000] [78.0/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:39:41,683 - Train: 7.80% [385400/4942000] [78.0/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 16:40:06,558 - ==> Total time: 1 day, 22:42:45 Eta: 23 days, 0:10:05 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 16:40:16,346 - Train: 7.80% [385500/4942000] [78.0/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 16:40:49,122 - Train: 7.80% [385600/4942000] [78.0/1000.0] [batch_t 0.337 (0.328)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-04 16:41:21,856 - Train: 7.80% [385700/4942000] [78.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 16:41:54,637 - Train: 7.81% [385800/4942000] [78.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 16:42:27,449 - Train: 7.81% [385900/4942000] [78.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 16:43:00,206 - Train: 7.81% [386000/4942000] [78.1/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 16:43:33,007 - Train: 7.81% [386100/4942000] [78.1/1000.0] [batch_t 0.333 (0.328)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-04 16:44:05,832 - Train: 7.81% [386200/4942000] [78.1/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 16:44:38,650 - Train: 7.82% [386300/4942000] [78.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 16:45:11,508 - Train: 7.82% [386400/4942000] [78.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 16:45:44,397 - Train: 7.82% [386500/4942000] [78.2/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 16:46:17,318 - Train: 7.82% [386600/4942000] [78.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 16:46:50,187 - Train: 7.82% [386700/4942000] [78.2/1000.0] [batch_t 0.337 (0.329)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-04 16:47:23,247 - Train: 7.83% [386800/4942000] [78.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 16:47:56,029 - Train: 7.83% [386900/4942000] [78.3/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 16:48:28,865 - Train: 7.83% [387000/4942000] [78.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 16:49:01,724 - Train: 7.83% [387100/4942000] [78.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 16:49:34,575 - Train: 7.83% [387200/4942000] [78.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 16:50:07,486 - Train: 7.84% [387300/4942000] [78.4/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 16:50:40,427 - Train: 7.84% [387400/4942000] [78.4/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 16:51:13,266 - Train: 7.84% [387500/4942000] [78.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:51:46,150 - Train: 7.84% [387600/4942000] [78.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 16:52:19,010 - Train: 7.85% [387700/4942000] [78.5/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 16:52:51,849 - Train: 7.85% [387800/4942000] [78.5/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 16:53:25,601 - Train: 7.85% [387900/4942000] [78.5/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 16:53:58,423 - Train: 7.85% [388000/4942000] [78.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 16:54:31,299 - Train: 7.85% [388100/4942000] [78.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-04 16:55:04,288 - Train: 7.86% [388200/4942000] [78.6/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 16:55:37,152 - Train: 7.86% [388300/4942000] [78.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 16:56:11,167 - Train: 7.86% [388400/4942000] [78.6/1000.0] [batch_t 0.331 (0.340)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 16:56:43,962 - Train: 7.86% [388500/4942000] [78.6/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 16:57:16,804 - Train: 7.86% [388600/4942000] [78.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-04 16:57:49,669 - Train: 7.87% [388700/4942000] [78.7/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 16:58:22,482 - Train: 7.87% [388800/4942000] [78.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 16:58:55,315 - Train: 7.87% [388900/4942000] [78.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 16:59:28,813 - Train: 7.87% [389000/4942000] [78.7/1000.0] [batch_t 0.326 (0.335)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 17:00:01,630 - Train: 7.87% [389100/4942000] [78.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 17:00:34,519 - Train: 7.88% [389200/4942000] [78.8/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 17:01:08,169 - Train: 7.88% [389300/4942000] [78.8/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 17:01:40,983 - Train: 7.88% [389400/4942000] [78.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 17:02:13,863 - Train: 7.88% [389500/4942000] [78.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 17:02:46,834 - Train: 7.88% [389600/4942000] [78.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-04 17:03:20,293 - Train: 7.89% [389700/4942000] [78.9/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 17:03:53,096 - Train: 7.89% [389800/4942000] [78.9/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 17:04:27,031 - Train: 7.89% [389900/4942000] [78.9/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 17:04:59,846 - Train: 7.89% [390000/4942000] [78.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 17:05:35,120 - Train: 7.89% [390100/4942000] [78.9/1000.0] [batch_t 0.324 (0.353)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 17:06:09,122 - Train: 7.90% [390200/4942000] [79.0/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 17:06:41,882 - Train: 7.90% [390300/4942000] [79.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 17:07:15,634 - Train: 7.90% [390400/4942000] [79.0/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 17:07:21,549 - ==> Total time: 1 day, 23:10:00 Eta: 22 days, 21:52:55 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 17:07:50,356 - Train: 7.90% [390500/4942000] [79.0/1000.0] [batch_t 0.322 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 17:08:24,500 - Train: 7.90% [390600/4942000] [79.0/1000.0] [batch_t 0.324 (0.341)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 17:08:57,226 - Train: 7.91% [390700/4942000] [79.1/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 17:09:32,218 - Train: 7.91% [390800/4942000] [79.1/1000.0] [batch_t 0.324 (0.350)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 17:10:04,918 - Train: 7.91% [390900/4942000] [79.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 17:10:38,684 - Train: 7.91% [391000/4942000] [79.1/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 17:11:11,896 - Train: 7.91% [391100/4942000] [79.1/1000.0] [batch_t 0.334 (0.332)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 17:11:44,619 - Train: 7.92% [391200/4942000] [79.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 17:12:17,380 - Train: 7.92% [391300/4942000] [79.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 17:12:50,079 - Train: 7.92% [391400/4942000] [79.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 17:13:22,825 - Train: 7.92% [391500/4942000] [79.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 17:13:55,683 - Train: 7.92% [391600/4942000] [79.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 17:14:28,517 - Train: 7.93% [391700/4942000] [79.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 17:15:01,275 - Train: 7.93% [391800/4942000] [79.3/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 17:15:35,447 - Train: 7.93% [391900/4942000] [79.3/1000.0] [batch_t 0.328 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 17:16:10,715 - Train: 7.93% [392000/4942000] [79.3/1000.0] [batch_t 0.327 (0.353)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 17:16:43,512 - Train: 7.93% [392100/4942000] [79.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 17:17:16,326 - Train: 7.94% [392200/4942000] [79.4/1000.0] [batch_t 0.321 (0.328)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-04 17:17:49,106 - Train: 7.94% [392300/4942000] [79.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 17:18:22,938 - Train: 7.94% [392400/4942000] [79.4/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 17:18:55,703 - Train: 7.94% [392500/4942000] [79.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 17:19:28,515 - Train: 7.94% [392600/4942000] [79.4/1000.0] [batch_t 0.332 (0.328)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:20:01,357 - Train: 7.95% [392700/4942000] [79.5/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 17:20:34,938 - Train: 7.95% [392800/4942000] [79.5/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 17:21:07,717 - Train: 7.95% [392900/4942000] [79.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 17:21:40,520 - Train: 7.95% [393000/4942000] [79.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 17:22:13,316 - Train: 7.95% [393100/4942000] [79.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 17:22:46,088 - Train: 7.96% [393200/4942000] [79.6/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:23:19,008 - Train: 7.96% [393300/4942000] [79.6/1000.0] [batch_t 0.320 (0.329)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-04 17:23:51,762 - Train: 7.96% [393400/4942000] [79.6/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 17:24:24,573 - Train: 7.96% [393500/4942000] [79.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 17:24:57,390 - Train: 7.96% [393600/4942000] [79.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 17:25:31,212 - Train: 7.97% [393700/4942000] [79.7/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 17:26:04,190 - Train: 7.97% [393800/4942000] [79.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:26:39,128 - Train: 7.97% [393900/4942000] [79.7/1000.0] [batch_t 0.327 (0.349)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 17:27:13,815 - Train: 7.97% [394000/4942000] [79.7/1000.0] [batch_t 0.331 (0.347)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:27:46,638 - Train: 7.97% [394100/4942000] [79.7/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 17:28:19,374 - Train: 7.98% [394200/4942000] [79.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 17:28:52,239 - Train: 7.98% [394300/4942000] [79.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 17:29:25,702 - Train: 7.98% [394400/4942000] [79.8/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 17:29:58,447 - Train: 7.98% [394500/4942000] [79.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 17:30:31,621 - Train: 7.98% [394600/4942000] [79.8/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 17:31:04,481 - Train: 7.99% [394700/4942000] [79.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-04 17:31:38,727 - Train: 7.99% [394800/4942000] [79.9/1000.0] [batch_t 0.327 (0.342)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 17:32:11,529 - Train: 7.99% [394900/4942000] [79.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 17:32:44,303 - Train: 7.99% [395000/4942000] [79.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 17:33:17,668 - Train: 7.99% [395100/4942000] [79.9/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 17:33:50,523 - Train: 8.00% [395200/4942000] [80.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 17:34:24,471 - Train: 8.00% [395300/4942000] [80.0/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 17:34:44,155 - ==> Total time: 1 day, 23:37:23 Eta: 22 days, 19:39:58 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 17:34:59,258 - Train: 8.00% [395400/4942000] [80.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:35:32,767 - Train: 8.00% [395500/4942000] [80.0/1000.0] [batch_t 0.331 (0.335)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:36:05,508 - Train: 8.00% [395600/4942000] [80.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 17:36:38,262 - Train: 8.01% [395700/4942000] [80.1/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 17:37:11,804 - Train: 8.01% [395800/4942000] [80.1/1000.0] [batch_t 0.322 (0.335)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-04 17:37:44,539 - Train: 8.01% [395900/4942000] [80.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 17:38:17,502 - Train: 8.01% [396000/4942000] [80.1/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 17:38:50,267 - Train: 8.01% [396100/4942000] [80.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 17:39:24,095 - Train: 8.02% [396200/4942000] [80.2/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 17:39:56,824 - Train: 8.02% [396300/4942000] [80.2/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 17:40:30,137 - Train: 8.02% [396400/4942000] [80.2/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 17:41:02,877 - Train: 8.02% [396500/4942000] [80.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 17:41:36,171 - Train: 8.03% [396600/4942000] [80.3/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 17:42:14,344 - Train: 8.03% [396700/4942000] [80.3/1000.0] [batch_t 0.328 (0.382)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 17:42:47,037 - Train: 8.03% [396800/4942000] [80.3/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 17:43:20,476 - Train: 8.03% [396900/4942000] [80.3/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 17:43:53,211 - Train: 8.03% [397000/4942000] [80.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 17:44:26,188 - Train: 8.04% [397100/4942000] [80.4/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 17:44:59,000 - Train: 8.04% [397200/4942000] [80.4/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-04 17:45:33,821 - Train: 8.04% [397300/4942000] [80.4/1000.0] [batch_t 0.325 (0.348)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 17:46:09,733 - Train: 8.04% [397400/4942000] [80.4/1000.0] [batch_t 0.325 (0.359)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 17:46:42,453 - Train: 8.04% [397500/4942000] [80.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 17:47:15,181 - Train: 8.05% [397600/4942000] [80.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 17:47:47,924 - Train: 8.05% [397700/4942000] [80.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 17:48:24,440 - Train: 8.05% [397800/4942000] [80.5/1000.0] [batch_t 0.331 (0.365)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:48:57,178 - Train: 8.05% [397900/4942000] [80.5/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:49:31,903 - Train: 8.05% [398000/4942000] [80.5/1000.0] [batch_t 0.327 (0.347)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 17:50:08,093 - Train: 8.06% [398100/4942000] [80.6/1000.0] [batch_t 0.326 (0.362)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 17:50:40,863 - Train: 8.06% [398200/4942000] [80.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 17:51:15,975 - Train: 8.06% [398300/4942000] [80.6/1000.0] [batch_t 0.328 (0.351)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 17:51:48,711 - Train: 8.06% [398400/4942000] [80.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 17:52:23,573 - Train: 8.06% [398500/4942000] [80.6/1000.0] [batch_t 0.327 (0.349)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 17:52:56,309 - Train: 8.07% [398600/4942000] [80.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 17:53:29,368 - Train: 8.07% [398700/4942000] [80.7/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 17:54:02,113 - Train: 8.07% [398800/4942000] [80.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 17:54:38,728 - Train: 8.07% [398900/4942000] [80.7/1000.0] [batch_t 0.328 (0.366)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 17:55:11,413 - Train: 8.07% [399000/4942000] [80.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 17:55:44,176 - Train: 8.08% [399100/4942000] [80.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 17:56:19,419 - Train: 8.08% [399200/4942000] [80.8/1000.0] [batch_t 0.326 (0.352)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 17:56:52,198 - Train: 8.08% [399300/4942000] [80.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 17:57:25,778 - Train: 8.08% [399400/4942000] [80.8/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 17:57:58,614 - Train: 8.08% [399500/4942000] [80.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 17:58:31,338 - Train: 8.09% [399600/4942000] [80.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 17:59:04,100 - Train: 8.09% [399700/4942000] [80.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:59:36,869 - Train: 8.09% [399800/4942000] [80.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 18:00:11,939 - Train: 8.09% [399900/4942000] [80.9/1000.0] [batch_t 0.327 (0.351)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 18:00:44,714 - Train: 8.09% [400000/4942000] [80.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 18:01:18,117 - Train: 8.10% [400100/4942000] [81.0/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 18:01:50,880 - Train: 8.10% [400200/4942000] [81.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 18:02:24,343 - Train: 8.10% [400300/4942000] [81.0/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 18:02:25,004 - ==> Total time: 2 days, 0:05:04 Eta: 22 days, 17:33:04 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 18:02:59,054 - Train: 8.10% [400400/4942000] [81.0/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 18:03:32,726 - Train: 8.10% [400500/4942000] [81.0/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 18:04:05,759 - Train: 8.11% [400600/4942000] [81.1/1000.0] [batch_t 0.593 (0.330)] [data_t 0.266] [optim_t 0.327] [lr 0.005000] 2024-04-04 18:04:38,938 - Train: 8.11% [400700/4942000] [81.1/1000.0] [batch_t 0.326 (0.332)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 18:05:11,904 - Train: 8.11% [400800/4942000] [81.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 18:05:44,712 - Train: 8.11% [400900/4942000] [81.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-04 18:06:18,292 - Train: 8.11% [401000/4942000] [81.1/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 18:06:51,101 - Train: 8.12% [401100/4942000] [81.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 18:07:23,922 - Train: 8.12% [401200/4942000] [81.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-04 18:07:56,760 - Train: 8.12% [401300/4942000] [81.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 18:08:30,059 - Train: 8.12% [401400/4942000] [81.2/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 18:09:02,887 - Train: 8.12% [401500/4942000] [81.2/1000.0] [batch_t 0.336 (0.328)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-04 18:09:35,731 - Train: 8.13% [401600/4942000] [81.3/1000.0] [batch_t 0.334 (0.328)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-04 18:10:09,275 - Train: 8.13% [401700/4942000] [81.3/1000.0] [batch_t 0.324 (0.335)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 18:10:42,077 - Train: 8.13% [401800/4942000] [81.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 18:11:15,952 - Train: 8.13% [401900/4942000] [81.3/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 18:11:48,745 - Train: 8.13% [402000/4942000] [81.3/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 18:12:22,411 - Train: 8.14% [402100/4942000] [81.4/1000.0] [batch_t 0.330 (0.337)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-04 18:12:55,209 - Train: 8.14% [402200/4942000] [81.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 18:13:29,960 - Train: 8.14% [402300/4942000] [81.4/1000.0] [batch_t 0.326 (0.347)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 18:14:02,775 - Train: 8.14% [402400/4942000] [81.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 18:14:37,018 - Train: 8.14% [402500/4942000] [81.4/1000.0] [batch_t 0.333 (0.342)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 18:15:10,440 - Train: 8.15% [402600/4942000] [81.5/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 18:15:43,263 - Train: 8.15% [402700/4942000] [81.5/1000.0] [batch_t 0.331 (0.328)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:16:16,600 - Train: 8.15% [402800/4942000] [81.5/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 18:16:49,346 - Train: 8.15% [402900/4942000] [81.5/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 18:17:23,127 - Train: 8.15% [403000/4942000] [81.5/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 18:17:55,857 - Train: 8.16% [403100/4942000] [81.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 18:18:29,433 - Train: 8.16% [403200/4942000] [81.6/1000.0] [batch_t 0.326 (0.336)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 18:19:02,192 - Train: 8.16% [403300/4942000] [81.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 18:19:36,246 - Train: 8.16% [403400/4942000] [81.6/1000.0] [batch_t 0.325 (0.340)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 18:20:09,410 - Train: 8.16% [403500/4942000] [81.6/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 18:20:42,136 - Train: 8.17% [403600/4942000] [81.7/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 18:21:15,781 - Train: 8.17% [403700/4942000] [81.7/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 18:21:48,535 - Train: 8.17% [403800/4942000] [81.7/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 18:22:21,318 - Train: 8.17% [403900/4942000] [81.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 18:22:54,080 - Train: 8.17% [404000/4942000] [81.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 18:23:27,288 - Train: 8.18% [404100/4942000] [81.8/1000.0] [batch_t 0.326 (0.332)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 18:24:00,101 - Train: 8.18% [404200/4942000] [81.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 18:24:34,079 - Train: 8.18% [404300/4942000] [81.8/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 18:25:07,533 - Train: 8.18% [404400/4942000] [81.8/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 18:25:40,346 - Train: 8.18% [404500/4942000] [81.8/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 18:26:14,048 - Train: 8.19% [404600/4942000] [81.9/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 18:26:46,810 - Train: 8.19% [404700/4942000] [81.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 18:27:19,569 - Train: 8.19% [404800/4942000] [81.9/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 18:27:52,320 - Train: 8.19% [404900/4942000] [81.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 18:28:25,817 - Train: 8.20% [405000/4942000] [82.0/1000.0] [batch_t 0.326 (0.335)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 18:28:58,765 - Train: 8.20% [405100/4942000] [82.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 18:29:32,467 - Train: 8.20% [405200/4942000] [82.0/1000.0] [batch_t 0.324 (0.337)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 18:29:46,906 - ==> Total time: 2 days, 0:32:26 Eta: 22 days, 15:25:03 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 18:30:07,045 - Train: 8.20% [405300/4942000] [82.0/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 18:30:39,810 - Train: 8.20% [405400/4942000] [82.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-04 18:31:12,534 - Train: 8.21% [405500/4942000] [82.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 18:31:45,272 - Train: 8.21% [405600/4942000] [82.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-04 18:32:17,997 - Train: 8.21% [405700/4942000] [82.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 18:32:50,740 - Train: 8.21% [405800/4942000] [82.1/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 18:33:24,114 - Train: 8.21% [405900/4942000] [82.1/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 18:33:56,918 - Train: 8.22% [406000/4942000] [82.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 18:34:30,522 - Train: 8.22% [406100/4942000] [82.2/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 18:35:03,309 - Train: 8.22% [406200/4942000] [82.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 18:35:36,811 - Train: 8.22% [406300/4942000] [82.2/1000.0] [batch_t 0.325 (0.335)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 18:36:09,620 - Train: 8.22% [406400/4942000] [82.2/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-04 18:36:42,509 - Train: 8.23% [406500/4942000] [82.3/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 18:37:16,173 - Train: 8.23% [406600/4942000] [82.3/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 18:37:48,942 - Train: 8.23% [406700/4942000] [82.3/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 18:38:23,051 - Train: 8.23% [406800/4942000] [82.3/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 18:38:55,815 - Train: 8.23% [406900/4942000] [82.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 18:39:29,646 - Train: 8.24% [407000/4942000] [82.4/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 18:40:02,422 - Train: 8.24% [407100/4942000] [82.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 18:40:35,187 - Train: 8.24% [407200/4942000] [82.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-04 18:41:07,943 - Train: 8.24% [407300/4942000] [82.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 18:41:40,671 - Train: 8.24% [407400/4942000] [82.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 18:42:13,452 - Train: 8.25% [407500/4942000] [82.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 18:42:46,235 - Train: 8.25% [407600/4942000] [82.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 18:43:19,321 - Train: 8.25% [407700/4942000] [82.5/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 18:43:52,060 - Train: 8.25% [407800/4942000] [82.5/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 18:44:24,951 - Train: 8.25% [407900/4942000] [82.5/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 18:44:57,704 - Train: 8.26% [408000/4942000] [82.6/1000.0] [batch_t 0.318 (0.327)] [data_t 0.002] [optim_t 0.316] [lr 0.005000] 2024-04-04 18:45:30,475 - Train: 8.26% [408100/4942000] [82.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 18:46:03,220 - Train: 8.26% [408200/4942000] [82.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 18:46:35,956 - Train: 8.26% [408300/4942000] [82.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 18:47:08,730 - Train: 8.26% [408400/4942000] [82.6/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 18:47:41,509 - Train: 8.27% [408500/4942000] [82.7/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 18:48:15,066 - Train: 8.27% [408600/4942000] [82.7/1000.0] [batch_t 0.326 (0.335)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 18:48:47,801 - Train: 8.27% [408700/4942000] [82.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 18:49:22,404 - Train: 8.27% [408800/4942000] [82.7/1000.0] [batch_t 0.337 (0.346)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-04 18:49:55,242 - Train: 8.27% [408900/4942000] [82.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 18:50:28,073 - Train: 8.28% [409000/4942000] [82.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 18:51:00,822 - Train: 8.28% [409100/4942000] [82.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 18:51:33,569 - Train: 8.28% [409200/4942000] [82.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 18:52:06,389 - Train: 8.28% [409300/4942000] [82.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 18:52:39,182 - Train: 8.28% [409400/4942000] [82.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 18:53:11,974 - Train: 8.29% [409500/4942000] [82.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 18:53:44,742 - Train: 8.29% [409600/4942000] [82.9/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 18:54:18,625 - Train: 8.29% [409700/4942000] [82.9/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 18:54:51,387 - Train: 8.29% [409800/4942000] [82.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 18:55:24,158 - Train: 8.29% [409900/4942000] [82.9/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 18:55:56,941 - Train: 8.30% [410000/4942000] [83.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 18:56:29,680 - Train: 8.30% [410100/4942000] [83.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 18:56:57,841 - ==> Total time: 2 days, 0:59:37 Eta: 22 days, 13:17:27 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 18:57:04,548 - Train: 8.30% [410200/4942000] [83.0/1000.0] [batch_t 0.329 (0.334)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 18:57:37,727 - Train: 8.30% [410300/4942000] [83.0/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 18:58:10,517 - Train: 8.30% [410400/4942000] [83.0/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-04 18:58:43,297 - Train: 8.31% [410500/4942000] [83.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 18:59:17,236 - Train: 8.31% [410600/4942000] [83.1/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 18:59:49,978 - Train: 8.31% [410700/4942000] [83.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:00:23,734 - Train: 8.31% [410800/4942000] [83.1/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 19:00:56,467 - Train: 8.31% [410900/4942000] [83.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 19:01:30,902 - Train: 8.32% [411000/4942000] [83.2/1000.0] [batch_t 0.330 (0.344)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 19:02:03,673 - Train: 8.32% [411100/4942000] [83.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:02:36,990 - Train: 8.32% [411200/4942000] [83.2/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:03:11,927 - Train: 8.32% [411300/4942000] [83.2/1000.0] [batch_t 0.328 (0.349)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 19:03:44,723 - Train: 8.32% [411400/4942000] [83.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 19:04:18,833 - Train: 8.33% [411500/4942000] [83.3/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 19:04:51,606 - Train: 8.33% [411600/4942000] [83.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:05:25,187 - Train: 8.33% [411700/4942000] [83.3/1000.0] [batch_t 0.326 (0.336)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 19:05:58,012 - Train: 8.33% [411800/4942000] [83.3/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 19:06:31,745 - Train: 8.33% [411900/4942000] [83.3/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 19:07:05,462 - Train: 8.34% [412000/4942000] [83.4/1000.0] [batch_t 1.280 (0.337)] [data_t 0.954] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:07:38,312 - Train: 8.34% [412100/4942000] [83.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 19:08:11,106 - Train: 8.34% [412200/4942000] [83.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:08:43,783 - Train: 8.34% [412300/4942000] [83.4/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 19:09:16,716 - Train: 8.34% [412400/4942000] [83.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 19:09:49,464 - Train: 8.35% [412500/4942000] [83.5/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 19:10:24,261 - Train: 8.35% [412600/4942000] [83.5/1000.0] [batch_t 0.326 (0.348)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 19:10:57,013 - Train: 8.35% [412700/4942000] [83.5/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 19:11:30,668 - Train: 8.35% [412800/4942000] [83.5/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 19:12:03,397 - Train: 8.35% [412900/4942000] [83.5/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 19:12:36,130 - Train: 8.36% [413000/4942000] [83.6/1000.0] [batch_t 0.317 (0.327)] [data_t 0.002] [optim_t 0.316] [lr 0.005000] 2024-04-04 19:13:08,881 - Train: 8.36% [413100/4942000] [83.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 19:13:41,654 - Train: 8.36% [413200/4942000] [83.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:14:14,427 - Train: 8.36% [413300/4942000] [83.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:14:47,130 - Train: 8.37% [413400/4942000] [83.7/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 19:15:20,286 - Train: 8.37% [413500/4942000] [83.7/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 19:15:53,028 - Train: 8.37% [413600/4942000] [83.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 19:16:25,738 - Train: 8.37% [413700/4942000] [83.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 19:16:58,465 - Train: 8.37% [413800/4942000] [83.7/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 19:17:32,350 - Train: 8.38% [413900/4942000] [83.8/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 19:18:07,273 - Train: 8.38% [414000/4942000] [83.8/1000.0] [batch_t 2.511 (0.349)] [data_t 2.185] [optim_t 0.326] [lr 0.005000] 2024-04-04 19:18:40,373 - Train: 8.38% [414100/4942000] [83.8/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 19:19:13,144 - Train: 8.38% [414200/4942000] [83.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 19:19:45,912 - Train: 8.38% [414300/4942000] [83.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 19:20:18,662 - Train: 8.39% [414400/4942000] [83.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 19:20:51,456 - Train: 8.39% [414500/4942000] [83.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:21:24,252 - Train: 8.39% [414600/4942000] [83.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 19:21:57,019 - Train: 8.39% [414700/4942000] [83.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 19:22:29,815 - Train: 8.39% [414800/4942000] [83.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 19:23:02,568 - Train: 8.40% [414900/4942000] [84.0/1000.0] [batch_t 0.324 (0.327)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-04 19:23:35,462 - Train: 8.40% [415000/4942000] [84.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 19:24:08,227 - Train: 8.40% [415100/4942000] [84.0/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 19:24:17,409 - ==> Total time: 2 days, 1:26:56 Eta: 22 days, 11:13:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 19:24:43,216 - Train: 8.40% [415200/4942000] [84.0/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 19:25:16,476 - Train: 8.40% [415300/4942000] [84.0/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 19:25:49,272 - Train: 8.41% [415400/4942000] [84.1/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 19:26:22,014 - Train: 8.41% [415500/4942000] [84.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 19:26:54,790 - Train: 8.41% [415600/4942000] [84.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 19:27:27,515 - Train: 8.41% [415700/4942000] [84.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 19:28:00,240 - Train: 8.41% [415800/4942000] [84.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 19:28:32,955 - Train: 8.42% [415900/4942000] [84.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 19:29:06,709 - Train: 8.42% [416000/4942000] [84.2/1000.0] [batch_t 0.325 (0.337)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 19:29:39,462 - Train: 8.42% [416100/4942000] [84.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 19:30:12,867 - Train: 8.42% [416200/4942000] [84.2/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 19:30:45,726 - Train: 8.42% [416300/4942000] [84.2/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 19:31:18,533 - Train: 8.43% [416400/4942000] [84.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:31:51,336 - Train: 8.43% [416500/4942000] [84.3/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 19:32:24,076 - Train: 8.43% [416600/4942000] [84.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 19:32:56,826 - Train: 8.43% [416700/4942000] [84.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 19:33:30,310 - Train: 8.43% [416800/4942000] [84.3/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 19:34:03,103 - Train: 8.44% [416900/4942000] [84.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 19:34:35,854 - Train: 8.44% [417000/4942000] [84.4/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 19:35:09,426 - Train: 8.44% [417100/4942000] [84.4/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 19:35:42,169 - Train: 8.44% [417200/4942000] [84.4/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 19:36:14,992 - Train: 8.44% [417300/4942000] [84.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 19:36:47,717 - Train: 8.45% [417400/4942000] [84.5/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 19:37:20,615 - Train: 8.45% [417500/4942000] [84.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:37:53,416 - Train: 8.45% [417600/4942000] [84.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 19:38:27,422 - Train: 8.45% [417700/4942000] [84.5/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:39:00,190 - Train: 8.45% [417800/4942000] [84.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-04 19:39:34,203 - Train: 8.46% [417900/4942000] [84.6/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 19:40:07,014 - Train: 8.46% [418000/4942000] [84.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 19:40:39,770 - Train: 8.46% [418100/4942000] [84.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 19:41:13,541 - Train: 8.46% [418200/4942000] [84.6/1000.0] [batch_t 0.325 (0.338)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 19:41:46,294 - Train: 8.46% [418300/4942000] [84.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 19:42:19,690 - Train: 8.47% [418400/4942000] [84.7/1000.0] [batch_t 0.326 (0.334)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 19:42:52,417 - Train: 8.47% [418500/4942000] [84.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 19:43:25,741 - Train: 8.47% [418600/4942000] [84.7/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:43:58,463 - Train: 8.47% [418700/4942000] [84.7/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 19:44:31,914 - Train: 8.47% [418800/4942000] [84.7/1000.0] [batch_t 0.323 (0.334)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 19:45:04,682 - Train: 8.48% [418900/4942000] [84.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 19:45:37,574 - Train: 8.48% [419000/4942000] [84.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:46:12,396 - Train: 8.48% [419100/4942000] [84.8/1000.0] [batch_t 0.327 (0.348)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 19:46:45,162 - Train: 8.48% [419200/4942000] [84.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 19:47:17,866 - Train: 8.48% [419300/4942000] [84.8/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 19:47:50,650 - Train: 8.49% [419400/4942000] [84.9/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:48:23,903 - Train: 8.49% [419500/4942000] [84.9/1000.0] [batch_t 0.325 (0.332)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 19:48:56,632 - Train: 8.49% [419600/4942000] [84.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 19:49:29,374 - Train: 8.49% [419700/4942000] [84.9/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 19:50:02,103 - Train: 8.49% [419800/4942000] [84.9/1000.0] [batch_t 0.320 (0.327)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-04 19:50:34,870 - Train: 8.50% [419900/4942000] [85.0/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 19:51:07,586 - Train: 8.50% [420000/4942000] [85.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 19:51:30,487 - ==> Total time: 2 days, 1:54:09 Eta: 22 days, 9:11:15 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 19:51:42,200 - Train: 8.50% [420100/4942000] [85.0/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 19:52:14,941 - Train: 8.50% [420200/4942000] [85.0/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 19:52:47,684 - Train: 8.50% [420300/4942000] [85.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 19:53:22,683 - Train: 8.51% [420400/4942000] [85.1/1000.0] [batch_t 0.330 (0.350)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 19:53:55,474 - Train: 8.51% [420500/4942000] [85.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:54:28,684 - Train: 8.51% [420600/4942000] [85.1/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 19:55:01,378 - Train: 8.51% [420700/4942000] [85.1/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 19:55:34,971 - Train: 8.51% [420800/4942000] [85.1/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 19:56:09,254 - Train: 8.52% [420900/4942000] [85.2/1000.0] [batch_t 0.327 (0.343)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 19:56:42,070 - Train: 8.52% [421000/4942000] [85.2/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:57:16,122 - Train: 8.52% [421100/4942000] [85.2/1000.0] [batch_t 0.330 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:57:48,823 - Train: 8.52% [421200/4942000] [85.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 19:58:21,583 - Train: 8.52% [421300/4942000] [85.2/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:58:54,362 - Train: 8.53% [421400/4942000] [85.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 19:59:27,131 - Train: 8.53% [421500/4942000] [85.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 19:59:59,910 - Train: 8.53% [421600/4942000] [85.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 20:00:33,860 - Train: 8.53% [421700/4942000] [85.3/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:01:06,674 - Train: 8.54% [421800/4942000] [85.4/1000.0] [batch_t 0.320 (0.328)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-04 20:01:39,375 - Train: 8.54% [421900/4942000] [85.4/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 20:02:14,105 - Train: 8.54% [422000/4942000] [85.4/1000.0] [batch_t 0.325 (0.347)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 20:02:46,848 - Train: 8.54% [422100/4942000] [85.4/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 20:03:20,135 - Train: 8.54% [422200/4942000] [85.4/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 20:03:52,840 - Train: 8.55% [422300/4942000] [85.5/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 20:04:25,621 - Train: 8.55% [422400/4942000] [85.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 20:04:58,336 - Train: 8.55% [422500/4942000] [85.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 20:05:31,086 - Train: 8.55% [422600/4942000] [85.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 20:06:03,829 - Train: 8.55% [422700/4942000] [85.5/1000.0] [batch_t 0.330 (0.327)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:06:36,581 - Train: 8.56% [422800/4942000] [85.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 20:07:09,272 - Train: 8.56% [422900/4942000] [85.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:07:47,236 - Train: 8.56% [423000/4942000] [85.6/1000.0] [batch_t 0.325 (0.380)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 20:08:19,976 - Train: 8.56% [423100/4942000] [85.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:08:52,710 - Train: 8.56% [423200/4942000] [85.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 20:09:25,541 - Train: 8.57% [423300/4942000] [85.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 20:09:58,295 - Train: 8.57% [423400/4942000] [85.7/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 20:10:32,083 - Train: 8.57% [423500/4942000] [85.7/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 20:11:04,802 - Train: 8.57% [423600/4942000] [85.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 20:11:40,368 - Train: 8.57% [423700/4942000] [85.7/1000.0] [batch_t 0.325 (0.356)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 20:12:13,208 - Train: 8.58% [423800/4942000] [85.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 20:12:45,918 - Train: 8.58% [423900/4942000] [85.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 20:13:18,630 - Train: 8.58% [424000/4942000] [85.8/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 20:13:51,334 - Train: 8.58% [424100/4942000] [85.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:14:24,035 - Train: 8.58% [424200/4942000] [85.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:14:56,829 - Train: 8.59% [424300/4942000] [85.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:15:29,570 - Train: 8.59% [424400/4942000] [85.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 20:16:02,312 - Train: 8.59% [424500/4942000] [85.9/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-04 20:16:35,568 - Train: 8.59% [424600/4942000] [85.9/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 20:17:09,333 - Train: 8.59% [424700/4942000] [85.9/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 20:17:42,149 - Train: 8.60% [424800/4942000] [86.0/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 20:18:14,958 - Train: 8.60% [424900/4942000] [86.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:18:47,713 - Train: 8.60% [425000/4942000] [86.0/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 20:18:51,663 - ==> Total time: 2 days, 2:21:30 Eta: 22 days, 7:12:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 20:19:22,477 - Train: 8.60% [425100/4942000] [86.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 20:19:55,268 - Train: 8.60% [425200/4942000] [86.0/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:20:29,308 - Train: 8.61% [425300/4942000] [86.1/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 20:21:02,040 - Train: 8.61% [425400/4942000] [86.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 20:21:35,324 - Train: 8.61% [425500/4942000] [86.1/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:22:09,595 - Train: 8.61% [425600/4942000] [86.1/1000.0] [batch_t 0.329 (0.343)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:22:42,335 - Train: 8.61% [425700/4942000] [86.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 20:23:15,829 - Train: 8.62% [425800/4942000] [86.2/1000.0] [batch_t 0.324 (0.335)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 20:23:48,602 - Train: 8.62% [425900/4942000] [86.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 20:24:22,511 - Train: 8.62% [426000/4942000] [86.2/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 20:24:55,267 - Train: 8.62% [426100/4942000] [86.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 20:25:29,086 - Train: 8.62% [426200/4942000] [86.2/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 20:26:01,827 - Train: 8.63% [426300/4942000] [86.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 20:26:36,657 - Train: 8.63% [426400/4942000] [86.3/1000.0] [batch_t 0.330 (0.348)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:27:10,373 - Train: 8.63% [426500/4942000] [86.3/1000.0] [batch_t 0.324 (0.337)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 20:27:43,112 - Train: 8.63% [426600/4942000] [86.3/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 20:28:16,504 - Train: 8.63% [426700/4942000] [86.3/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 20:28:49,230 - Train: 8.64% [426800/4942000] [86.4/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 20:29:22,697 - Train: 8.64% [426900/4942000] [86.4/1000.0] [batch_t 0.322 (0.335)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-04 20:29:55,413 - Train: 8.64% [427000/4942000] [86.4/1000.0] [batch_t 0.318 (0.327)] [data_t 0.002] [optim_t 0.316] [lr 0.005000] 2024-04-04 20:30:29,187 - Train: 8.64% [427100/4942000] [86.4/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 20:31:01,903 - Train: 8.64% [427200/4942000] [86.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 20:31:34,689 - Train: 8.65% [427300/4942000] [86.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 20:32:07,486 - Train: 8.65% [427400/4942000] [86.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 20:32:40,505 - Train: 8.65% [427500/4942000] [86.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:33:13,250 - Train: 8.65% [427600/4942000] [86.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 20:33:46,032 - Train: 8.65% [427700/4942000] [86.5/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 20:34:19,363 - Train: 8.66% [427800/4942000] [86.6/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 20:34:52,157 - Train: 8.66% [427900/4942000] [86.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 20:35:24,940 - Train: 8.66% [428000/4942000] [86.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 20:35:57,707 - Train: 8.66% [428100/4942000] [86.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 20:36:30,527 - Train: 8.66% [428200/4942000] [86.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 20:37:03,353 - Train: 8.67% [428300/4942000] [86.7/1000.0] [batch_t 0.320 (0.328)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-04 20:37:36,144 - Train: 8.67% [428400/4942000] [86.7/1000.0] [batch_t 0.320 (0.328)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-04 20:38:08,905 - Train: 8.67% [428500/4942000] [86.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 20:38:41,685 - Train: 8.67% [428600/4942000] [86.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 20:39:15,516 - Train: 8.67% [428700/4942000] [86.7/1000.0] [batch_t 0.324 (0.338)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 20:39:48,356 - Train: 8.68% [428800/4942000] [86.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 20:40:21,270 - Train: 8.68% [428900/4942000] [86.8/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 20:40:54,088 - Train: 8.68% [429000/4942000] [86.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-04 20:41:28,192 - Train: 8.68% [429100/4942000] [86.8/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 20:42:01,038 - Train: 8.68% [429200/4942000] [86.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 20:42:34,807 - Train: 8.69% [429300/4942000] [86.9/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 20:43:09,180 - Train: 8.69% [429400/4942000] [86.9/1000.0] [batch_t 0.324 (0.344)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 20:43:42,013 - Train: 8.69% [429500/4942000] [86.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 20:44:14,834 - Train: 8.69% [429600/4942000] [86.9/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:44:47,614 - Train: 8.69% [429700/4942000] [86.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-04 20:45:22,154 - Train: 8.70% [429800/4942000] [87.0/1000.0] [batch_t 0.327 (0.345)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 20:45:54,972 - Train: 8.70% [429900/4942000] [87.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 20:46:13,823 - ==> Total time: 2 days, 2:48:53 Eta: 22 days, 5:15:45 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 20:46:31,006 - Train: 8.70% [430000/4942000] [87.0/1000.0] [batch_t 0.333 (0.328)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-04 20:47:03,819 - Train: 8.70% [430100/4942000] [87.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:47:36,833 - Train: 8.70% [430200/4942000] [87.0/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 20:48:10,159 - Train: 8.71% [430300/4942000] [87.1/1000.0] [batch_t 0.326 (0.333)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 20:48:42,944 - Train: 8.71% [430400/4942000] [87.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 20:49:16,370 - Train: 8.71% [430500/4942000] [87.1/1000.0] [batch_t 0.324 (0.334)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 20:49:49,196 - Train: 8.71% [430600/4942000] [87.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 20:50:23,245 - Train: 8.72% [430700/4942000] [87.2/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 20:50:56,106 - Train: 8.72% [430800/4942000] [87.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:51:29,756 - Train: 8.72% [430900/4942000] [87.2/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:52:02,567 - Train: 8.72% [431000/4942000] [87.2/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-04 20:52:36,313 - Train: 8.72% [431100/4942000] [87.2/1000.0] [batch_t 0.325 (0.337)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 20:53:10,199 - Train: 8.73% [431200/4942000] [87.3/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 20:53:42,951 - Train: 8.73% [431300/4942000] [87.3/1000.0] [batch_t 0.330 (0.327)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:54:16,810 - Train: 8.73% [431400/4942000] [87.3/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 20:54:49,605 - Train: 8.73% [431500/4942000] [87.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 20:55:23,422 - Train: 8.73% [431600/4942000] [87.3/1000.0] [batch_t 0.328 (0.338)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-04 20:55:56,343 - Train: 8.74% [431700/4942000] [87.4/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 20:56:30,462 - Train: 8.74% [431800/4942000] [87.4/1000.0] [batch_t 0.328 (0.341)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 20:57:03,270 - Train: 8.74% [431900/4942000] [87.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 20:57:36,064 - Train: 8.74% [432000/4942000] [87.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 20:58:09,449 - Train: 8.74% [432100/4942000] [87.4/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 20:58:42,268 - Train: 8.75% [432200/4942000] [87.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 20:59:15,022 - Train: 8.75% [432300/4942000] [87.5/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 20:59:47,861 - Train: 8.75% [432400/4942000] [87.5/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 21:00:21,368 - Train: 8.75% [432500/4942000] [87.5/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 21:00:54,211 - Train: 8.75% [432600/4942000] [87.5/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 21:01:26,939 - Train: 8.76% [432700/4942000] [87.6/1000.0] [batch_t 0.334 (0.327)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 21:01:59,685 - Train: 8.76% [432800/4942000] [87.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 21:02:32,472 - Train: 8.76% [432900/4942000] [87.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 21:03:05,278 - Train: 8.76% [433000/4942000] [87.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 21:03:38,178 - Train: 8.76% [433100/4942000] [87.6/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 21:04:10,975 - Train: 8.77% [433200/4942000] [87.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-04 21:04:43,748 - Train: 8.77% [433300/4942000] [87.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 21:05:16,550 - Train: 8.77% [433400/4942000] [87.7/1000.0] [batch_t 0.333 (0.328)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-04 21:05:49,306 - Train: 8.77% [433500/4942000] [87.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 21:06:22,300 - Train: 8.77% [433600/4942000] [87.7/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 21:06:55,078 - Train: 8.78% [433700/4942000] [87.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 21:07:27,849 - Train: 8.78% [433800/4942000] [87.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 21:08:00,588 - Train: 8.78% [433900/4942000] [87.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 21:08:33,387 - Train: 8.78% [434000/4942000] [87.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 21:09:07,318 - Train: 8.78% [434100/4942000] [87.8/1000.0] [batch_t 0.408 (0.339)] [data_t 0.085] [optim_t 0.324] [lr 0.005000] 2024-04-04 21:09:40,090 - Train: 8.79% [434200/4942000] [87.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 21:10:12,908 - Train: 8.79% [434300/4942000] [87.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:10:45,698 - Train: 8.79% [434400/4942000] [87.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 21:11:18,437 - Train: 8.79% [434500/4942000] [87.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 21:11:51,301 - Train: 8.79% [434600/4942000] [87.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 21:12:24,138 - Train: 8.80% [434700/4942000] [88.0/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 21:12:56,926 - Train: 8.80% [434800/4942000] [88.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-04 21:13:28,368 - ==> Total time: 2 days, 3:16:07 Eta: 22 days, 3:19:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 21:13:31,651 - Train: 8.80% [434900/4942000] [88.0/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 21:14:05,485 - Train: 8.80% [435000/4942000] [88.0/1000.0] [batch_t 1.412 (0.338)] [data_t 1.089] [optim_t 0.323] [lr 0.005000] 2024-04-04 21:14:38,470 - Train: 8.80% [435100/4942000] [88.0/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 21:15:11,237 - Train: 8.81% [435200/4942000] [88.1/1000.0] [batch_t 0.323 (0.328)] [data_t 0.003] [optim_t 0.320] [lr 0.005000] 2024-04-04 21:15:44,030 - Train: 8.81% [435300/4942000] [88.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 21:16:16,762 - Train: 8.81% [435400/4942000] [88.1/1000.0] [batch_t 0.334 (0.327)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 21:16:49,572 - Train: 8.81% [435500/4942000] [88.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:17:24,236 - Train: 8.81% [435600/4942000] [88.1/1000.0] [batch_t 0.329 (0.347)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 21:17:56,996 - Train: 8.82% [435700/4942000] [88.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 21:18:31,031 - Train: 8.82% [435800/4942000] [88.2/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 21:19:19,735 - Train: 8.82% [435900/4942000] [88.2/1000.0] [batch_t 0.324 (0.487)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 21:19:52,503 - Train: 8.82% [436000/4942000] [88.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 21:20:26,649 - Train: 8.82% [436100/4942000] [88.2/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 21:20:59,470 - Train: 8.83% [436200/4942000] [88.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 21:21:32,541 - Train: 8.83% [436300/4942000] [88.3/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:22:05,489 - Train: 8.83% [436400/4942000] [88.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 21:22:38,322 - Train: 8.83% [436500/4942000] [88.3/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:23:11,183 - Train: 8.83% [436600/4942000] [88.3/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 21:23:44,070 - Train: 8.84% [436700/4942000] [88.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 21:24:29,844 - Train: 8.84% [436800/4942000] [88.4/1000.0] [batch_t 0.329 (0.458)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 21:25:02,651 - Train: 8.84% [436900/4942000] [88.4/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 21:25:35,407 - Train: 8.84% [437000/4942000] [88.4/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 21:26:08,825 - Train: 8.84% [437100/4942000] [88.4/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 21:26:41,610 - Train: 8.85% [437200/4942000] [88.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:27:15,485 - Train: 8.85% [437300/4942000] [88.5/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 21:27:48,424 - Train: 8.85% [437400/4942000] [88.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 21:28:21,273 - Train: 8.85% [437500/4942000] [88.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:28:54,056 - Train: 8.85% [437600/4942000] [88.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:29:27,566 - Train: 8.86% [437700/4942000] [88.6/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 21:30:00,321 - Train: 8.86% [437800/4942000] [88.6/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 21:30:33,526 - Train: 8.86% [437900/4942000] [88.6/1000.0] [batch_t 0.321 (0.332)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-04 21:31:08,425 - Train: 8.86% [438000/4942000] [88.6/1000.0] [batch_t 0.326 (0.349)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 21:31:41,286 - Train: 8.86% [438100/4942000] [88.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 21:32:15,676 - Train: 8.87% [438200/4942000] [88.7/1000.0] [batch_t 0.328 (0.344)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 21:32:48,436 - Train: 8.87% [438300/4942000] [88.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 21:33:22,395 - Train: 8.87% [438400/4942000] [88.7/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 21:33:55,241 - Train: 8.87% [438500/4942000] [88.7/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 21:34:31,341 - Train: 8.87% [438600/4942000] [88.7/1000.0] [batch_t 0.326 (0.361)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 21:35:04,172 - Train: 8.88% [438700/4942000] [88.8/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 21:35:37,676 - Train: 8.88% [438800/4942000] [88.8/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 21:36:11,525 - Train: 8.88% [438900/4942000] [88.8/1000.0] [batch_t 0.325 (0.338)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 21:36:44,293 - Train: 8.88% [439000/4942000] [88.8/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 21:37:17,781 - Train: 8.89% [439100/4942000] [88.9/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 21:37:52,707 - Train: 8.89% [439200/4942000] [88.9/1000.0] [batch_t 0.327 (0.349)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 21:38:25,932 - Train: 8.89% [439300/4942000] [88.9/1000.0] [batch_t 0.324 (0.332)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 21:38:58,732 - Train: 8.89% [439400/4942000] [88.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 21:39:32,413 - Train: 8.89% [439500/4942000] [88.9/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 21:40:06,638 - Train: 8.90% [439600/4942000] [89.0/1000.0] [batch_t 0.328 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 21:40:42,492 - Train: 8.90% [439700/4942000] [89.0/1000.0] [batch_t 0.327 (0.358)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 21:41:16,234 - Train: 8.90% [439800/4942000] [89.0/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 21:41:28,736 - ==> Total time: 2 days, 3:44:07 Eta: 22 days, 1:33:45 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 21:41:51,438 - Train: 8.90% [439900/4942000] [89.0/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 21:42:25,913 - Train: 8.90% [440000/4942000] [89.0/1000.0] [batch_t 0.327 (0.345)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 21:42:58,825 - Train: 8.91% [440100/4942000] [89.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 21:43:32,700 - Train: 8.91% [440200/4942000] [89.1/1000.0] [batch_t 0.325 (0.339)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 21:44:14,362 - Train: 8.91% [440300/4942000] [89.1/1000.0] [batch_t 0.325 (0.417)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 21:44:47,161 - Train: 8.91% [440400/4942000] [89.1/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 21:45:21,086 - Train: 8.91% [440500/4942000] [89.1/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 21:45:53,832 - Train: 8.92% [440600/4942000] [89.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 21:46:27,758 - Train: 8.92% [440700/4942000] [89.2/1000.0] [batch_t 0.323 (0.339)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 21:47:00,519 - Train: 8.92% [440800/4942000] [89.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 21:47:33,993 - Train: 8.92% [440900/4942000] [89.2/1000.0] [batch_t 0.324 (0.335)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 21:48:07,034 - Train: 8.92% [441000/4942000] [89.2/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 21:48:39,820 - Train: 8.93% [441100/4942000] [89.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 21:49:13,726 - Train: 8.93% [441200/4942000] [89.3/1000.0] [batch_t 0.325 (0.339)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 21:49:46,527 - Train: 8.93% [441300/4942000] [89.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-04 21:50:22,048 - Train: 8.93% [441400/4942000] [89.3/1000.0] [batch_t 0.327 (0.355)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 21:50:54,991 - Train: 8.93% [441500/4942000] [89.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-04 21:51:29,003 - Train: 8.94% [441600/4942000] [89.4/1000.0] [batch_t 0.335 (0.340)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 21:52:01,768 - Train: 8.94% [441700/4942000] [89.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 21:52:35,278 - Train: 8.94% [441800/4942000] [89.4/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 21:53:08,896 - Train: 8.94% [441900/4942000] [89.4/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 21:53:41,666 - Train: 8.94% [442000/4942000] [89.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 21:54:16,654 - Train: 8.95% [442100/4942000] [89.5/1000.0] [batch_t 0.325 (0.350)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-04 21:54:49,471 - Train: 8.95% [442200/4942000] [89.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 21:55:22,746 - Train: 8.95% [442300/4942000] [89.5/1000.0] [batch_t 0.326 (0.333)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 21:55:55,526 - Train: 8.95% [442400/4942000] [89.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 21:56:29,148 - Train: 8.95% [442500/4942000] [89.5/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 21:57:01,956 - Train: 8.96% [442600/4942000] [89.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 21:57:35,000 - Train: 8.96% [442700/4942000] [89.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 21:58:09,842 - Train: 8.96% [442800/4942000] [89.6/1000.0] [batch_t 0.326 (0.348)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 21:58:42,597 - Train: 8.96% [442900/4942000] [89.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 21:59:16,364 - Train: 8.96% [443000/4942000] [89.6/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 21:59:49,160 - Train: 8.97% [443100/4942000] [89.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 22:00:22,130 - Train: 8.97% [443200/4942000] [89.7/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 22:00:54,866 - Train: 8.97% [443300/4942000] [89.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 22:01:28,225 - Train: 8.97% [443400/4942000] [89.7/1000.0] [batch_t 0.324 (0.334)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 22:02:00,981 - Train: 8.97% [443500/4942000] [89.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 22:02:34,296 - Train: 8.98% [443600/4942000] [89.8/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 22:03:07,686 - Train: 8.98% [443700/4942000] [89.8/1000.0] [batch_t 0.325 (0.334)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-04 22:03:40,457 - Train: 8.98% [443800/4942000] [89.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 22:04:14,053 - Train: 8.98% [443900/4942000] [89.8/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:04:49,376 - Train: 8.98% [444000/4942000] [89.8/1000.0] [batch_t 0.324 (0.353)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 22:05:23,191 - Train: 8.99% [444100/4942000] [89.9/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 22:05:55,947 - Train: 8.99% [444200/4942000] [89.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 22:06:29,939 - Train: 8.99% [444300/4942000] [89.9/1000.0] [batch_t 0.319 (0.340)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-04 22:07:02,839 - Train: 8.99% [444400/4942000] [89.9/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 22:07:36,291 - Train: 8.99% [444500/4942000] [89.9/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 22:08:10,208 - Train: 9.00% [444600/4942000] [90.0/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 22:08:43,011 - Train: 9.00% [444700/4942000] [90.0/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 22:09:10,424 - ==> Total time: 2 days, 4:11:49 Eta: 21 days, 23:46:14 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 22:09:19,317 - Train: 9.00% [444800/4942000] [90.0/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 22:09:52,132 - Train: 9.00% [444900/4942000] [90.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:10:26,000 - Train: 9.00% [445000/4942000] [90.0/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 22:10:58,808 - Train: 9.01% [445100/4942000] [90.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 22:11:33,507 - Train: 9.01% [445200/4942000] [90.1/1000.0] [batch_t 0.327 (0.347)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 22:12:07,935 - Train: 9.01% [445300/4942000] [90.1/1000.0] [batch_t 0.327 (0.344)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 22:12:40,717 - Train: 9.01% [445400/4942000] [90.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 22:13:14,763 - Train: 9.01% [445500/4942000] [90.1/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 22:13:47,520 - Train: 9.02% [445600/4942000] [90.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 22:14:21,232 - Train: 9.02% [445700/4942000] [90.2/1000.0] [batch_t 0.326 (0.337)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 22:14:54,029 - Train: 9.02% [445800/4942000] [90.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 22:15:28,292 - Train: 9.02% [445900/4942000] [90.2/1000.0] [batch_t 0.326 (0.343)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 22:16:01,080 - Train: 9.02% [446000/4942000] [90.2/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 22:16:34,971 - Train: 9.03% [446100/4942000] [90.3/1000.0] [batch_t 0.321 (0.339)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-04 22:17:10,101 - Train: 9.03% [446200/4942000] [90.3/1000.0] [batch_t 0.330 (0.351)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:17:42,886 - Train: 9.03% [446300/4942000] [90.3/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 22:18:16,469 - Train: 9.03% [446400/4942000] [90.3/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:18:49,274 - Train: 9.03% [446500/4942000] [90.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 22:19:23,195 - Train: 9.04% [446600/4942000] [90.4/1000.0] [batch_t 0.324 (0.339)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 22:19:56,037 - Train: 9.04% [446700/4942000] [90.4/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 22:20:29,850 - Train: 9.04% [446800/4942000] [90.4/1000.0] [batch_t 0.325 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 22:21:02,699 - Train: 9.04% [446900/4942000] [90.4/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 22:21:35,490 - Train: 9.04% [447000/4942000] [90.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 22:22:08,933 - Train: 9.05% [447100/4942000] [90.5/1000.0] [batch_t 0.328 (0.334)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 22:22:41,821 - Train: 9.05% [447200/4942000] [90.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 22:23:14,932 - Train: 9.05% [447300/4942000] [90.5/1000.0] [batch_t 0.323 (0.331)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 22:23:47,764 - Train: 9.05% [447400/4942000] [90.5/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 22:24:20,703 - Train: 9.06% [447500/4942000] [90.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 22:24:53,563 - Train: 9.06% [447600/4942000] [90.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 22:25:27,097 - Train: 9.06% [447700/4942000] [90.6/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 22:25:59,871 - Train: 9.06% [447800/4942000] [90.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 22:26:32,699 - Train: 9.06% [447900/4942000] [90.6/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 22:27:05,536 - Train: 9.07% [448000/4942000] [90.7/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 22:27:38,418 - Train: 9.07% [448100/4942000] [90.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 22:28:12,016 - Train: 9.07% [448200/4942000] [90.7/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 22:28:44,800 - Train: 9.07% [448300/4942000] [90.7/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 22:29:18,466 - Train: 9.07% [448400/4942000] [90.7/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 22:29:51,226 - Train: 9.08% [448500/4942000] [90.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 22:30:24,137 - Train: 9.08% [448600/4942000] [90.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 22:30:56,912 - Train: 9.08% [448700/4942000] [90.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 22:31:30,565 - Train: 9.08% [448800/4942000] [90.8/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 22:32:03,437 - Train: 9.08% [448900/4942000] [90.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-04 22:32:38,581 - Train: 9.09% [449000/4942000] [90.9/1000.0] [batch_t 0.330 (0.351)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:33:12,607 - Train: 9.09% [449100/4942000] [90.9/1000.0] [batch_t 0.335 (0.340)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 22:33:45,373 - Train: 9.09% [449200/4942000] [90.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 22:34:18,110 - Train: 9.09% [449300/4942000] [90.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 22:34:50,877 - Train: 9.09% [449400/4942000] [90.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 22:35:24,833 - Train: 9.10% [449500/4942000] [91.0/1000.0] [batch_t 0.332 (0.339)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 22:35:57,582 - Train: 9.10% [449600/4942000] [91.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 22:36:31,787 - Train: 9.10% [449700/4942000] [91.0/1000.0] [batch_t 0.330 (0.342)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:36:39,010 - ==> Total time: 2 days, 4:39:18 Eta: 21 days, 21:58:18 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 22:37:06,694 - Train: 9.10% [449800/4942000] [91.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 22:37:39,648 - Train: 9.10% [449900/4942000] [91.0/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 22:38:14,209 - Train: 9.11% [450000/4942000] [91.1/1000.0] [batch_t 0.329 (0.346)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-04 22:38:46,940 - Train: 9.11% [450100/4942000] [91.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 22:39:20,417 - Train: 9.11% [450200/4942000] [91.1/1000.0] [batch_t 0.333 (0.335)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 22:39:53,259 - Train: 9.11% [450300/4942000] [91.1/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 22:40:26,143 - Train: 9.11% [450400/4942000] [91.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 22:40:58,900 - Train: 9.12% [450500/4942000] [91.2/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:41:31,681 - Train: 9.12% [450600/4942000] [91.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 22:42:04,465 - Train: 9.12% [450700/4942000] [91.2/1000.0] [batch_t 0.341 (0.328)] [data_t 0.013] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:42:38,117 - Train: 9.12% [450800/4942000] [91.2/1000.0] [batch_t 0.326 (0.336)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 22:43:11,623 - Train: 9.12% [450900/4942000] [91.2/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 22:43:44,462 - Train: 9.13% [451000/4942000] [91.3/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 22:44:19,022 - Train: 9.13% [451100/4942000] [91.3/1000.0] [batch_t 0.329 (0.346)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 22:44:51,800 - Train: 9.13% [451200/4942000] [91.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-04 22:45:26,078 - Train: 9.13% [451300/4942000] [91.3/1000.0] [batch_t 0.327 (0.343)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 22:45:59,026 - Train: 9.13% [451400/4942000] [91.3/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 22:46:32,939 - Train: 9.14% [451500/4942000] [91.4/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 22:47:06,387 - Train: 9.14% [451600/4942000] [91.4/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 22:47:39,198 - Train: 9.14% [451700/4942000] [91.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 22:48:12,712 - Train: 9.14% [451800/4942000] [91.4/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 22:48:45,516 - Train: 9.14% [451900/4942000] [91.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 22:49:19,315 - Train: 9.15% [452000/4942000] [91.5/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 22:49:52,117 - Train: 9.15% [452100/4942000] [91.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 22:50:25,994 - Train: 9.15% [452200/4942000] [91.5/1000.0] [batch_t 0.322 (0.339)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-04 22:50:58,826 - Train: 9.15% [452300/4942000] [91.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 22:51:32,748 - Train: 9.15% [452400/4942000] [91.5/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 22:52:06,856 - Train: 9.16% [452500/4942000] [91.6/1000.0] [batch_t 0.399 (0.341)] [data_t 0.073] [optim_t 0.326] [lr 0.005000] 2024-04-04 22:52:39,829 - Train: 9.16% [452600/4942000] [91.6/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 22:53:12,683 - Train: 9.16% [452700/4942000] [91.6/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 22:53:45,632 - Train: 9.16% [452800/4942000] [91.6/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 22:54:20,412 - Train: 9.16% [452900/4942000] [91.6/1000.0] [batch_t 0.329 (0.348)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 22:54:53,181 - Train: 9.17% [453000/4942000] [91.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 22:55:25,934 - Train: 9.17% [453100/4942000] [91.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 22:55:58,731 - Train: 9.17% [453200/4942000] [91.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 22:56:32,411 - Train: 9.17% [453300/4942000] [91.7/1000.0] [batch_t 0.326 (0.337)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-04 22:57:06,791 - Train: 9.17% [453400/4942000] [91.7/1000.0] [batch_t 1.809 (0.344)] [data_t 1.481] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:57:39,617 - Train: 9.18% [453500/4942000] [91.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-04 22:58:13,911 - Train: 9.18% [453600/4942000] [91.8/1000.0] [batch_t 0.328 (0.343)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 22:58:46,703 - Train: 9.18% [453700/4942000] [91.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 22:59:21,222 - Train: 9.18% [453800/4942000] [91.8/1000.0] [batch_t 0.330 (0.345)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:59:58,058 - Train: 9.18% [453900/4942000] [91.8/1000.0] [batch_t 0.327 (0.368)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 23:00:37,369 - Train: 9.19% [454000/4942000] [91.9/1000.0] [batch_t 0.330 (0.393)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 23:01:12,954 - Train: 9.19% [454100/4942000] [91.9/1000.0] [batch_t 0.326 (0.356)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-04 23:01:45,779 - Train: 9.19% [454200/4942000] [91.9/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 23:02:20,079 - Train: 9.19% [454300/4942000] [91.9/1000.0] [batch_t 0.328 (0.343)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 23:02:52,904 - Train: 9.19% [454400/4942000] [91.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 23:03:28,533 - Train: 9.20% [454500/4942000] [92.0/1000.0] [batch_t 0.326 (0.356)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 23:04:01,292 - Train: 9.20% [454600/4942000] [92.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 23:04:22,466 - ==> Total time: 2 days, 5:07:01 Eta: 21 days, 20:14:34 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 23:04:36,151 - Train: 9.20% [454700/4942000] [92.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 23:05:12,142 - Train: 9.20% [454800/4942000] [92.0/1000.0] [batch_t 0.327 (0.360)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 23:05:45,044 - Train: 9.20% [454900/4942000] [92.0/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-04 23:06:36,023 - Train: 9.21% [455000/4942000] [92.1/1000.0] [batch_t 0.333 (0.510)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-04 23:07:11,938 - Train: 9.21% [455100/4942000] [92.1/1000.0] [batch_t 0.328 (0.359)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 23:07:44,830 - Train: 9.21% [455200/4942000] [92.1/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 23:08:24,698 - Train: 9.21% [455300/4942000] [92.1/1000.0] [batch_t 0.328 (0.399)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 23:08:57,597 - Train: 9.21% [455400/4942000] [92.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 23:09:37,967 - Train: 9.22% [455500/4942000] [92.2/1000.0] [batch_t 0.327 (0.404)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 23:10:20,371 - Train: 9.22% [455600/4942000] [92.2/1000.0] [batch_t 0.477 (0.424)] [data_t 0.154] [optim_t 0.323] [lr 0.005000] 2024-04-04 23:11:34,898 - Train: 9.22% [455700/4942000] [92.2/1000.0] [batch_t 1.874 (0.745)] [data_t 1.552] [optim_t 0.322] [lr 0.005000] 2024-04-04 23:12:22,819 - Train: 9.22% [455800/4942000] [92.2/1000.0] [batch_t 0.327 (0.479)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 23:12:55,636 - Train: 9.23% [455900/4942000] [92.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 23:13:39,141 - Train: 9.23% [456000/4942000] [92.3/1000.0] [batch_t 0.328 (0.435)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 23:14:20,370 - Train: 9.23% [456100/4942000] [92.3/1000.0] [batch_t 0.328 (0.412)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 23:14:54,005 - Train: 9.23% [456200/4942000] [92.3/1000.0] [batch_t 0.326 (0.336)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 23:15:30,364 - Train: 9.23% [456300/4942000] [92.3/1000.0] [batch_t 0.328 (0.363)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 23:16:03,128 - Train: 9.24% [456400/4942000] [92.4/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 23:16:38,109 - Train: 9.24% [456500/4942000] [92.4/1000.0] [batch_t 0.328 (0.350)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 23:17:13,237 - Train: 9.24% [456600/4942000] [92.4/1000.0] [batch_t 0.327 (0.351)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 23:17:46,054 - Train: 9.24% [456700/4942000] [92.4/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 23:18:20,629 - Train: 9.24% [456800/4942000] [92.4/1000.0] [batch_t 0.328 (0.346)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 23:18:53,399 - Train: 9.25% [456900/4942000] [92.5/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 23:19:28,603 - Train: 9.25% [457000/4942000] [92.5/1000.0] [batch_t 0.327 (0.352)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 23:20:01,381 - Train: 9.25% [457100/4942000] [92.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 23:20:36,117 - Train: 9.25% [457200/4942000] [92.5/1000.0] [batch_t 0.325 (0.347)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 23:21:11,235 - Train: 9.25% [457300/4942000] [92.5/1000.0] [batch_t 0.327 (0.351)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 23:21:43,956 - Train: 9.26% [457400/4942000] [92.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 23:22:17,788 - Train: 9.26% [457500/4942000] [92.6/1000.0] [batch_t 0.331 (0.338)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 23:22:50,549 - Train: 9.26% [457600/4942000] [92.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 23:23:25,412 - Train: 9.26% [457700/4942000] [92.6/1000.0] [batch_t 0.328 (0.349)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 23:23:58,331 - Train: 9.26% [457800/4942000] [92.6/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 23:24:32,726 - Train: 9.27% [457900/4942000] [92.7/1000.0] [batch_t 0.327 (0.344)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 23:25:06,658 - Train: 9.27% [458000/4942000] [92.7/1000.0] [batch_t 0.333 (0.339)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 23:25:39,582 - Train: 9.27% [458100/4942000] [92.7/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 23:26:13,214 - Train: 9.27% [458200/4942000] [92.7/1000.0] [batch_t 0.332 (0.336)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-04 23:26:46,034 - Train: 9.27% [458300/4942000] [92.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 23:27:19,983 - Train: 9.28% [458400/4942000] [92.8/1000.0] [batch_t 0.324 (0.339)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 23:27:52,872 - Train: 9.28% [458500/4942000] [92.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 23:28:28,526 - Train: 9.28% [458600/4942000] [92.8/1000.0] [batch_t 0.327 (0.356)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 23:29:01,258 - Train: 9.28% [458700/4942000] [92.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 23:29:34,915 - Train: 9.28% [458800/4942000] [92.8/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 23:30:11,083 - Train: 9.29% [458900/4942000] [92.9/1000.0] [batch_t 0.327 (0.362)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 23:30:43,850 - Train: 9.29% [459000/4942000] [92.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 23:31:17,090 - Train: 9.29% [459100/4942000] [92.9/1000.0] [batch_t 0.326 (0.332)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 23:31:49,814 - Train: 9.29% [459200/4942000] [92.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 23:32:25,471 - Train: 9.29% [459300/4942000] [92.9/1000.0] [batch_t 0.324 (0.356)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 23:32:58,217 - Train: 9.30% [459400/4942000] [93.0/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 23:33:32,193 - Train: 9.30% [459500/4942000] [93.0/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 23:34:04,973 - Train: 9.30% [459600/4942000] [93.0/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 23:34:08,399 - ==> Total time: 2 days, 5:36:47 Eta: 21 days, 18:52:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-04 23:34:41,004 - Train: 9.30% [459700/4942000] [93.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 23:35:15,344 - Train: 9.30% [459800/4942000] [93.0/1000.0] [batch_t 0.329 (0.343)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 23:35:48,139 - Train: 9.31% [459900/4942000] [93.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 23:36:21,592 - Train: 9.31% [460000/4942000] [93.1/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 23:36:54,366 - Train: 9.31% [460100/4942000] [93.1/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 23:37:28,593 - Train: 9.31% [460200/4942000] [93.1/1000.0] [batch_t 0.328 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 23:38:01,346 - Train: 9.31% [460300/4942000] [93.1/1000.0] [batch_t 0.331 (0.327)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-04 23:38:35,407 - Train: 9.32% [460400/4942000] [93.2/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 23:39:09,143 - Train: 9.32% [460500/4942000] [93.2/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 23:39:41,933 - Train: 9.32% [460600/4942000] [93.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 23:40:15,368 - Train: 9.32% [460700/4942000] [93.2/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 23:40:48,143 - Train: 9.32% [460800/4942000] [93.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 23:41:21,371 - Train: 9.33% [460900/4942000] [93.3/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 23:41:54,117 - Train: 9.33% [461000/4942000] [93.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 23:42:27,554 - Train: 9.33% [461100/4942000] [93.3/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 23:43:00,289 - Train: 9.33% [461200/4942000] [93.3/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 23:43:33,915 - Train: 9.33% [461300/4942000] [93.3/1000.0] [batch_t 0.325 (0.336)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 23:44:07,696 - Train: 9.34% [461400/4942000] [93.4/1000.0] [batch_t 0.331 (0.338)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 23:44:40,425 - Train: 9.34% [461500/4942000] [93.4/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 23:45:15,738 - Train: 9.34% [461600/4942000] [93.4/1000.0] [batch_t 0.326 (0.353)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 23:45:48,499 - Train: 9.34% [461700/4942000] [93.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 23:46:22,241 - Train: 9.34% [461800/4942000] [93.4/1000.0] [batch_t 0.326 (0.337)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 23:46:55,062 - Train: 9.35% [461900/4942000] [93.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 23:47:28,594 - Train: 9.35% [462000/4942000] [93.5/1000.0] [batch_t 0.328 (0.335)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-04 23:48:01,486 - Train: 9.35% [462100/4942000] [93.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 23:48:35,785 - Train: 9.35% [462200/4942000] [93.5/1000.0] [batch_t 0.328 (0.343)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 23:49:09,635 - Train: 9.35% [462300/4942000] [93.5/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 23:49:42,479 - Train: 9.36% [462400/4942000] [93.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 23:50:15,853 - Train: 9.36% [462500/4942000] [93.6/1000.0] [batch_t 0.320 (0.334)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-04 23:50:48,697 - Train: 9.36% [462600/4942000] [93.6/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 23:51:21,726 - Train: 9.36% [462700/4942000] [93.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 23:51:54,571 - Train: 9.36% [462800/4942000] [93.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 23:52:28,370 - Train: 9.37% [462900/4942000] [93.7/1000.0] [batch_t 0.325 (0.338)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 23:53:01,194 - Train: 9.37% [463000/4942000] [93.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 23:53:35,257 - Train: 9.37% [463100/4942000] [93.7/1000.0] [batch_t 0.333 (0.341)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 23:54:08,065 - Train: 9.37% [463200/4942000] [93.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 23:54:40,938 - Train: 9.37% [463300/4942000] [93.7/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 23:55:14,963 - Train: 9.38% [463400/4942000] [93.8/1000.0] [batch_t 0.330 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 23:55:47,768 - Train: 9.38% [463500/4942000] [93.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-04 23:56:21,306 - Train: 9.38% [463600/4942000] [93.8/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 23:56:54,153 - Train: 9.38% [463700/4942000] [93.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 23:57:27,484 - Train: 9.38% [463800/4942000] [93.8/1000.0] [batch_t 0.338 (0.333)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-04 23:58:00,338 - Train: 9.39% [463900/4942000] [93.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 23:58:34,878 - Train: 9.39% [464000/4942000] [93.9/1000.0] [batch_t 0.325 (0.345)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-04 23:59:10,095 - Train: 9.39% [464100/4942000] [93.9/1000.0] [batch_t 0.329 (0.352)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 23:59:43,184 - Train: 9.39% [464200/4942000] [93.9/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 00:00:17,086 - Train: 9.39% [464300/4942000] [93.9/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:00:49,946 - Train: 9.40% [464400/4942000] [94.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 00:01:23,691 - Train: 9.40% [464500/4942000] [94.0/1000.0] [batch_t 0.325 (0.337)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 00:01:39,476 - ==> Total time: 2 days, 6:04:18 Eta: 21 days, 17:09:38 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-05 00:01:58,935 - Train: 9.40% [464600/4942000] [94.0/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:02:33,650 - Train: 9.40% [464700/4942000] [94.0/1000.0] [batch_t 0.328 (0.347)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:03:07,400 - Train: 9.41% [464800/4942000] [94.1/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 00:03:40,236 - Train: 9.41% [464900/4942000] [94.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 00:04:14,087 - Train: 9.41% [465000/4942000] [94.1/1000.0] [batch_t 0.323 (0.338)] [data_t 0.003] [optim_t 0.321] [lr 0.005000] 2024-04-05 00:04:46,944 - Train: 9.41% [465100/4942000] [94.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-05 00:05:22,560 - Train: 9.41% [465200/4942000] [94.1/1000.0] [batch_t 0.329 (0.356)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:05:55,340 - Train: 9.42% [465300/4942000] [94.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 00:06:29,442 - Train: 9.42% [465400/4942000] [94.2/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 00:07:02,387 - Train: 9.42% [465500/4942000] [94.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-05 00:07:36,525 - Train: 9.42% [465600/4942000] [94.2/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 00:08:10,875 - Train: 9.42% [465700/4942000] [94.2/1000.0] [batch_t 0.330 (0.343)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 00:08:43,759 - Train: 9.43% [465800/4942000] [94.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 00:09:17,447 - Train: 9.43% [465900/4942000] [94.3/1000.0] [batch_t 0.322 (0.337)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-05 00:09:50,221 - Train: 9.43% [466000/4942000] [94.3/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 00:10:24,641 - Train: 9.43% [466100/4942000] [94.3/1000.0] [batch_t 0.328 (0.344)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:10:57,445 - Train: 9.43% [466200/4942000] [94.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 00:11:31,241 - Train: 9.44% [466300/4942000] [94.4/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 00:12:04,045 - Train: 9.44% [466400/4942000] [94.4/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-05 00:12:37,119 - Train: 9.44% [466500/4942000] [94.4/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 00:13:10,469 - Train: 9.44% [466600/4942000] [94.4/1000.0] [batch_t 0.325 (0.333)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 00:13:43,211 - Train: 9.44% [466700/4942000] [94.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 00:14:16,909 - Train: 9.45% [466800/4942000] [94.5/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 00:14:49,826 - Train: 9.45% [466900/4942000] [94.5/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-05 00:15:24,030 - Train: 9.45% [467000/4942000] [94.5/1000.0] [batch_t 0.328 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:15:56,767 - Train: 9.45% [467100/4942000] [94.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 00:16:30,608 - Train: 9.45% [467200/4942000] [94.5/1000.0] [batch_t 0.325 (0.338)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 00:17:03,404 - Train: 9.46% [467300/4942000] [94.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-05 00:17:36,383 - Train: 9.46% [467400/4942000] [94.6/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 00:18:09,928 - Train: 9.46% [467500/4942000] [94.6/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 00:18:42,792 - Train: 9.46% [467600/4942000] [94.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 00:19:17,319 - Train: 9.46% [467700/4942000] [94.6/1000.0] [batch_t 0.327 (0.345)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 00:19:50,058 - Train: 9.47% [467800/4942000] [94.7/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 00:20:23,462 - Train: 9.47% [467900/4942000] [94.7/1000.0] [batch_t 0.324 (0.334)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 00:20:56,279 - Train: 9.47% [468000/4942000] [94.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 00:21:30,164 - Train: 9.47% [468100/4942000] [94.7/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 00:22:02,986 - Train: 9.47% [468200/4942000] [94.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 00:22:36,053 - Train: 9.48% [468300/4942000] [94.8/1000.0] [batch_t 0.333 (0.331)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 00:23:09,970 - Train: 9.48% [468400/4942000] [94.8/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 00:23:42,800 - Train: 9.48% [468500/4942000] [94.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 00:24:17,992 - Train: 9.48% [468600/4942000] [94.8/1000.0] [batch_t 0.328 (0.352)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:24:50,786 - Train: 9.48% [468700/4942000] [94.8/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 00:25:23,571 - Train: 9.49% [468800/4942000] [94.9/1000.0] [batch_t 0.331 (0.328)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-05 00:25:56,375 - Train: 9.49% [468900/4942000] [94.9/1000.0] [batch_t 0.325 (0.328)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-05 00:26:29,606 - Train: 9.49% [469000/4942000] [94.9/1000.0] [batch_t 0.325 (0.332)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 00:27:02,374 - Train: 9.49% [469100/4942000] [94.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:27:36,233 - Train: 9.49% [469200/4942000] [94.9/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:28:09,326 - Train: 9.50% [469300/4942000] [95.0/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 00:28:42,063 - Train: 9.50% [469400/4942000] [95.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 00:29:11,876 - ==> Total time: 2 days, 6:31:51 Eta: 21 days, 15:28:41 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-05 00:29:17,025 - Train: 9.50% [469500/4942000] [95.0/1000.0] [batch_t 0.325 (0.335)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 00:29:49,828 - Train: 9.50% [469600/4942000] [95.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 00:30:22,702 - Train: 9.50% [469700/4942000] [95.0/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 00:30:55,472 - Train: 9.51% [469800/4942000] [95.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 00:31:29,539 - Train: 9.51% [469900/4942000] [95.1/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:32:02,326 - Train: 9.51% [470000/4942000] [95.1/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 00:32:37,125 - Train: 9.51% [470100/4942000] [95.1/1000.0] [batch_t 0.328 (0.348)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:33:11,054 - Train: 9.51% [470200/4942000] [95.1/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 00:33:43,816 - Train: 9.52% [470300/4942000] [95.2/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 00:34:16,559 - Train: 9.52% [470400/4942000] [95.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 00:34:49,307 - Train: 9.52% [470500/4942000] [95.2/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 00:35:23,954 - Train: 9.52% [470600/4942000] [95.2/1000.0] [batch_t 0.337 (0.346)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-05 00:35:56,829 - Train: 9.52% [470700/4942000] [95.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 00:36:31,218 - Train: 9.53% [470800/4942000] [95.3/1000.0] [batch_t 0.329 (0.344)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 00:37:04,030 - Train: 9.53% [470900/4942000] [95.3/1000.0] [batch_t 0.333 (0.328)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-05 00:37:39,300 - Train: 9.53% [471000/4942000] [95.3/1000.0] [batch_t 0.324 (0.353)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 00:38:12,438 - Train: 9.53% [471100/4942000] [95.3/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 00:38:45,258 - Train: 9.53% [471200/4942000] [95.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:39:18,218 - Train: 9.54% [471300/4942000] [95.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 00:39:51,070 - Train: 9.54% [471400/4942000] [95.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 00:40:25,709 - Train: 9.54% [471500/4942000] [95.4/1000.0] [batch_t 0.327 (0.346)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 00:40:58,462 - Train: 9.54% [471600/4942000] [95.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 00:41:31,203 - Train: 9.54% [471700/4942000] [95.4/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 00:42:03,932 - Train: 9.55% [471800/4942000] [95.5/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 00:42:38,823 - Train: 9.55% [471900/4942000] [95.5/1000.0] [batch_t 0.327 (0.349)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 00:43:11,569 - Train: 9.55% [472000/4942000] [95.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:43:44,321 - Train: 9.55% [472100/4942000] [95.5/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 00:44:18,568 - Train: 9.55% [472200/4942000] [95.5/1000.0] [batch_t 0.324 (0.342)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 00:44:51,341 - Train: 9.56% [472300/4942000] [95.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 00:45:25,189 - Train: 9.56% [472400/4942000] [95.6/1000.0] [batch_t 0.322 (0.338)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-05 00:45:58,115 - Train: 9.56% [472500/4942000] [95.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:46:30,867 - Train: 9.56% [472600/4942000] [95.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:47:03,805 - Train: 9.56% [472700/4942000] [95.6/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 00:47:37,234 - Train: 9.57% [472800/4942000] [95.7/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 00:48:11,476 - Train: 9.57% [472900/4942000] [95.7/1000.0] [batch_t 0.327 (0.342)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 00:48:44,369 - Train: 9.57% [473000/4942000] [95.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 00:49:18,504 - Train: 9.57% [473100/4942000] [95.7/1000.0] [batch_t 0.331 (0.341)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-05 00:49:51,409 - Train: 9.58% [473200/4942000] [95.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:50:24,960 - Train: 9.58% [473300/4942000] [95.8/1000.0] [batch_t 0.332 (0.335)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-05 00:50:57,764 - Train: 9.58% [473400/4942000] [95.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 00:51:31,867 - Train: 9.58% [473500/4942000] [95.8/1000.0] [batch_t 0.331 (0.341)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 00:52:04,708 - Train: 9.58% [473600/4942000] [95.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 00:52:38,343 - Train: 9.59% [473700/4942000] [95.9/1000.0] [batch_t 0.331 (0.336)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-05 00:53:11,504 - Train: 9.59% [473800/4942000] [95.9/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:53:44,334 - Train: 9.59% [473900/4942000] [95.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 00:54:18,316 - Train: 9.59% [474000/4942000] [95.9/1000.0] [batch_t 0.326 (0.340)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 00:54:51,212 - Train: 9.59% [474100/4942000] [95.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 00:55:25,119 - Train: 9.60% [474200/4942000] [96.0/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 00:55:58,066 - Train: 9.60% [474300/4942000] [96.0/1000.0] [batch_t 0.326 (0.329)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-05 00:56:30,979 - Train: 9.60% [474400/4942000] [96.0/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 00:56:41,503 - ==> Total time: 2 days, 6:59:20 Eta: 21 days, 13:48:49 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-05 00:57:06,183 - Train: 9.60% [474500/4942000] [96.0/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 00:57:38,997 - Train: 9.60% [474600/4942000] [96.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:58:12,939 - Train: 9.61% [474700/4942000] [96.1/1000.0] [batch_t 0.331 (0.339)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-05 00:58:45,791 - Train: 9.61% [474800/4942000] [96.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:59:19,572 - Train: 9.61% [474900/4942000] [96.1/1000.0] [batch_t 0.328 (0.338)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:59:52,359 - Train: 9.61% [475000/4942000] [96.1/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 01:00:25,168 - Train: 9.61% [475100/4942000] [96.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 01:00:58,006 - Train: 9.62% [475200/4942000] [96.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:01:32,762 - Train: 9.62% [475300/4942000] [96.2/1000.0] [batch_t 0.327 (0.347)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 01:02:05,861 - Train: 9.62% [475400/4942000] [96.2/1000.0] [batch_t 0.323 (0.331)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-05 01:02:38,640 - Train: 9.62% [475500/4942000] [96.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 01:03:11,966 - Train: 9.62% [475600/4942000] [96.2/1000.0] [batch_t 0.331 (0.333)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 01:03:44,756 - Train: 9.63% [475700/4942000] [96.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 01:04:19,879 - Train: 9.63% [475800/4942000] [96.3/1000.0] [batch_t 0.328 (0.351)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:04:52,807 - Train: 9.63% [475900/4942000] [96.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:05:25,633 - Train: 9.63% [476000/4942000] [96.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 01:05:58,449 - Train: 9.63% [476100/4942000] [96.3/1000.0] [batch_t 0.336 (0.328)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-05 01:06:33,368 - Train: 9.64% [476200/4942000] [96.4/1000.0] [batch_t 0.327 (0.349)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:07:07,348 - Train: 9.64% [476300/4942000] [96.4/1000.0] [batch_t 0.334 (0.340)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 01:07:40,252 - Train: 9.64% [476400/4942000] [96.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-05 01:08:13,459 - Train: 9.64% [476500/4942000] [96.4/1000.0] [batch_t 0.339 (0.332)] [data_t 0.002] [optim_t 0.337] [lr 0.005000] 2024-04-05 01:08:46,307 - Train: 9.64% [476600/4942000] [96.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 01:09:20,442 - Train: 9.65% [476700/4942000] [96.5/1000.0] [batch_t 0.324 (0.341)] [data_t 0.003] [optim_t 0.321] [lr 0.005000] 2024-04-05 01:09:53,428 - Train: 9.65% [476800/4942000] [96.5/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-05 01:10:27,498 - Train: 9.65% [476900/4942000] [96.5/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:11:00,286 - Train: 9.65% [477000/4942000] [96.5/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 01:11:33,060 - Train: 9.65% [477100/4942000] [96.5/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 01:12:06,630 - Train: 9.66% [477200/4942000] [96.6/1000.0] [batch_t 0.324 (0.336)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 01:12:39,427 - Train: 9.66% [477300/4942000] [96.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 01:13:12,259 - Train: 9.66% [477400/4942000] [96.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 01:13:45,125 - Train: 9.66% [477500/4942000] [96.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 01:14:18,112 - Train: 9.66% [477600/4942000] [96.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 01:14:50,923 - Train: 9.67% [477700/4942000] [96.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 01:15:24,916 - Train: 9.67% [477800/4942000] [96.7/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 01:15:57,678 - Train: 9.67% [477900/4942000] [96.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:16:31,488 - Train: 9.67% [478000/4942000] [96.7/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 01:17:04,297 - Train: 9.67% [478100/4942000] [96.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 01:17:38,514 - Train: 9.68% [478200/4942000] [96.8/1000.0] [batch_t 0.324 (0.342)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 01:18:11,771 - Train: 9.68% [478300/4942000] [96.8/1000.0] [batch_t 0.322 (0.332)] [data_t 0.003] [optim_t 0.320] [lr 0.005000] 2024-04-05 01:18:44,608 - Train: 9.68% [478400/4942000] [96.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:19:19,714 - Train: 9.68% [478500/4942000] [96.8/1000.0] [batch_t 0.326 (0.351)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 01:19:52,530 - Train: 9.68% [478600/4942000] [96.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:20:25,884 - Train: 9.69% [478700/4942000] [96.9/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 01:20:58,678 - Train: 9.69% [478800/4942000] [96.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 01:21:31,781 - Train: 9.69% [478900/4942000] [96.9/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 01:22:06,691 - Train: 9.69% [479000/4942000] [96.9/1000.0] [batch_t 2.401 (0.349)] [data_t 2.079] [optim_t 0.322] [lr 0.005000] 2024-04-05 01:22:39,479 - Train: 9.69% [479100/4942000] [96.9/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 01:23:12,255 - Train: 9.70% [479200/4942000] [97.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:23:45,092 - Train: 9.70% [479300/4942000] [97.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:24:10,718 - ==> Total time: 2 days, 7:26:49 Eta: 21 days, 12:10:24 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-05 01:24:21,333 - Train: 9.70% [479400/4942000] [97.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:24:54,338 - Train: 9.70% [479500/4942000] [97.0/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 01:25:27,999 - Train: 9.70% [479600/4942000] [97.0/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:26:00,791 - Train: 9.71% [479700/4942000] [97.1/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 01:26:34,502 - Train: 9.71% [479800/4942000] [97.1/1000.0] [batch_t 0.331 (0.337)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 01:27:07,339 - Train: 9.71% [479900/4942000] [97.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 01:27:40,091 - Train: 9.71% [480000/4942000] [97.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:28:12,952 - Train: 9.71% [480100/4942000] [97.1/1000.0] [batch_t 0.325 (0.329)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-05 01:28:45,720 - Train: 9.72% [480200/4942000] [97.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 01:29:19,101 - Train: 9.72% [480300/4942000] [97.2/1000.0] [batch_t 0.325 (0.334)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 01:29:51,922 - Train: 9.72% [480400/4942000] [97.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 01:30:24,856 - Train: 9.72% [480500/4942000] [97.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 01:30:57,674 - Train: 9.72% [480600/4942000] [97.2/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 01:31:31,617 - Train: 9.73% [480700/4942000] [97.3/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 01:32:04,490 - Train: 9.73% [480800/4942000] [97.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 01:32:38,270 - Train: 9.73% [480900/4942000] [97.3/1000.0] [batch_t 0.331 (0.338)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 01:33:11,241 - Train: 9.73% [481000/4942000] [97.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 01:33:44,039 - Train: 9.73% [481100/4942000] [97.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:34:16,833 - Train: 9.74% [481200/4942000] [97.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:34:49,667 - Train: 9.74% [481300/4942000] [97.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 01:35:23,345 - Train: 9.74% [481400/4942000] [97.4/1000.0] [batch_t 0.325 (0.337)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 01:35:56,153 - Train: 9.74% [481500/4942000] [97.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 01:36:28,973 - Train: 9.75% [481600/4942000] [97.5/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 01:37:01,781 - Train: 9.75% [481700/4942000] [97.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:37:35,841 - Train: 9.75% [481800/4942000] [97.5/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:38:08,633 - Train: 9.75% [481900/4942000] [97.5/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 01:38:41,394 - Train: 9.75% [482000/4942000] [97.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:39:14,388 - Train: 9.76% [482100/4942000] [97.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:39:47,188 - Train: 9.76% [482200/4942000] [97.6/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 01:40:19,964 - Train: 9.76% [482300/4942000] [97.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 01:40:52,813 - Train: 9.76% [482400/4942000] [97.6/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 01:41:25,813 - Train: 9.76% [482500/4942000] [97.6/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 01:41:58,673 - Train: 9.77% [482600/4942000] [97.7/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 01:42:31,544 - Train: 9.77% [482700/4942000] [97.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-05 01:43:04,293 - Train: 9.77% [482800/4942000] [97.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 01:43:37,044 - Train: 9.77% [482900/4942000] [97.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 01:44:09,894 - Train: 9.77% [483000/4942000] [97.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:44:42,660 - Train: 9.78% [483100/4942000] [97.8/1000.0] [batch_t 0.331 (0.328)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-05 01:45:15,468 - Train: 9.78% [483200/4942000] [97.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 01:45:48,246 - Train: 9.78% [483300/4942000] [97.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:46:20,999 - Train: 9.78% [483400/4942000] [97.8/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 01:46:53,823 - Train: 9.78% [483500/4942000] [97.8/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 01:47:27,028 - Train: 9.79% [483600/4942000] [97.9/1000.0] [batch_t 0.324 (0.332)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-05 01:47:59,868 - Train: 9.79% [483700/4942000] [97.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 01:48:32,770 - Train: 9.79% [483800/4942000] [97.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 01:49:05,839 - Train: 9.79% [483900/4942000] [97.9/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:49:38,701 - Train: 9.79% [484000/4942000] [97.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:50:11,474 - Train: 9.80% [484100/4942000] [98.0/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 01:50:44,293 - Train: 9.80% [484200/4942000] [98.0/1000.0] [batch_t 0.323 (0.328)] [data_t 0.003] [optim_t 0.320] [lr 0.005000] 2024-04-05 01:51:17,372 - Train: 9.80% [484300/4942000] [98.0/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:51:22,639 - ==> Total time: 2 days, 7:54:01 Eta: 21 days, 10:30:46 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-05 01:51:52,055 - Train: 9.80% [484400/4942000] [98.0/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 01:52:25,122 - Train: 9.80% [484500/4942000] [98.0/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 01:52:57,948 - Train: 9.81% [484600/4942000] [98.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-05 01:53:31,581 - Train: 9.81% [484700/4942000] [98.1/1000.0] [batch_t 0.331 (0.336)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 01:54:04,407 - Train: 9.81% [484800/4942000] [98.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:54:37,276 - Train: 9.81% [484900/4942000] [98.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:55:10,462 - Train: 9.81% [485000/4942000] [98.1/1000.0] [batch_t 0.325 (0.332)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 01:55:43,314 - Train: 9.82% [485100/4942000] [98.2/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 01:56:16,888 - Train: 9.82% [485200/4942000] [98.2/1000.0] [batch_t 0.326 (0.336)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 01:56:49,666 - Train: 9.82% [485300/4942000] [98.2/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 01:57:23,951 - Train: 9.82% [485400/4942000] [98.2/1000.0] [batch_t 0.324 (0.343)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 01:57:56,705 - Train: 9.82% [485500/4942000] [98.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 01:58:30,405 - Train: 9.83% [485600/4942000] [98.3/1000.0] [batch_t 0.323 (0.337)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 01:59:03,260 - Train: 9.83% [485700/4942000] [98.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 01:59:36,893 - Train: 9.83% [485800/4942000] [98.3/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 02:00:10,688 - Train: 9.83% [485900/4942000] [98.3/1000.0] [batch_t 0.324 (0.338)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 02:00:43,541 - Train: 9.83% [486000/4942000] [98.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:01:17,624 - Train: 9.84% [486100/4942000] [98.4/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 02:01:50,486 - Train: 9.84% [486200/4942000] [98.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-05 02:02:24,306 - Train: 9.84% [486300/4942000] [98.4/1000.0] [batch_t 0.332 (0.338)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 02:02:57,110 - Train: 9.84% [486400/4942000] [98.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-05 02:03:32,586 - Train: 9.84% [486500/4942000] [98.4/1000.0] [batch_t 0.329 (0.355)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 02:04:05,561 - Train: 9.85% [486600/4942000] [98.5/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 02:04:38,395 - Train: 9.85% [486700/4942000] [98.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 02:05:11,228 - Train: 9.85% [486800/4942000] [98.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 02:05:44,066 - Train: 9.85% [486900/4942000] [98.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:06:18,383 - Train: 9.85% [487000/4942000] [98.5/1000.0] [batch_t 0.326 (0.343)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 02:06:51,192 - Train: 9.86% [487100/4942000] [98.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-05 02:07:25,426 - Train: 9.86% [487200/4942000] [98.6/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 02:07:58,235 - Train: 9.86% [487300/4942000] [98.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 02:08:31,522 - Train: 9.86% [487400/4942000] [98.6/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 02:09:04,265 - Train: 9.86% [487500/4942000] [98.6/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 02:09:39,661 - Train: 9.87% [487600/4942000] [98.7/1000.0] [batch_t 0.335 (0.354)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 02:10:13,408 - Train: 9.87% [487700/4942000] [98.7/1000.0] [batch_t 0.325 (0.337)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 02:10:46,155 - Train: 9.87% [487800/4942000] [98.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:11:18,952 - Train: 9.87% [487900/4942000] [98.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 02:11:51,864 - Train: 9.87% [488000/4942000] [98.7/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 02:12:24,697 - Train: 9.88% [488100/4942000] [98.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 02:12:57,496 - Train: 9.88% [488200/4942000] [98.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 02:13:32,136 - Train: 9.88% [488300/4942000] [98.8/1000.0] [batch_t 0.329 (0.346)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 02:14:04,935 - Train: 9.88% [488400/4942000] [98.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:14:37,800 - Train: 9.88% [488500/4942000] [98.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:15:10,597 - Train: 9.89% [488600/4942000] [98.9/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-05 02:15:43,404 - Train: 9.89% [488700/4942000] [98.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:16:16,307 - Train: 9.89% [488800/4942000] [98.9/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 02:16:49,146 - Train: 9.89% [488900/4942000] [98.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 02:17:22,642 - Train: 9.89% [489000/4942000] [98.9/1000.0] [batch_t 0.323 (0.335)] [data_t 0.003] [optim_t 0.320] [lr 0.005000] 2024-04-05 02:17:55,391 - Train: 9.90% [489100/4942000] [99.0/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 02:18:29,477 - Train: 9.90% [489200/4942000] [99.0/1000.0] [batch_t 0.338 (0.341)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-05 02:18:48,539 - ==> Total time: 2 days, 8:21:27 Eta: 21 days, 8:54:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-05 02:19:04,226 - Train: 9.90% [489300/4942000] [99.0/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:19:36,988 - Train: 9.90% [489400/4942000] [99.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-05 02:20:10,860 - Train: 9.90% [489500/4942000] [99.0/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 02:20:43,637 - Train: 9.91% [489600/4942000] [99.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:21:17,099 - Train: 9.91% [489700/4942000] [99.1/1000.0] [batch_t 0.326 (0.335)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 02:21:49,869 - Train: 9.91% [489800/4942000] [99.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:22:22,619 - Train: 9.91% [489900/4942000] [99.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 02:22:55,465 - Train: 9.92% [490000/4942000] [99.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 02:23:29,222 - Train: 9.92% [490100/4942000] [99.2/1000.0] [batch_t 0.331 (0.337)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 02:24:02,000 - Train: 9.92% [490200/4942000] [99.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-05 02:24:35,839 - Train: 9.92% [490300/4942000] [99.2/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 02:25:09,937 - Train: 9.92% [490400/4942000] [99.2/1000.0] [batch_t 0.325 (0.341)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 02:25:42,689 - Train: 9.93% [490500/4942000] [99.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:26:15,674 - Train: 9.93% [490600/4942000] [99.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 02:26:48,422 - Train: 9.93% [490700/4942000] [99.3/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 02:27:22,410 - Train: 9.93% [490800/4942000] [99.3/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 02:27:55,168 - Train: 9.93% [490900/4942000] [99.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 02:28:29,293 - Train: 9.94% [491000/4942000] [99.4/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:29:02,060 - Train: 9.94% [491100/4942000] [99.4/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 02:29:35,355 - Train: 9.94% [491200/4942000] [99.4/1000.0] [batch_t 0.326 (0.333)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 02:30:09,293 - Train: 9.94% [491300/4942000] [99.4/1000.0] [batch_t 0.328 (0.339)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:30:42,127 - Train: 9.94% [491400/4942000] [99.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 02:31:15,399 - Train: 9.95% [491500/4942000] [99.5/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 02:31:48,305 - Train: 9.95% [491600/4942000] [99.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-05 02:32:21,188 - Train: 9.95% [491700/4942000] [99.5/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 02:32:54,099 - Train: 9.95% [491800/4942000] [99.5/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 02:33:27,948 - Train: 9.95% [491900/4942000] [99.5/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 02:34:00,786 - Train: 9.96% [492000/4942000] [99.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 02:34:33,700 - Train: 9.96% [492100/4942000] [99.6/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 02:35:07,651 - Train: 9.96% [492200/4942000] [99.6/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:35:40,495 - Train: 9.96% [492300/4942000] [99.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:36:13,342 - Train: 9.96% [492400/4942000] [99.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 02:36:46,204 - Train: 9.97% [492500/4942000] [99.7/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 02:37:19,261 - Train: 9.97% [492600/4942000] [99.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:37:52,145 - Train: 9.97% [492700/4942000] [99.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 02:38:25,950 - Train: 9.97% [492800/4942000] [99.7/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 02:38:58,857 - Train: 9.97% [492900/4942000] [99.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 02:39:31,653 - Train: 9.98% [493000/4942000] [99.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:40:04,490 - Train: 9.98% [493100/4942000] [99.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 02:40:37,272 - Train: 9.98% [493200/4942000] [99.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 02:41:10,091 - Train: 9.98% [493300/4942000] [99.8/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 02:41:42,898 - Train: 9.98% [493400/4942000] [99.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 02:42:16,537 - Train: 9.99% [493500/4942000] [99.9/1000.0] [batch_t 0.318 (0.336)] [data_t 0.002] [optim_t 0.316] [lr 0.005000] 2024-04-05 02:42:49,417 - Train: 9.99% [493600/4942000] [99.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 02:43:22,433 - Train: 9.99% [493700/4942000] [99.9/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:43:55,189 - Train: 9.99% [493800/4942000] [99.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 02:44:28,009 - Train: 9.99% [493900/4942000] [99.9/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 02:45:00,812 - Train: 10.00% [494000/4942000] [100.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:45:33,527 - Train: 10.00% [494100/4942000] [100.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 02:46:06,276 - Train: 10.00% [494200/4942000] [100.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:46:14,967 - Test: 16.13% [50/310] [batch_t 0.152 (0.160)] 2024-04-05 02:46:22,574 - Test: 32.26% [100/310] [batch_t 0.157 (0.156)] 2024-04-05 02:46:30,143 - Test: 48.39% [150/310] [batch_t 0.153 (0.154)] 2024-04-05 02:46:37,705 - Test: 64.52% [200/310] [batch_t 0.158 (0.154)] 2024-04-05 02:46:45,263 - Test: 80.65% [250/310] [batch_t 0.155 (0.153)] 2024-04-05 02:46:52,781 - Test: 96.77% [300/310] [batch_t 0.150 (0.153)] 2024-04-05 02:46:54,206 - Test: 100.00% [310/310] [batch_t 0.081 (0.152)] 2024-04-05 03:12:23,364 - ==> Metric Time for coco : 0.004 (mAUROC_sp_max) 0.001 (mAP_sp_max) 0.001 (mF1_max_sp_max) 359.695 (mAUROC_px) 277.172 (mAP_px) 32.059 (mF1_max_px) 780.316 (mAUPRO_px) 11.085 (mF1_px_0.2_0.8_0.1) 12.118 (mAcc_px_0.2_0.8_0.1) 12.080 (mIoU_px_0.2_0.8_0.1) 35.360 (mIoU_max_px) 2024-04-05 03:12:23,917 - | Name | mAUROC_sp_max | mAUROC_sp_max (Max) | mAP_sp_max | mAP_sp_max (Max) | mF1_max_sp_max | mF1_max_sp_max (Max) | mAUROC_px | mAUROC_px (Max) | mAP_px | mAP_px (Max) | mF1_max_px | mF1_max_px (Max) | mAUPRO_px | mAUPRO_px (Max) | mF1_px_0.2_0.8_0.1 | mF1_px_0.2_0.8_0.1 (Max) | mAcc_px_0.2_0.8_0.1 | mAcc_px_0.2_0.8_0.1 (Max) | mIoU_px_0.2_0.8_0.1 | mIoU_px_0.2_0.8_0.1 (Max) | mIoU_max_px | mIoU_max_px (Max) | |:------:|:---------------:|:---------------------:|:------------:|:------------------:|:----------------:|:----------------------:|:-----------:|:------------------:|:--------:|:------------------:|:------------:|:------------------:|:-----------:|:------------------:|:--------------------:|:--------------------------:|:---------------------:|:---------------------------:|:---------------------:|:---------------------------:|:-------------:|:-------------------:| | coco | 66.234 | 66.882 (50 epoch) | 46.681 | 46.681 (100 epoch) | 53.663 | 54.576 (50 epoch) | 71.503 | 71.503 (100 epoch) | 14.361 | 14.361 (100 epoch) | 22.013 | 22.013 (100 epoch) | 42.329 | 44.441 (50 epoch) | 11.792 | 11.792 (50 epoch) | 44.178 | 44.665 (50 epoch) | 6.432 | 6.432 (100 epoch) | 12.367 | 12.367 (100 epoch) | 2024-04-05 03:12:24,455 - ==> Total time: 2 days, 9:15:03 Eta: 21 days, 11:15:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-05 03:13:43,378 - Train: 10.00% [494300/4942000] [100.0/1000.0] [batch_t 0.771 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 03:14:59,687 - Train: 10.00% [494400/4942000] [100.0/1000.0] [batch_t 0.768 (0.763)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-05 03:16:16,039 - Train: 10.01% [494500/4942000] [100.1/1000.0] [batch_t 0.768 (0.763)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 03:17:32,506 - Train: 10.01% [494600/4942000] [100.1/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 03:18:48,862 - Train: 10.01% [494700/4942000] [100.1/1000.0] [batch_t 0.771 (0.763)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-05 03:20:05,259 - Train: 10.01% [494800/4942000] [100.1/1000.0] [batch_t 0.768 (0.764)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 03:21:21,583 - Train: 10.01% [494900/4942000] [100.1/1000.0] [batch_t 0.774 (0.763)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-05 03:22:38,089 - Train: 10.02% [495000/4942000] [100.2/1000.0] [batch_t 0.766 (0.765)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-05 03:23:54,629 - Train: 10.02% [495100/4942000] [100.2/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 03:25:11,195 - Train: 10.02% [495200/4942000] [100.2/1000.0] [batch_t 0.763 (0.766)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-05 03:26:27,770 - Train: 10.02% [495300/4942000] [100.2/1000.0] [batch_t 0.763 (0.766)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-05 03:27:44,193 - Train: 10.02% [495400/4942000] [100.2/1000.0] [batch_t 0.757 (0.764)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-05 03:29:00,595 - Train: 10.03% [495500/4942000] [100.3/1000.0] [batch_t 0.777 (0.764)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-05 03:30:17,170 - Train: 10.03% [495600/4942000] [100.3/1000.0] [batch_t 0.766 (0.766)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 03:31:33,749 - Train: 10.03% [495700/4942000] [100.3/1000.0] [batch_t 0.771 (0.766)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-05 03:32:50,141 - Train: 10.03% [495800/4942000] [100.3/1000.0] [batch_t 0.771 (0.764)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-05 03:34:06,580 - Train: 10.03% [495900/4942000] [100.3/1000.0] [batch_t 0.767 (0.764)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-05 03:35:23,052 - Train: 10.04% [496000/4942000] [100.4/1000.0] [batch_t 0.749 (0.765)] [data_t 0.003] [optim_t 0.746] [lr 0.005000] 2024-04-05 03:36:39,551 - Train: 10.04% [496100/4942000] [100.4/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-05 03:37:56,042 - Train: 10.04% [496200/4942000] [100.4/1000.0] [batch_t 0.774 (0.765)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-05 03:39:12,576 - Train: 10.04% [496300/4942000] [100.4/1000.0] [batch_t 0.757 (0.765)] [data_t 0.002] [optim_t 0.754] [lr 0.005000] 2024-04-05 03:40:29,195 - Train: 10.04% [496400/4942000] [100.4/1000.0] [batch_t 0.768 (0.766)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 03:41:45,694 - Train: 10.05% [496500/4942000] [100.5/1000.0] [batch_t 0.764 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 03:43:02,279 - Train: 10.05% [496600/4942000] [100.5/1000.0] [batch_t 0.768 (0.766)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-05 03:44:18,654 - Train: 10.05% [496700/4942000] [100.5/1000.0] [batch_t 0.765 (0.764)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-05 03:45:35,209 - Train: 10.05% [496800/4942000] [100.5/1000.0] [batch_t 0.779 (0.765)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-05 03:46:51,702 - Train: 10.05% [496900/4942000] [100.5/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 03:48:08,250 - Train: 10.06% [497000/4942000] [100.6/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 03:49:24,777 - Train: 10.06% [497100/4942000] [100.6/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 03:50:41,301 - Train: 10.06% [497200/4942000] [100.6/1000.0] [batch_t 0.770 (0.765)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-05 03:51:57,853 - Train: 10.06% [497300/4942000] [100.6/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 03:53:14,250 - Train: 10.06% [497400/4942000] [100.6/1000.0] [batch_t 0.769 (0.764)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 03:54:30,729 - Train: 10.07% [497500/4942000] [100.7/1000.0] [batch_t 0.771 (0.765)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-05 03:55:47,264 - Train: 10.07% [497600/4942000] [100.7/1000.0] [batch_t 0.769 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 03:57:03,736 - Train: 10.07% [497700/4942000] [100.7/1000.0] [batch_t 0.774 (0.765)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-05 03:58:20,256 - Train: 10.07% [497800/4942000] [100.7/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-05 03:59:36,734 - Train: 10.07% [497900/4942000] [100.7/1000.0] [batch_t 0.758 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-05 04:00:53,247 - Train: 10.08% [498000/4942000] [100.8/1000.0] [batch_t 0.758 (0.765)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-05 04:02:09,676 - Train: 10.08% [498100/4942000] [100.8/1000.0] [batch_t 0.757 (0.764)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-05 04:03:26,229 - Train: 10.08% [498200/4942000] [100.8/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 04:04:42,684 - Train: 10.08% [498300/4942000] [100.8/1000.0] [batch_t 0.777 (0.764)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-05 04:05:59,172 - Train: 10.08% [498400/4942000] [100.8/1000.0] [batch_t 0.762 (0.765)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-05 04:07:15,680 - Train: 10.09% [498500/4942000] [100.9/1000.0] [batch_t 0.758 (0.765)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 04:08:32,192 - Train: 10.09% [498600/4942000] [100.9/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 04:09:48,687 - Train: 10.09% [498700/4942000] [100.9/1000.0] [batch_t 0.754 (0.765)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-05 04:11:05,048 - Train: 10.09% [498800/4942000] [100.9/1000.0] [batch_t 0.769 (0.764)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 04:12:21,427 - Train: 10.10% [498900/4942000] [101.0/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-05 04:13:37,845 - Train: 10.10% [499000/4942000] [101.0/1000.0] [batch_t 0.764 (0.764)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-05 04:14:54,426 - Train: 10.10% [499100/4942000] [101.0/1000.0] [batch_t 0.782 (0.766)] [data_t 0.003] [optim_t 0.780] [lr 0.005000] 2024-04-05 04:15:26,567 - ==> Total time: 2 days, 10:18:05 Eta: 21 days, 14:56:31 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-05 04:16:12,991 - Train: 10.10% [499200/4942000] [101.0/1000.0] [batch_t 0.776 (0.764)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-05 04:17:29,544 - Train: 10.10% [499300/4942000] [101.0/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 04:18:46,065 - Train: 10.11% [499400/4942000] [101.1/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 04:20:02,558 - Train: 10.11% [499500/4942000] [101.1/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 04:21:18,960 - Train: 10.11% [499600/4942000] [101.1/1000.0] [batch_t 0.776 (0.764)] [data_t 0.002] [optim_t 0.773] [lr 0.005000] 2024-04-05 04:22:35,419 - Train: 10.11% [499700/4942000] [101.1/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-05 04:23:51,997 - Train: 10.11% [499800/4942000] [101.1/1000.0] [batch_t 0.777 (0.766)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-05 04:25:08,588 - Train: 10.12% [499900/4942000] [101.2/1000.0] [batch_t 0.764 (0.766)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 04:26:25,055 - Train: 10.12% [500000/4942000] [101.2/1000.0] [batch_t 0.754 (0.765)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-05 04:27:41,575 - Train: 10.12% [500100/4942000] [101.2/1000.0] [batch_t 0.759 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-05 04:28:57,979 - Train: 10.12% [500200/4942000] [101.2/1000.0] [batch_t 0.755 (0.764)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-05 04:30:14,514 - Train: 10.12% [500300/4942000] [101.2/1000.0] [batch_t 0.779 (0.765)] [data_t 0.002] [optim_t 0.776] [lr 0.005000] 2024-04-05 04:31:31,009 - Train: 10.13% [500400/4942000] [101.3/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 04:32:47,555 - Train: 10.13% [500500/4942000] [101.3/1000.0] [batch_t 0.766 (0.765)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-05 04:34:04,130 - Train: 10.13% [500600/4942000] [101.3/1000.0] [batch_t 0.763 (0.766)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-05 04:35:20,678 - Train: 10.13% [500700/4942000] [101.3/1000.0] [batch_t 0.749 (0.765)] [data_t 0.003] [optim_t 0.746] [lr 0.005000] 2024-04-05 04:36:37,145 - Train: 10.13% [500800/4942000] [101.3/1000.0] [batch_t 0.761 (0.765)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-05 04:37:53,663 - Train: 10.14% [500900/4942000] [101.4/1000.0] [batch_t 0.771 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-05 04:39:10,186 - Train: 10.14% [501000/4942000] [101.4/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 04:40:26,706 - Train: 10.14% [501100/4942000] [101.4/1000.0] [batch_t 0.778 (0.765)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-05 04:41:43,138 - Train: 10.14% [501200/4942000] [101.4/1000.0] [batch_t 0.772 (0.764)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-05 04:42:59,635 - Train: 10.14% [501300/4942000] [101.4/1000.0] [batch_t 0.764 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 04:44:16,069 - Train: 10.15% [501400/4942000] [101.5/1000.0] [batch_t 0.767 (0.764)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-05 04:45:32,567 - Train: 10.15% [501500/4942000] [101.5/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-05 04:46:49,026 - Train: 10.15% [501600/4942000] [101.5/1000.0] [batch_t 0.777 (0.764)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-05 04:48:05,475 - Train: 10.15% [501700/4942000] [101.5/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 04:49:22,076 - Train: 10.15% [501800/4942000] [101.5/1000.0] [batch_t 0.764 (0.766)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 04:50:38,531 - Train: 10.16% [501900/4942000] [101.6/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 04:51:55,104 - Train: 10.16% [502000/4942000] [101.6/1000.0] [batch_t 0.768 (0.766)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 04:53:11,447 - Train: 10.16% [502100/4942000] [101.6/1000.0] [batch_t 0.757 (0.763)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 04:54:27,883 - Train: 10.16% [502200/4942000] [101.6/1000.0] [batch_t 0.760 (0.764)] [data_t 0.002] [optim_t 0.758] [lr 0.005000] 2024-04-05 04:55:44,183 - Train: 10.16% [502300/4942000] [101.6/1000.0] [batch_t 0.749 (0.763)] [data_t 0.002] [optim_t 0.747] [lr 0.005000] 2024-04-05 04:57:00,459 - Train: 10.17% [502400/4942000] [101.7/1000.0] [batch_t 0.764 (0.763)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 04:58:17,016 - Train: 10.17% [502500/4942000] [101.7/1000.0] [batch_t 0.769 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 04:59:33,447 - Train: 10.17% [502600/4942000] [101.7/1000.0] [batch_t 0.769 (0.764)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-05 05:00:49,880 - Train: 10.17% [502700/4942000] [101.7/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 05:02:06,297 - Train: 10.17% [502800/4942000] [101.7/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 05:03:22,933 - Train: 10.18% [502900/4942000] [101.8/1000.0] [batch_t 0.767 (0.766)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 05:04:39,399 - Train: 10.18% [503000/4942000] [101.8/1000.0] [batch_t 0.754 (0.765)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-05 05:05:55,872 - Train: 10.18% [503100/4942000] [101.8/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 05:07:12,416 - Train: 10.18% [503200/4942000] [101.8/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 05:08:28,980 - Train: 10.18% [503300/4942000] [101.8/1000.0] [batch_t 0.767 (0.766)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 05:09:45,613 - Train: 10.19% [503400/4942000] [101.9/1000.0] [batch_t 0.754 (0.766)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-05 05:11:02,125 - Train: 10.19% [503500/4942000] [101.9/1000.0] [batch_t 0.776 (0.765)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-05 05:12:18,542 - Train: 10.19% [503600/4942000] [101.9/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-05 05:13:35,008 - Train: 10.19% [503700/4942000] [101.9/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 05:14:51,528 - Train: 10.19% [503800/4942000] [101.9/1000.0] [batch_t 0.771 (0.765)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-05 05:16:08,047 - Train: 10.20% [503900/4942000] [102.0/1000.0] [batch_t 0.783 (0.765)] [data_t 0.003] [optim_t 0.780] [lr 0.005000] 2024-04-05 05:17:24,656 - Train: 10.20% [504000/4942000] [102.0/1000.0] [batch_t 0.754 (0.766)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-05 05:18:28,893 - ==> Total time: 2 days, 11:21:08 Eta: 21 days, 18:31:57 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-05 05:18:43,036 - Train: 10.20% [504100/4942000] [102.0/1000.0] [batch_t 0.768 (0.766)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 05:19:59,502 - Train: 10.20% [504200/4942000] [102.0/1000.0] [batch_t 0.779 (0.765)] [data_t 0.002] [optim_t 0.777] [lr 0.005000] 2024-04-05 05:21:15,947 - Train: 10.20% [504300/4942000] [102.0/1000.0] [batch_t 0.767 (0.764)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-05 05:22:32,452 - Train: 10.21% [504400/4942000] [102.1/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-05 05:23:48,906 - Train: 10.21% [504500/4942000] [102.1/1000.0] [batch_t 0.773 (0.764)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 05:25:05,341 - Train: 10.21% [504600/4942000] [102.1/1000.0] [batch_t 0.754 (0.764)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-05 05:26:22,030 - Train: 10.21% [504700/4942000] [102.1/1000.0] [batch_t 0.767 (0.767)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 05:27:38,527 - Train: 10.21% [504800/4942000] [102.1/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-05 05:28:54,954 - Train: 10.22% [504900/4942000] [102.2/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 05:30:11,335 - Train: 10.22% [505000/4942000] [102.2/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 05:31:27,707 - Train: 10.22% [505100/4942000] [102.2/1000.0] [batch_t 0.754 (0.764)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-05 05:32:44,268 - Train: 10.22% [505200/4942000] [102.2/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 05:34:00,771 - Train: 10.22% [505300/4942000] [102.2/1000.0] [batch_t 0.758 (0.765)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-05 05:35:17,175 - Train: 10.23% [505400/4942000] [102.3/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 05:36:33,583 - Train: 10.23% [505500/4942000] [102.3/1000.0] [batch_t 0.771 (0.764)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-05 05:37:49,854 - Train: 10.23% [505600/4942000] [102.3/1000.0] [batch_t 0.771 (0.763)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-05 05:39:06,363 - Train: 10.23% [505700/4942000] [102.3/1000.0] [batch_t 0.753 (0.765)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-05 05:40:23,014 - Train: 10.23% [505800/4942000] [102.3/1000.0] [batch_t 0.773 (0.766)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 05:41:39,514 - Train: 10.24% [505900/4942000] [102.4/1000.0] [batch_t 0.778 (0.765)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-05 05:42:55,978 - Train: 10.24% [506000/4942000] [102.4/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 05:44:12,392 - Train: 10.24% [506100/4942000] [102.4/1000.0] [batch_t 0.757 (0.764)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-05 05:45:29,184 - Train: 10.24% [506200/4942000] [102.4/1000.0] [batch_t 0.769 (0.768)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 05:46:45,801 - Train: 10.24% [506300/4942000] [102.4/1000.0] [batch_t 0.758 (0.766)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 05:48:02,289 - Train: 10.25% [506400/4942000] [102.5/1000.0] [batch_t 0.779 (0.765)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-05 05:49:18,756 - Train: 10.25% [506500/4942000] [102.5/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 05:50:35,294 - Train: 10.25% [506600/4942000] [102.5/1000.0] [batch_t 0.774 (0.765)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-05 05:51:51,808 - Train: 10.25% [506700/4942000] [102.5/1000.0] [batch_t 0.759 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-05 05:53:08,289 - Train: 10.25% [506800/4942000] [102.5/1000.0] [batch_t 0.775 (0.765)] [data_t 0.002] [optim_t 0.772] [lr 0.005000] 2024-04-05 05:54:24,716 - Train: 10.26% [506900/4942000] [102.6/1000.0] [batch_t 0.768 (0.764)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-05 05:55:41,031 - Train: 10.26% [507000/4942000] [102.6/1000.0] [batch_t 0.768 (0.763)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 05:56:57,500 - Train: 10.26% [507100/4942000] [102.6/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-05 05:58:14,058 - Train: 10.26% [507200/4942000] [102.6/1000.0] [batch_t 0.759 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-05 05:59:30,494 - Train: 10.27% [507300/4942000] [102.7/1000.0] [batch_t 0.750 (0.764)] [data_t 0.002] [optim_t 0.747] [lr 0.005000] 2024-04-05 06:00:46,978 - Train: 10.27% [507400/4942000] [102.7/1000.0] [batch_t 0.753 (0.765)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-05 06:02:03,480 - Train: 10.27% [507500/4942000] [102.7/1000.0] [batch_t 0.764 (0.765)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-05 06:03:19,860 - Train: 10.27% [507600/4942000] [102.7/1000.0] [batch_t 0.761 (0.764)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-05 06:04:36,256 - Train: 10.27% [507700/4942000] [102.7/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 06:05:52,789 - Train: 10.28% [507800/4942000] [102.8/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 06:07:09,248 - Train: 10.28% [507900/4942000] [102.8/1000.0] [batch_t 0.773 (0.764)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 06:08:25,768 - Train: 10.28% [508000/4942000] [102.8/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-05 06:09:42,312 - Train: 10.28% [508100/4942000] [102.8/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-05 06:10:58,783 - Train: 10.28% [508200/4942000] [102.8/1000.0] [batch_t 0.762 (0.765)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-05 06:12:15,221 - Train: 10.29% [508300/4942000] [102.9/1000.0] [batch_t 0.773 (0.764)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 06:13:31,499 - Train: 10.29% [508400/4942000] [102.9/1000.0] [batch_t 0.763 (0.763)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-05 06:14:47,817 - Train: 10.29% [508500/4942000] [102.9/1000.0] [batch_t 0.774 (0.763)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-05 06:16:04,242 - Train: 10.29% [508600/4942000] [102.9/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 06:17:20,831 - Train: 10.29% [508700/4942000] [102.9/1000.0] [batch_t 0.772 (0.766)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 06:18:37,298 - Train: 10.30% [508800/4942000] [103.0/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 06:19:53,744 - Train: 10.30% [508900/4942000] [103.0/1000.0] [batch_t 0.766 (0.764)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-05 06:21:10,279 - Train: 10.30% [509000/4942000] [103.0/1000.0] [batch_t 0.769 (0.765)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-05 06:21:30,206 - ==> Total time: 2 days, 12:24:09 Eta: 21 days, 22:01:49 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-05 06:22:29,058 - Train: 10.30% [509100/4942000] [103.0/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 06:23:45,509 - Train: 10.30% [509200/4942000] [103.0/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 06:25:01,968 - Train: 10.31% [509300/4942000] [103.1/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 06:26:18,541 - Train: 10.31% [509400/4942000] [103.1/1000.0] [batch_t 0.768 (0.766)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 06:27:34,973 - Train: 10.31% [509500/4942000] [103.1/1000.0] [batch_t 0.778 (0.764)] [data_t 0.002] [optim_t 0.776] [lr 0.005000] 2024-04-05 06:28:51,511 - Train: 10.31% [509600/4942000] [103.1/1000.0] [batch_t 0.764 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 06:30:08,087 - Train: 10.31% [509700/4942000] [103.1/1000.0] [batch_t 0.773 (0.766)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-05 06:31:24,553 - Train: 10.32% [509800/4942000] [103.2/1000.0] [batch_t 0.753 (0.765)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-05 06:32:41,111 - Train: 10.32% [509900/4942000] [103.2/1000.0] [batch_t 0.770 (0.765)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-05 06:33:57,586 - Train: 10.32% [510000/4942000] [103.2/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 06:35:13,937 - Train: 10.32% [510100/4942000] [103.2/1000.0] [batch_t 0.757 (0.763)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-05 06:36:30,350 - Train: 10.32% [510200/4942000] [103.2/1000.0] [batch_t 0.760 (0.764)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-05 06:37:47,033 - Train: 10.33% [510300/4942000] [103.3/1000.0] [batch_t 0.770 (0.767)] [data_t 0.004] [optim_t 0.766] [lr 0.005000] 2024-04-05 06:39:03,623 - Train: 10.33% [510400/4942000] [103.3/1000.0] [batch_t 0.784 (0.766)] [data_t 0.003] [optim_t 0.781] [lr 0.005000] 2024-04-05 06:40:20,011 - Train: 10.33% [510500/4942000] [103.3/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 06:41:36,516 - Train: 10.33% [510600/4942000] [103.3/1000.0] [batch_t 0.752 (0.765)] [data_t 0.003] [optim_t 0.749] [lr 0.005000] 2024-04-05 06:42:52,977 - Train: 10.33% [510700/4942000] [103.3/1000.0] [batch_t 0.771 (0.765)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-05 06:44:09,459 - Train: 10.34% [510800/4942000] [103.4/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-05 06:45:25,925 - Train: 10.34% [510900/4942000] [103.4/1000.0] [batch_t 0.764 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 06:46:42,468 - Train: 10.34% [511000/4942000] [103.4/1000.0] [batch_t 0.769 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 06:47:58,985 - Train: 10.34% [511100/4942000] [103.4/1000.0] [batch_t 0.771 (0.765)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-05 06:49:15,353 - Train: 10.34% [511200/4942000] [103.4/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 06:50:31,846 - Train: 10.35% [511300/4942000] [103.5/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-05 06:51:48,566 - Train: 10.35% [511400/4942000] [103.5/1000.0] [batch_t 0.763 (0.767)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 06:53:05,048 - Train: 10.35% [511500/4942000] [103.5/1000.0] [batch_t 0.757 (0.765)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 06:54:21,435 - Train: 10.35% [511600/4942000] [103.5/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-05 06:55:37,971 - Train: 10.35% [511700/4942000] [103.5/1000.0] [batch_t 0.765 (0.765)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-05 06:56:54,495 - Train: 10.36% [511800/4942000] [103.6/1000.0] [batch_t 0.771 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-05 06:58:11,013 - Train: 10.36% [511900/4942000] [103.6/1000.0] [batch_t 0.771 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 06:59:27,438 - Train: 10.36% [512000/4942000] [103.6/1000.0] [batch_t 0.769 (0.764)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 07:00:43,870 - Train: 10.36% [512100/4942000] [103.6/1000.0] [batch_t 0.773 (0.764)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 07:02:00,400 - Train: 10.36% [512200/4942000] [103.6/1000.0] [batch_t 0.755 (0.765)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-05 07:03:16,925 - Train: 10.37% [512300/4942000] [103.7/1000.0] [batch_t 0.766 (0.765)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-05 07:04:33,324 - Train: 10.37% [512400/4942000] [103.7/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 07:05:49,737 - Train: 10.37% [512500/4942000] [103.7/1000.0] [batch_t 0.758 (0.764)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-05 07:07:06,244 - Train: 10.37% [512600/4942000] [103.7/1000.0] [batch_t 0.776 (0.765)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-05 07:08:22,824 - Train: 10.37% [512700/4942000] [103.7/1000.0] [batch_t 0.764 (0.766)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 07:09:39,289 - Train: 10.38% [512800/4942000] [103.8/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 07:10:55,711 - Train: 10.38% [512900/4942000] [103.8/1000.0] [batch_t 0.769 (0.764)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-05 07:12:12,217 - Train: 10.38% [513000/4942000] [103.8/1000.0] [batch_t 0.774 (0.765)] [data_t 0.003] [optim_t 0.772] [lr 0.005000] 2024-04-05 07:13:28,660 - Train: 10.38% [513100/4942000] [103.8/1000.0] [batch_t 0.777 (0.764)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-05 07:14:45,156 - Train: 10.38% [513200/4942000] [103.8/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 07:16:01,624 - Train: 10.39% [513300/4942000] [103.9/1000.0] [batch_t 0.759 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-05 07:17:18,173 - Train: 10.39% [513400/4942000] [103.9/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-05 07:18:34,593 - Train: 10.39% [513500/4942000] [103.9/1000.0] [batch_t 0.775 (0.764)] [data_t 0.003] [optim_t 0.772] [lr 0.005000] 2024-04-05 07:19:51,049 - Train: 10.39% [513600/4942000] [103.9/1000.0] [batch_t 0.760 (0.764)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-05 07:21:07,464 - Train: 10.39% [513700/4942000] [103.9/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 07:22:23,891 - Train: 10.40% [513800/4942000] [104.0/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 07:23:40,451 - Train: 10.40% [513900/4942000] [104.0/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-05 07:24:32,369 - ==> Total time: 2 days, 13:27:11 Eta: 22 days, 1:26:34 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-05 07:24:58,802 - Train: 10.40% [514000/4942000] [104.0/1000.0] [batch_t 0.758 (0.767)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 07:26:15,310 - Train: 10.40% [514100/4942000] [104.0/1000.0] [batch_t 0.762 (0.765)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-05 07:27:31,753 - Train: 10.40% [514200/4942000] [104.0/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 07:28:48,243 - Train: 10.41% [514300/4942000] [104.1/1000.0] [batch_t 0.754 (0.765)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-05 07:30:04,779 - Train: 10.41% [514400/4942000] [104.1/1000.0] [batch_t 0.749 (0.765)] [data_t 0.002] [optim_t 0.747] [lr 0.005000] 2024-04-05 07:31:21,364 - Train: 10.41% [514500/4942000] [104.1/1000.0] [batch_t 0.760 (0.766)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-05 07:32:37,811 - Train: 10.41% [514600/4942000] [104.1/1000.0] [batch_t 0.779 (0.764)] [data_t 0.002] [optim_t 0.776] [lr 0.005000] 2024-04-05 07:33:54,337 - Train: 10.41% [514700/4942000] [104.1/1000.0] [batch_t 0.771 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 07:35:10,736 - Train: 10.42% [514800/4942000] [104.2/1000.0] [batch_t 0.773 (0.764)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 07:36:27,381 - Train: 10.42% [514900/4942000] [104.2/1000.0] [batch_t 0.748 (0.766)] [data_t 0.003] [optim_t 0.745] [lr 0.005000] 2024-04-05 07:37:43,733 - Train: 10.42% [515000/4942000] [104.2/1000.0] [batch_t 0.767 (0.763)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 07:39:00,261 - Train: 10.42% [515100/4942000] [104.2/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 07:40:16,750 - Train: 10.42% [515200/4942000] [104.2/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 07:41:33,222 - Train: 10.43% [515300/4942000] [104.3/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 07:42:49,555 - Train: 10.43% [515400/4942000] [104.3/1000.0] [batch_t 0.766 (0.763)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-05 07:44:06,114 - Train: 10.43% [515500/4942000] [104.3/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-05 07:45:22,618 - Train: 10.43% [515600/4942000] [104.3/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 07:46:39,071 - Train: 10.44% [515700/4942000] [104.4/1000.0] [batch_t 0.773 (0.764)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 07:47:55,530 - Train: 10.44% [515800/4942000] [104.4/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 07:49:12,077 - Train: 10.44% [515900/4942000] [104.4/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 07:50:28,554 - Train: 10.44% [516000/4942000] [104.4/1000.0] [batch_t 0.757 (0.765)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-05 07:51:44,977 - Train: 10.44% [516100/4942000] [104.4/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 07:53:01,300 - Train: 10.45% [516200/4942000] [104.5/1000.0] [batch_t 0.763 (0.763)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-05 07:54:17,779 - Train: 10.45% [516300/4942000] [104.5/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 07:55:34,292 - Train: 10.45% [516400/4942000] [104.5/1000.0] [batch_t 0.758 (0.765)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 07:56:50,737 - Train: 10.45% [516500/4942000] [104.5/1000.0] [batch_t 0.749 (0.764)] [data_t 0.003] [optim_t 0.745] [lr 0.005000] 2024-04-05 07:58:07,227 - Train: 10.45% [516600/4942000] [104.5/1000.0] [batch_t 0.764 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 07:59:23,716 - Train: 10.46% [516700/4942000] [104.6/1000.0] [batch_t 0.777 (0.765)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-05 08:00:40,266 - Train: 10.46% [516800/4942000] [104.6/1000.0] [batch_t 0.753 (0.765)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-05 08:01:56,949 - Train: 10.46% [516900/4942000] [104.6/1000.0] [batch_t 0.771 (0.767)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 08:03:13,556 - Train: 10.46% [517000/4942000] [104.6/1000.0] [batch_t 0.763 (0.766)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 08:04:30,225 - Train: 10.46% [517100/4942000] [104.6/1000.0] [batch_t 0.757 (0.767)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-05 08:05:46,698 - Train: 10.47% [517200/4942000] [104.7/1000.0] [batch_t 0.776 (0.765)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-05 08:07:03,122 - Train: 10.47% [517300/4942000] [104.7/1000.0] [batch_t 0.748 (0.764)] [data_t 0.002] [optim_t 0.745] [lr 0.005000] 2024-04-05 08:08:19,479 - Train: 10.47% [517400/4942000] [104.7/1000.0] [batch_t 0.769 (0.763)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 08:09:36,036 - Train: 10.47% [517500/4942000] [104.7/1000.0] [batch_t 0.773 (0.765)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-05 08:10:52,503 - Train: 10.47% [517600/4942000] [104.7/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 08:12:08,969 - Train: 10.48% [517700/4942000] [104.8/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-05 08:13:25,336 - Train: 10.48% [517800/4942000] [104.8/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 08:14:41,817 - Train: 10.48% [517900/4942000] [104.8/1000.0] [batch_t 0.758 (0.765)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 08:15:58,323 - Train: 10.48% [518000/4942000] [104.8/1000.0] [batch_t 0.777 (0.765)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-05 08:17:14,819 - Train: 10.48% [518100/4942000] [104.8/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 08:18:31,272 - Train: 10.49% [518200/4942000] [104.9/1000.0] [batch_t 0.762 (0.764)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-05 08:19:47,761 - Train: 10.49% [518300/4942000] [104.9/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-05 08:21:04,092 - Train: 10.49% [518400/4942000] [104.9/1000.0] [batch_t 0.759 (0.763)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-05 08:22:20,550 - Train: 10.49% [518500/4942000] [104.9/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 08:23:36,900 - Train: 10.49% [518600/4942000] [104.9/1000.0] [batch_t 0.758 (0.763)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 08:24:53,397 - Train: 10.50% [518700/4942000] [105.0/1000.0] [batch_t 0.769 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 08:26:09,978 - Train: 10.50% [518800/4942000] [105.0/1000.0] [batch_t 0.764 (0.766)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-05 08:27:26,505 - Train: 10.50% [518900/4942000] [105.0/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 08:27:34,183 - ==> Total time: 2 days, 14:30:13 Eta: 22 days, 4:46:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-05 08:28:45,156 - Train: 10.50% [519000/4942000] [105.0/1000.0] [batch_t 0.758 (0.765)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 08:30:01,496 - Train: 10.50% [519100/4942000] [105.0/1000.0] [batch_t 0.768 (0.763)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 08:31:18,003 - Train: 10.51% [519200/4942000] [105.1/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 08:32:34,580 - Train: 10.51% [519300/4942000] [105.1/1000.0] [batch_t 0.766 (0.766)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-05 08:33:51,003 - Train: 10.51% [519400/4942000] [105.1/1000.0] [batch_t 0.766 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 08:35:07,461 - Train: 10.51% [519500/4942000] [105.1/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 08:36:23,863 - Train: 10.51% [519600/4942000] [105.1/1000.0] [batch_t 0.778 (0.764)] [data_t 0.002] [optim_t 0.776] [lr 0.005000] 2024-04-05 08:37:40,374 - Train: 10.52% [519700/4942000] [105.2/1000.0] [batch_t 0.773 (0.765)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-05 08:38:56,879 - Train: 10.52% [519800/4942000] [105.2/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 08:40:13,253 - Train: 10.52% [519900/4942000] [105.2/1000.0] [batch_t 0.760 (0.764)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-05 08:41:29,729 - Train: 10.52% [520000/4942000] [105.2/1000.0] [batch_t 0.774 (0.765)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-05 08:42:46,254 - Train: 10.52% [520100/4942000] [105.2/1000.0] [batch_t 0.757 (0.765)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-05 08:44:02,654 - Train: 10.53% [520200/4942000] [105.3/1000.0] [batch_t 0.755 (0.764)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-05 08:45:19,089 - Train: 10.53% [520300/4942000] [105.3/1000.0] [batch_t 0.766 (0.764)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-05 08:46:35,499 - Train: 10.53% [520400/4942000] [105.3/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 08:47:51,972 - Train: 10.53% [520500/4942000] [105.3/1000.0] [batch_t 0.756 (0.765)] [data_t 0.003] [optim_t 0.753] [lr 0.005000] 2024-04-05 08:49:08,468 - Train: 10.53% [520600/4942000] [105.3/1000.0] [batch_t 0.765 (0.765)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-05 08:50:24,769 - Train: 10.54% [520700/4942000] [105.4/1000.0] [batch_t 0.772 (0.763)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 08:51:41,224 - Train: 10.54% [520800/4942000] [105.4/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 08:52:57,784 - Train: 10.54% [520900/4942000] [105.4/1000.0] [batch_t 0.756 (0.765)] [data_t 0.003] [optim_t 0.753] [lr 0.005000] 2024-04-05 08:54:14,216 - Train: 10.54% [521000/4942000] [105.4/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 08:55:30,593 - Train: 10.54% [521100/4942000] [105.4/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 08:56:47,124 - Train: 10.55% [521200/4942000] [105.5/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 08:58:03,621 - Train: 10.55% [521300/4942000] [105.5/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-05 08:59:20,038 - Train: 10.55% [521400/4942000] [105.5/1000.0] [batch_t 0.774 (0.764)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-05 09:00:36,394 - Train: 10.55% [521500/4942000] [105.5/1000.0] [batch_t 0.773 (0.763)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-05 09:01:52,815 - Train: 10.55% [521600/4942000] [105.5/1000.0] [batch_t 0.772 (0.764)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 09:03:09,258 - Train: 10.56% [521700/4942000] [105.6/1000.0] [batch_t 0.772 (0.764)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 09:04:25,722 - Train: 10.56% [521800/4942000] [105.6/1000.0] [batch_t 0.758 (0.765)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-05 09:05:42,257 - Train: 10.56% [521900/4942000] [105.6/1000.0] [batch_t 0.769 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 09:06:58,788 - Train: 10.56% [522000/4942000] [105.6/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 09:08:15,209 - Train: 10.56% [522100/4942000] [105.6/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 09:09:31,727 - Train: 10.57% [522200/4942000] [105.7/1000.0] [batch_t 0.767 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-05 09:10:48,103 - Train: 10.57% [522300/4942000] [105.7/1000.0] [batch_t 0.761 (0.764)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-05 09:12:04,564 - Train: 10.57% [522400/4942000] [105.7/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 09:13:21,246 - Train: 10.57% [522500/4942000] [105.7/1000.0] [batch_t 0.757 (0.767)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-05 09:14:37,852 - Train: 10.57% [522600/4942000] [105.7/1000.0] [batch_t 0.768 (0.766)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 09:15:54,416 - Train: 10.58% [522700/4942000] [105.8/1000.0] [batch_t 0.767 (0.766)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-05 09:17:10,792 - Train: 10.58% [522800/4942000] [105.8/1000.0] [batch_t 0.757 (0.764)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-05 09:18:27,297 - Train: 10.58% [522900/4942000] [105.8/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-05 09:19:43,939 - Train: 10.58% [523000/4942000] [105.8/1000.0] [batch_t 0.752 (0.766)] [data_t 0.003] [optim_t 0.749] [lr 0.005000] 2024-04-05 09:21:00,402 - Train: 10.58% [523100/4942000] [105.8/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 09:22:16,731 - Train: 10.59% [523200/4942000] [105.9/1000.0] [batch_t 0.778 (0.763)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-05 09:23:33,285 - Train: 10.59% [523300/4942000] [105.9/1000.0] [batch_t 0.761 (0.765)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-05 09:24:49,673 - Train: 10.59% [523400/4942000] [105.9/1000.0] [batch_t 0.776 (0.764)] [data_t 0.002] [optim_t 0.773] [lr 0.005000] 2024-04-05 09:26:06,117 - Train: 10.59% [523500/4942000] [105.9/1000.0] [batch_t 0.774 (0.764)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-05 09:27:22,603 - Train: 10.59% [523600/4942000] [105.9/1000.0] [batch_t 0.765 (0.765)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-05 09:28:39,106 - Train: 10.60% [523700/4942000] [106.0/1000.0] [batch_t 0.757 (0.765)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-05 09:29:55,621 - Train: 10.60% [523800/4942000] [106.0/1000.0] [batch_t 0.770 (0.765)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-05 09:30:35,353 - ==> Total time: 2 days, 15:33:14 Eta: 22 days, 8:00:44 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-05 09:31:13,990 - Train: 10.60% [523900/4942000] [106.0/1000.0] [batch_t 0.774 (0.765)] [data_t 0.002] [optim_t 0.772] [lr 0.005000] 2024-04-05 09:32:30,413 - Train: 10.60% [524000/4942000] [106.0/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 09:33:47,032 - Train: 10.61% [524100/4942000] [106.1/1000.0] [batch_t 0.757 (0.766)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 09:35:03,522 - Train: 10.61% [524200/4942000] [106.1/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 09:36:19,972 - Train: 10.61% [524300/4942000] [106.1/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 09:37:36,390 - Train: 10.61% [524400/4942000] [106.1/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-05 09:38:53,028 - Train: 10.61% [524500/4942000] [106.1/1000.0] [batch_t 0.775 (0.766)] [data_t 0.002] [optim_t 0.772] [lr 0.005000] 2024-04-05 09:40:09,437 - Train: 10.62% [524600/4942000] [106.2/1000.0] [batch_t 0.754 (0.764)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-05 09:41:26,049 - Train: 10.62% [524700/4942000] [106.2/1000.0] [batch_t 0.760 (0.766)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-05 09:42:42,731 - Train: 10.62% [524800/4942000] [106.2/1000.0] [batch_t 0.776 (0.767)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-05 09:43:59,111 - Train: 10.62% [524900/4942000] [106.2/1000.0] [batch_t 0.760 (0.764)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-05 09:45:15,567 - Train: 10.62% [525000/4942000] [106.2/1000.0] [batch_t 0.758 (0.764)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-05 09:46:32,111 - Train: 10.63% [525100/4942000] [106.3/1000.0] [batch_t 0.773 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 09:47:48,609 - Train: 10.63% [525200/4942000] [106.3/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 09:49:05,123 - Train: 10.63% [525300/4942000] [106.3/1000.0] [batch_t 0.770 (0.765)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-05 09:50:21,593 - Train: 10.63% [525400/4942000] [106.3/1000.0] [batch_t 0.754 (0.765)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-05 09:51:38,042 - Train: 10.63% [525500/4942000] [106.3/1000.0] [batch_t 0.770 (0.764)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-05 09:52:54,600 - Train: 10.64% [525600/4942000] [106.4/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 09:54:10,991 - Train: 10.64% [525700/4942000] [106.4/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-05 09:55:29,009 - Train: 10.64% [525800/4942000] [106.4/1000.0] [batch_t 0.743 (0.780)] [data_t 0.003] [optim_t 0.740] [lr 0.005000] 2024-04-05 09:56:46,440 - Train: 10.64% [525900/4942000] [106.4/1000.0] [batch_t 0.758 (0.774)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-05 09:58:02,949 - Train: 10.64% [526000/4942000] [106.4/1000.0] [batch_t 0.778 (0.765)] [data_t 0.002] [optim_t 0.776] [lr 0.005000] 2024-04-05 09:59:39,578 - Train: 10.65% [526100/4942000] [106.5/1000.0] [batch_t 0.769 (0.966)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 10:01:09,007 - Train: 10.65% [526200/4942000] [106.5/1000.0] [batch_t 0.762 (0.894)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 10:02:25,474 - Train: 10.65% [526300/4942000] [106.5/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 10:03:41,985 - Train: 10.65% [526400/4942000] [106.5/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 10:04:58,531 - Train: 10.65% [526500/4942000] [106.5/1000.0] [batch_t 0.759 (0.765)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-05 10:06:14,990 - Train: 10.66% [526600/4942000] [106.6/1000.0] [batch_t 0.769 (0.764)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-05 10:07:32,297 - Train: 10.66% [526700/4942000] [106.6/1000.0] [batch_t 0.777 (0.773)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-05 10:08:48,647 - Train: 10.66% [526800/4942000] [106.6/1000.0] [batch_t 0.751 (0.763)] [data_t 0.002] [optim_t 0.748] [lr 0.005000] 2024-04-05 10:10:05,030 - Train: 10.66% [526900/4942000] [106.6/1000.0] [batch_t 0.769 (0.764)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 10:11:21,520 - Train: 10.66% [527000/4942000] [106.6/1000.0] [batch_t 0.765 (0.765)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-05 10:12:38,160 - Train: 10.67% [527100/4942000] [106.7/1000.0] [batch_t 0.767 (0.766)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 10:13:54,628 - Train: 10.67% [527200/4942000] [106.7/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 10:15:11,019 - Train: 10.67% [527300/4942000] [106.7/1000.0] [batch_t 0.762 (0.764)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-05 10:16:27,470 - Train: 10.67% [527400/4942000] [106.7/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 10:17:43,925 - Train: 10.67% [527500/4942000] [106.7/1000.0] [batch_t 0.749 (0.764)] [data_t 0.003] [optim_t 0.746] [lr 0.005000] 2024-04-05 10:19:00,580 - Train: 10.68% [527600/4942000] [106.8/1000.0] [batch_t 0.763 (0.766)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 10:20:17,047 - Train: 10.68% [527700/4942000] [106.8/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 10:21:33,616 - Train: 10.68% [527800/4942000] [106.8/1000.0] [batch_t 0.763 (0.766)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-05 10:22:50,015 - Train: 10.68% [527900/4942000] [106.8/1000.0] [batch_t 0.753 (0.764)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-05 10:24:06,435 - Train: 10.68% [528000/4942000] [106.8/1000.0] [batch_t 0.768 (0.764)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 10:25:22,891 - Train: 10.69% [528100/4942000] [106.9/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 10:26:39,504 - Train: 10.69% [528200/4942000] [106.9/1000.0] [batch_t 0.764 (0.766)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-05 10:27:55,991 - Train: 10.69% [528300/4942000] [106.9/1000.0] [batch_t 0.769 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 10:29:12,543 - Train: 10.69% [528400/4942000] [106.9/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 10:30:29,090 - Train: 10.69% [528500/4942000] [106.9/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 10:31:45,580 - Train: 10.70% [528600/4942000] [107.0/1000.0] [batch_t 0.754 (0.765)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-05 10:33:02,079 - Train: 10.70% [528700/4942000] [107.0/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 10:34:14,058 - ==> Total time: 2 days, 16:36:53 Eta: 22 days, 11:15:42 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-05 10:34:20,583 - Train: 10.70% [528800/4942000] [107.0/1000.0] [batch_t 0.761 (0.769)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-05 10:35:37,074 - Train: 10.70% [528900/4942000] [107.0/1000.0] [batch_t 0.769 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 10:36:53,417 - Train: 10.70% [529000/4942000] [107.0/1000.0] [batch_t 0.776 (0.763)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-05 10:38:09,885 - Train: 10.71% [529100/4942000] [107.1/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 10:39:26,374 - Train: 10.71% [529200/4942000] [107.1/1000.0] [batch_t 0.772 (0.765)] [data_t 0.004] [optim_t 0.769] [lr 0.005000] 2024-04-05 10:40:42,959 - Train: 10.71% [529300/4942000] [107.1/1000.0] [batch_t 0.773 (0.766)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 10:41:59,394 - Train: 10.71% [529400/4942000] [107.1/1000.0] [batch_t 0.767 (0.764)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-05 10:43:15,940 - Train: 10.71% [529500/4942000] [107.1/1000.0] [batch_t 0.767 (0.765)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-05 10:44:32,405 - Train: 10.72% [529600/4942000] [107.2/1000.0] [batch_t 0.764 (0.765)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-05 10:45:48,707 - Train: 10.72% [529700/4942000] [107.2/1000.0] [batch_t 0.767 (0.763)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 10:47:05,246 - Train: 10.72% [529800/4942000] [107.2/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-05 10:48:21,735 - Train: 10.72% [529900/4942000] [107.2/1000.0] [batch_t 0.760 (0.765)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-05 10:49:38,183 - Train: 10.72% [530000/4942000] [107.2/1000.0] [batch_t 0.766 (0.764)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-05 10:50:54,608 - Train: 10.73% [530100/4942000] [107.3/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 10:52:11,039 - Train: 10.73% [530200/4942000] [107.3/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 10:53:27,528 - Train: 10.73% [530300/4942000] [107.3/1000.0] [batch_t 0.753 (0.765)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-05 10:54:44,106 - Train: 10.73% [530400/4942000] [107.3/1000.0] [batch_t 0.764 (0.766)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-05 10:56:00,631 - Train: 10.73% [530500/4942000] [107.3/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 10:57:17,132 - Train: 10.74% [530600/4942000] [107.4/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 10:58:33,544 - Train: 10.74% [530700/4942000] [107.4/1000.0] [batch_t 0.777 (0.764)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-05 10:59:49,982 - Train: 10.74% [530800/4942000] [107.4/1000.0] [batch_t 0.774 (0.764)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-05 11:01:06,431 - Train: 10.74% [530900/4942000] [107.4/1000.0] [batch_t 0.766 (0.764)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-05 11:02:22,908 - Train: 10.74% [531000/4942000] [107.4/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-05 11:03:39,431 - Train: 10.75% [531100/4942000] [107.5/1000.0] [batch_t 0.775 (0.765)] [data_t 0.003] [optim_t 0.772] [lr 0.005000] 2024-04-05 11:04:55,974 - Train: 10.75% [531200/4942000] [107.5/1000.0] [batch_t 0.778 (0.765)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-05 11:06:12,474 - Train: 10.75% [531300/4942000] [107.5/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 11:07:28,947 - Train: 10.75% [531400/4942000] [107.5/1000.0] [batch_t 0.762 (0.765)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-05 11:08:45,526 - Train: 10.75% [531500/4942000] [107.5/1000.0] [batch_t 0.762 (0.766)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 11:10:01,939 - Train: 10.76% [531600/4942000] [107.6/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 11:11:18,426 - Train: 10.76% [531700/4942000] [107.6/1000.0] [batch_t 0.778 (0.765)] [data_t 0.002] [optim_t 0.776] [lr 0.005000] 2024-04-05 11:12:34,954 - Train: 10.76% [531800/4942000] [107.6/1000.0] [batch_t 0.770 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 11:13:51,593 - Train: 10.76% [531900/4942000] [107.6/1000.0] [batch_t 0.768 (0.766)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-05 11:15:08,176 - Train: 10.76% [532000/4942000] [107.6/1000.0] [batch_t 0.762 (0.766)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 11:16:24,671 - Train: 10.77% [532100/4942000] [107.7/1000.0] [batch_t 0.758 (0.765)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-05 11:17:41,227 - Train: 10.77% [532200/4942000] [107.7/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 11:18:57,662 - Train: 10.77% [532300/4942000] [107.7/1000.0] [batch_t 0.771 (0.764)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-05 11:20:14,142 - Train: 10.77% [532400/4942000] [107.7/1000.0] [batch_t 0.767 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-05 11:21:30,691 - Train: 10.77% [532500/4942000] [107.7/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 11:22:47,151 - Train: 10.78% [532600/4942000] [107.8/1000.0] [batch_t 0.758 (0.765)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-05 11:24:03,671 - Train: 10.78% [532700/4942000] [107.8/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 11:25:20,212 - Train: 10.78% [532800/4942000] [107.8/1000.0] [batch_t 0.773 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 11:26:36,682 - Train: 10.78% [532900/4942000] [107.8/1000.0] [batch_t 0.769 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 11:27:53,240 - Train: 10.79% [533000/4942000] [107.9/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-05 11:29:09,720 - Train: 10.79% [533100/4942000] [107.9/1000.0] [batch_t 0.769 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 11:30:26,255 - Train: 10.79% [533200/4942000] [107.9/1000.0] [batch_t 0.753 (0.765)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-05 11:31:42,746 - Train: 10.79% [533300/4942000] [107.9/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-05 11:32:59,331 - Train: 10.79% [533400/4942000] [107.9/1000.0] [batch_t 0.759 (0.766)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-05 11:34:15,880 - Train: 10.80% [533500/4942000] [108.0/1000.0] [batch_t 0.759 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-05 11:35:32,472 - Train: 10.80% [533600/4942000] [108.0/1000.0] [batch_t 0.765 (0.766)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-05 11:36:48,945 - Train: 10.80% [533700/4942000] [108.0/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 11:37:16,530 - ==> Total time: 2 days, 17:39:55 Eta: 22 days, 14:20:53 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-05 11:38:07,457 - Train: 10.80% [533800/4942000] [108.0/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 11:39:23,926 - Train: 10.80% [533900/4942000] [108.0/1000.0] [batch_t 0.770 (0.765)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-05 11:40:40,398 - Train: 10.81% [534000/4942000] [108.1/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 11:41:56,845 - Train: 10.81% [534100/4942000] [108.1/1000.0] [batch_t 0.762 (0.764)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-05 11:43:13,354 - Train: 10.81% [534200/4942000] [108.1/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-05 11:44:29,807 - Train: 10.81% [534300/4942000] [108.1/1000.0] [batch_t 0.761 (0.764)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-05 11:45:46,273 - Train: 10.81% [534400/4942000] [108.1/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 11:47:02,747 - Train: 10.82% [534500/4942000] [108.2/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 11:48:19,277 - Train: 10.82% [534600/4942000] [108.2/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 11:49:35,784 - Train: 10.82% [534700/4942000] [108.2/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 11:50:52,269 - Train: 10.82% [534800/4942000] [108.2/1000.0] [batch_t 0.759 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-05 11:52:08,713 - Train: 10.82% [534900/4942000] [108.2/1000.0] [batch_t 0.777 (0.764)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-05 11:53:25,180 - Train: 10.83% [535000/4942000] [108.3/1000.0] [batch_t 0.764 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 11:54:41,648 - Train: 10.83% [535100/4942000] [108.3/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 11:55:58,111 - Train: 10.83% [535200/4942000] [108.3/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 11:57:14,514 - Train: 10.83% [535300/4942000] [108.3/1000.0] [batch_t 0.754 (0.764)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-05 11:58:30,951 - Train: 10.83% [535400/4942000] [108.3/1000.0] [batch_t 0.769 (0.764)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 11:59:47,462 - Train: 10.84% [535500/4942000] [108.4/1000.0] [batch_t 0.764 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 12:01:03,938 - Train: 10.84% [535600/4942000] [108.4/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 12:02:20,439 - Train: 10.84% [535700/4942000] [108.4/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-05 12:03:36,976 - Train: 10.84% [535800/4942000] [108.4/1000.0] [batch_t 0.764 (0.765)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-05 12:04:53,511 - Train: 10.84% [535900/4942000] [108.4/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-05 12:06:09,911 - Train: 10.85% [536000/4942000] [108.5/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 12:07:26,317 - Train: 10.85% [536100/4942000] [108.5/1000.0] [batch_t 0.769 (0.764)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-05 12:08:42,849 - Train: 10.85% [536200/4942000] [108.5/1000.0] [batch_t 0.757 (0.765)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-05 12:09:59,371 - Train: 10.85% [536300/4942000] [108.5/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-05 12:11:15,741 - Train: 10.85% [536400/4942000] [108.5/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 12:12:32,061 - Train: 10.86% [536500/4942000] [108.6/1000.0] [batch_t 0.771 (0.763)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-05 12:13:48,484 - Train: 10.86% [536600/4942000] [108.6/1000.0] [batch_t 0.777 (0.764)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-05 12:15:04,972 - Train: 10.86% [536700/4942000] [108.6/1000.0] [batch_t 0.778 (0.765)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-05 12:16:21,442 - Train: 10.86% [536800/4942000] [108.6/1000.0] [batch_t 0.760 (0.765)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-05 12:17:38,011 - Train: 10.86% [536900/4942000] [108.6/1000.0] [batch_t 0.765 (0.766)] [data_t 0.002] [optim_t 0.763] [lr 0.005000] 2024-04-05 12:18:54,590 - Train: 10.87% [537000/4942000] [108.7/1000.0] [batch_t 0.759 (0.766)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-05 12:20:11,095 - Train: 10.87% [537100/4942000] [108.7/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 12:21:27,593 - Train: 10.87% [537200/4942000] [108.7/1000.0] [batch_t 0.769 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 12:22:44,028 - Train: 10.87% [537300/4942000] [108.7/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 12:24:00,478 - Train: 10.87% [537400/4942000] [108.7/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 12:25:17,036 - Train: 10.88% [537500/4942000] [108.8/1000.0] [batch_t 0.773 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 12:26:33,470 - Train: 10.88% [537600/4942000] [108.8/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 12:27:49,924 - Train: 10.88% [537700/4942000] [108.8/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-05 12:29:06,346 - Train: 10.88% [537800/4942000] [108.8/1000.0] [batch_t 0.777 (0.764)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-05 12:30:22,947 - Train: 10.88% [537900/4942000] [108.8/1000.0] [batch_t 0.768 (0.766)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 12:31:39,373 - Train: 10.89% [538000/4942000] [108.9/1000.0] [batch_t 0.782 (0.764)] [data_t 0.002] [optim_t 0.779] [lr 0.005000] 2024-04-05 12:32:55,868 - Train: 10.89% [538100/4942000] [108.9/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-05 12:34:12,409 - Train: 10.89% [538200/4942000] [108.9/1000.0] [batch_t 0.754 (0.765)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-05 12:35:28,942 - Train: 10.89% [538300/4942000] [108.9/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 12:36:45,313 - Train: 10.89% [538400/4942000] [108.9/1000.0] [batch_t 0.754 (0.764)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-05 12:38:01,773 - Train: 10.90% [538500/4942000] [109.0/1000.0] [batch_t 0.769 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 12:39:18,236 - Train: 10.90% [538600/4942000] [109.0/1000.0] [batch_t 0.760 (0.765)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-05 12:40:17,879 - ==> Total time: 2 days, 18:42:57 Eta: 22 days, 17:21:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-05 12:40:36,613 - Train: 10.90% [538700/4942000] [109.0/1000.0] [batch_t 0.748 (0.764)] [data_t 0.003] [optim_t 0.745] [lr 0.005000] 2024-04-05 12:41:53,097 - Train: 10.90% [538800/4942000] [109.0/1000.0] [batch_t 0.758 (0.765)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 12:43:09,445 - Train: 10.90% [538900/4942000] [109.0/1000.0] [batch_t 0.778 (0.763)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-05 12:44:25,970 - Train: 10.91% [539000/4942000] [109.1/1000.0] [batch_t 0.781 (0.765)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-05 12:45:42,484 - Train: 10.91% [539100/4942000] [109.1/1000.0] [batch_t 0.773 (0.765)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-05 12:46:58,964 - Train: 10.91% [539200/4942000] [109.1/1000.0] [batch_t 0.764 (0.765)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-05 12:48:15,464 - Train: 10.91% [539300/4942000] [109.1/1000.0] [batch_t 0.769 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 12:49:31,860 - Train: 10.91% [539400/4942000] [109.1/1000.0] [batch_t 0.771 (0.764)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-05 12:50:48,182 - Train: 10.92% [539500/4942000] [109.2/1000.0] [batch_t 0.754 (0.763)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-05 12:52:04,608 - Train: 10.92% [539600/4942000] [109.2/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 12:53:21,118 - Train: 10.92% [539700/4942000] [109.2/1000.0] [batch_t 0.749 (0.765)] [data_t 0.003] [optim_t 0.746] [lr 0.005000] 2024-04-05 12:54:37,532 - Train: 10.92% [539800/4942000] [109.2/1000.0] [batch_t 0.773 (0.764)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-05 12:55:54,043 - Train: 10.92% [539900/4942000] [109.2/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 12:57:10,612 - Train: 10.93% [540000/4942000] [109.3/1000.0] [batch_t 0.768 (0.766)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 12:58:27,199 - Train: 10.93% [540100/4942000] [109.3/1000.0] [batch_t 0.772 (0.766)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 12:59:43,738 - Train: 10.93% [540200/4942000] [109.3/1000.0] [batch_t 0.758 (0.765)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-05 13:01:00,211 - Train: 10.93% [540300/4942000] [109.3/1000.0] [batch_t 0.764 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 13:02:16,748 - Train: 10.93% [540400/4942000] [109.3/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 13:03:33,339 - Train: 10.94% [540500/4942000] [109.4/1000.0] [batch_t 0.770 (0.766)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-05 13:04:49,772 - Train: 10.94% [540600/4942000] [109.4/1000.0] [batch_t 0.755 (0.764)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-05 13:06:06,285 - Train: 10.94% [540700/4942000] [109.4/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-05 13:07:22,758 - Train: 10.94% [540800/4942000] [109.4/1000.0] [batch_t 0.762 (0.765)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-05 13:08:39,215 - Train: 10.94% [540900/4942000] [109.4/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 13:09:55,699 - Train: 10.95% [541000/4942000] [109.5/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 13:11:12,369 - Train: 10.95% [541100/4942000] [109.5/1000.0] [batch_t 0.763 (0.767)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-05 13:12:28,925 - Train: 10.95% [541200/4942000] [109.5/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 13:13:45,520 - Train: 10.95% [541300/4942000] [109.5/1000.0] [batch_t 0.765 (0.766)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-05 13:15:02,181 - Train: 10.96% [541400/4942000] [109.6/1000.0] [batch_t 0.782 (0.767)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-05 13:16:18,694 - Train: 10.96% [541500/4942000] [109.6/1000.0] [batch_t 0.770 (0.765)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-05 13:17:35,018 - Train: 10.96% [541600/4942000] [109.6/1000.0] [batch_t 0.767 (0.763)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 13:18:51,407 - Train: 10.96% [541700/4942000] [109.6/1000.0] [batch_t 0.779 (0.764)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-05 13:20:07,792 - Train: 10.96% [541800/4942000] [109.6/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 13:21:24,339 - Train: 10.97% [541900/4942000] [109.7/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 13:22:40,871 - Train: 10.97% [542000/4942000] [109.7/1000.0] [batch_t 0.771 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-05 13:23:57,396 - Train: 10.97% [542100/4942000] [109.7/1000.0] [batch_t 0.764 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 13:25:14,001 - Train: 10.97% [542200/4942000] [109.7/1000.0] [batch_t 0.774 (0.766)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-05 13:26:30,530 - Train: 10.97% [542300/4942000] [109.7/1000.0] [batch_t 0.764 (0.765)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-05 13:27:46,977 - Train: 10.98% [542400/4942000] [109.8/1000.0] [batch_t 0.762 (0.764)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-05 13:29:03,523 - Train: 10.98% [542500/4942000] [109.8/1000.0] [batch_t 0.754 (0.765)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-05 13:30:20,000 - Train: 10.98% [542600/4942000] [109.8/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 13:31:37,522 - Train: 10.98% [542700/4942000] [109.8/1000.0] [batch_t 0.767 (0.775)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 13:32:54,007 - Train: 10.98% [542800/4942000] [109.8/1000.0] [batch_t 0.777 (0.765)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-05 13:34:10,392 - Train: 10.99% [542900/4942000] [109.9/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 13:35:26,920 - Train: 10.99% [543000/4942000] [109.9/1000.0] [batch_t 0.764 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 13:36:43,484 - Train: 10.99% [543100/4942000] [109.9/1000.0] [batch_t 0.777 (0.766)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-05 13:37:59,985 - Train: 10.99% [543200/4942000] [109.9/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 13:39:16,380 - Train: 10.99% [543300/4942000] [109.9/1000.0] [batch_t 0.773 (0.764)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-05 13:40:32,737 - Train: 11.00% [543400/4942000] [110.0/1000.0] [batch_t 0.772 (0.763)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 13:41:49,112 - Train: 11.00% [543500/4942000] [110.0/1000.0] [batch_t 0.750 (0.764)] [data_t 0.002] [optim_t 0.748] [lr 0.005000] 2024-04-05 13:43:05,575 - Train: 11.00% [543600/4942000] [110.0/1000.0] [batch_t 0.784 (0.765)] [data_t 0.002] [optim_t 0.781] [lr 0.005000] 2024-04-05 13:43:20,855 - ==> Total time: 2 days, 19:46:00 Eta: 22 days, 20:17:38 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-05 13:44:24,326 - Train: 11.00% [543700/4942000] [110.0/1000.0] [batch_t 0.766 (0.766)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-05 13:45:40,746 - Train: 11.00% [543800/4942000] [110.0/1000.0] [batch_t 0.778 (0.764)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-05 13:46:57,086 - Train: 11.01% [543900/4942000] [110.1/1000.0] [batch_t 0.754 (0.763)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-05 13:48:13,517 - Train: 11.01% [544000/4942000] [110.1/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 13:49:30,091 - Train: 11.01% [544100/4942000] [110.1/1000.0] [batch_t 0.772 (0.766)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 13:50:46,689 - Train: 11.01% [544200/4942000] [110.1/1000.0] [batch_t 0.777 (0.766)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-05 13:52:03,127 - Train: 11.01% [544300/4942000] [110.1/1000.0] [batch_t 0.747 (0.764)] [data_t 0.003] [optim_t 0.744] [lr 0.005000] 2024-04-05 13:53:19,501 - Train: 11.02% [544400/4942000] [110.2/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 13:54:35,965 - Train: 11.02% [544500/4942000] [110.2/1000.0] [batch_t 0.769 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 13:55:52,342 - Train: 11.02% [544600/4942000] [110.2/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 13:57:08,800 - Train: 11.02% [544700/4942000] [110.2/1000.0] [batch_t 0.768 (0.764)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-05 13:58:25,329 - Train: 11.02% [544800/4942000] [110.2/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 13:59:41,726 - Train: 11.03% [544900/4942000] [110.3/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 14:00:58,262 - Train: 11.03% [545000/4942000] [110.3/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 14:02:14,873 - Train: 11.03% [545100/4942000] [110.3/1000.0] [batch_t 0.772 (0.766)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 14:03:31,353 - Train: 11.03% [545200/4942000] [110.3/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 14:04:47,853 - Train: 11.03% [545300/4942000] [110.3/1000.0] [batch_t 0.770 (0.765)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-05 14:06:04,333 - Train: 11.04% [545400/4942000] [110.4/1000.0] [batch_t 0.764 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 14:07:20,804 - Train: 11.04% [545500/4942000] [110.4/1000.0] [batch_t 0.769 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 14:08:37,262 - Train: 11.04% [545600/4942000] [110.4/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 14:09:53,715 - Train: 11.04% [545700/4942000] [110.4/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-05 14:11:10,245 - Train: 11.04% [545800/4942000] [110.4/1000.0] [batch_t 0.767 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-05 14:12:26,627 - Train: 11.05% [545900/4942000] [110.5/1000.0] [batch_t 0.769 (0.764)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 14:13:43,169 - Train: 11.05% [546000/4942000] [110.5/1000.0] [batch_t 0.776 (0.765)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-05 14:14:59,644 - Train: 11.05% [546100/4942000] [110.5/1000.0] [batch_t 0.756 (0.765)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-05 14:16:16,101 - Train: 11.05% [546200/4942000] [110.5/1000.0] [batch_t 0.753 (0.764)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-05 14:17:32,637 - Train: 11.05% [546300/4942000] [110.5/1000.0] [batch_t 0.758 (0.765)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 14:18:49,179 - Train: 11.06% [546400/4942000] [110.6/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 14:20:05,574 - Train: 11.06% [546500/4942000] [110.6/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-05 14:21:21,989 - Train: 11.06% [546600/4942000] [110.6/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 14:22:38,467 - Train: 11.06% [546700/4942000] [110.6/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 14:23:55,035 - Train: 11.06% [546800/4942000] [110.6/1000.0] [batch_t 0.771 (0.766)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-05 14:25:11,359 - Train: 11.07% [546900/4942000] [110.7/1000.0] [batch_t 0.775 (0.763)] [data_t 0.003] [optim_t 0.772] [lr 0.005000] 2024-04-05 14:26:27,819 - Train: 11.07% [547000/4942000] [110.7/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 14:27:44,442 - Train: 11.07% [547100/4942000] [110.7/1000.0] [batch_t 0.768 (0.766)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 14:29:00,869 - Train: 11.07% [547200/4942000] [110.7/1000.0] [batch_t 0.769 (0.764)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 14:30:17,392 - Train: 11.07% [547300/4942000] [110.7/1000.0] [batch_t 0.752 (0.765)] [data_t 0.003] [optim_t 0.748] [lr 0.005000] 2024-04-05 14:31:33,746 - Train: 11.08% [547400/4942000] [110.8/1000.0] [batch_t 0.769 (0.763)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 14:32:50,276 - Train: 11.08% [547500/4942000] [110.8/1000.0] [batch_t 0.753 (0.765)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-05 14:34:06,839 - Train: 11.08% [547600/4942000] [110.8/1000.0] [batch_t 0.750 (0.766)] [data_t 0.002] [optim_t 0.747] [lr 0.005000] 2024-04-05 14:35:23,323 - Train: 11.08% [547700/4942000] [110.8/1000.0] [batch_t 0.766 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 14:36:39,732 - Train: 11.08% [547800/4942000] [110.8/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 14:37:56,175 - Train: 11.09% [547900/4942000] [110.9/1000.0] [batch_t 0.757 (0.764)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-05 14:39:12,595 - Train: 11.09% [548000/4942000] [110.9/1000.0] [batch_t 0.767 (0.764)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-05 14:40:29,040 - Train: 11.09% [548100/4942000] [110.9/1000.0] [batch_t 0.776 (0.764)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-05 14:41:45,491 - Train: 11.09% [548200/4942000] [110.9/1000.0] [batch_t 0.768 (0.764)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-05 14:43:02,044 - Train: 11.09% [548300/4942000] [110.9/1000.0] [batch_t 0.759 (0.765)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-05 14:44:18,571 - Train: 11.10% [548400/4942000] [111.0/1000.0] [batch_t 0.769 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 14:45:35,077 - Train: 11.10% [548500/4942000] [111.0/1000.0] [batch_t 0.746 (0.765)] [data_t 0.002] [optim_t 0.744] [lr 0.005000] 2024-04-05 14:46:22,348 - ==> Total time: 2 days, 20:49:01 Eta: 22 days, 23:09:24 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-05 14:46:53,410 - Train: 11.10% [548600/4942000] [111.0/1000.0] [batch_t 0.759 (0.765)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-05 14:48:09,953 - Train: 11.10% [548700/4942000] [111.0/1000.0] [batch_t 0.769 (0.765)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-05 14:49:26,528 - Train: 11.10% [548800/4942000] [111.0/1000.0] [batch_t 0.773 (0.766)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-05 14:50:42,949 - Train: 11.11% [548900/4942000] [111.1/1000.0] [batch_t 0.750 (0.764)] [data_t 0.002] [optim_t 0.747] [lr 0.005000] 2024-04-05 14:51:59,386 - Train: 11.11% [549000/4942000] [111.1/1000.0] [batch_t 0.773 (0.764)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 14:53:15,765 - Train: 11.11% [549100/4942000] [111.1/1000.0] [batch_t 0.754 (0.764)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-05 14:54:32,033 - Train: 11.11% [549200/4942000] [111.1/1000.0] [batch_t 0.764 (0.763)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-05 14:55:48,443 - Train: 11.11% [549300/4942000] [111.1/1000.0] [batch_t 0.766 (0.764)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-05 14:57:04,980 - Train: 11.12% [549400/4942000] [111.2/1000.0] [batch_t 0.754 (0.765)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-05 14:58:21,433 - Train: 11.12% [549500/4942000] [111.2/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 14:59:37,979 - Train: 11.12% [549600/4942000] [111.2/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 15:00:54,599 - Train: 11.12% [549700/4942000] [111.2/1000.0] [batch_t 0.765 (0.766)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-05 15:02:10,978 - Train: 11.13% [549800/4942000] [111.3/1000.0] [batch_t 0.762 (0.764)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-05 15:03:27,443 - Train: 11.13% [549900/4942000] [111.3/1000.0] [batch_t 0.759 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-05 15:04:43,962 - Train: 11.13% [550000/4942000] [111.3/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 15:06:00,475 - Train: 11.13% [550100/4942000] [111.3/1000.0] [batch_t 0.753 (0.765)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-05 15:07:17,055 - Train: 11.13% [550200/4942000] [111.3/1000.0] [batch_t 0.765 (0.766)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-05 15:08:33,563 - Train: 11.14% [550300/4942000] [111.4/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 15:09:49,978 - Train: 11.14% [550400/4942000] [111.4/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-05 15:11:06,473 - Train: 11.14% [550500/4942000] [111.4/1000.0] [batch_t 0.776 (0.765)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-05 15:12:22,861 - Train: 11.14% [550600/4942000] [111.4/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 15:13:39,314 - Train: 11.14% [550700/4942000] [111.4/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 15:14:55,800 - Train: 11.15% [550800/4942000] [111.5/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-05 15:16:12,067 - Train: 11.15% [550900/4942000] [111.5/1000.0] [batch_t 0.764 (0.763)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-05 15:17:28,496 - Train: 11.15% [551000/4942000] [111.5/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 15:18:45,004 - Train: 11.15% [551100/4942000] [111.5/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 15:20:01,480 - Train: 11.15% [551200/4942000] [111.5/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 15:21:17,775 - Train: 11.16% [551300/4942000] [111.6/1000.0] [batch_t 0.761 (0.763)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-05 15:22:34,283 - Train: 11.16% [551400/4942000] [111.6/1000.0] [batch_t 0.766 (0.765)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-05 15:23:50,709 - Train: 11.16% [551500/4942000] [111.6/1000.0] [batch_t 0.773 (0.764)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-05 15:25:07,235 - Train: 11.16% [551600/4942000] [111.6/1000.0] [batch_t 0.759 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-05 15:26:23,836 - Train: 11.16% [551700/4942000] [111.6/1000.0] [batch_t 0.763 (0.766)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 15:27:40,372 - Train: 11.17% [551800/4942000] [111.7/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 15:28:56,837 - Train: 11.17% [551900/4942000] [111.7/1000.0] [batch_t 0.758 (0.765)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-05 15:30:13,307 - Train: 11.17% [552000/4942000] [111.7/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 15:31:29,755 - Train: 11.17% [552100/4942000] [111.7/1000.0] [batch_t 0.770 (0.764)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-05 15:32:46,328 - Train: 11.17% [552200/4942000] [111.7/1000.0] [batch_t 0.769 (0.766)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 15:34:02,879 - Train: 11.18% [552300/4942000] [111.8/1000.0] [batch_t 0.766 (0.765)] [data_t 0.002] [optim_t 0.763] [lr 0.005000] 2024-04-05 15:35:19,418 - Train: 11.18% [552400/4942000] [111.8/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 15:36:35,913 - Train: 11.18% [552500/4942000] [111.8/1000.0] [batch_t 0.765 (0.765)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-05 15:37:52,459 - Train: 11.18% [552600/4942000] [111.8/1000.0] [batch_t 0.774 (0.765)] [data_t 0.002] [optim_t 0.772] [lr 0.005000] 2024-04-05 15:39:08,911 - Train: 11.18% [552700/4942000] [111.8/1000.0] [batch_t 0.755 (0.764)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-05 15:40:25,282 - Train: 11.19% [552800/4942000] [111.9/1000.0] [batch_t 0.764 (0.764)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-05 15:41:41,855 - Train: 11.19% [552900/4942000] [111.9/1000.0] [batch_t 0.768 (0.766)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 15:42:58,229 - Train: 11.19% [553000/4942000] [111.9/1000.0] [batch_t 0.766 (0.764)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-05 15:44:14,689 - Train: 11.19% [553100/4942000] [111.9/1000.0] [batch_t 0.756 (0.765)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-05 15:45:31,300 - Train: 11.19% [553200/4942000] [111.9/1000.0] [batch_t 0.753 (0.766)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-05 15:46:47,739 - Train: 11.20% [553300/4942000] [112.0/1000.0] [batch_t 0.755 (0.764)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-05 15:48:04,255 - Train: 11.20% [553400/4942000] [112.0/1000.0] [batch_t 0.769 (0.765)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-05 15:49:20,650 - Train: 11.20% [553500/4942000] [112.0/1000.0] [batch_t 0.764 (0.764)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-05 15:49:23,728 - ==> Total time: 2 days, 21:52:02 Eta: 23 days, 1:56:57 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-05 15:50:39,153 - Train: 11.20% [553600/4942000] [112.0/1000.0] [batch_t 0.757 (0.767)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-05 15:51:55,731 - Train: 11.20% [553700/4942000] [112.0/1000.0] [batch_t 0.758 (0.766)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 15:53:12,261 - Train: 11.21% [553800/4942000] [112.1/1000.0] [batch_t 0.778 (0.765)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-05 15:54:28,762 - Train: 11.21% [553900/4942000] [112.1/1000.0] [batch_t 0.767 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-05 15:55:45,385 - Train: 11.21% [554000/4942000] [112.1/1000.0] [batch_t 0.777 (0.766)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-05 15:57:01,953 - Train: 11.21% [554100/4942000] [112.1/1000.0] [batch_t 0.755 (0.766)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-05 15:58:18,507 - Train: 11.21% [554200/4942000] [112.1/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 15:59:35,095 - Train: 11.22% [554300/4942000] [112.2/1000.0] [batch_t 0.763 (0.766)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 16:00:51,687 - Train: 11.22% [554400/4942000] [112.2/1000.0] [batch_t 0.755 (0.766)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-05 16:02:08,112 - Train: 11.22% [554500/4942000] [112.2/1000.0] [batch_t 0.750 (0.764)] [data_t 0.003] [optim_t 0.747] [lr 0.005000] 2024-04-05 16:03:24,595 - Train: 11.22% [554600/4942000] [112.2/1000.0] [batch_t 0.752 (0.765)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-05 16:04:41,106 - Train: 11.22% [554700/4942000] [112.2/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 16:05:57,725 - Train: 11.23% [554800/4942000] [112.3/1000.0] [batch_t 0.764 (0.766)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 16:07:14,330 - Train: 11.23% [554900/4942000] [112.3/1000.0] [batch_t 0.774 (0.766)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-05 16:08:30,997 - Train: 11.23% [555000/4942000] [112.3/1000.0] [batch_t 0.753 (0.767)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-05 16:09:47,476 - Train: 11.23% [555100/4942000] [112.3/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 16:11:04,000 - Train: 11.23% [555200/4942000] [112.3/1000.0] [batch_t 0.778 (0.765)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-05 16:12:20,475 - Train: 11.24% [555300/4942000] [112.4/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 16:13:37,113 - Train: 11.24% [555400/4942000] [112.4/1000.0] [batch_t 0.773 (0.766)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 16:14:53,607 - Train: 11.24% [555500/4942000] [112.4/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 16:16:10,113 - Train: 11.24% [555600/4942000] [112.4/1000.0] [batch_t 0.766 (0.765)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-05 16:17:26,561 - Train: 11.24% [555700/4942000] [112.4/1000.0] [batch_t 0.773 (0.764)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 16:18:43,003 - Train: 11.25% [555800/4942000] [112.5/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-05 16:19:59,351 - Train: 11.25% [555900/4942000] [112.5/1000.0] [batch_t 0.758 (0.763)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 16:21:15,862 - Train: 11.25% [556000/4942000] [112.5/1000.0] [batch_t 0.757 (0.765)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-05 16:22:32,270 - Train: 11.25% [556100/4942000] [112.5/1000.0] [batch_t 0.754 (0.764)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-05 16:23:48,731 - Train: 11.25% [556200/4942000] [112.5/1000.0] [batch_t 0.745 (0.765)] [data_t 0.002] [optim_t 0.743] [lr 0.005000] 2024-04-05 16:25:05,146 - Train: 11.26% [556300/4942000] [112.6/1000.0] [batch_t 0.755 (0.764)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-05 16:26:21,565 - Train: 11.26% [556400/4942000] [112.6/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 16:27:38,110 - Train: 11.26% [556500/4942000] [112.6/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 16:28:54,665 - Train: 11.26% [556600/4942000] [112.6/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 16:30:11,090 - Train: 11.26% [556700/4942000] [112.6/1000.0] [batch_t 0.778 (0.764)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-05 16:31:27,650 - Train: 11.27% [556800/4942000] [112.7/1000.0] [batch_t 0.759 (0.766)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-05 16:32:44,116 - Train: 11.27% [556900/4942000] [112.7/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 16:34:00,717 - Train: 11.27% [557000/4942000] [112.7/1000.0] [batch_t 0.769 (0.766)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-05 16:35:17,018 - Train: 11.27% [557100/4942000] [112.7/1000.0] [batch_t 0.764 (0.763)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 16:36:33,441 - Train: 11.27% [557200/4942000] [112.7/1000.0] [batch_t 0.773 (0.764)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-05 16:37:50,068 - Train: 11.28% [557300/4942000] [112.8/1000.0] [batch_t 0.768 (0.766)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 16:39:06,452 - Train: 11.28% [557400/4942000] [112.8/1000.0] [batch_t 0.755 (0.764)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-05 16:40:23,070 - Train: 11.28% [557500/4942000] [112.8/1000.0] [batch_t 0.772 (0.766)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-05 16:41:39,605 - Train: 11.28% [557600/4942000] [112.8/1000.0] [batch_t 0.767 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-05 16:42:56,207 - Train: 11.28% [557700/4942000] [112.8/1000.0] [batch_t 0.762 (0.766)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 16:44:12,736 - Train: 11.29% [557800/4942000] [112.9/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 16:45:29,333 - Train: 11.29% [557900/4942000] [112.9/1000.0] [batch_t 0.772 (0.766)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 16:46:45,838 - Train: 11.29% [558000/4942000] [112.9/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 16:48:02,303 - Train: 11.29% [558100/4942000] [112.9/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 16:49:18,797 - Train: 11.30% [558200/4942000] [113.0/1000.0] [batch_t 0.750 (0.765)] [data_t 0.003] [optim_t 0.748] [lr 0.005000] 2024-04-05 16:50:35,355 - Train: 11.30% [558300/4942000] [113.0/1000.0] [batch_t 0.771 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-05 16:51:52,036 - Train: 11.30% [558400/4942000] [113.0/1000.0] [batch_t 0.778 (0.767)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-05 16:52:27,236 - ==> Total time: 2 days, 22:55:06 Eta: 23 days, 4:40:42 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-05 16:53:10,546 - Train: 11.30% [558500/4942000] [113.0/1000.0] [batch_t 0.747 (0.765)] [data_t 0.003] [optim_t 0.745] [lr 0.005000] 2024-04-05 16:54:27,216 - Train: 11.30% [558600/4942000] [113.0/1000.0] [batch_t 0.777 (0.767)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-05 16:55:43,834 - Train: 11.31% [558700/4942000] [113.1/1000.0] [batch_t 0.771 (0.766)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-05 16:57:00,356 - Train: 11.31% [558800/4942000] [113.1/1000.0] [batch_t 0.759 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-05 16:58:16,740 - Train: 11.31% [558900/4942000] [113.1/1000.0] [batch_t 0.772 (0.764)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 16:59:33,246 - Train: 11.31% [559000/4942000] [113.1/1000.0] [batch_t 0.767 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-05 17:00:49,671 - Train: 11.31% [559100/4942000] [113.1/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-05 17:02:06,219 - Train: 11.32% [559200/4942000] [113.2/1000.0] [batch_t 0.759 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-05 17:03:22,577 - Train: 11.32% [559300/4942000] [113.2/1000.0] [batch_t 0.772 (0.763)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 17:04:38,973 - Train: 11.32% [559400/4942000] [113.2/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 17:05:55,498 - Train: 11.32% [559500/4942000] [113.2/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 17:07:12,058 - Train: 11.32% [559600/4942000] [113.2/1000.0] [batch_t 0.749 (0.765)] [data_t 0.003] [optim_t 0.746] [lr 0.005000] 2024-04-05 17:08:28,527 - Train: 11.33% [559700/4942000] [113.3/1000.0] [batch_t 0.778 (0.765)] [data_t 0.002] [optim_t 0.776] [lr 0.005000] 2024-04-05 17:09:45,056 - Train: 11.33% [559800/4942000] [113.3/1000.0] [batch_t 0.755 (0.765)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-05 17:11:01,655 - Train: 11.33% [559900/4942000] [113.3/1000.0] [batch_t 0.757 (0.766)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-05 17:12:18,016 - Train: 11.33% [560000/4942000] [113.3/1000.0] [batch_t 0.764 (0.764)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-05 17:13:34,424 - Train: 11.33% [560100/4942000] [113.3/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 17:14:50,909 - Train: 11.34% [560200/4942000] [113.4/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-05 17:16:07,342 - Train: 11.34% [560300/4942000] [113.4/1000.0] [batch_t 0.754 (0.764)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-05 17:17:23,715 - Train: 11.34% [560400/4942000] [113.4/1000.0] [batch_t 0.767 (0.764)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-05 17:18:40,100 - Train: 11.34% [560500/4942000] [113.4/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 17:19:56,749 - Train: 11.34% [560600/4942000] [113.4/1000.0] [batch_t 0.768 (0.766)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 17:21:13,251 - Train: 11.35% [560700/4942000] [113.5/1000.0] [batch_t 0.749 (0.765)] [data_t 0.002] [optim_t 0.747] [lr 0.005000] 2024-04-05 17:22:29,728 - Train: 11.35% [560800/4942000] [113.5/1000.0] [batch_t 0.764 (0.765)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-05 17:23:46,208 - Train: 11.35% [560900/4942000] [113.5/1000.0] [batch_t 0.779 (0.765)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-05 17:25:02,619 - Train: 11.35% [561000/4942000] [113.5/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 17:26:19,113 - Train: 11.35% [561100/4942000] [113.5/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 17:27:35,545 - Train: 11.36% [561200/4942000] [113.6/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 17:28:52,177 - Train: 11.36% [561300/4942000] [113.6/1000.0] [batch_t 0.782 (0.766)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-05 17:30:08,657 - Train: 11.36% [561400/4942000] [113.6/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-05 17:31:25,095 - Train: 11.36% [561500/4942000] [113.6/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 17:32:41,595 - Train: 11.36% [561600/4942000] [113.6/1000.0] [batch_t 0.769 (0.765)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-05 17:33:58,067 - Train: 11.37% [561700/4942000] [113.7/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 17:35:14,478 - Train: 11.37% [561800/4942000] [113.7/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 17:36:30,940 - Train: 11.37% [561900/4942000] [113.7/1000.0] [batch_t 0.755 (0.765)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-05 17:37:47,554 - Train: 11.37% [562000/4942000] [113.7/1000.0] [batch_t 0.773 (0.766)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 17:39:03,904 - Train: 11.37% [562100/4942000] [113.7/1000.0] [batch_t 0.758 (0.763)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-05 17:40:20,321 - Train: 11.38% [562200/4942000] [113.8/1000.0] [batch_t 0.766 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 17:41:36,763 - Train: 11.38% [562300/4942000] [113.8/1000.0] [batch_t 0.777 (0.764)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-05 17:42:53,292 - Train: 11.38% [562400/4942000] [113.8/1000.0] [batch_t 0.758 (0.765)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-05 17:44:09,627 - Train: 11.38% [562500/4942000] [113.8/1000.0] [batch_t 0.771 (0.763)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 17:45:26,067 - Train: 11.38% [562600/4942000] [113.8/1000.0] [batch_t 0.777 (0.764)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-05 17:46:42,530 - Train: 11.39% [562700/4942000] [113.9/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-05 17:47:58,977 - Train: 11.39% [562800/4942000] [113.9/1000.0] [batch_t 0.765 (0.764)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-05 17:49:15,461 - Train: 11.39% [562900/4942000] [113.9/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 17:50:32,071 - Train: 11.39% [563000/4942000] [113.9/1000.0] [batch_t 0.772 (0.766)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 17:51:48,470 - Train: 11.39% [563100/4942000] [113.9/1000.0] [batch_t 0.775 (0.764)] [data_t 0.003] [optim_t 0.772] [lr 0.005000] 2024-04-05 17:53:04,923 - Train: 11.40% [563200/4942000] [114.0/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 17:54:21,363 - Train: 11.40% [563300/4942000] [114.0/1000.0] [batch_t 0.779 (0.764)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-05 17:55:28,601 - ==> Total time: 2 days, 23:58:07 Eta: 23 days, 7:20:12 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-05 17:55:40,004 - Train: 11.40% [563400/4942000] [114.0/1000.0] [batch_t 0.759 (0.772)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-05 17:56:56,509 - Train: 11.40% [563500/4942000] [114.0/1000.0] [batch_t 0.758 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-05 17:58:12,977 - Train: 11.40% [563600/4942000] [114.0/1000.0] [batch_t 0.761 (0.765)] [data_t 0.002] [optim_t 0.758] [lr 0.005000] 2024-04-05 17:59:29,374 - Train: 11.41% [563700/4942000] [114.1/1000.0] [batch_t 0.776 (0.764)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-05 18:00:45,729 - Train: 11.41% [563800/4942000] [114.1/1000.0] [batch_t 0.762 (0.763)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 18:02:02,038 - Train: 11.41% [563900/4942000] [114.1/1000.0] [batch_t 0.757 (0.763)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 18:03:18,452 - Train: 11.41% [564000/4942000] [114.1/1000.0] [batch_t 0.771 (0.764)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-05 18:04:34,825 - Train: 11.41% [564100/4942000] [114.1/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 18:05:51,256 - Train: 11.42% [564200/4942000] [114.2/1000.0] [batch_t 0.769 (0.764)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 18:07:07,886 - Train: 11.42% [564300/4942000] [114.2/1000.0] [batch_t 0.774 (0.766)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-05 18:08:24,498 - Train: 11.42% [564400/4942000] [114.2/1000.0] [batch_t 0.768 (0.766)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 18:09:40,854 - Train: 11.42% [564500/4942000] [114.2/1000.0] [batch_t 0.766 (0.763)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-05 18:10:57,394 - Train: 11.42% [564600/4942000] [114.2/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 18:12:13,944 - Train: 11.43% [564700/4942000] [114.3/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 18:13:30,275 - Train: 11.43% [564800/4942000] [114.3/1000.0] [batch_t 0.763 (0.763)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-05 18:14:46,801 - Train: 11.43% [564900/4942000] [114.3/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 18:16:03,321 - Train: 11.43% [565000/4942000] [114.3/1000.0] [batch_t 0.761 (0.765)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-05 18:17:19,636 - Train: 11.43% [565100/4942000] [114.3/1000.0] [batch_t 0.767 (0.763)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 18:18:36,127 - Train: 11.44% [565200/4942000] [114.4/1000.0] [batch_t 0.759 (0.765)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-05 18:19:52,607 - Train: 11.44% [565300/4942000] [114.4/1000.0] [batch_t 0.764 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 18:21:09,023 - Train: 11.44% [565400/4942000] [114.4/1000.0] [batch_t 0.777 (0.764)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-05 18:22:26,176 - Train: 11.44% [565500/4942000] [114.4/1000.0] [batch_t 0.770 (0.771)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-05 18:23:43,265 - Train: 11.44% [565600/4942000] [114.4/1000.0] [batch_t 0.765 (0.771)] [data_t 0.002] [optim_t 0.763] [lr 0.005000] 2024-04-05 18:24:59,576 - Train: 11.45% [565700/4942000] [114.5/1000.0] [batch_t 0.778 (0.763)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-05 18:26:17,202 - Train: 11.45% [565800/4942000] [114.5/1000.0] [batch_t 0.771 (0.776)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-05 18:27:33,734 - Train: 11.45% [565900/4942000] [114.5/1000.0] [batch_t 0.776 (0.765)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-05 18:28:50,321 - Train: 11.45% [566000/4942000] [114.5/1000.0] [batch_t 0.763 (0.766)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-05 18:30:06,895 - Train: 11.45% [566100/4942000] [114.5/1000.0] [batch_t 0.754 (0.766)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-05 18:31:23,304 - Train: 11.46% [566200/4942000] [114.6/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 18:32:39,793 - Train: 11.46% [566300/4942000] [114.6/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 18:33:56,205 - Train: 11.46% [566400/4942000] [114.6/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 18:35:12,744 - Train: 11.46% [566500/4942000] [114.6/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 18:36:29,291 - Train: 11.46% [566600/4942000] [114.6/1000.0] [batch_t 0.778 (0.765)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-05 18:37:45,650 - Train: 11.47% [566700/4942000] [114.7/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-05 18:39:02,175 - Train: 11.47% [566800/4942000] [114.7/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 18:40:18,699 - Train: 11.47% [566900/4942000] [114.7/1000.0] [batch_t 0.758 (0.765)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 18:41:35,230 - Train: 11.47% [567000/4942000] [114.7/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 18:42:51,733 - Train: 11.48% [567100/4942000] [114.8/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 18:44:08,236 - Train: 11.48% [567200/4942000] [114.8/1000.0] [batch_t 0.769 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 18:45:24,608 - Train: 11.48% [567300/4942000] [114.8/1000.0] [batch_t 0.753 (0.764)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-05 18:46:41,075 - Train: 11.48% [567400/4942000] [114.8/1000.0] [batch_t 0.759 (0.765)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-05 18:47:57,393 - Train: 11.48% [567500/4942000] [114.8/1000.0] [batch_t 0.750 (0.763)] [data_t 0.002] [optim_t 0.747] [lr 0.005000] 2024-04-05 18:49:13,991 - Train: 11.49% [567600/4942000] [114.9/1000.0] [batch_t 0.769 (0.766)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 18:50:30,562 - Train: 11.49% [567700/4942000] [114.9/1000.0] [batch_t 0.764 (0.766)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-05 18:51:46,983 - Train: 11.49% [567800/4942000] [114.9/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 18:53:03,512 - Train: 11.49% [567900/4942000] [114.9/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 18:54:19,975 - Train: 11.49% [568000/4942000] [114.9/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 18:55:36,439 - Train: 11.50% [568100/4942000] [115.0/1000.0] [batch_t 0.773 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 18:56:52,785 - Train: 11.50% [568200/4942000] [115.0/1000.0] [batch_t 0.756 (0.763)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-05 18:58:09,150 - Train: 11.50% [568300/4942000] [115.0/1000.0] [batch_t 0.754 (0.764)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-05 18:58:32,171 - ==> Total time: 3 days, 1:01:11 Eta: 23 days, 9:56:06 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-05 18:59:27,678 - Train: 11.50% [568400/4942000] [115.0/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 19:00:44,150 - Train: 11.50% [568500/4942000] [115.0/1000.0] [batch_t 0.757 (0.765)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-05 19:02:00,492 - Train: 11.51% [568600/4942000] [115.1/1000.0] [batch_t 0.763 (0.763)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 19:03:17,393 - Train: 11.51% [568700/4942000] [115.1/1000.0] [batch_t 0.772 (0.769)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 19:04:33,752 - Train: 11.51% [568800/4942000] [115.1/1000.0] [batch_t 0.772 (0.763)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 19:05:50,273 - Train: 11.51% [568900/4942000] [115.1/1000.0] [batch_t 0.771 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 19:07:06,737 - Train: 11.51% [569000/4942000] [115.1/1000.0] [batch_t 0.776 (0.765)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-05 19:08:23,183 - Train: 11.52% [569100/4942000] [115.2/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-05 19:09:39,772 - Train: 11.52% [569200/4942000] [115.2/1000.0] [batch_t 0.774 (0.766)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-05 19:10:56,213 - Train: 11.52% [569300/4942000] [115.2/1000.0] [batch_t 0.752 (0.764)] [data_t 0.003] [optim_t 0.749] [lr 0.005000] 2024-04-05 19:12:20,393 - Train: 11.52% [569400/4942000] [115.2/1000.0] [batch_t 0.762 (0.842)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-05 19:13:38,603 - Train: 11.52% [569500/4942000] [115.2/1000.0] [batch_t 0.772 (0.782)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-05 19:14:57,322 - Train: 11.53% [569600/4942000] [115.3/1000.0] [batch_t 0.771 (0.787)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-05 19:16:16,670 - Train: 11.53% [569700/4942000] [115.3/1000.0] [batch_t 0.763 (0.793)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 19:17:33,166 - Train: 11.53% [569800/4942000] [115.3/1000.0] [batch_t 0.771 (0.765)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-05 19:18:49,499 - Train: 11.53% [569900/4942000] [115.3/1000.0] [batch_t 0.760 (0.763)] [data_t 0.002] [optim_t 0.758] [lr 0.005000] 2024-04-05 19:20:06,076 - Train: 11.53% [570000/4942000] [115.3/1000.0] [batch_t 0.773 (0.766)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 19:21:22,594 - Train: 11.54% [570100/4942000] [115.4/1000.0] [batch_t 0.759 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-05 19:22:39,174 - Train: 11.54% [570200/4942000] [115.4/1000.0] [batch_t 0.773 (0.766)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 19:23:55,702 - Train: 11.54% [570300/4942000] [115.4/1000.0] [batch_t 0.754 (0.765)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-05 19:25:13,061 - Train: 11.54% [570400/4942000] [115.4/1000.0] [batch_t 0.761 (0.773)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-05 19:26:30,176 - Train: 11.54% [570500/4942000] [115.4/1000.0] [batch_t 0.771 (0.771)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-05 19:27:46,633 - Train: 11.55% [570600/4942000] [115.5/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 19:29:17,969 - Train: 11.55% [570700/4942000] [115.5/1000.0] [batch_t 0.748 (0.913)] [data_t 0.003] [optim_t 0.745] [lr 0.005000] 2024-04-05 19:30:52,338 - Train: 11.55% [570800/4942000] [115.5/1000.0] [batch_t 0.772 (0.944)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 19:32:21,719 - Train: 11.55% [570900/4942000] [115.5/1000.0] [batch_t 0.764 (0.894)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-05 19:33:38,091 - Train: 11.55% [571000/4942000] [115.5/1000.0] [batch_t 0.762 (0.764)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-05 19:35:01,638 - Train: 11.56% [571100/4942000] [115.6/1000.0] [batch_t 0.749 (0.835)] [data_t 0.003] [optim_t 0.747] [lr 0.005000] 2024-04-05 19:36:31,390 - Train: 11.56% [571200/4942000] [115.6/1000.0] [batch_t 0.817 (0.897)] [data_t 0.043] [optim_t 0.774] [lr 0.005000] 2024-04-05 19:37:51,865 - Train: 11.56% [571300/4942000] [115.6/1000.0] [batch_t 0.768 (0.805)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 19:39:12,408 - Train: 11.56% [571400/4942000] [115.6/1000.0] [batch_t 0.768 (0.805)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 19:40:54,855 - Train: 11.56% [571500/4942000] [115.6/1000.0] [batch_t 0.759 (1.024)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-05 19:42:11,170 - Train: 11.57% [571600/4942000] [115.7/1000.0] [batch_t 0.767 (0.763)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 19:43:27,616 - Train: 11.57% [571700/4942000] [115.7/1000.0] [batch_t 0.753 (0.764)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-05 19:44:44,116 - Train: 11.57% [571800/4942000] [115.7/1000.0] [batch_t 0.754 (0.765)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-05 19:46:00,547 - Train: 11.57% [571900/4942000] [115.7/1000.0] [batch_t 0.759 (0.764)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-05 19:47:17,138 - Train: 11.57% [572000/4942000] [115.7/1000.0] [batch_t 0.776 (0.766)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-05 19:48:33,631 - Train: 11.58% [572100/4942000] [115.8/1000.0] [batch_t 0.765 (0.765)] [data_t 0.002] [optim_t 0.763] [lr 0.005000] 2024-04-05 19:49:50,174 - Train: 11.58% [572200/4942000] [115.8/1000.0] [batch_t 0.759 (0.765)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-05 19:51:06,568 - Train: 11.58% [572300/4942000] [115.8/1000.0] [batch_t 0.753 (0.764)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-05 19:52:23,112 - Train: 11.58% [572400/4942000] [115.8/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-05 19:53:39,680 - Train: 11.58% [572500/4942000] [115.8/1000.0] [batch_t 0.773 (0.766)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 19:54:56,121 - Train: 11.59% [572600/4942000] [115.9/1000.0] [batch_t 0.773 (0.764)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 19:56:12,634 - Train: 11.59% [572700/4942000] [115.9/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-05 19:57:29,138 - Train: 11.59% [572800/4942000] [115.9/1000.0] [batch_t 0.751 (0.765)] [data_t 0.003] [optim_t 0.749] [lr 0.005000] 2024-04-05 19:58:45,773 - Train: 11.59% [572900/4942000] [115.9/1000.0] [batch_t 0.769 (0.766)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 20:00:02,332 - Train: 11.59% [573000/4942000] [115.9/1000.0] [batch_t 0.754 (0.765)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-05 20:01:18,778 - Train: 11.60% [573100/4942000] [116.0/1000.0] [batch_t 0.753 (0.764)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-05 20:02:35,277 - Train: 11.60% [573200/4942000] [116.0/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-05 20:03:30,372 - ==> Total time: 3 days, 2:06:09 Eta: 23 days, 12:42:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-05 20:03:54,811 - Train: 11.60% [573300/4942000] [116.0/1000.0] [batch_t 0.767 (0.790)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-05 20:05:11,199 - Train: 11.60% [573400/4942000] [116.0/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-05 20:06:27,621 - Train: 11.60% [573500/4942000] [116.0/1000.0] [batch_t 0.764 (0.764)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-05 20:07:44,142 - Train: 11.61% [573600/4942000] [116.1/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-05 20:09:00,553 - Train: 11.61% [573700/4942000] [116.1/1000.0] [batch_t 0.771 (0.764)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-05 20:10:17,026 - Train: 11.61% [573800/4942000] [116.1/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 20:11:33,533 - Train: 11.61% [573900/4942000] [116.1/1000.0] [batch_t 0.765 (0.765)] [data_t 0.002] [optim_t 0.763] [lr 0.005000] 2024-04-05 20:12:50,095 - Train: 11.61% [574000/4942000] [116.1/1000.0] [batch_t 0.772 (0.766)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 20:14:06,461 - Train: 11.62% [574100/4942000] [116.2/1000.0] [batch_t 0.771 (0.764)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-05 20:15:22,997 - Train: 11.62% [574200/4942000] [116.2/1000.0] [batch_t 0.774 (0.765)] [data_t 0.002] [optim_t 0.772] [lr 0.005000] 2024-04-05 20:16:39,477 - Train: 11.62% [574300/4942000] [116.2/1000.0] [batch_t 0.773 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 20:17:55,836 - Train: 11.62% [574400/4942000] [116.2/1000.0] [batch_t 0.758 (0.763)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-05 20:19:12,343 - Train: 11.62% [574500/4942000] [116.2/1000.0] [batch_t 0.753 (0.765)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-05 20:20:28,784 - Train: 11.63% [574600/4942000] [116.3/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 20:21:45,345 - Train: 11.63% [574700/4942000] [116.3/1000.0] [batch_t 0.753 (0.766)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-05 20:23:01,848 - Train: 11.63% [574800/4942000] [116.3/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-05 20:24:18,370 - Train: 11.63% [574900/4942000] [116.3/1000.0] [batch_t 0.774 (0.765)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-05 20:25:34,863 - Train: 11.63% [575000/4942000] [116.3/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 20:26:51,385 - Train: 11.64% [575100/4942000] [116.4/1000.0] [batch_t 0.766 (0.765)] [data_t 0.002] [optim_t 0.763] [lr 0.005000] 2024-04-05 20:28:07,961 - Train: 11.64% [575200/4942000] [116.4/1000.0] [batch_t 0.762 (0.766)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 20:29:24,417 - Train: 11.64% [575300/4942000] [116.4/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 20:30:40,815 - Train: 11.64% [575400/4942000] [116.4/1000.0] [batch_t 0.768 (0.764)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 20:31:57,334 - Train: 11.65% [575500/4942000] [116.5/1000.0] [batch_t 0.777 (0.765)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-05 20:33:13,675 - Train: 11.65% [575600/4942000] [116.5/1000.0] [batch_t 0.757 (0.763)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-05 20:34:29,992 - Train: 11.65% [575700/4942000] [116.5/1000.0] [batch_t 0.762 (0.763)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 20:35:46,516 - Train: 11.65% [575800/4942000] [116.5/1000.0] [batch_t 0.764 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 20:37:03,051 - Train: 11.65% [575900/4942000] [116.5/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 20:38:19,457 - Train: 11.66% [576000/4942000] [116.6/1000.0] [batch_t 0.767 (0.764)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-05 20:39:36,024 - Train: 11.66% [576100/4942000] [116.6/1000.0] [batch_t 0.773 (0.766)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-05 20:40:52,508 - Train: 11.66% [576200/4942000] [116.6/1000.0] [batch_t 0.753 (0.765)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-05 20:42:08,949 - Train: 11.66% [576300/4942000] [116.6/1000.0] [batch_t 0.766 (0.764)] [data_t 0.006] [optim_t 0.760] [lr 0.005000] 2024-04-05 20:43:25,243 - Train: 11.66% [576400/4942000] [116.6/1000.0] [batch_t 0.768 (0.763)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 20:44:41,728 - Train: 11.67% [576500/4942000] [116.7/1000.0] [batch_t 0.771 (0.765)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-05 20:45:58,263 - Train: 11.67% [576600/4942000] [116.7/1000.0] [batch_t 0.757 (0.765)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-05 20:47:14,625 - Train: 11.67% [576700/4942000] [116.7/1000.0] [batch_t 0.766 (0.764)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-05 20:48:31,093 - Train: 11.67% [576800/4942000] [116.7/1000.0] [batch_t 0.761 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 20:49:47,600 - Train: 11.67% [576900/4942000] [116.7/1000.0] [batch_t 0.753 (0.765)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-05 20:51:04,077 - Train: 11.68% [577000/4942000] [116.8/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 20:52:20,546 - Train: 11.68% [577100/4942000] [116.8/1000.0] [batch_t 0.761 (0.765)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-05 20:53:36,991 - Train: 11.68% [577200/4942000] [116.8/1000.0] [batch_t 0.761 (0.764)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-05 20:54:53,502 - Train: 11.68% [577300/4942000] [116.8/1000.0] [batch_t 0.759 (0.765)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-05 20:56:09,944 - Train: 11.68% [577400/4942000] [116.8/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 20:57:26,374 - Train: 11.69% [577500/4942000] [116.9/1000.0] [batch_t 0.763 (0.764)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-05 20:58:42,951 - Train: 11.69% [577600/4942000] [116.9/1000.0] [batch_t 0.754 (0.766)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-05 21:00:00,380 - Train: 11.69% [577700/4942000] [116.9/1000.0] [batch_t 0.759 (0.774)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-05 21:01:16,668 - Train: 11.69% [577800/4942000] [116.9/1000.0] [batch_t 0.757 (0.763)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-05 21:02:33,116 - Train: 11.69% [577900/4942000] [116.9/1000.0] [batch_t 0.767 (0.764)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-05 21:03:49,574 - Train: 11.70% [578000/4942000] [117.0/1000.0] [batch_t 0.768 (0.764)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 21:05:05,961 - Train: 11.70% [578100/4942000] [117.0/1000.0] [batch_t 0.771 (0.764)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-05 21:06:22,441 - Train: 11.70% [578200/4942000] [117.0/1000.0] [batch_t 0.758 (0.765)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 21:06:33,198 - ==> Total time: 3 days, 3:09:12 Eta: 23 days, 15:11:01 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-05 21:07:41,181 - Train: 11.70% [578300/4942000] [117.0/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-05 21:08:57,761 - Train: 11.70% [578400/4942000] [117.0/1000.0] [batch_t 0.774 (0.766)] [data_t 0.002] [optim_t 0.772] [lr 0.005000] 2024-04-05 21:10:14,367 - Train: 11.71% [578500/4942000] [117.1/1000.0] [batch_t 0.763 (0.766)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 21:11:30,726 - Train: 11.71% [578600/4942000] [117.1/1000.0] [batch_t 0.771 (0.764)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-05 21:12:47,197 - Train: 11.71% [578700/4942000] [117.1/1000.0] [batch_t 0.761 (0.765)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-05 21:14:03,681 - Train: 11.71% [578800/4942000] [117.1/1000.0] [batch_t 0.761 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 21:15:20,286 - Train: 11.71% [578900/4942000] [117.1/1000.0] [batch_t 0.749 (0.766)] [data_t 0.003] [optim_t 0.746] [lr 0.005000] 2024-04-05 21:16:36,757 - Train: 11.72% [579000/4942000] [117.2/1000.0] [batch_t 0.779 (0.765)] [data_t 0.002] [optim_t 0.777] [lr 0.005000] 2024-04-05 21:17:53,312 - Train: 11.72% [579100/4942000] [117.2/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 21:19:09,822 - Train: 11.72% [579200/4942000] [117.2/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 21:20:26,184 - Train: 11.72% [579300/4942000] [117.2/1000.0] [batch_t 0.771 (0.764)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-05 21:21:42,757 - Train: 11.72% [579400/4942000] [117.2/1000.0] [batch_t 0.763 (0.766)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 21:22:59,176 - Train: 11.73% [579500/4942000] [117.3/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 21:24:15,702 - Train: 11.73% [579600/4942000] [117.3/1000.0] [batch_t 0.766 (0.765)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-05 21:25:32,210 - Train: 11.73% [579700/4942000] [117.3/1000.0] [batch_t 0.754 (0.765)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-05 21:26:48,615 - Train: 11.73% [579800/4942000] [117.3/1000.0] [batch_t 0.755 (0.764)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-05 21:28:05,111 - Train: 11.73% [579900/4942000] [117.3/1000.0] [batch_t 0.767 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-05 21:29:21,504 - Train: 11.74% [580000/4942000] [117.4/1000.0] [batch_t 0.776 (0.764)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-05 21:30:38,026 - Train: 11.74% [580100/4942000] [117.4/1000.0] [batch_t 0.757 (0.765)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-05 21:31:54,471 - Train: 11.74% [580200/4942000] [117.4/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 21:33:11,061 - Train: 11.74% [580300/4942000] [117.4/1000.0] [batch_t 0.758 (0.766)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-05 21:34:27,530 - Train: 11.74% [580400/4942000] [117.4/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-05 21:35:43,943 - Train: 11.75% [580500/4942000] [117.5/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 21:37:00,444 - Train: 11.75% [580600/4942000] [117.5/1000.0] [batch_t 0.765 (0.765)] [data_t 0.002] [optim_t 0.763] [lr 0.005000] 2024-04-05 21:38:16,896 - Train: 11.75% [580700/4942000] [117.5/1000.0] [batch_t 0.760 (0.764)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-05 21:39:33,496 - Train: 11.75% [580800/4942000] [117.5/1000.0] [batch_t 0.767 (0.766)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-05 21:40:50,075 - Train: 11.75% [580900/4942000] [117.5/1000.0] [batch_t 0.774 (0.766)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-05 21:42:06,521 - Train: 11.76% [581000/4942000] [117.6/1000.0] [batch_t 0.756 (0.764)] [data_t 0.003] [optim_t 0.753] [lr 0.005000] 2024-04-05 21:43:22,920 - Train: 11.76% [581100/4942000] [117.6/1000.0] [batch_t 0.772 (0.764)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 21:44:39,405 - Train: 11.76% [581200/4942000] [117.6/1000.0] [batch_t 0.766 (0.765)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-05 21:45:55,993 - Train: 11.76% [581300/4942000] [117.6/1000.0] [batch_t 0.758 (0.766)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 21:47:12,489 - Train: 11.76% [581400/4942000] [117.6/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 21:48:29,012 - Train: 11.77% [581500/4942000] [117.7/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 21:49:45,398 - Train: 11.77% [581600/4942000] [117.7/1000.0] [batch_t 0.763 (0.764)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-05 21:51:01,859 - Train: 11.77% [581700/4942000] [117.7/1000.0] [batch_t 0.760 (0.765)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-05 21:52:18,283 - Train: 11.77% [581800/4942000] [117.7/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 21:53:34,739 - Train: 11.77% [581900/4942000] [117.7/1000.0] [batch_t 0.753 (0.764)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-05 21:54:51,266 - Train: 11.78% [582000/4942000] [117.8/1000.0] [batch_t 0.753 (0.765)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-05 21:56:07,747 - Train: 11.78% [582100/4942000] [117.8/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 21:57:24,257 - Train: 11.78% [582200/4942000] [117.8/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 21:58:40,707 - Train: 11.78% [582300/4942000] [117.8/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-05 21:59:57,217 - Train: 11.78% [582400/4942000] [117.8/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 22:01:13,709 - Train: 11.79% [582500/4942000] [117.9/1000.0] [batch_t 0.764 (0.765)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-05 22:02:30,238 - Train: 11.79% [582600/4942000] [117.9/1000.0] [batch_t 0.773 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 22:03:46,765 - Train: 11.79% [582700/4942000] [117.9/1000.0] [batch_t 0.766 (0.765)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-05 22:05:03,404 - Train: 11.79% [582800/4942000] [117.9/1000.0] [batch_t 0.763 (0.766)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 22:06:19,899 - Train: 11.79% [582900/4942000] [117.9/1000.0] [batch_t 0.781 (0.765)] [data_t 0.002] [optim_t 0.779] [lr 0.005000] 2024-04-05 22:07:36,328 - Train: 11.80% [583000/4942000] [118.0/1000.0] [batch_t 0.776 (0.764)] [data_t 0.002] [optim_t 0.773] [lr 0.005000] 2024-04-05 22:08:52,931 - Train: 11.80% [583100/4942000] [118.0/1000.0] [batch_t 0.758 (0.766)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-05 22:09:35,788 - ==> Total time: 3 days, 4:12:14 Eta: 23 days, 17:35:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-05 22:10:11,739 - Train: 11.80% [583200/4942000] [118.0/1000.0] [batch_t 0.778 (0.764)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-05 22:11:28,153 - Train: 11.80% [583300/4942000] [118.0/1000.0] [batch_t 0.753 (0.764)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-05 22:12:44,611 - Train: 11.80% [583400/4942000] [118.0/1000.0] [batch_t 0.772 (0.764)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-05 22:14:01,063 - Train: 11.81% [583500/4942000] [118.1/1000.0] [batch_t 0.769 (0.764)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-05 22:15:17,416 - Train: 11.81% [583600/4942000] [118.1/1000.0] [batch_t 0.756 (0.763)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-05 22:16:33,879 - Train: 11.81% [583700/4942000] [118.1/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-05 22:17:50,423 - Train: 11.81% [583800/4942000] [118.1/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 22:19:06,889 - Train: 11.82% [583900/4942000] [118.2/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 22:20:23,343 - Train: 11.82% [584000/4942000] [118.2/1000.0] [batch_t 0.750 (0.764)] [data_t 0.002] [optim_t 0.748] [lr 0.005000] 2024-04-05 22:21:39,882 - Train: 11.82% [584100/4942000] [118.2/1000.0] [batch_t 0.753 (0.765)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-05 22:22:56,430 - Train: 11.82% [584200/4942000] [118.2/1000.0] [batch_t 0.782 (0.765)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-05 22:24:12,793 - Train: 11.82% [584300/4942000] [118.2/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 22:25:29,168 - Train: 11.83% [584400/4942000] [118.3/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-05 22:26:45,661 - Train: 11.83% [584500/4942000] [118.3/1000.0] [batch_t 0.770 (0.765)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-05 22:28:02,315 - Train: 11.83% [584600/4942000] [118.3/1000.0] [batch_t 0.779 (0.766)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-05 22:29:18,799 - Train: 11.83% [584700/4942000] [118.3/1000.0] [batch_t 0.769 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 22:30:35,219 - Train: 11.83% [584800/4942000] [118.3/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 22:31:51,605 - Train: 11.84% [584900/4942000] [118.4/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 22:33:08,025 - Train: 11.84% [585000/4942000] [118.4/1000.0] [batch_t 0.773 (0.764)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 22:34:24,485 - Train: 11.84% [585100/4942000] [118.4/1000.0] [batch_t 0.773 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-05 22:35:41,070 - Train: 11.84% [585200/4942000] [118.4/1000.0] [batch_t 0.768 (0.766)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 22:36:57,486 - Train: 11.84% [585300/4942000] [118.4/1000.0] [batch_t 0.760 (0.764)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-05 22:38:13,924 - Train: 11.85% [585400/4942000] [118.5/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 22:39:30,460 - Train: 11.85% [585500/4942000] [118.5/1000.0] [batch_t 0.781 (0.765)] [data_t 0.003] [optim_t 0.778] [lr 0.005000] 2024-04-05 22:40:46,904 - Train: 11.85% [585600/4942000] [118.5/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 22:42:03,306 - Train: 11.85% [585700/4942000] [118.5/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 22:43:19,693 - Train: 11.85% [585800/4942000] [118.5/1000.0] [batch_t 0.765 (0.764)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-05 22:44:36,307 - Train: 11.86% [585900/4942000] [118.6/1000.0] [batch_t 0.761 (0.766)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 22:45:52,897 - Train: 11.86% [586000/4942000] [118.6/1000.0] [batch_t 0.768 (0.766)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 22:47:09,420 - Train: 11.86% [586100/4942000] [118.6/1000.0] [batch_t 0.770 (0.765)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-05 22:48:25,900 - Train: 11.86% [586200/4942000] [118.6/1000.0] [batch_t 0.779 (0.765)] [data_t 0.002] [optim_t 0.776] [lr 0.005000] 2024-04-05 22:49:42,273 - Train: 11.86% [586300/4942000] [118.6/1000.0] [batch_t 0.753 (0.764)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-05 22:50:58,783 - Train: 11.87% [586400/4942000] [118.7/1000.0] [batch_t 0.778 (0.765)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-05 22:52:15,228 - Train: 11.87% [586500/4942000] [118.7/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 22:53:31,737 - Train: 11.87% [586600/4942000] [118.7/1000.0] [batch_t 0.771 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 22:54:48,275 - Train: 11.87% [586700/4942000] [118.7/1000.0] [batch_t 0.765 (0.765)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-05 22:56:04,904 - Train: 11.87% [586800/4942000] [118.7/1000.0] [batch_t 0.762 (0.766)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 22:57:21,446 - Train: 11.88% [586900/4942000] [118.8/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 22:58:37,802 - Train: 11.88% [587000/4942000] [118.8/1000.0] [batch_t 0.772 (0.763)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 22:59:54,244 - Train: 11.88% [587100/4942000] [118.8/1000.0] [batch_t 0.767 (0.764)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-05 23:01:10,873 - Train: 11.88% [587200/4942000] [118.8/1000.0] [batch_t 0.760 (0.766)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-05 23:02:27,383 - Train: 11.88% [587300/4942000] [118.8/1000.0] [batch_t 0.771 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 23:03:43,828 - Train: 11.89% [587400/4942000] [118.9/1000.0] [batch_t 0.749 (0.764)] [data_t 0.002] [optim_t 0.747] [lr 0.005000] 2024-04-05 23:05:00,400 - Train: 11.89% [587500/4942000] [118.9/1000.0] [batch_t 0.764 (0.766)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 23:06:16,970 - Train: 11.89% [587600/4942000] [118.9/1000.0] [batch_t 0.773 (0.766)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 23:07:33,535 - Train: 11.89% [587700/4942000] [118.9/1000.0] [batch_t 0.763 (0.766)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-05 23:08:49,990 - Train: 11.89% [587800/4942000] [118.9/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 23:10:06,532 - Train: 11.90% [587900/4942000] [119.0/1000.0] [batch_t 0.766 (0.765)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-05 23:11:22,897 - Train: 11.90% [588000/4942000] [119.0/1000.0] [batch_t 0.757 (0.764)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-05 23:12:37,816 - ==> Total time: 3 days, 5:15:17 Eta: 23 days, 19:56:40 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-05 23:12:41,339 - Train: 11.90% [588100/4942000] [119.0/1000.0] [batch_t 0.755 (0.795)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-05 23:13:57,841 - Train: 11.90% [588200/4942000] [119.0/1000.0] [batch_t 0.752 (0.765)] [data_t 0.002] [optim_t 0.749] [lr 0.005000] 2024-04-05 23:15:14,209 - Train: 11.90% [588300/4942000] [119.0/1000.0] [batch_t 0.773 (0.764)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 23:16:30,680 - Train: 11.91% [588400/4942000] [119.1/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 23:17:47,187 - Train: 11.91% [588500/4942000] [119.1/1000.0] [batch_t 0.771 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-05 23:19:03,719 - Train: 11.91% [588600/4942000] [119.1/1000.0] [batch_t 0.775 (0.765)] [data_t 0.002] [optim_t 0.772] [lr 0.005000] 2024-04-05 23:20:20,270 - Train: 11.91% [588700/4942000] [119.1/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 23:21:36,800 - Train: 11.91% [588800/4942000] [119.1/1000.0] [batch_t 0.759 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-05 23:22:53,153 - Train: 11.92% [588900/4942000] [119.2/1000.0] [batch_t 0.762 (0.763)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-05 23:24:09,682 - Train: 11.92% [589000/4942000] [119.2/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-05 23:25:26,167 - Train: 11.92% [589100/4942000] [119.2/1000.0] [batch_t 0.773 (0.765)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-05 23:26:42,573 - Train: 11.92% [589200/4942000] [119.2/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 23:27:58,933 - Train: 11.92% [589300/4942000] [119.2/1000.0] [batch_t 0.782 (0.764)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-05 23:29:15,488 - Train: 11.93% [589400/4942000] [119.3/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-05 23:30:31,819 - Train: 11.93% [589500/4942000] [119.3/1000.0] [batch_t 0.766 (0.763)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-05 23:31:48,161 - Train: 11.93% [589600/4942000] [119.3/1000.0] [batch_t 0.767 (0.763)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-05 23:33:04,518 - Train: 11.93% [589700/4942000] [119.3/1000.0] [batch_t 0.768 (0.763)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 23:34:21,022 - Train: 11.93% [589800/4942000] [119.3/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 23:35:37,409 - Train: 11.94% [589900/4942000] [119.4/1000.0] [batch_t 0.767 (0.764)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-05 23:36:53,938 - Train: 11.94% [590000/4942000] [119.4/1000.0] [batch_t 0.758 (0.765)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-05 23:38:10,539 - Train: 11.94% [590100/4942000] [119.4/1000.0] [batch_t 0.767 (0.766)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-05 23:39:27,037 - Train: 11.94% [590200/4942000] [119.4/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-05 23:40:43,459 - Train: 11.94% [590300/4942000] [119.4/1000.0] [batch_t 0.766 (0.764)] [data_t 0.002] [optim_t 0.763] [lr 0.005000] 2024-04-05 23:41:59,905 - Train: 11.95% [590400/4942000] [119.5/1000.0] [batch_t 0.766 (0.764)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-05 23:43:16,333 - Train: 11.95% [590500/4942000] [119.5/1000.0] [batch_t 0.773 (0.764)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-05 23:44:32,882 - Train: 11.95% [590600/4942000] [119.5/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-05 23:45:49,281 - Train: 11.95% [590700/4942000] [119.5/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 23:47:05,686 - Train: 11.95% [590800/4942000] [119.5/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-05 23:48:22,115 - Train: 11.96% [590900/4942000] [119.6/1000.0] [batch_t 0.778 (0.764)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-05 23:49:38,576 - Train: 11.96% [591000/4942000] [119.6/1000.0] [batch_t 0.767 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-05 23:50:55,065 - Train: 11.96% [591100/4942000] [119.6/1000.0] [batch_t 0.769 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-05 23:52:11,609 - Train: 11.96% [591200/4942000] [119.6/1000.0] [batch_t 0.771 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-05 23:53:28,026 - Train: 11.96% [591300/4942000] [119.6/1000.0] [batch_t 0.777 (0.764)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-05 23:54:44,427 - Train: 11.97% [591400/4942000] [119.7/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-05 23:56:01,057 - Train: 11.97% [591500/4942000] [119.7/1000.0] [batch_t 0.768 (0.766)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-05 23:57:17,417 - Train: 11.97% [591600/4942000] [119.7/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-05 23:58:33,953 - Train: 11.97% [591700/4942000] [119.7/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-05 23:59:50,433 - Train: 11.97% [591800/4942000] [119.7/1000.0] [batch_t 0.766 (0.765)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-06 00:01:07,008 - Train: 11.98% [591900/4942000] [119.8/1000.0] [batch_t 0.770 (0.766)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-06 00:02:23,462 - Train: 11.98% [592000/4942000] [119.8/1000.0] [batch_t 0.771 (0.764)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-06 00:03:40,039 - Train: 11.98% [592100/4942000] [119.8/1000.0] [batch_t 0.768 (0.766)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 00:04:56,492 - Train: 11.98% [592200/4942000] [119.8/1000.0] [batch_t 0.771 (0.764)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 00:06:12,835 - Train: 11.99% [592300/4942000] [119.9/1000.0] [batch_t 0.768 (0.763)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 00:07:29,232 - Train: 11.99% [592400/4942000] [119.9/1000.0] [batch_t 0.771 (0.764)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 00:08:45,712 - Train: 11.99% [592500/4942000] [119.9/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 00:10:02,260 - Train: 11.99% [592600/4942000] [119.9/1000.0] [batch_t 0.758 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-06 00:11:18,753 - Train: 11.99% [592700/4942000] [119.9/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 00:12:35,237 - Train: 12.00% [592800/4942000] [120.0/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 00:13:51,600 - Train: 12.00% [592900/4942000] [120.0/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 00:15:08,018 - Train: 12.00% [593000/4942000] [120.0/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 00:15:38,637 - ==> Total time: 3 days, 6:18:17 Eta: 23 days, 22:14:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-06 00:16:26,707 - Train: 12.00% [593100/4942000] [120.0/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 00:17:43,165 - Train: 12.00% [593200/4942000] [120.0/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 00:18:59,604 - Train: 12.01% [593300/4942000] [120.1/1000.0] [batch_t 0.764 (0.764)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-06 00:20:15,973 - Train: 12.01% [593400/4942000] [120.1/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 00:21:32,432 - Train: 12.01% [593500/4942000] [120.1/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 00:22:48,659 - Train: 12.01% [593600/4942000] [120.1/1000.0] [batch_t 0.760 (0.762)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-06 00:24:05,053 - Train: 12.01% [593700/4942000] [120.1/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 00:25:21,500 - Train: 12.02% [593800/4942000] [120.2/1000.0] [batch_t 0.753 (0.764)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-06 00:26:37,938 - Train: 12.02% [593900/4942000] [120.2/1000.0] [batch_t 0.770 (0.764)] [data_t 0.004] [optim_t 0.766] [lr 0.005000] 2024-04-06 00:27:54,288 - Train: 12.02% [594000/4942000] [120.2/1000.0] [batch_t 0.757 (0.763)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-06 00:29:10,722 - Train: 12.02% [594100/4942000] [120.2/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-06 00:30:27,314 - Train: 12.02% [594200/4942000] [120.2/1000.0] [batch_t 0.762 (0.766)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 00:31:43,815 - Train: 12.03% [594300/4942000] [120.3/1000.0] [batch_t 0.769 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 00:33:00,239 - Train: 12.03% [594400/4942000] [120.3/1000.0] [batch_t 0.767 (0.764)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 00:34:16,703 - Train: 12.03% [594500/4942000] [120.3/1000.0] [batch_t 0.762 (0.765)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-06 00:35:33,217 - Train: 12.03% [594600/4942000] [120.3/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-06 00:36:49,637 - Train: 12.03% [594700/4942000] [120.3/1000.0] [batch_t 0.740 (0.764)] [data_t 0.003] [optim_t 0.737] [lr 0.005000] 2024-04-06 00:38:06,047 - Train: 12.04% [594800/4942000] [120.4/1000.0] [batch_t 0.765 (0.764)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-06 00:39:22,525 - Train: 12.04% [594900/4942000] [120.4/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 00:40:38,929 - Train: 12.04% [595000/4942000] [120.4/1000.0] [batch_t 0.767 (0.764)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 00:41:55,283 - Train: 12.04% [595100/4942000] [120.4/1000.0] [batch_t 0.771 (0.763)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-06 00:43:11,775 - Train: 12.04% [595200/4942000] [120.4/1000.0] [batch_t 0.774 (0.765)] [data_t 0.002] [optim_t 0.772] [lr 0.005000] 2024-04-06 00:44:28,212 - Train: 12.05% [595300/4942000] [120.5/1000.0] [batch_t 0.769 (0.764)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-06 00:45:44,638 - Train: 12.05% [595400/4942000] [120.5/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 00:47:01,111 - Train: 12.05% [595500/4942000] [120.5/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 00:48:17,526 - Train: 12.05% [595600/4942000] [120.5/1000.0] [batch_t 0.776 (0.764)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-06 00:49:34,112 - Train: 12.05% [595700/4942000] [120.5/1000.0] [batch_t 0.763 (0.766)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 00:50:50,625 - Train: 12.06% [595800/4942000] [120.6/1000.0] [batch_t 0.770 (0.765)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-06 00:52:06,974 - Train: 12.06% [595900/4942000] [120.6/1000.0] [batch_t 0.773 (0.763)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 00:53:23,470 - Train: 12.06% [596000/4942000] [120.6/1000.0] [batch_t 0.761 (0.765)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-06 00:54:39,939 - Train: 12.06% [596100/4942000] [120.6/1000.0] [batch_t 0.774 (0.765)] [data_t 0.002] [optim_t 0.772] [lr 0.005000] 2024-04-06 00:55:56,374 - Train: 12.06% [596200/4942000] [120.6/1000.0] [batch_t 0.758 (0.764)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-06 00:57:12,747 - Train: 12.07% [596300/4942000] [120.7/1000.0] [batch_t 0.771 (0.764)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-06 00:58:29,306 - Train: 12.07% [596400/4942000] [120.7/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 00:59:45,705 - Train: 12.07% [596500/4942000] [120.7/1000.0] [batch_t 0.764 (0.764)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 01:01:02,317 - Train: 12.07% [596600/4942000] [120.7/1000.0] [batch_t 0.767 (0.766)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 01:02:18,817 - Train: 12.07% [596700/4942000] [120.7/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 01:03:35,295 - Train: 12.08% [596800/4942000] [120.8/1000.0] [batch_t 0.753 (0.765)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-06 01:04:51,659 - Train: 12.08% [596900/4942000] [120.8/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 01:06:08,274 - Train: 12.08% [597000/4942000] [120.8/1000.0] [batch_t 0.768 (0.766)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 01:07:24,750 - Train: 12.08% [597100/4942000] [120.8/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 01:08:41,267 - Train: 12.08% [597200/4942000] [120.8/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 01:09:57,779 - Train: 12.09% [597300/4942000] [120.9/1000.0] [batch_t 0.779 (0.765)] [data_t 0.002] [optim_t 0.777] [lr 0.005000] 2024-04-06 01:11:14,336 - Train: 12.09% [597400/4942000] [120.9/1000.0] [batch_t 0.774 (0.765)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-06 01:12:30,810 - Train: 12.09% [597500/4942000] [120.9/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 01:13:47,274 - Train: 12.09% [597600/4942000] [120.9/1000.0] [batch_t 0.748 (0.765)] [data_t 0.003] [optim_t 0.745] [lr 0.005000] 2024-04-06 01:15:03,745 - Train: 12.09% [597700/4942000] [120.9/1000.0] [batch_t 0.754 (0.765)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-06 01:16:20,089 - Train: 12.10% [597800/4942000] [121.0/1000.0] [batch_t 0.754 (0.763)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-06 01:17:36,519 - Train: 12.10% [597900/4942000] [121.0/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 01:18:39,341 - ==> Total time: 3 days, 7:21:18 Eta: 24 days, 0:28:21 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-06 01:18:55,311 - Train: 12.10% [598000/4942000] [121.0/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 01:20:11,669 - Train: 12.10% [598100/4942000] [121.0/1000.0] [batch_t 0.767 (0.763)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 01:21:28,085 - Train: 12.10% [598200/4942000] [121.0/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 01:22:44,425 - Train: 12.11% [598300/4942000] [121.1/1000.0] [batch_t 0.769 (0.763)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 01:24:00,883 - Train: 12.11% [598400/4942000] [121.1/1000.0] [batch_t 0.773 (0.764)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-06 01:25:17,342 - Train: 12.11% [598500/4942000] [121.1/1000.0] [batch_t 0.768 (0.764)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 01:26:33,735 - Train: 12.11% [598600/4942000] [121.1/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 01:27:50,240 - Train: 12.11% [598700/4942000] [121.1/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 01:29:06,739 - Train: 12.12% [598800/4942000] [121.2/1000.0] [batch_t 0.757 (0.765)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-06 01:30:23,339 - Train: 12.12% [598900/4942000] [121.2/1000.0] [batch_t 0.760 (0.766)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-06 01:31:39,702 - Train: 12.12% [599000/4942000] [121.2/1000.0] [batch_t 0.778 (0.764)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 01:32:56,046 - Train: 12.12% [599100/4942000] [121.2/1000.0] [batch_t 0.760 (0.763)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-06 01:34:12,583 - Train: 12.12% [599200/4942000] [121.2/1000.0] [batch_t 0.758 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-06 01:35:28,942 - Train: 12.13% [599300/4942000] [121.3/1000.0] [batch_t 0.763 (0.763)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-06 01:36:45,399 - Train: 12.13% [599400/4942000] [121.3/1000.0] [batch_t 0.756 (0.764)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-06 01:38:01,848 - Train: 12.13% [599500/4942000] [121.3/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 01:39:18,401 - Train: 12.13% [599600/4942000] [121.3/1000.0] [batch_t 0.765 (0.765)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-06 01:40:34,894 - Train: 12.13% [599700/4942000] [121.3/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 01:41:51,352 - Train: 12.14% [599800/4942000] [121.4/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 01:43:07,906 - Train: 12.14% [599900/4942000] [121.4/1000.0] [batch_t 0.759 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-06 01:44:24,456 - Train: 12.14% [600000/4942000] [121.4/1000.0] [batch_t 0.766 (0.765)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-06 01:45:40,892 - Train: 12.14% [600100/4942000] [121.4/1000.0] [batch_t 0.759 (0.764)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-06 01:46:57,249 - Train: 12.14% [600200/4942000] [121.4/1000.0] [batch_t 0.763 (0.763)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 01:48:13,532 - Train: 12.15% [600300/4942000] [121.5/1000.0] [batch_t 0.762 (0.763)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-06 01:49:29,945 - Train: 12.15% [600400/4942000] [121.5/1000.0] [batch_t 0.758 (0.764)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-06 01:50:46,421 - Train: 12.15% [600500/4942000] [121.5/1000.0] [batch_t 0.759 (0.765)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-06 01:52:02,825 - Train: 12.15% [600600/4942000] [121.5/1000.0] [batch_t 0.782 (0.764)] [data_t 0.002] [optim_t 0.780] [lr 0.005000] 2024-04-06 01:53:19,183 - Train: 12.15% [600700/4942000] [121.5/1000.0] [batch_t 0.763 (0.763)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 01:54:35,649 - Train: 12.16% [600800/4942000] [121.6/1000.0] [batch_t 0.773 (0.765)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-06 01:55:52,045 - Train: 12.16% [600900/4942000] [121.6/1000.0] [batch_t 0.769 (0.764)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 01:57:08,512 - Train: 12.16% [601000/4942000] [121.6/1000.0] [batch_t 0.769 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 01:58:25,123 - Train: 12.16% [601100/4942000] [121.6/1000.0] [batch_t 0.759 (0.766)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-06 01:59:41,559 - Train: 12.17% [601200/4942000] [121.7/1000.0] [batch_t 0.763 (0.764)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 02:00:58,033 - Train: 12.17% [601300/4942000] [121.7/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 02:02:14,387 - Train: 12.17% [601400/4942000] [121.7/1000.0] [batch_t 0.772 (0.763)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 02:03:30,743 - Train: 12.17% [601500/4942000] [121.7/1000.0] [batch_t 0.768 (0.763)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 02:04:47,338 - Train: 12.17% [601600/4942000] [121.7/1000.0] [batch_t 0.762 (0.766)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 02:06:03,846 - Train: 12.18% [601700/4942000] [121.8/1000.0] [batch_t 0.775 (0.765)] [data_t 0.003] [optim_t 0.772] [lr 0.005000] 2024-04-06 02:07:20,301 - Train: 12.18% [601800/4942000] [121.8/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 02:08:36,684 - Train: 12.18% [601900/4942000] [121.8/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 02:09:53,175 - Train: 12.18% [602000/4942000] [121.8/1000.0] [batch_t 0.765 (0.765)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-06 02:11:09,721 - Train: 12.18% [602100/4942000] [121.8/1000.0] [batch_t 0.761 (0.765)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-06 02:12:26,192 - Train: 12.19% [602200/4942000] [121.9/1000.0] [batch_t 0.764 (0.765)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-06 02:13:42,645 - Train: 12.19% [602300/4942000] [121.9/1000.0] [batch_t 0.767 (0.764)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 02:14:59,041 - Train: 12.19% [602400/4942000] [121.9/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 02:16:15,537 - Train: 12.19% [602500/4942000] [121.9/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 02:17:32,180 - Train: 12.19% [602600/4942000] [121.9/1000.0] [batch_t 0.753 (0.766)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-06 02:18:48,756 - Train: 12.20% [602700/4942000] [122.0/1000.0] [batch_t 0.757 (0.766)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-06 02:20:05,242 - Train: 12.20% [602800/4942000] [122.0/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 02:21:21,691 - Train: 12.20% [602900/4942000] [122.0/1000.0] [batch_t 0.748 (0.764)] [data_t 0.002] [optim_t 0.746] [lr 0.005000] 2024-04-06 02:21:40,085 - ==> Total time: 3 days, 8:24:19 Eta: 24 days, 2:39:17 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-06 02:22:39,980 - Train: 12.20% [603000/4942000] [122.0/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 02:23:56,432 - Train: 12.20% [603100/4942000] [122.0/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 02:25:12,868 - Train: 12.21% [603200/4942000] [122.1/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-06 02:26:29,086 - Train: 12.21% [603300/4942000] [122.1/1000.0] [batch_t 0.759 (0.762)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-06 02:27:45,636 - Train: 12.21% [603400/4942000] [122.1/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 02:29:02,227 - Train: 12.21% [603500/4942000] [122.1/1000.0] [batch_t 0.768 (0.766)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 02:30:18,700 - Train: 12.21% [603600/4942000] [122.1/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-06 02:31:35,192 - Train: 12.22% [603700/4942000] [122.2/1000.0] [batch_t 0.762 (0.765)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-06 02:32:51,628 - Train: 12.22% [603800/4942000] [122.2/1000.0] [batch_t 0.773 (0.764)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-06 02:34:08,043 - Train: 12.22% [603900/4942000] [122.2/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 02:35:24,537 - Train: 12.22% [604000/4942000] [122.2/1000.0] [batch_t 0.764 (0.765)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-06 02:36:40,886 - Train: 12.22% [604100/4942000] [122.2/1000.0] [batch_t 0.759 (0.763)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-06 02:37:57,344 - Train: 12.23% [604200/4942000] [122.3/1000.0] [batch_t 0.776 (0.764)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-06 02:39:13,873 - Train: 12.23% [604300/4942000] [122.3/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 02:40:30,354 - Train: 12.23% [604400/4942000] [122.3/1000.0] [batch_t 0.773 (0.765)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-06 02:41:46,778 - Train: 12.23% [604500/4942000] [122.3/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 02:43:03,207 - Train: 12.23% [604600/4942000] [122.3/1000.0] [batch_t 0.773 (0.764)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-06 02:44:19,630 - Train: 12.24% [604700/4942000] [122.4/1000.0] [batch_t 0.774 (0.764)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-06 02:45:36,060 - Train: 12.24% [604800/4942000] [122.4/1000.0] [batch_t 0.754 (0.764)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-06 02:46:52,573 - Train: 12.24% [604900/4942000] [122.4/1000.0] [batch_t 0.766 (0.765)] [data_t 0.002] [optim_t 0.763] [lr 0.005000] 2024-04-06 02:48:08,978 - Train: 12.24% [605000/4942000] [122.4/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 02:49:25,507 - Train: 12.24% [605100/4942000] [122.4/1000.0] [batch_t 0.776 (0.765)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 02:50:41,996 - Train: 12.25% [605200/4942000] [122.5/1000.0] [batch_t 0.754 (0.765)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-06 02:51:58,533 - Train: 12.25% [605300/4942000] [122.5/1000.0] [batch_t 0.777 (0.765)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-06 02:53:15,041 - Train: 12.25% [605400/4942000] [122.5/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 02:54:31,571 - Train: 12.25% [605500/4942000] [122.5/1000.0] [batch_t 0.759 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-06 02:55:47,971 - Train: 12.25% [605600/4942000] [122.5/1000.0] [batch_t 0.753 (0.764)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-06 02:57:04,402 - Train: 12.26% [605700/4942000] [122.6/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 02:58:20,832 - Train: 12.26% [605800/4942000] [122.6/1000.0] [batch_t 0.778 (0.764)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 02:59:37,375 - Train: 12.26% [605900/4942000] [122.6/1000.0] [batch_t 0.774 (0.765)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-06 03:00:53,832 - Train: 12.26% [606000/4942000] [122.6/1000.0] [batch_t 0.754 (0.764)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-06 03:02:10,148 - Train: 12.26% [606100/4942000] [122.6/1000.0] [batch_t 0.778 (0.763)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-06 03:03:26,598 - Train: 12.27% [606200/4942000] [122.7/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 03:04:43,016 - Train: 12.27% [606300/4942000] [122.7/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 03:05:59,493 - Train: 12.27% [606400/4942000] [122.7/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 03:07:16,005 - Train: 12.27% [606500/4942000] [122.7/1000.0] [batch_t 0.766 (0.765)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-06 03:08:32,450 - Train: 12.27% [606600/4942000] [122.7/1000.0] [batch_t 0.778 (0.764)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 03:09:48,870 - Train: 12.28% [606700/4942000] [122.8/1000.0] [batch_t 0.768 (0.764)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 03:11:05,443 - Train: 12.28% [606800/4942000] [122.8/1000.0] [batch_t 0.762 (0.766)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 03:12:22,011 - Train: 12.28% [606900/4942000] [122.8/1000.0] [batch_t 0.752 (0.766)] [data_t 0.003] [optim_t 0.749] [lr 0.005000] 2024-04-06 03:13:38,558 - Train: 12.28% [607000/4942000] [122.8/1000.0] [batch_t 0.750 (0.765)] [data_t 0.003] [optim_t 0.747] [lr 0.005000] 2024-04-06 03:14:55,009 - Train: 12.28% [607100/4942000] [122.8/1000.0] [batch_t 0.760 (0.764)] [data_t 0.002] [optim_t 0.758] [lr 0.005000] 2024-04-06 03:16:11,547 - Train: 12.29% [607200/4942000] [122.9/1000.0] [batch_t 0.753 (0.765)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-06 03:17:28,023 - Train: 12.29% [607300/4942000] [122.9/1000.0] [batch_t 0.752 (0.765)] [data_t 0.003] [optim_t 0.749] [lr 0.005000] 2024-04-06 03:18:44,516 - Train: 12.29% [607400/4942000] [122.9/1000.0] [batch_t 0.760 (0.765)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-06 03:20:01,085 - Train: 12.29% [607500/4942000] [122.9/1000.0] [batch_t 0.763 (0.766)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 03:21:17,585 - Train: 12.29% [607600/4942000] [122.9/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 03:22:33,989 - Train: 12.30% [607700/4942000] [123.0/1000.0] [batch_t 0.768 (0.764)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 03:23:50,457 - Train: 12.30% [607800/4942000] [123.0/1000.0] [batch_t 0.778 (0.765)] [data_t 0.002] [optim_t 0.776] [lr 0.005000] 2024-04-06 03:24:41,109 - ==> Total time: 3 days, 9:27:20 Eta: 24 days, 4:47:07 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-06 03:25:09,071 - Train: 12.30% [607900/4942000] [123.0/1000.0] [batch_t 0.771 (0.762)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-06 03:26:25,503 - Train: 12.30% [608000/4942000] [123.0/1000.0] [batch_t 0.771 (0.764)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 03:27:41,927 - Train: 12.30% [608100/4942000] [123.0/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 03:28:58,484 - Train: 12.31% [608200/4942000] [123.1/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 03:30:14,935 - Train: 12.31% [608300/4942000] [123.1/1000.0] [batch_t 0.766 (0.764)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-06 03:31:31,378 - Train: 12.31% [608400/4942000] [123.1/1000.0] [batch_t 0.748 (0.764)] [data_t 0.003] [optim_t 0.745] [lr 0.005000] 2024-04-06 03:32:47,954 - Train: 12.31% [608500/4942000] [123.1/1000.0] [batch_t 0.768 (0.766)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 03:34:04,479 - Train: 12.31% [608600/4942000] [123.1/1000.0] [batch_t 0.759 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-06 03:35:20,886 - Train: 12.32% [608700/4942000] [123.2/1000.0] [batch_t 0.772 (0.764)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-06 03:36:37,257 - Train: 12.32% [608800/4942000] [123.2/1000.0] [batch_t 0.761 (0.764)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 03:37:53,686 - Train: 12.32% [608900/4942000] [123.2/1000.0] [batch_t 0.762 (0.764)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-06 03:39:10,281 - Train: 12.32% [609000/4942000] [123.2/1000.0] [batch_t 0.769 (0.766)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 03:40:26,705 - Train: 12.32% [609100/4942000] [123.2/1000.0] [batch_t 0.752 (0.764)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-06 03:41:43,335 - Train: 12.33% [609200/4942000] [123.3/1000.0] [batch_t 0.768 (0.766)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 03:42:59,865 - Train: 12.33% [609300/4942000] [123.3/1000.0] [batch_t 0.766 (0.765)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-06 03:44:16,444 - Train: 12.33% [609400/4942000] [123.3/1000.0] [batch_t 0.768 (0.766)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 03:45:32,916 - Train: 12.33% [609500/4942000] [123.3/1000.0] [batch_t 0.745 (0.765)] [data_t 0.003] [optim_t 0.742] [lr 0.005000] 2024-04-06 03:46:49,382 - Train: 12.34% [609600/4942000] [123.4/1000.0] [batch_t 0.782 (0.765)] [data_t 0.002] [optim_t 0.780] [lr 0.005000] 2024-04-06 03:48:05,778 - Train: 12.34% [609700/4942000] [123.4/1000.0] [batch_t 0.752 (0.764)] [data_t 0.003] [optim_t 0.749] [lr 0.005000] 2024-04-06 03:49:22,379 - Train: 12.34% [609800/4942000] [123.4/1000.0] [batch_t 0.767 (0.766)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 03:50:38,840 - Train: 12.34% [609900/4942000] [123.4/1000.0] [batch_t 0.777 (0.765)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-06 03:51:55,210 - Train: 12.34% [610000/4942000] [123.4/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 03:53:11,729 - Train: 12.35% [610100/4942000] [123.5/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 03:54:28,266 - Train: 12.35% [610200/4942000] [123.5/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 03:55:44,580 - Train: 12.35% [610300/4942000] [123.5/1000.0] [batch_t 0.767 (0.763)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-06 03:57:01,044 - Train: 12.35% [610400/4942000] [123.5/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 03:58:17,456 - Train: 12.35% [610500/4942000] [123.5/1000.0] [batch_t 0.753 (0.764)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-06 03:59:33,845 - Train: 12.36% [610600/4942000] [123.6/1000.0] [batch_t 0.763 (0.764)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 04:00:50,387 - Train: 12.36% [610700/4942000] [123.6/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 04:02:06,731 - Train: 12.36% [610800/4942000] [123.6/1000.0] [batch_t 0.771 (0.763)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-06 04:03:23,080 - Train: 12.36% [610900/4942000] [123.6/1000.0] [batch_t 0.772 (0.763)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-06 04:04:39,515 - Train: 12.36% [611000/4942000] [123.6/1000.0] [batch_t 0.778 (0.764)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 04:05:56,058 - Train: 12.37% [611100/4942000] [123.7/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 04:07:12,390 - Train: 12.37% [611200/4942000] [123.7/1000.0] [batch_t 0.764 (0.763)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 04:08:28,866 - Train: 12.37% [611300/4942000] [123.7/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 04:09:45,445 - Train: 12.37% [611400/4942000] [123.7/1000.0] [batch_t 0.761 (0.766)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-06 04:11:01,816 - Train: 12.37% [611500/4942000] [123.7/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 04:12:18,309 - Train: 12.38% [611600/4942000] [123.8/1000.0] [batch_t 0.766 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 04:13:34,899 - Train: 12.38% [611700/4942000] [123.8/1000.0] [batch_t 0.760 (0.766)] [data_t 0.002] [optim_t 0.758] [lr 0.005000] 2024-04-06 04:14:51,392 - Train: 12.38% [611800/4942000] [123.8/1000.0] [batch_t 0.764 (0.765)] [data_t 0.004] [optim_t 0.760] [lr 0.005000] 2024-04-06 04:16:07,862 - Train: 12.38% [611900/4942000] [123.8/1000.0] [batch_t 0.767 (0.765)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-06 04:17:24,247 - Train: 12.38% [612000/4942000] [123.8/1000.0] [batch_t 0.767 (0.764)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 04:18:40,811 - Train: 12.39% [612100/4942000] [123.9/1000.0] [batch_t 0.777 (0.766)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 04:19:57,310 - Train: 12.39% [612200/4942000] [123.9/1000.0] [batch_t 0.751 (0.765)] [data_t 0.002] [optim_t 0.748] [lr 0.005000] 2024-04-06 04:21:13,845 - Train: 12.39% [612300/4942000] [123.9/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 04:22:30,342 - Train: 12.39% [612400/4942000] [123.9/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 04:23:46,764 - Train: 12.39% [612500/4942000] [123.9/1000.0] [batch_t 0.768 (0.764)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 04:25:03,350 - Train: 12.40% [612600/4942000] [124.0/1000.0] [batch_t 0.777 (0.766)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 04:26:19,806 - Train: 12.40% [612700/4942000] [124.0/1000.0] [batch_t 0.764 (0.764)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 04:27:36,229 - Train: 12.40% [612800/4942000] [124.0/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 04:27:42,420 - ==> Total time: 3 days, 10:30:21 Eta: 24 days, 6:51:53 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-06 04:28:55,033 - Train: 12.40% [612900/4942000] [124.0/1000.0] [batch_t 0.764 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 04:30:11,582 - Train: 12.40% [613000/4942000] [124.0/1000.0] [batch_t 0.769 (0.765)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-06 04:31:27,791 - Train: 12.41% [613100/4942000] [124.1/1000.0] [batch_t 0.757 (0.762)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-06 04:32:44,159 - Train: 12.41% [613200/4942000] [124.1/1000.0] [batch_t 0.769 (0.764)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 04:34:00,642 - Train: 12.41% [613300/4942000] [124.1/1000.0] [batch_t 0.754 (0.765)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-06 04:35:17,114 - Train: 12.41% [613400/4942000] [124.1/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 04:36:33,517 - Train: 12.41% [613500/4942000] [124.1/1000.0] [batch_t 0.753 (0.764)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-06 04:37:49,827 - Train: 12.42% [613600/4942000] [124.2/1000.0] [batch_t 0.772 (0.763)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-06 04:39:06,235 - Train: 12.42% [613700/4942000] [124.2/1000.0] [batch_t 0.774 (0.764)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-06 04:40:22,804 - Train: 12.42% [613800/4942000] [124.2/1000.0] [batch_t 0.774 (0.766)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-06 04:41:39,312 - Train: 12.42% [613900/4942000] [124.2/1000.0] [batch_t 0.769 (0.765)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-06 04:42:55,817 - Train: 12.42% [614000/4942000] [124.2/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 04:44:12,257 - Train: 12.43% [614100/4942000] [124.3/1000.0] [batch_t 0.777 (0.764)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 04:45:28,661 - Train: 12.43% [614200/4942000] [124.3/1000.0] [batch_t 0.755 (0.764)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-06 04:46:45,026 - Train: 12.43% [614300/4942000] [124.3/1000.0] [batch_t 0.767 (0.764)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-06 04:48:01,467 - Train: 12.43% [614400/4942000] [124.3/1000.0] [batch_t 0.767 (0.764)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 04:49:17,912 - Train: 12.43% [614500/4942000] [124.3/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 04:50:34,387 - Train: 12.44% [614600/4942000] [124.4/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 04:51:50,766 - Train: 12.44% [614700/4942000] [124.4/1000.0] [batch_t 0.769 (0.764)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-06 04:53:07,278 - Train: 12.44% [614800/4942000] [124.4/1000.0] [batch_t 0.759 (0.765)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-06 04:54:23,810 - Train: 12.44% [614900/4942000] [124.4/1000.0] [batch_t 0.769 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 04:55:40,268 - Train: 12.44% [615000/4942000] [124.4/1000.0] [batch_t 0.774 (0.764)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-06 04:56:56,780 - Train: 12.45% [615100/4942000] [124.5/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 04:58:13,305 - Train: 12.45% [615200/4942000] [124.5/1000.0] [batch_t 0.753 (0.765)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-06 04:59:29,889 - Train: 12.45% [615300/4942000] [124.5/1000.0] [batch_t 0.757 (0.766)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-06 05:00:46,510 - Train: 12.45% [615400/4942000] [124.5/1000.0] [batch_t 0.778 (0.766)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-06 05:02:03,043 - Train: 12.45% [615500/4942000] [124.5/1000.0] [batch_t 0.762 (0.765)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-06 05:03:19,531 - Train: 12.46% [615600/4942000] [124.6/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 05:04:35,882 - Train: 12.46% [615700/4942000] [124.6/1000.0] [batch_t 0.757 (0.763)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-06 05:05:52,321 - Train: 12.46% [615800/4942000] [124.6/1000.0] [batch_t 0.758 (0.764)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-06 05:07:08,899 - Train: 12.46% [615900/4942000] [124.6/1000.0] [batch_t 0.758 (0.766)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-06 05:08:25,314 - Train: 12.46% [616000/4942000] [124.6/1000.0] [batch_t 0.772 (0.764)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 05:09:41,891 - Train: 12.47% [616100/4942000] [124.7/1000.0] [batch_t 0.765 (0.766)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-06 05:10:58,312 - Train: 12.47% [616200/4942000] [124.7/1000.0] [batch_t 0.759 (0.764)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-06 05:12:14,726 - Train: 12.47% [616300/4942000] [124.7/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 05:13:31,280 - Train: 12.47% [616400/4942000] [124.7/1000.0] [batch_t 0.752 (0.765)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-06 05:14:47,630 - Train: 12.47% [616500/4942000] [124.7/1000.0] [batch_t 0.772 (0.763)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-06 05:16:04,106 - Train: 12.48% [616600/4942000] [124.8/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-06 05:17:20,577 - Train: 12.48% [616700/4942000] [124.8/1000.0] [batch_t 0.764 (0.765)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-06 05:18:37,230 - Train: 12.48% [616800/4942000] [124.8/1000.0] [batch_t 0.768 (0.766)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 05:19:53,838 - Train: 12.48% [616900/4942000] [124.8/1000.0] [batch_t 0.771 (0.766)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-06 05:21:10,356 - Train: 12.48% [617000/4942000] [124.8/1000.0] [batch_t 0.758 (0.765)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-06 05:22:26,778 - Train: 12.49% [617100/4942000] [124.9/1000.0] [batch_t 0.757 (0.764)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-06 05:23:43,221 - Train: 12.49% [617200/4942000] [124.9/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 05:24:59,684 - Train: 12.49% [617300/4942000] [124.9/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 05:26:16,137 - Train: 12.49% [617400/4942000] [124.9/1000.0] [batch_t 0.768 (0.764)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 05:27:32,601 - Train: 12.49% [617500/4942000] [124.9/1000.0] [batch_t 0.765 (0.765)] [data_t 0.002] [optim_t 0.763] [lr 0.005000] 2024-04-06 05:28:49,168 - Train: 12.50% [617600/4942000] [125.0/1000.0] [batch_t 0.769 (0.766)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 05:30:05,659 - Train: 12.50% [617700/4942000] [125.0/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 05:30:43,818 - ==> Total time: 3 days, 11:33:23 Eta: 24 days, 8:53:41 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-06 05:31:23,833 - Train: 12.50% [617800/4942000] [125.0/1000.0] [batch_t 0.763 (0.763)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 05:32:40,309 - Train: 12.50% [617900/4942000] [125.0/1000.0] [batch_t 0.766 (0.765)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-06 05:33:56,796 - Train: 12.51% [618000/4942000] [125.1/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 05:35:13,341 - Train: 12.51% [618100/4942000] [125.1/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 05:36:29,767 - Train: 12.51% [618200/4942000] [125.1/1000.0] [batch_t 0.766 (0.764)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-06 05:37:46,263 - Train: 12.51% [618300/4942000] [125.1/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 05:39:02,735 - Train: 12.51% [618400/4942000] [125.1/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 05:40:19,229 - Train: 12.52% [618500/4942000] [125.2/1000.0] [batch_t 0.764 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 05:41:35,769 - Train: 12.52% [618600/4942000] [125.2/1000.0] [batch_t 0.761 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 05:42:52,282 - Train: 12.52% [618700/4942000] [125.2/1000.0] [batch_t 0.773 (0.765)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-06 05:44:08,764 - Train: 12.52% [618800/4942000] [125.2/1000.0] [batch_t 0.759 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-06 05:45:25,260 - Train: 12.52% [618900/4942000] [125.2/1000.0] [batch_t 0.770 (0.765)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-06 05:46:41,691 - Train: 12.53% [619000/4942000] [125.3/1000.0] [batch_t 0.761 (0.764)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-06 05:47:58,193 - Train: 12.53% [619100/4942000] [125.3/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 05:49:14,707 - Train: 12.53% [619200/4942000] [125.3/1000.0] [batch_t 0.769 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 05:50:31,007 - Train: 12.53% [619300/4942000] [125.3/1000.0] [batch_t 0.763 (0.763)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 05:51:47,540 - Train: 12.53% [619400/4942000] [125.3/1000.0] [batch_t 0.776 (0.765)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-06 05:53:03,988 - Train: 12.54% [619500/4942000] [125.4/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 05:54:20,475 - Train: 12.54% [619600/4942000] [125.4/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 05:55:37,092 - Train: 12.54% [619700/4942000] [125.4/1000.0] [batch_t 0.762 (0.766)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 05:56:53,508 - Train: 12.54% [619800/4942000] [125.4/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-06 05:58:09,962 - Train: 12.54% [619900/4942000] [125.4/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 05:59:26,425 - Train: 12.55% [620000/4942000] [125.5/1000.0] [batch_t 0.765 (0.765)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-06 06:00:42,840 - Train: 12.55% [620100/4942000] [125.5/1000.0] [batch_t 0.773 (0.764)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 06:01:59,251 - Train: 12.55% [620200/4942000] [125.5/1000.0] [batch_t 0.762 (0.764)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-06 06:03:15,801 - Train: 12.55% [620300/4942000] [125.5/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 06:04:32,176 - Train: 12.55% [620400/4942000] [125.5/1000.0] [batch_t 0.747 (0.764)] [data_t 0.003] [optim_t 0.745] [lr 0.005000] 2024-04-06 06:05:48,564 - Train: 12.56% [620500/4942000] [125.6/1000.0] [batch_t 0.773 (0.764)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 06:07:05,026 - Train: 12.56% [620600/4942000] [125.6/1000.0] [batch_t 0.770 (0.765)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-06 06:08:21,560 - Train: 12.56% [620700/4942000] [125.6/1000.0] [batch_t 0.773 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-06 06:09:37,988 - Train: 12.56% [620800/4942000] [125.6/1000.0] [batch_t 0.754 (0.764)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-06 06:10:54,397 - Train: 12.56% [620900/4942000] [125.6/1000.0] [batch_t 0.755 (0.764)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-06 06:12:10,817 - Train: 12.57% [621000/4942000] [125.7/1000.0] [batch_t 0.759 (0.764)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-06 06:13:27,237 - Train: 12.57% [621100/4942000] [125.7/1000.0] [batch_t 0.770 (0.764)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-06 06:14:43,769 - Train: 12.57% [621200/4942000] [125.7/1000.0] [batch_t 0.757 (0.765)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-06 06:16:00,243 - Train: 12.57% [621300/4942000] [125.7/1000.0] [batch_t 0.766 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 06:17:16,713 - Train: 12.57% [621400/4942000] [125.7/1000.0] [batch_t 0.766 (0.765)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-06 06:18:33,137 - Train: 12.58% [621500/4942000] [125.8/1000.0] [batch_t 0.766 (0.764)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-06 06:19:49,557 - Train: 12.58% [621600/4942000] [125.8/1000.0] [batch_t 0.753 (0.764)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-06 06:21:06,055 - Train: 12.58% [621700/4942000] [125.8/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 06:22:22,671 - Train: 12.58% [621800/4942000] [125.8/1000.0] [batch_t 0.752 (0.766)] [data_t 0.003] [optim_t 0.749] [lr 0.005000] 2024-04-06 06:23:39,047 - Train: 12.58% [621900/4942000] [125.8/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 06:24:55,522 - Train: 12.59% [622000/4942000] [125.9/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 06:26:11,910 - Train: 12.59% [622100/4942000] [125.9/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 06:27:28,499 - Train: 12.59% [622200/4942000] [125.9/1000.0] [batch_t 0.767 (0.766)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 06:28:44,954 - Train: 12.59% [622300/4942000] [125.9/1000.0] [batch_t 0.773 (0.764)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-06 06:30:01,449 - Train: 12.59% [622400/4942000] [125.9/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 06:31:18,028 - Train: 12.60% [622500/4942000] [126.0/1000.0] [batch_t 0.774 (0.766)] [data_t 0.002] [optim_t 0.772] [lr 0.005000] 2024-04-06 06:32:34,429 - Train: 12.60% [622600/4942000] [126.0/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 06:33:44,769 - ==> Total time: 3 days, 12:36:23 Eta: 24 days, 10:52:29 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-06 06:33:53,024 - Train: 12.60% [622700/4942000] [126.0/1000.0] [batch_t 0.763 (0.766)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-06 06:35:09,411 - Train: 12.60% [622800/4942000] [126.0/1000.0] [batch_t 0.759 (0.764)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-06 06:36:25,760 - Train: 12.60% [622900/4942000] [126.0/1000.0] [batch_t 0.765 (0.763)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-06 06:37:42,161 - Train: 12.61% [623000/4942000] [126.1/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 06:38:58,694 - Train: 12.61% [623100/4942000] [126.1/1000.0] [batch_t 0.766 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 06:40:15,207 - Train: 12.61% [623200/4942000] [126.1/1000.0] [batch_t 0.771 (0.765)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-06 06:41:31,751 - Train: 12.61% [623300/4942000] [126.1/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 06:42:48,183 - Train: 12.61% [623400/4942000] [126.1/1000.0] [batch_t 0.766 (0.764)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-06 06:44:04,622 - Train: 12.62% [623500/4942000] [126.2/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 06:45:21,164 - Train: 12.62% [623600/4942000] [126.2/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 06:46:37,632 - Train: 12.62% [623700/4942000] [126.2/1000.0] [batch_t 0.764 (0.765)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-06 06:47:54,185 - Train: 12.62% [623800/4942000] [126.2/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 06:49:10,784 - Train: 12.62% [623900/4942000] [126.2/1000.0] [batch_t 0.766 (0.766)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-06 06:50:27,153 - Train: 12.63% [624000/4942000] [126.3/1000.0] [batch_t 0.765 (0.764)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-06 06:51:43,549 - Train: 12.63% [624100/4942000] [126.3/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 06:53:00,101 - Train: 12.63% [624200/4942000] [126.3/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-06 06:54:16,676 - Train: 12.63% [624300/4942000] [126.3/1000.0] [batch_t 0.773 (0.766)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 06:55:33,379 - Train: 12.63% [624400/4942000] [126.3/1000.0] [batch_t 0.776 (0.767)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-06 06:56:49,973 - Train: 12.64% [624500/4942000] [126.4/1000.0] [batch_t 0.774 (0.766)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-06 06:58:06,579 - Train: 12.64% [624600/4942000] [126.4/1000.0] [batch_t 0.784 (0.766)] [data_t 0.003] [optim_t 0.781] [lr 0.005000] 2024-04-06 06:59:23,143 - Train: 12.64% [624700/4942000] [126.4/1000.0] [batch_t 0.753 (0.766)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-06 07:00:39,675 - Train: 12.64% [624800/4942000] [126.4/1000.0] [batch_t 0.759 (0.765)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-06 07:01:56,111 - Train: 12.64% [624900/4942000] [126.4/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 07:03:12,599 - Train: 12.65% [625000/4942000] [126.5/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 07:04:29,072 - Train: 12.65% [625100/4942000] [126.5/1000.0] [batch_t 0.759 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-06 07:05:45,692 - Train: 12.65% [625200/4942000] [126.5/1000.0] [batch_t 0.773 (0.766)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-06 07:07:02,102 - Train: 12.65% [625300/4942000] [126.5/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 07:08:18,713 - Train: 12.65% [625400/4942000] [126.5/1000.0] [batch_t 0.778 (0.766)] [data_t 0.002] [optim_t 0.776] [lr 0.005000] 2024-04-06 07:09:35,226 - Train: 12.66% [625500/4942000] [126.6/1000.0] [batch_t 0.771 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 07:10:51,763 - Train: 12.66% [625600/4942000] [126.6/1000.0] [batch_t 0.781 (0.765)] [data_t 0.003] [optim_t 0.778] [lr 0.005000] 2024-04-06 07:12:08,215 - Train: 12.66% [625700/4942000] [126.6/1000.0] [batch_t 0.753 (0.764)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-06 07:13:24,657 - Train: 12.66% [625800/4942000] [126.6/1000.0] [batch_t 0.759 (0.764)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-06 07:14:40,961 - Train: 12.66% [625900/4942000] [126.6/1000.0] [batch_t 0.755 (0.763)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-06 07:15:57,463 - Train: 12.67% [626000/4942000] [126.7/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 07:17:13,943 - Train: 12.67% [626100/4942000] [126.7/1000.0] [batch_t 0.769 (0.765)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-06 07:18:30,317 - Train: 12.67% [626200/4942000] [126.7/1000.0] [batch_t 0.769 (0.764)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 07:19:46,850 - Train: 12.67% [626300/4942000] [126.7/1000.0] [batch_t 0.758 (0.765)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-06 07:21:03,345 - Train: 12.68% [626400/4942000] [126.8/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 07:22:19,758 - Train: 12.68% [626500/4942000] [126.8/1000.0] [batch_t 0.776 (0.764)] [data_t 0.002] [optim_t 0.773] [lr 0.005000] 2024-04-06 07:23:36,192 - Train: 12.68% [626600/4942000] [126.8/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 07:24:52,502 - Train: 12.68% [626700/4942000] [126.8/1000.0] [batch_t 0.753 (0.763)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-06 07:26:08,925 - Train: 12.68% [626800/4942000] [126.8/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 07:27:25,490 - Train: 12.69% [626900/4942000] [126.9/1000.0] [batch_t 0.772 (0.766)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 07:28:41,911 - Train: 12.69% [627000/4942000] [126.9/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 07:29:58,434 - Train: 12.69% [627100/4942000] [126.9/1000.0] [batch_t 0.769 (0.765)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-06 07:31:14,821 - Train: 12.69% [627200/4942000] [126.9/1000.0] [batch_t 0.766 (0.764)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-06 07:32:31,124 - Train: 12.69% [627300/4942000] [126.9/1000.0] [batch_t 0.762 (0.763)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 07:33:47,588 - Train: 12.70% [627400/4942000] [127.0/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 07:35:04,008 - Train: 12.70% [627500/4942000] [127.0/1000.0] [batch_t 0.778 (0.764)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 07:36:20,503 - Train: 12.70% [627600/4942000] [127.0/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 07:36:46,535 - ==> Total time: 3 days, 13:39:25 Eta: 24 days, 12:48:30 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-06 07:37:38,892 - Train: 12.70% [627700/4942000] [127.0/1000.0] [batch_t 0.772 (0.764)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 07:38:55,336 - Train: 12.70% [627800/4942000] [127.0/1000.0] [batch_t 0.757 (0.764)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-06 07:40:11,819 - Train: 12.71% [627900/4942000] [127.1/1000.0] [batch_t 0.755 (0.765)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-06 07:41:28,178 - Train: 12.71% [628000/4942000] [127.1/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 07:42:44,689 - Train: 12.71% [628100/4942000] [127.1/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 07:44:01,217 - Train: 12.71% [628200/4942000] [127.1/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 07:45:17,762 - Train: 12.71% [628300/4942000] [127.1/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 07:46:34,157 - Train: 12.72% [628400/4942000] [127.2/1000.0] [batch_t 0.742 (0.764)] [data_t 0.002] [optim_t 0.740] [lr 0.005000] 2024-04-06 07:47:50,631 - Train: 12.72% [628500/4942000] [127.2/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 07:49:06,985 - Train: 12.72% [628600/4942000] [127.2/1000.0] [batch_t 0.764 (0.763)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 07:50:23,515 - Train: 12.72% [628700/4942000] [127.2/1000.0] [batch_t 0.769 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 07:51:39,781 - Train: 12.72% [628800/4942000] [127.2/1000.0] [batch_t 0.760 (0.763)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-06 07:52:56,261 - Train: 12.73% [628900/4942000] [127.3/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 07:54:12,785 - Train: 12.73% [629000/4942000] [127.3/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-06 07:55:29,328 - Train: 12.73% [629100/4942000] [127.3/1000.0] [batch_t 0.764 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 07:56:45,892 - Train: 12.73% [629200/4942000] [127.3/1000.0] [batch_t 0.769 (0.766)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 07:58:02,424 - Train: 12.73% [629300/4942000] [127.3/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 07:59:18,844 - Train: 12.74% [629400/4942000] [127.4/1000.0] [batch_t 0.778 (0.764)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-06 08:00:35,124 - Train: 12.74% [629500/4942000] [127.4/1000.0] [batch_t 0.768 (0.763)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 08:01:51,660 - Train: 12.74% [629600/4942000] [127.4/1000.0] [batch_t 0.759 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-06 08:03:08,064 - Train: 12.74% [629700/4942000] [127.4/1000.0] [batch_t 0.765 (0.764)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-06 08:04:24,408 - Train: 12.74% [629800/4942000] [127.4/1000.0] [batch_t 0.782 (0.763)] [data_t 0.002] [optim_t 0.779] [lr 0.005000] 2024-04-06 08:05:40,927 - Train: 12.75% [629900/4942000] [127.5/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-06 08:06:57,412 - Train: 12.75% [630000/4942000] [127.5/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 08:08:13,874 - Train: 12.75% [630100/4942000] [127.5/1000.0] [batch_t 0.776 (0.765)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-06 08:09:30,300 - Train: 12.75% [630200/4942000] [127.5/1000.0] [batch_t 0.757 (0.764)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-06 08:10:46,832 - Train: 12.75% [630300/4942000] [127.5/1000.0] [batch_t 0.764 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 08:12:03,276 - Train: 12.76% [630400/4942000] [127.6/1000.0] [batch_t 0.761 (0.764)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 08:13:19,721 - Train: 12.76% [630500/4942000] [127.6/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 08:14:36,135 - Train: 12.76% [630600/4942000] [127.6/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 08:15:52,828 - Train: 12.76% [630700/4942000] [127.6/1000.0] [batch_t 0.772 (0.767)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 08:17:09,364 - Train: 12.76% [630800/4942000] [127.6/1000.0] [batch_t 0.771 (0.765)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-06 08:18:25,783 - Train: 12.77% [630900/4942000] [127.7/1000.0] [batch_t 0.757 (0.764)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-06 08:19:42,091 - Train: 12.77% [631000/4942000] [127.7/1000.0] [batch_t 0.771 (0.763)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-06 08:20:58,487 - Train: 12.77% [631100/4942000] [127.7/1000.0] [batch_t 0.769 (0.764)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-06 08:22:14,894 - Train: 12.77% [631200/4942000] [127.7/1000.0] [batch_t 0.773 (0.764)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 08:23:31,436 - Train: 12.77% [631300/4942000] [127.7/1000.0] [batch_t 0.758 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-06 08:24:47,949 - Train: 12.78% [631400/4942000] [127.8/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-06 08:26:04,448 - Train: 12.78% [631500/4942000] [127.8/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 08:27:20,824 - Train: 12.78% [631600/4942000] [127.8/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 08:28:37,218 - Train: 12.78% [631700/4942000] [127.8/1000.0] [batch_t 0.754 (0.764)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-06 08:29:53,582 - Train: 12.78% [631800/4942000] [127.8/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 08:31:10,106 - Train: 12.79% [631900/4942000] [127.9/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 08:32:26,559 - Train: 12.79% [632000/4942000] [127.9/1000.0] [batch_t 0.752 (0.764)] [data_t 0.003] [optim_t 0.749] [lr 0.005000] 2024-04-06 08:33:43,037 - Train: 12.79% [632100/4942000] [127.9/1000.0] [batch_t 0.759 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-06 08:34:59,419 - Train: 12.79% [632200/4942000] [127.9/1000.0] [batch_t 0.747 (0.764)] [data_t 0.002] [optim_t 0.745] [lr 0.005000] 2024-04-06 08:36:15,911 - Train: 12.79% [632300/4942000] [127.9/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 08:37:32,344 - Train: 12.80% [632400/4942000] [128.0/1000.0] [batch_t 0.771 (0.764)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 08:38:48,691 - Train: 12.80% [632500/4942000] [128.0/1000.0] [batch_t 0.755 (0.763)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-06 08:39:46,770 - ==> Total time: 3 days, 14:42:25 Eta: 24 days, 14:41:34 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-06 08:40:06,991 - Train: 12.80% [632600/4942000] [128.0/1000.0] [batch_t 0.758 (0.768)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-06 08:41:23,474 - Train: 12.80% [632700/4942000] [128.0/1000.0] [batch_t 0.770 (0.765)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-06 08:42:39,957 - Train: 12.80% [632800/4942000] [128.0/1000.0] [batch_t 0.766 (0.765)] [data_t 0.002] [optim_t 0.763] [lr 0.005000] 2024-04-06 08:43:56,510 - Train: 12.81% [632900/4942000] [128.1/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 08:45:13,024 - Train: 12.81% [633000/4942000] [128.1/1000.0] [batch_t 0.759 (0.765)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-06 08:46:29,476 - Train: 12.81% [633100/4942000] [128.1/1000.0] [batch_t 0.776 (0.764)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-06 08:47:45,988 - Train: 12.81% [633200/4942000] [128.1/1000.0] [batch_t 0.773 (0.765)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-06 08:49:02,363 - Train: 12.81% [633300/4942000] [128.1/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 08:50:18,751 - Train: 12.82% [633400/4942000] [128.2/1000.0] [batch_t 0.772 (0.764)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-06 08:51:35,242 - Train: 12.82% [633500/4942000] [128.2/1000.0] [batch_t 0.766 (0.765)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-06 08:52:51,769 - Train: 12.82% [633600/4942000] [128.2/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 08:54:08,202 - Train: 12.82% [633700/4942000] [128.2/1000.0] [batch_t 0.771 (0.764)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-06 08:55:24,659 - Train: 12.82% [633800/4942000] [128.2/1000.0] [batch_t 0.758 (0.764)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-06 08:56:41,089 - Train: 12.83% [633900/4942000] [128.3/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-06 08:57:57,657 - Train: 12.83% [634000/4942000] [128.3/1000.0] [batch_t 0.764 (0.766)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 08:59:14,134 - Train: 12.83% [634100/4942000] [128.3/1000.0] [batch_t 0.769 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 09:00:30,620 - Train: 12.83% [634200/4942000] [128.3/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 09:01:47,119 - Train: 12.83% [634300/4942000] [128.3/1000.0] [batch_t 0.749 (0.765)] [data_t 0.003] [optim_t 0.746] [lr 0.005000] 2024-04-06 09:03:03,576 - Train: 12.84% [634400/4942000] [128.4/1000.0] [batch_t 0.777 (0.764)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-06 09:04:20,147 - Train: 12.84% [634500/4942000] [128.4/1000.0] [batch_t 0.772 (0.766)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 09:05:36,702 - Train: 12.84% [634600/4942000] [128.4/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 09:06:53,107 - Train: 12.84% [634700/4942000] [128.4/1000.0] [batch_t 0.776 (0.764)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-06 09:08:09,506 - Train: 12.85% [634800/4942000] [128.5/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 09:09:26,047 - Train: 12.85% [634900/4942000] [128.5/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 09:10:42,444 - Train: 12.85% [635000/4942000] [128.5/1000.0] [batch_t 0.759 (0.764)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-06 09:11:58,875 - Train: 12.85% [635100/4942000] [128.5/1000.0] [batch_t 0.768 (0.764)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 09:13:15,395 - Train: 12.85% [635200/4942000] [128.5/1000.0] [batch_t 0.760 (0.765)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-06 09:14:32,006 - Train: 12.86% [635300/4942000] [128.6/1000.0] [batch_t 0.769 (0.766)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 09:15:48,592 - Train: 12.86% [635400/4942000] [128.6/1000.0] [batch_t 0.769 (0.766)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 09:17:05,087 - Train: 12.86% [635500/4942000] [128.6/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 09:18:21,539 - Train: 12.86% [635600/4942000] [128.6/1000.0] [batch_t 0.776 (0.764)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 09:19:37,983 - Train: 12.86% [635700/4942000] [128.6/1000.0] [batch_t 0.754 (0.764)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-06 09:20:54,434 - Train: 12.87% [635800/4942000] [128.7/1000.0] [batch_t 0.764 (0.764)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 09:22:10,978 - Train: 12.87% [635900/4942000] [128.7/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 09:23:27,577 - Train: 12.87% [636000/4942000] [128.7/1000.0] [batch_t 0.762 (0.766)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-06 09:24:44,043 - Train: 12.87% [636100/4942000] [128.7/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 09:26:00,344 - Train: 12.87% [636200/4942000] [128.7/1000.0] [batch_t 0.765 (0.763)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-06 09:27:16,887 - Train: 12.88% [636300/4942000] [128.8/1000.0] [batch_t 0.754 (0.765)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-06 09:28:33,385 - Train: 12.88% [636400/4942000] [128.8/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 09:29:49,920 - Train: 12.88% [636500/4942000] [128.8/1000.0] [batch_t 0.758 (0.765)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-06 09:31:06,399 - Train: 12.88% [636600/4942000] [128.8/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 09:32:22,740 - Train: 12.88% [636700/4942000] [128.8/1000.0] [batch_t 0.766 (0.763)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-06 09:33:39,148 - Train: 12.89% [636800/4942000] [128.9/1000.0] [batch_t 0.758 (0.764)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-06 09:34:55,638 - Train: 12.89% [636900/4942000] [128.9/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-06 09:36:12,166 - Train: 12.89% [637000/4942000] [128.9/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 09:37:28,552 - Train: 12.89% [637100/4942000] [128.9/1000.0] [batch_t 0.758 (0.764)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-06 09:38:45,097 - Train: 12.89% [637200/4942000] [128.9/1000.0] [batch_t 0.767 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 09:40:01,553 - Train: 12.90% [637300/4942000] [129.0/1000.0] [batch_t 0.777 (0.764)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-06 09:41:17,977 - Train: 12.90% [637400/4942000] [129.0/1000.0] [batch_t 0.768 (0.764)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 09:42:34,390 - Train: 12.90% [637500/4942000] [129.0/1000.0] [batch_t 0.758 (0.764)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-06 09:42:48,164 - ==> Total time: 3 days, 15:45:27 Eta: 24 days, 16:32:01 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-06 09:43:52,784 - Train: 12.90% [637600/4942000] [129.0/1000.0] [batch_t 0.759 (0.766)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-06 09:45:09,186 - Train: 12.90% [637700/4942000] [129.0/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 09:46:25,673 - Train: 12.91% [637800/4942000] [129.1/1000.0] [batch_t 0.767 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 09:47:42,168 - Train: 12.91% [637900/4942000] [129.1/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 09:48:58,598 - Train: 12.91% [638000/4942000] [129.1/1000.0] [batch_t 0.778 (0.764)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-06 09:50:15,114 - Train: 12.91% [638100/4942000] [129.1/1000.0] [batch_t 0.774 (0.765)] [data_t 0.004] [optim_t 0.770] [lr 0.005000] 2024-04-06 09:51:31,527 - Train: 12.91% [638200/4942000] [129.1/1000.0] [batch_t 0.771 (0.764)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-06 09:52:47,958 - Train: 12.92% [638300/4942000] [129.2/1000.0] [batch_t 0.768 (0.764)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 09:54:04,453 - Train: 12.92% [638400/4942000] [129.2/1000.0] [batch_t 0.761 (0.765)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-06 09:55:20,996 - Train: 12.92% [638500/4942000] [129.2/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 09:56:37,395 - Train: 12.92% [638600/4942000] [129.2/1000.0] [batch_t 0.757 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-06 09:57:54,037 - Train: 12.92% [638700/4942000] [129.2/1000.0] [batch_t 0.751 (0.766)] [data_t 0.002] [optim_t 0.749] [lr 0.005000] 2024-04-06 09:59:10,556 - Train: 12.93% [638800/4942000] [129.3/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 10:00:26,933 - Train: 12.93% [638900/4942000] [129.3/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 10:01:43,479 - Train: 12.93% [639000/4942000] [129.3/1000.0] [batch_t 0.757 (0.765)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-06 10:02:59,915 - Train: 12.93% [639100/4942000] [129.3/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 10:04:16,408 - Train: 12.93% [639200/4942000] [129.3/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 10:05:32,923 - Train: 12.94% [639300/4942000] [129.4/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 10:06:49,475 - Train: 12.94% [639400/4942000] [129.4/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 10:08:05,985 - Train: 12.94% [639500/4942000] [129.4/1000.0] [batch_t 0.782 (0.765)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-06 10:09:22,341 - Train: 12.94% [639600/4942000] [129.4/1000.0] [batch_t 0.771 (0.763)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 10:10:38,920 - Train: 12.94% [639700/4942000] [129.4/1000.0] [batch_t 0.762 (0.766)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 10:11:55,249 - Train: 12.95% [639800/4942000] [129.5/1000.0] [batch_t 0.771 (0.763)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-06 10:13:11,630 - Train: 12.95% [639900/4942000] [129.5/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 10:14:28,081 - Train: 12.95% [640000/4942000] [129.5/1000.0] [batch_t 0.762 (0.764)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-06 10:15:44,591 - Train: 12.95% [640100/4942000] [129.5/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 10:17:00,934 - Train: 12.95% [640200/4942000] [129.5/1000.0] [batch_t 0.764 (0.763)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-06 10:18:17,430 - Train: 12.96% [640300/4942000] [129.6/1000.0] [batch_t 0.776 (0.765)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-06 10:19:33,830 - Train: 12.96% [640400/4942000] [129.6/1000.0] [batch_t 0.753 (0.764)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-06 10:20:50,451 - Train: 12.96% [640500/4942000] [129.6/1000.0] [batch_t 0.781 (0.766)] [data_t 0.003] [optim_t 0.778] [lr 0.005000] 2024-04-06 10:22:07,017 - Train: 12.96% [640600/4942000] [129.6/1000.0] [batch_t 0.771 (0.766)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-06 10:23:23,432 - Train: 12.96% [640700/4942000] [129.6/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 10:24:39,813 - Train: 12.97% [640800/4942000] [129.7/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-06 10:25:56,318 - Train: 12.97% [640900/4942000] [129.7/1000.0] [batch_t 0.753 (0.765)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-06 10:27:12,671 - Train: 12.97% [641000/4942000] [129.7/1000.0] [batch_t 0.753 (0.763)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-06 10:28:29,045 - Train: 12.97% [641100/4942000] [129.7/1000.0] [batch_t 0.778 (0.764)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 10:29:45,480 - Train: 12.97% [641200/4942000] [129.7/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 10:31:01,890 - Train: 12.98% [641300/4942000] [129.8/1000.0] [batch_t 0.752 (0.764)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-06 10:32:18,331 - Train: 12.98% [641400/4942000] [129.8/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 10:33:34,939 - Train: 12.98% [641500/4942000] [129.8/1000.0] [batch_t 0.764 (0.766)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 10:34:51,349 - Train: 12.98% [641600/4942000] [129.8/1000.0] [batch_t 0.762 (0.764)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-06 10:36:07,967 - Train: 12.98% [641700/4942000] [129.8/1000.0] [batch_t 0.771 (0.766)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 10:37:24,369 - Train: 12.99% [641800/4942000] [129.9/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 10:38:40,893 - Train: 12.99% [641900/4942000] [129.9/1000.0] [batch_t 0.764 (0.765)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-06 10:39:57,287 - Train: 12.99% [642000/4942000] [129.9/1000.0] [batch_t 0.765 (0.764)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-06 10:41:13,699 - Train: 12.99% [642100/4942000] [129.9/1000.0] [batch_t 0.761 (0.764)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 10:42:30,174 - Train: 12.99% [642200/4942000] [129.9/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 10:43:46,665 - Train: 13.00% [642300/4942000] [130.0/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 10:45:03,002 - Train: 13.00% [642400/4942000] [130.0/1000.0] [batch_t 0.757 (0.763)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-06 10:45:48,827 - ==> Total time: 3 days, 16:48:28 Eta: 24 days, 18:19:44 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-06 10:46:21,436 - Train: 13.00% [642500/4942000] [130.0/1000.0] [batch_t 0.753 (0.765)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-06 10:47:37,934 - Train: 13.00% [642600/4942000] [130.0/1000.0] [batch_t 0.769 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 10:48:54,490 - Train: 13.00% [642700/4942000] [130.0/1000.0] [batch_t 0.761 (0.765)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-06 10:50:10,916 - Train: 13.01% [642800/4942000] [130.1/1000.0] [batch_t 0.777 (0.764)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-06 10:51:27,396 - Train: 13.01% [642900/4942000] [130.1/1000.0] [batch_t 0.758 (0.765)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-06 10:52:43,926 - Train: 13.01% [643000/4942000] [130.1/1000.0] [batch_t 0.758 (0.765)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-06 10:54:00,433 - Train: 13.01% [643100/4942000] [130.1/1000.0] [batch_t 0.778 (0.765)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-06 10:55:16,743 - Train: 13.01% [643200/4942000] [130.1/1000.0] [batch_t 0.757 (0.763)] [data_t 0.002] [optim_t 0.754] [lr 0.005000] 2024-04-06 10:56:33,194 - Train: 13.02% [643300/4942000] [130.2/1000.0] [batch_t 0.773 (0.764)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-06 10:57:49,564 - Train: 13.02% [643400/4942000] [130.2/1000.0] [batch_t 0.771 (0.764)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-06 10:59:06,005 - Train: 13.02% [643500/4942000] [130.2/1000.0] [batch_t 0.769 (0.764)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 11:00:22,505 - Train: 13.02% [643600/4942000] [130.2/1000.0] [batch_t 0.758 (0.765)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-06 11:01:38,987 - Train: 13.03% [643700/4942000] [130.3/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 11:02:55,419 - Train: 13.03% [643800/4942000] [130.3/1000.0] [batch_t 0.757 (0.764)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-06 11:04:11,926 - Train: 13.03% [643900/4942000] [130.3/1000.0] [batch_t 0.759 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-06 11:05:28,394 - Train: 13.03% [644000/4942000] [130.3/1000.0] [batch_t 0.765 (0.765)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-06 11:06:45,002 - Train: 13.03% [644100/4942000] [130.3/1000.0] [batch_t 0.752 (0.766)] [data_t 0.003] [optim_t 0.749] [lr 0.005000] 2024-04-06 11:08:01,502 - Train: 13.04% [644200/4942000] [130.4/1000.0] [batch_t 0.764 (0.765)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-06 11:09:17,841 - Train: 13.04% [644300/4942000] [130.4/1000.0] [batch_t 0.757 (0.763)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-06 11:10:34,274 - Train: 13.04% [644400/4942000] [130.4/1000.0] [batch_t 0.776 (0.764)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 11:11:50,800 - Train: 13.04% [644500/4942000] [130.4/1000.0] [batch_t 0.764 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 11:13:07,324 - Train: 13.04% [644600/4942000] [130.4/1000.0] [batch_t 0.779 (0.765)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-06 11:14:23,854 - Train: 13.05% [644700/4942000] [130.5/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 11:15:40,470 - Train: 13.05% [644800/4942000] [130.5/1000.0] [batch_t 0.780 (0.766)] [data_t 0.003] [optim_t 0.777] [lr 0.005000] 2024-04-06 11:16:57,560 - Train: 13.05% [644900/4942000] [130.5/1000.0] [batch_t 0.778 (0.771)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 11:18:15,262 - Train: 13.05% [645000/4942000] [130.5/1000.0] [batch_t 0.787 (0.777)] [data_t 0.003] [optim_t 0.784] [lr 0.005000] 2024-04-06 11:19:32,886 - Train: 13.05% [645100/4942000] [130.5/1000.0] [batch_t 0.786 (0.776)] [data_t 0.003] [optim_t 0.783] [lr 0.005000] 2024-04-06 11:20:50,616 - Train: 13.06% [645200/4942000] [130.6/1000.0] [batch_t 0.788 (0.777)] [data_t 0.003] [optim_t 0.784] [lr 0.005000] 2024-04-06 11:22:08,341 - Train: 13.06% [645300/4942000] [130.6/1000.0] [batch_t 0.782 (0.777)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-06 11:23:26,077 - Train: 13.06% [645400/4942000] [130.6/1000.0] [batch_t 0.772 (0.777)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 11:24:43,715 - Train: 13.06% [645500/4942000] [130.6/1000.0] [batch_t 0.782 (0.776)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-06 11:26:01,465 - Train: 13.06% [645600/4942000] [130.6/1000.0] [batch_t 0.777 (0.777)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 11:27:19,184 - Train: 13.07% [645700/4942000] [130.7/1000.0] [batch_t 0.782 (0.777)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-06 11:28:36,838 - Train: 13.07% [645800/4942000] [130.7/1000.0] [batch_t 0.778 (0.776)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 11:29:54,571 - Train: 13.07% [645900/4942000] [130.7/1000.0] [batch_t 0.774 (0.777)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-06 11:31:12,254 - Train: 13.07% [646000/4942000] [130.7/1000.0] [batch_t 0.775 (0.777)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-06 11:32:30,139 - Train: 13.07% [646100/4942000] [130.7/1000.0] [batch_t 0.769 (0.779)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 11:33:47,848 - Train: 13.08% [646200/4942000] [130.8/1000.0] [batch_t 0.787 (0.777)] [data_t 0.003] [optim_t 0.784] [lr 0.005000] 2024-04-06 11:35:05,588 - Train: 13.08% [646300/4942000] [130.8/1000.0] [batch_t 0.782 (0.777)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-06 11:36:23,371 - Train: 13.08% [646400/4942000] [130.8/1000.0] [batch_t 0.778 (0.778)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 11:37:41,073 - Train: 13.08% [646500/4942000] [130.8/1000.0] [batch_t 0.782 (0.777)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-06 11:38:58,751 - Train: 13.08% [646600/4942000] [130.8/1000.0] [batch_t 0.784 (0.777)] [data_t 0.003] [optim_t 0.781] [lr 0.005000] 2024-04-06 11:40:16,399 - Train: 13.09% [646700/4942000] [130.9/1000.0] [batch_t 0.782 (0.776)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-06 11:41:33,814 - Train: 13.09% [646800/4942000] [130.9/1000.0] [batch_t 0.766 (0.774)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-06 11:42:50,259 - Train: 13.09% [646900/4942000] [130.9/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 11:44:06,672 - Train: 13.09% [647000/4942000] [130.9/1000.0] [batch_t 0.775 (0.764)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-06 11:45:23,214 - Train: 13.09% [647100/4942000] [130.9/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 11:46:40,573 - Train: 13.10% [647200/4942000] [131.0/1000.0] [batch_t 0.754 (0.773)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-06 11:47:57,111 - Train: 13.10% [647300/4942000] [131.0/1000.0] [batch_t 0.769 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 11:49:28,084 - Train: 13.10% [647400/4942000] [131.0/1000.0] [batch_t 0.779 (0.910)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-06 11:49:30,189 - ==> Total time: 3 days, 17:52:09 Eta: 24 days, 20:09:20 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-06 11:51:01,862 - Train: 13.10% [647500/4942000] [131.0/1000.0] [batch_t 0.767 (0.902)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 11:52:18,362 - Train: 13.10% [647600/4942000] [131.0/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 11:53:34,823 - Train: 13.11% [647700/4942000] [131.1/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 11:54:51,337 - Train: 13.11% [647800/4942000] [131.1/1000.0] [batch_t 0.771 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 11:56:07,802 - Train: 13.11% [647900/4942000] [131.1/1000.0] [batch_t 0.776 (0.765)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-06 11:57:24,363 - Train: 13.11% [648000/4942000] [131.1/1000.0] [batch_t 0.758 (0.766)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-06 11:58:40,831 - Train: 13.11% [648100/4942000] [131.1/1000.0] [batch_t 0.776 (0.765)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-06 11:59:57,333 - Train: 13.12% [648200/4942000] [131.2/1000.0] [batch_t 0.774 (0.765)] [data_t 0.004] [optim_t 0.770] [lr 0.005000] 2024-04-06 12:01:13,811 - Train: 13.12% [648300/4942000] [131.2/1000.0] [batch_t 0.766 (0.765)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-06 12:02:30,266 - Train: 13.12% [648400/4942000] [131.2/1000.0] [batch_t 0.777 (0.764)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 12:03:46,798 - Train: 13.12% [648500/4942000] [131.2/1000.0] [batch_t 0.771 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 12:05:03,374 - Train: 13.12% [648600/4942000] [131.2/1000.0] [batch_t 0.763 (0.766)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 12:06:19,764 - Train: 13.13% [648700/4942000] [131.3/1000.0] [batch_t 0.777 (0.764)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 12:07:36,262 - Train: 13.13% [648800/4942000] [131.3/1000.0] [batch_t 0.756 (0.765)] [data_t 0.002] [optim_t 0.754] [lr 0.005000] 2024-04-06 12:08:52,832 - Train: 13.13% [648900/4942000] [131.3/1000.0] [batch_t 0.772 (0.766)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-06 12:10:09,302 - Train: 13.13% [649000/4942000] [131.3/1000.0] [batch_t 0.775 (0.765)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-06 12:11:25,852 - Train: 13.13% [649100/4942000] [131.3/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 12:12:42,345 - Train: 13.14% [649200/4942000] [131.4/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 12:13:58,728 - Train: 13.14% [649300/4942000] [131.4/1000.0] [batch_t 0.769 (0.764)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 12:15:15,166 - Train: 13.14% [649400/4942000] [131.4/1000.0] [batch_t 0.771 (0.764)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 12:16:31,690 - Train: 13.14% [649500/4942000] [131.4/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 12:17:48,009 - Train: 13.14% [649600/4942000] [131.4/1000.0] [batch_t 0.767 (0.763)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 12:19:04,475 - Train: 13.15% [649700/4942000] [131.5/1000.0] [batch_t 0.776 (0.765)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 12:20:20,711 - Train: 13.15% [649800/4942000] [131.5/1000.0] [batch_t 0.762 (0.762)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-06 12:21:37,182 - Train: 13.15% [649900/4942000] [131.5/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 12:22:53,602 - Train: 13.15% [650000/4942000] [131.5/1000.0] [batch_t 0.761 (0.764)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-06 12:24:10,008 - Train: 13.15% [650100/4942000] [131.5/1000.0] [batch_t 0.759 (0.764)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-06 12:25:26,304 - Train: 13.16% [650200/4942000] [131.6/1000.0] [batch_t 0.759 (0.763)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-06 12:26:42,781 - Train: 13.16% [650300/4942000] [131.6/1000.0] [batch_t 0.759 (0.765)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-06 12:27:59,298 - Train: 13.16% [650400/4942000] [131.6/1000.0] [batch_t 0.758 (0.765)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-06 12:29:15,708 - Train: 13.16% [650500/4942000] [131.6/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 12:30:32,202 - Train: 13.16% [650600/4942000] [131.6/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 12:31:48,555 - Train: 13.17% [650700/4942000] [131.7/1000.0] [batch_t 0.745 (0.763)] [data_t 0.003] [optim_t 0.742] [lr 0.005000] 2024-04-06 12:33:04,903 - Train: 13.17% [650800/4942000] [131.7/1000.0] [batch_t 0.762 (0.763)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 12:34:21,361 - Train: 13.17% [650900/4942000] [131.7/1000.0] [batch_t 0.772 (0.764)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-06 12:35:37,810 - Train: 13.17% [651000/4942000] [131.7/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 12:36:54,305 - Train: 13.17% [651100/4942000] [131.7/1000.0] [batch_t 0.776 (0.765)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-06 12:38:10,833 - Train: 13.18% [651200/4942000] [131.8/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 12:39:27,270 - Train: 13.18% [651300/4942000] [131.8/1000.0] [batch_t 0.771 (0.764)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-06 12:40:43,687 - Train: 13.18% [651400/4942000] [131.8/1000.0] [batch_t 0.755 (0.764)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-06 12:42:00,312 - Train: 13.18% [651500/4942000] [131.8/1000.0] [batch_t 0.777 (0.766)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 12:43:16,721 - Train: 13.18% [651600/4942000] [131.8/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-06 12:44:33,285 - Train: 13.19% [651700/4942000] [131.9/1000.0] [batch_t 0.763 (0.766)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 12:45:49,789 - Train: 13.19% [651800/4942000] [131.9/1000.0] [batch_t 0.767 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 12:47:06,278 - Train: 13.19% [651900/4942000] [131.9/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 12:48:22,555 - Train: 13.19% [652000/4942000] [131.9/1000.0] [batch_t 0.770 (0.763)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-06 12:49:39,167 - Train: 13.20% [652100/4942000] [132.0/1000.0] [batch_t 0.776 (0.766)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-06 12:50:55,649 - Train: 13.20% [652200/4942000] [132.0/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 12:52:11,973 - Train: 13.20% [652300/4942000] [132.0/1000.0] [batch_t 0.757 (0.763)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-06 12:52:45,558 - ==> Total time: 3 days, 18:55:24 Eta: 24 days, 21:53:28 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-06 12:53:30,317 - Train: 13.20% [652400/4942000] [132.0/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 12:54:46,785 - Train: 13.20% [652500/4942000] [132.0/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 12:56:03,262 - Train: 13.21% [652600/4942000] [132.1/1000.0] [batch_t 0.782 (0.765)] [data_t 0.002] [optim_t 0.779] [lr 0.005000] 2024-04-06 12:57:19,774 - Train: 13.21% [652700/4942000] [132.1/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 12:58:36,227 - Train: 13.21% [652800/4942000] [132.1/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 12:59:52,586 - Train: 13.21% [652900/4942000] [132.1/1000.0] [batch_t 0.763 (0.764)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 13:01:09,049 - Train: 13.21% [653000/4942000] [132.1/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 13:02:25,613 - Train: 13.22% [653100/4942000] [132.2/1000.0] [batch_t 0.767 (0.766)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 13:03:41,892 - Train: 13.22% [653200/4942000] [132.2/1000.0] [batch_t 0.763 (0.763)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 13:04:58,380 - Train: 13.22% [653300/4942000] [132.2/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 13:06:14,802 - Train: 13.22% [653400/4942000] [132.2/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-06 13:07:31,210 - Train: 13.22% [653500/4942000] [132.2/1000.0] [batch_t 0.755 (0.764)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-06 13:08:47,552 - Train: 13.23% [653600/4942000] [132.3/1000.0] [batch_t 0.752 (0.763)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-06 13:10:04,054 - Train: 13.23% [653700/4942000] [132.3/1000.0] [batch_t 0.771 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 13:11:20,574 - Train: 13.23% [653800/4942000] [132.3/1000.0] [batch_t 0.762 (0.765)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-06 13:12:37,035 - Train: 13.23% [653900/4942000] [132.3/1000.0] [batch_t 0.761 (0.765)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-06 13:13:53,537 - Train: 13.23% [654000/4942000] [132.3/1000.0] [batch_t 0.762 (0.765)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-06 13:15:10,143 - Train: 13.24% [654100/4942000] [132.4/1000.0] [batch_t 0.769 (0.766)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 13:16:26,636 - Train: 13.24% [654200/4942000] [132.4/1000.0] [batch_t 0.758 (0.765)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-06 13:17:43,145 - Train: 13.24% [654300/4942000] [132.4/1000.0] [batch_t 0.776 (0.765)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-06 13:18:59,546 - Train: 13.24% [654400/4942000] [132.4/1000.0] [batch_t 0.754 (0.764)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-06 13:20:16,171 - Train: 13.24% [654500/4942000] [132.4/1000.0] [batch_t 0.769 (0.766)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-06 13:21:32,598 - Train: 13.25% [654600/4942000] [132.5/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-06 13:22:48,983 - Train: 13.25% [654700/4942000] [132.5/1000.0] [batch_t 0.773 (0.764)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 13:24:05,590 - Train: 13.25% [654800/4942000] [132.5/1000.0] [batch_t 0.773 (0.766)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 13:25:22,026 - Train: 13.25% [654900/4942000] [132.5/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-06 13:26:38,493 - Train: 13.25% [655000/4942000] [132.5/1000.0] [batch_t 0.771 (0.765)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-06 13:27:54,992 - Train: 13.26% [655100/4942000] [132.6/1000.0] [batch_t 0.765 (0.765)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-06 13:29:11,530 - Train: 13.26% [655200/4942000] [132.6/1000.0] [batch_t 0.767 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 13:30:27,946 - Train: 13.26% [655300/4942000] [132.6/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 13:31:44,343 - Train: 13.26% [655400/4942000] [132.6/1000.0] [batch_t 0.777 (0.764)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-06 13:33:00,715 - Train: 13.26% [655500/4942000] [132.6/1000.0] [batch_t 0.775 (0.764)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-06 13:34:17,086 - Train: 13.27% [655600/4942000] [132.7/1000.0] [batch_t 0.765 (0.764)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-06 13:35:33,537 - Train: 13.27% [655700/4942000] [132.7/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-06 13:36:50,091 - Train: 13.27% [655800/4942000] [132.7/1000.0] [batch_t 0.778 (0.765)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 13:38:06,598 - Train: 13.27% [655900/4942000] [132.7/1000.0] [batch_t 0.760 (0.765)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-06 13:39:22,989 - Train: 13.27% [656000/4942000] [132.7/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 13:40:39,428 - Train: 13.28% [656100/4942000] [132.8/1000.0] [batch_t 0.767 (0.764)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 13:41:55,920 - Train: 13.28% [656200/4942000] [132.8/1000.0] [batch_t 0.751 (0.765)] [data_t 0.002] [optim_t 0.749] [lr 0.005000] 2024-04-06 13:43:12,342 - Train: 13.28% [656300/4942000] [132.8/1000.0] [batch_t 0.752 (0.764)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-06 13:44:28,653 - Train: 13.28% [656400/4942000] [132.8/1000.0] [batch_t 0.769 (0.763)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-06 13:45:45,187 - Train: 13.28% [656500/4942000] [132.8/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 13:47:01,555 - Train: 13.29% [656600/4942000] [132.9/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 13:48:17,906 - Train: 13.29% [656700/4942000] [132.9/1000.0] [batch_t 0.767 (0.763)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 13:49:34,375 - Train: 13.29% [656800/4942000] [132.9/1000.0] [batch_t 0.771 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 13:50:50,946 - Train: 13.29% [656900/4942000] [132.9/1000.0] [batch_t 0.772 (0.766)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 13:52:07,365 - Train: 13.29% [657000/4942000] [132.9/1000.0] [batch_t 0.771 (0.764)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 13:53:23,881 - Train: 13.30% [657100/4942000] [133.0/1000.0] [batch_t 0.764 (0.765)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-06 13:54:40,457 - Train: 13.30% [657200/4942000] [133.0/1000.0] [batch_t 0.776 (0.766)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 13:55:46,286 - ==> Total time: 3 days, 19:58:25 Eta: 24 days, 23:33:29 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-06 13:55:58,897 - Train: 13.30% [657300/4942000] [133.0/1000.0] [batch_t 0.754 (0.762)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-06 13:57:15,364 - Train: 13.30% [657400/4942000] [133.0/1000.0] [batch_t 0.762 (0.765)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-06 13:58:31,929 - Train: 13.30% [657500/4942000] [133.0/1000.0] [batch_t 0.766 (0.766)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 13:59:48,441 - Train: 13.31% [657600/4942000] [133.1/1000.0] [batch_t 0.767 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 14:01:04,948 - Train: 13.31% [657700/4942000] [133.1/1000.0] [batch_t 0.777 (0.765)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-06 14:02:21,459 - Train: 13.31% [657800/4942000] [133.1/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 14:03:37,918 - Train: 13.31% [657900/4942000] [133.1/1000.0] [batch_t 0.776 (0.764)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 14:04:54,456 - Train: 13.31% [658000/4942000] [133.1/1000.0] [batch_t 0.758 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-06 14:06:10,862 - Train: 13.32% [658100/4942000] [133.2/1000.0] [batch_t 0.764 (0.764)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 14:07:27,446 - Train: 13.32% [658200/4942000] [133.2/1000.0] [batch_t 0.769 (0.766)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-06 14:08:43,938 - Train: 13.32% [658300/4942000] [133.2/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 14:10:00,383 - Train: 13.32% [658400/4942000] [133.2/1000.0] [batch_t 0.766 (0.764)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-06 14:11:16,875 - Train: 13.32% [658500/4942000] [133.2/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 14:12:33,395 - Train: 13.33% [658600/4942000] [133.3/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 14:13:49,855 - Train: 13.33% [658700/4942000] [133.3/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 14:15:06,279 - Train: 13.33% [658800/4942000] [133.3/1000.0] [batch_t 0.759 (0.764)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-06 14:16:22,779 - Train: 13.33% [658900/4942000] [133.3/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 14:17:39,152 - Train: 13.33% [659000/4942000] [133.3/1000.0] [batch_t 0.769 (0.764)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 14:18:55,598 - Train: 13.34% [659100/4942000] [133.4/1000.0] [batch_t 0.763 (0.764)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 14:20:12,130 - Train: 13.34% [659200/4942000] [133.4/1000.0] [batch_t 0.771 (0.765)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-06 14:21:28,677 - Train: 13.34% [659300/4942000] [133.4/1000.0] [batch_t 0.766 (0.765)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-06 14:22:45,062 - Train: 13.34% [659400/4942000] [133.4/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 14:24:01,502 - Train: 13.34% [659500/4942000] [133.4/1000.0] [batch_t 0.763 (0.764)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 14:25:17,900 - Train: 13.35% [659600/4942000] [133.5/1000.0] [batch_t 0.764 (0.764)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-06 14:26:34,251 - Train: 13.35% [659700/4942000] [133.5/1000.0] [batch_t 0.769 (0.763)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 14:27:50,710 - Train: 13.35% [659800/4942000] [133.5/1000.0] [batch_t 0.769 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 14:29:07,304 - Train: 13.35% [659900/4942000] [133.5/1000.0] [batch_t 0.758 (0.766)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-06 14:30:23,696 - Train: 13.35% [660000/4942000] [133.5/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 14:31:40,100 - Train: 13.36% [660100/4942000] [133.6/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 14:32:56,596 - Train: 13.36% [660200/4942000] [133.6/1000.0] [batch_t 0.766 (0.765)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-06 14:34:13,091 - Train: 13.36% [660300/4942000] [133.6/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 14:35:29,651 - Train: 13.36% [660400/4942000] [133.6/1000.0] [batch_t 0.769 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 14:36:46,140 - Train: 13.37% [660500/4942000] [133.7/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-06 14:38:02,433 - Train: 13.37% [660600/4942000] [133.7/1000.0] [batch_t 0.753 (0.763)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-06 14:39:18,882 - Train: 13.37% [660700/4942000] [133.7/1000.0] [batch_t 0.762 (0.764)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-06 14:40:35,311 - Train: 13.37% [660800/4942000] [133.7/1000.0] [batch_t 0.775 (0.764)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-06 14:41:51,850 - Train: 13.37% [660900/4942000] [133.7/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 14:43:08,421 - Train: 13.38% [661000/4942000] [133.8/1000.0] [batch_t 0.772 (0.766)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 14:44:24,903 - Train: 13.38% [661100/4942000] [133.8/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 14:45:41,372 - Train: 13.38% [661200/4942000] [133.8/1000.0] [batch_t 0.764 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 14:46:57,764 - Train: 13.38% [661300/4942000] [133.8/1000.0] [batch_t 0.769 (0.764)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 14:48:14,300 - Train: 13.38% [661400/4942000] [133.8/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-06 14:49:30,761 - Train: 13.39% [661500/4942000] [133.9/1000.0] [batch_t 0.775 (0.765)] [data_t 0.003] [optim_t 0.772] [lr 0.005000] 2024-04-06 14:50:47,211 - Train: 13.39% [661600/4942000] [133.9/1000.0] [batch_t 0.766 (0.764)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-06 14:52:03,615 - Train: 13.39% [661700/4942000] [133.9/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 14:53:20,013 - Train: 13.39% [661800/4942000] [133.9/1000.0] [batch_t 0.757 (0.764)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-06 14:54:36,581 - Train: 13.39% [661900/4942000] [133.9/1000.0] [batch_t 0.765 (0.766)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-06 14:55:53,153 - Train: 13.40% [662000/4942000] [134.0/1000.0] [batch_t 0.767 (0.766)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 14:57:09,594 - Train: 13.40% [662100/4942000] [134.0/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 14:58:26,183 - Train: 13.40% [662200/4942000] [134.0/1000.0] [batch_t 0.773 (0.766)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-06 14:58:47,633 - ==> Total time: 3 days, 21:01:26 Eta: 25 days, 1:11:08 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-06 14:59:45,043 - Train: 13.40% [662300/4942000] [134.0/1000.0] [batch_t 0.763 (0.766)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 15:01:01,549 - Train: 13.40% [662400/4942000] [134.0/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-06 15:02:18,167 - Train: 13.41% [662500/4942000] [134.1/1000.0] [batch_t 0.765 (0.766)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-06 15:03:34,591 - Train: 13.41% [662600/4942000] [134.1/1000.0] [batch_t 0.772 (0.764)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 15:04:51,134 - Train: 13.41% [662700/4942000] [134.1/1000.0] [batch_t 0.771 (0.765)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-06 15:06:07,742 - Train: 13.41% [662800/4942000] [134.1/1000.0] [batch_t 0.780 (0.766)] [data_t 0.002] [optim_t 0.778] [lr 0.005000] 2024-04-06 15:07:24,478 - Train: 13.41% [662900/4942000] [134.1/1000.0] [batch_t 0.771 (0.767)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-06 15:08:40,951 - Train: 13.42% [663000/4942000] [134.2/1000.0] [batch_t 0.762 (0.765)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-06 15:09:57,520 - Train: 13.42% [663100/4942000] [134.2/1000.0] [batch_t 0.772 (0.766)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 15:11:13,881 - Train: 13.42% [663200/4942000] [134.2/1000.0] [batch_t 0.776 (0.764)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-06 15:12:30,361 - Train: 13.42% [663300/4942000] [134.2/1000.0] [batch_t 0.759 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-06 15:13:46,990 - Train: 13.42% [663400/4942000] [134.2/1000.0] [batch_t 0.758 (0.766)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-06 15:15:03,506 - Train: 13.43% [663500/4942000] [134.3/1000.0] [batch_t 0.774 (0.765)] [data_t 0.002] [optim_t 0.772] [lr 0.005000] 2024-04-06 15:16:28,387 - Train: 13.43% [663600/4942000] [134.3/1000.0] [batch_t 0.768 (0.849)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 15:17:44,854 - Train: 13.43% [663700/4942000] [134.3/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 15:19:01,313 - Train: 13.43% [663800/4942000] [134.3/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 15:20:17,819 - Train: 13.43% [663900/4942000] [134.3/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 15:21:34,358 - Train: 13.44% [664000/4942000] [134.4/1000.0] [batch_t 0.767 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 15:22:50,929 - Train: 13.44% [664100/4942000] [134.4/1000.0] [batch_t 0.754 (0.766)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-06 15:24:07,387 - Train: 13.44% [664200/4942000] [134.4/1000.0] [batch_t 0.759 (0.764)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-06 15:25:23,809 - Train: 13.44% [664300/4942000] [134.4/1000.0] [batch_t 0.755 (0.764)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-06 15:26:40,163 - Train: 13.44% [664400/4942000] [134.4/1000.0] [batch_t 0.762 (0.763)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-06 15:27:56,665 - Train: 13.45% [664500/4942000] [134.5/1000.0] [batch_t 0.776 (0.765)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-06 15:29:13,131 - Train: 13.45% [664600/4942000] [134.5/1000.0] [batch_t 0.769 (0.765)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-06 15:30:29,624 - Train: 13.45% [664700/4942000] [134.5/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 15:31:46,020 - Train: 13.45% [664800/4942000] [134.5/1000.0] [batch_t 0.768 (0.764)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 15:33:02,419 - Train: 13.45% [664900/4942000] [134.5/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 15:34:18,784 - Train: 13.46% [665000/4942000] [134.6/1000.0] [batch_t 0.764 (0.764)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 15:35:35,290 - Train: 13.46% [665100/4942000] [134.6/1000.0] [batch_t 0.764 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 15:36:51,746 - Train: 13.46% [665200/4942000] [134.6/1000.0] [batch_t 0.767 (0.764)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-06 15:38:08,214 - Train: 13.46% [665300/4942000] [134.6/1000.0] [batch_t 0.764 (0.765)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 15:39:24,682 - Train: 13.46% [665400/4942000] [134.6/1000.0] [batch_t 0.779 (0.765)] [data_t 0.003] [optim_t 0.777] [lr 0.005000] 2024-04-06 15:40:41,033 - Train: 13.47% [665500/4942000] [134.7/1000.0] [batch_t 0.755 (0.763)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-06 15:41:57,411 - Train: 13.47% [665600/4942000] [134.7/1000.0] [batch_t 0.774 (0.764)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-06 15:43:13,942 - Train: 13.47% [665700/4942000] [134.7/1000.0] [batch_t 0.778 (0.765)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 15:44:30,338 - Train: 13.47% [665800/4942000] [134.7/1000.0] [batch_t 0.773 (0.764)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-06 15:45:46,847 - Train: 13.47% [665900/4942000] [134.7/1000.0] [batch_t 0.777 (0.765)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-06 15:47:03,335 - Train: 13.48% [666000/4942000] [134.8/1000.0] [batch_t 0.782 (0.765)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-06 15:48:19,822 - Train: 13.48% [666100/4942000] [134.8/1000.0] [batch_t 0.757 (0.765)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-06 15:49:36,341 - Train: 13.48% [666200/4942000] [134.8/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 15:50:52,908 - Train: 13.48% [666300/4942000] [134.8/1000.0] [batch_t 0.769 (0.766)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 15:52:09,264 - Train: 13.48% [666400/4942000] [134.8/1000.0] [batch_t 0.752 (0.763)] [data_t 0.003] [optim_t 0.749] [lr 0.005000] 2024-04-06 15:53:25,785 - Train: 13.49% [666500/4942000] [134.9/1000.0] [batch_t 0.759 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-06 15:54:42,394 - Train: 13.49% [666600/4942000] [134.9/1000.0] [batch_t 0.767 (0.766)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 15:55:58,951 - Train: 13.49% [666700/4942000] [134.9/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 15:57:15,374 - Train: 13.49% [666800/4942000] [134.9/1000.0] [batch_t 0.768 (0.764)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 15:58:31,766 - Train: 13.49% [666900/4942000] [134.9/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 16:02:02,214 - Train: 13.50% [667000/4942000] [135.0/1000.0] [batch_t 0.771 (2.104)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-06 16:03:18,751 - Train: 13.50% [667100/4942000] [135.0/1000.0] [batch_t 0.761 (0.765)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-06 16:04:12,328 - ==> Total time: 3 days, 22:06:51 Eta: 25 days, 3:01:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-06 16:04:38,563 - Train: 13.50% [667200/4942000] [135.0/1000.0] [batch_t 0.765 (0.777)] [data_t 0.002] [optim_t 0.763] [lr 0.005000] 2024-04-06 16:05:55,003 - Train: 13.50% [667300/4942000] [135.0/1000.0] [batch_t 0.777 (0.764)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 16:07:11,428 - Train: 13.50% [667400/4942000] [135.0/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 16:08:27,971 - Train: 13.51% [667500/4942000] [135.1/1000.0] [batch_t 0.766 (0.765)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-06 16:09:44,479 - Train: 13.51% [667600/4942000] [135.1/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 16:11:00,949 - Train: 13.51% [667700/4942000] [135.1/1000.0] [batch_t 0.757 (0.765)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-06 16:12:17,211 - Train: 13.51% [667800/4942000] [135.1/1000.0] [batch_t 0.768 (0.763)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 16:13:33,781 - Train: 13.51% [667900/4942000] [135.1/1000.0] [batch_t 0.759 (0.766)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-06 16:14:50,102 - Train: 13.52% [668000/4942000] [135.2/1000.0] [batch_t 0.755 (0.763)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-06 16:16:06,627 - Train: 13.52% [668100/4942000] [135.2/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 16:17:23,064 - Train: 13.52% [668200/4942000] [135.2/1000.0] [batch_t 0.752 (0.764)] [data_t 0.003] [optim_t 0.749] [lr 0.005000] 2024-04-06 16:18:39,556 - Train: 13.52% [668300/4942000] [135.2/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 16:19:56,134 - Train: 13.52% [668400/4942000] [135.2/1000.0] [batch_t 0.754 (0.766)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-06 16:21:12,610 - Train: 13.53% [668500/4942000] [135.3/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 16:22:29,017 - Train: 13.53% [668600/4942000] [135.3/1000.0] [batch_t 0.769 (0.764)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 16:23:45,452 - Train: 13.53% [668700/4942000] [135.3/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 16:25:01,881 - Train: 13.53% [668800/4942000] [135.3/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-06 16:26:18,323 - Train: 13.54% [668900/4942000] [135.4/1000.0] [batch_t 0.776 (0.764)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-06 16:27:34,654 - Train: 13.54% [669000/4942000] [135.4/1000.0] [batch_t 0.763 (0.763)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 16:28:51,083 - Train: 13.54% [669100/4942000] [135.4/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 16:30:07,508 - Train: 13.54% [669200/4942000] [135.4/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 16:31:23,827 - Train: 13.54% [669300/4942000] [135.4/1000.0] [batch_t 0.781 (0.763)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-06 16:32:40,342 - Train: 13.55% [669400/4942000] [135.5/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 16:33:56,620 - Train: 13.55% [669500/4942000] [135.5/1000.0] [batch_t 0.762 (0.763)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 16:35:13,110 - Train: 13.55% [669600/4942000] [135.5/1000.0] [batch_t 0.769 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 16:36:29,582 - Train: 13.55% [669700/4942000] [135.5/1000.0] [batch_t 0.764 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 16:37:46,150 - Train: 13.55% [669800/4942000] [135.5/1000.0] [batch_t 0.774 (0.766)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-06 16:39:02,619 - Train: 13.56% [669900/4942000] [135.6/1000.0] [batch_t 0.753 (0.765)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-06 16:40:18,996 - Train: 13.56% [670000/4942000] [135.6/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 16:41:35,375 - Train: 13.56% [670100/4942000] [135.6/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 16:42:51,813 - Train: 13.56% [670200/4942000] [135.6/1000.0] [batch_t 0.773 (0.764)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-06 16:44:08,208 - Train: 13.56% [670300/4942000] [135.6/1000.0] [batch_t 0.763 (0.764)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-06 16:45:24,644 - Train: 13.57% [670400/4942000] [135.7/1000.0] [batch_t 0.773 (0.764)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-06 16:46:41,041 - Train: 13.57% [670500/4942000] [135.7/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 16:47:57,546 - Train: 13.57% [670600/4942000] [135.7/1000.0] [batch_t 0.767 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 16:49:13,897 - Train: 13.57% [670700/4942000] [135.7/1000.0] [batch_t 0.768 (0.763)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 16:50:30,399 - Train: 13.57% [670800/4942000] [135.7/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 16:51:46,765 - Train: 13.58% [670900/4942000] [135.8/1000.0] [batch_t 0.778 (0.764)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 16:53:03,200 - Train: 13.58% [671000/4942000] [135.8/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 16:54:19,528 - Train: 13.58% [671100/4942000] [135.8/1000.0] [batch_t 0.783 (0.763)] [data_t 0.003] [optim_t 0.780] [lr 0.005000] 2024-04-06 16:55:35,995 - Train: 13.58% [671200/4942000] [135.8/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 16:56:52,442 - Train: 13.58% [671300/4942000] [135.8/1000.0] [batch_t 0.771 (0.764)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 16:58:08,936 - Train: 13.59% [671400/4942000] [135.9/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-06 16:59:25,293 - Train: 13.59% [671500/4942000] [135.9/1000.0] [batch_t 0.766 (0.763)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-06 17:00:41,790 - Train: 13.59% [671600/4942000] [135.9/1000.0] [batch_t 0.788 (0.765)] [data_t 0.004] [optim_t 0.784] [lr 0.005000] 2024-04-06 17:01:58,235 - Train: 13.59% [671700/4942000] [135.9/1000.0] [batch_t 0.772 (0.764)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 17:03:14,648 - Train: 13.59% [671800/4942000] [135.9/1000.0] [batch_t 0.766 (0.764)] [data_t 0.002] [optim_t 0.763] [lr 0.005000] 2024-04-06 17:04:30,973 - Train: 13.60% [671900/4942000] [136.0/1000.0] [batch_t 0.768 (0.763)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 17:05:47,340 - Train: 13.60% [672000/4942000] [136.0/1000.0] [batch_t 0.759 (0.764)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-06 17:07:03,777 - Train: 13.60% [672100/4942000] [136.0/1000.0] [batch_t 0.774 (0.764)] [data_t 0.002] [optim_t 0.772] [lr 0.005000] 2024-04-06 17:07:12,995 - ==> Total time: 3 days, 23:09:52 Eta: 25 days, 4:34:27 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-06 17:08:22,188 - Train: 13.60% [672200/4942000] [136.0/1000.0] [batch_t 0.768 (0.763)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 17:09:38,657 - Train: 13.60% [672300/4942000] [136.0/1000.0] [batch_t 0.753 (0.765)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-06 17:10:55,034 - Train: 13.61% [672400/4942000] [136.1/1000.0] [batch_t 0.744 (0.764)] [data_t 0.002] [optim_t 0.742] [lr 0.005000] 2024-04-06 17:12:11,511 - Train: 13.61% [672500/4942000] [136.1/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 17:13:27,975 - Train: 13.61% [672600/4942000] [136.1/1000.0] [batch_t 0.760 (0.765)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-06 17:14:44,277 - Train: 13.61% [672700/4942000] [136.1/1000.0] [batch_t 0.768 (0.763)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 17:16:00,768 - Train: 13.61% [672800/4942000] [136.1/1000.0] [batch_t 0.780 (0.765)] [data_t 0.003] [optim_t 0.777] [lr 0.005000] 2024-04-06 17:17:17,257 - Train: 13.62% [672900/4942000] [136.2/1000.0] [batch_t 0.757 (0.765)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-06 17:18:33,851 - Train: 13.62% [673000/4942000] [136.2/1000.0] [batch_t 0.763 (0.766)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 17:19:50,312 - Train: 13.62% [673100/4942000] [136.2/1000.0] [batch_t 0.753 (0.765)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-06 17:21:06,652 - Train: 13.62% [673200/4942000] [136.2/1000.0] [batch_t 0.772 (0.763)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 17:22:23,158 - Train: 13.62% [673300/4942000] [136.2/1000.0] [batch_t 0.769 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 17:23:39,470 - Train: 13.63% [673400/4942000] [136.3/1000.0] [batch_t 0.755 (0.763)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-06 17:24:55,841 - Train: 13.63% [673500/4942000] [136.3/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 17:26:12,388 - Train: 13.63% [673600/4942000] [136.3/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-06 17:27:28,930 - Train: 13.63% [673700/4942000] [136.3/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 17:28:45,311 - Train: 13.63% [673800/4942000] [136.3/1000.0] [batch_t 0.777 (0.764)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 17:30:02,892 - Train: 13.64% [673900/4942000] [136.4/1000.0] [batch_t 0.768 (0.776)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 17:31:19,351 - Train: 13.64% [674000/4942000] [136.4/1000.0] [batch_t 0.767 (0.764)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 17:32:35,859 - Train: 13.64% [674100/4942000] [136.4/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 17:33:52,342 - Train: 13.64% [674200/4942000] [136.4/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 17:35:08,727 - Train: 13.64% [674300/4942000] [136.4/1000.0] [batch_t 0.772 (0.764)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-06 17:36:25,127 - Train: 13.65% [674400/4942000] [136.5/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 17:37:41,500 - Train: 13.65% [674500/4942000] [136.5/1000.0] [batch_t 0.766 (0.764)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-06 17:38:57,986 - Train: 13.65% [674600/4942000] [136.5/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 17:40:14,364 - Train: 13.65% [674700/4942000] [136.5/1000.0] [batch_t 0.763 (0.764)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 17:41:30,745 - Train: 13.65% [674800/4942000] [136.5/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 17:42:47,123 - Train: 13.66% [674900/4942000] [136.6/1000.0] [batch_t 0.754 (0.764)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-06 17:44:03,592 - Train: 13.66% [675000/4942000] [136.6/1000.0] [batch_t 0.759 (0.765)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-06 17:45:20,077 - Train: 13.66% [675100/4942000] [136.6/1000.0] [batch_t 0.758 (0.765)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-06 17:46:36,442 - Train: 13.66% [675200/4942000] [136.6/1000.0] [batch_t 0.777 (0.764)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-06 17:47:52,947 - Train: 13.66% [675300/4942000] [136.6/1000.0] [batch_t 0.758 (0.765)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-06 17:49:09,363 - Train: 13.67% [675400/4942000] [136.7/1000.0] [batch_t 0.757 (0.764)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-06 17:50:25,854 - Train: 13.67% [675500/4942000] [136.7/1000.0] [batch_t 0.766 (0.765)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-06 17:51:42,361 - Train: 13.67% [675600/4942000] [136.7/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-06 17:52:58,713 - Train: 13.67% [675700/4942000] [136.7/1000.0] [batch_t 0.778 (0.763)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-06 17:54:15,282 - Train: 13.67% [675800/4942000] [136.7/1000.0] [batch_t 0.754 (0.766)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-06 17:55:31,751 - Train: 13.68% [675900/4942000] [136.8/1000.0] [batch_t 0.754 (0.765)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-06 17:56:48,200 - Train: 13.68% [676000/4942000] [136.8/1000.0] [batch_t 0.763 (0.764)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 17:58:04,761 - Train: 13.68% [676100/4942000] [136.8/1000.0] [batch_t 0.773 (0.766)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-06 17:59:21,238 - Train: 13.68% [676200/4942000] [136.8/1000.0] [batch_t 0.755 (0.765)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-06 18:00:37,722 - Train: 13.68% [676300/4942000] [136.8/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 18:01:54,261 - Train: 13.69% [676400/4942000] [136.9/1000.0] [batch_t 0.764 (0.765)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-06 18:03:10,818 - Train: 13.69% [676500/4942000] [136.9/1000.0] [batch_t 0.762 (0.765)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-06 18:04:27,263 - Train: 13.69% [676600/4942000] [136.9/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 18:05:43,833 - Train: 13.69% [676700/4942000] [136.9/1000.0] [batch_t 0.777 (0.766)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 18:07:00,237 - Train: 13.69% [676800/4942000] [136.9/1000.0] [batch_t 0.769 (0.764)] [data_t 0.004] [optim_t 0.765] [lr 0.005000] 2024-04-06 18:08:16,693 - Train: 13.70% [676900/4942000] [137.0/1000.0] [batch_t 0.775 (0.764)] [data_t 0.003] [optim_t 0.772] [lr 0.005000] 2024-04-06 18:09:32,946 - Train: 13.70% [677000/4942000] [137.0/1000.0] [batch_t 0.772 (0.762)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 18:10:14,151 - ==> Total time: 4 days, 0:12:53 Eta: 25 days, 6:04:59 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-06 18:10:51,881 - Train: 13.70% [677100/4942000] [137.0/1000.0] [batch_t 0.767 (0.768)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 18:12:08,392 - Train: 13.70% [677200/4942000] [137.0/1000.0] [batch_t 0.769 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 18:13:24,826 - Train: 13.70% [677300/4942000] [137.0/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 18:14:41,248 - Train: 13.71% [677400/4942000] [137.1/1000.0] [batch_t 0.773 (0.764)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 18:15:57,647 - Train: 13.71% [677500/4942000] [137.1/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 18:17:14,165 - Train: 13.71% [677600/4942000] [137.1/1000.0] [batch_t 0.770 (0.765)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-06 18:18:30,655 - Train: 13.71% [677700/4942000] [137.1/1000.0] [batch_t 0.776 (0.765)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-06 18:19:47,094 - Train: 13.72% [677800/4942000] [137.2/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 18:21:03,522 - Train: 13.72% [677900/4942000] [137.2/1000.0] [batch_t 0.760 (0.764)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-06 18:22:19,941 - Train: 13.72% [678000/4942000] [137.2/1000.0] [batch_t 0.754 (0.764)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-06 18:23:36,528 - Train: 13.72% [678100/4942000] [137.2/1000.0] [batch_t 0.768 (0.766)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 18:24:53,059 - Train: 13.72% [678200/4942000] [137.2/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 18:26:09,416 - Train: 13.73% [678300/4942000] [137.3/1000.0] [batch_t 0.755 (0.763)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-06 18:27:25,845 - Train: 13.73% [678400/4942000] [137.3/1000.0] [batch_t 0.773 (0.764)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-06 18:28:42,375 - Train: 13.73% [678500/4942000] [137.3/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 18:29:58,884 - Train: 13.73% [678600/4942000] [137.3/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 18:31:15,471 - Train: 13.73% [678700/4942000] [137.3/1000.0] [batch_t 0.769 (0.766)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-06 18:32:31,947 - Train: 13.74% [678800/4942000] [137.4/1000.0] [batch_t 0.745 (0.765)] [data_t 0.002] [optim_t 0.743] [lr 0.005000] 2024-04-06 18:33:48,437 - Train: 13.74% [678900/4942000] [137.4/1000.0] [batch_t 0.751 (0.765)] [data_t 0.003] [optim_t 0.748] [lr 0.005000] 2024-04-06 18:35:04,852 - Train: 13.74% [679000/4942000] [137.4/1000.0] [batch_t 0.765 (0.764)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-06 18:36:21,359 - Train: 13.74% [679100/4942000] [137.4/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 18:37:37,935 - Train: 13.74% [679200/4942000] [137.4/1000.0] [batch_t 0.766 (0.766)] [data_t 0.002] [optim_t 0.763] [lr 0.005000] 2024-04-06 18:38:54,455 - Train: 13.75% [679300/4942000] [137.5/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 18:40:11,149 - Train: 13.75% [679400/4942000] [137.5/1000.0] [batch_t 0.771 (0.767)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 18:41:27,867 - Train: 13.75% [679500/4942000] [137.5/1000.0] [batch_t 0.768 (0.767)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 18:42:44,523 - Train: 13.75% [679600/4942000] [137.5/1000.0] [batch_t 0.768 (0.766)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 18:44:01,044 - Train: 13.75% [679700/4942000] [137.5/1000.0] [batch_t 0.766 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 18:45:17,609 - Train: 13.76% [679800/4942000] [137.6/1000.0] [batch_t 0.773 (0.766)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-06 18:46:34,200 - Train: 13.76% [679900/4942000] [137.6/1000.0] [batch_t 0.750 (0.766)] [data_t 0.003] [optim_t 0.746] [lr 0.005000] 2024-04-06 18:47:50,663 - Train: 13.76% [680000/4942000] [137.6/1000.0] [batch_t 0.759 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-06 18:49:07,050 - Train: 13.76% [680100/4942000] [137.6/1000.0] [batch_t 0.753 (0.764)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-06 18:50:23,542 - Train: 13.76% [680200/4942000] [137.6/1000.0] [batch_t 0.758 (0.765)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-06 18:51:40,075 - Train: 13.77% [680300/4942000] [137.7/1000.0] [batch_t 0.753 (0.765)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-06 18:53:40,804 - Train: 13.77% [680400/4942000] [137.7/1000.0] [batch_t 0.769 (1.207)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 18:55:04,733 - Train: 13.77% [680500/4942000] [137.7/1000.0] [batch_t 0.756 (0.839)] [data_t 0.002] [optim_t 0.754] [lr 0.005000] 2024-04-06 18:56:21,727 - Train: 13.77% [680600/4942000] [137.7/1000.0] [batch_t 0.772 (0.770)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-06 18:57:55,067 - Train: 13.77% [680700/4942000] [137.7/1000.0] [batch_t 0.754 (0.933)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-06 18:59:15,645 - Train: 13.78% [680800/4942000] [137.8/1000.0] [batch_t 0.778 (0.806)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 19:01:07,401 - Train: 13.78% [680900/4942000] [137.8/1000.0] [batch_t 0.763 (1.117)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 19:03:52,688 - Train: 13.78% [681000/4942000] [137.8/1000.0] [batch_t 0.776 (1.653)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-06 19:05:39,593 - Train: 13.78% [681100/4942000] [137.8/1000.0] [batch_t 0.769 (1.069)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 19:07:42,044 - Train: 13.78% [681200/4942000] [137.8/1000.0] [batch_t 0.773 (1.224)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-06 19:09:41,991 - Train: 13.79% [681300/4942000] [137.9/1000.0] [batch_t 0.755 (1.199)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-06 19:11:03,819 - Train: 13.79% [681400/4942000] [137.9/1000.0] [batch_t 0.772 (0.818)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 19:12:20,244 - Train: 13.79% [681500/4942000] [137.9/1000.0] [batch_t 0.761 (0.764)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 19:13:36,642 - Train: 13.79% [681600/4942000] [137.9/1000.0] [batch_t 0.773 (0.764)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-06 19:14:53,190 - Train: 13.79% [681700/4942000] [137.9/1000.0] [batch_t 0.773 (0.765)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-06 19:16:09,674 - Train: 13.80% [681800/4942000] [138.0/1000.0] [batch_t 0.754 (0.765)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-06 19:17:26,188 - Train: 13.80% [681900/4942000] [138.0/1000.0] [batch_t 0.757 (0.765)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-06 19:18:39,529 - ==> Total time: 4 days, 1:21:18 Eta: 25 days, 8:07:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-06 19:18:47,036 - Train: 13.80% [682000/4942000] [138.0/1000.0] [batch_t 0.772 (0.950)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 19:20:03,635 - Train: 13.80% [682100/4942000] [138.0/1000.0] [batch_t 0.781 (0.766)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-06 19:21:20,221 - Train: 13.80% [682200/4942000] [138.0/1000.0] [batch_t 0.765 (0.766)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-06 19:22:36,780 - Train: 13.81% [682300/4942000] [138.1/1000.0] [batch_t 0.763 (0.766)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 19:23:53,145 - Train: 13.81% [682400/4942000] [138.1/1000.0] [batch_t 0.752 (0.764)] [data_t 0.003] [optim_t 0.749] [lr 0.005000] 2024-04-06 19:25:09,566 - Train: 13.81% [682500/4942000] [138.1/1000.0] [batch_t 0.773 (0.764)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-06 19:26:25,986 - Train: 13.81% [682600/4942000] [138.1/1000.0] [batch_t 0.754 (0.764)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-06 19:27:42,430 - Train: 13.81% [682700/4942000] [138.1/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 19:28:59,042 - Train: 13.82% [682800/4942000] [138.2/1000.0] [batch_t 0.754 (0.766)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-06 19:30:15,465 - Train: 13.82% [682900/4942000] [138.2/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 19:31:31,925 - Train: 13.82% [683000/4942000] [138.2/1000.0] [batch_t 0.778 (0.765)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-06 19:32:48,455 - Train: 13.82% [683100/4942000] [138.2/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 19:34:04,845 - Train: 13.82% [683200/4942000] [138.2/1000.0] [batch_t 0.776 (0.764)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-06 19:35:21,293 - Train: 13.83% [683300/4942000] [138.3/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 19:36:37,873 - Train: 13.83% [683400/4942000] [138.3/1000.0] [batch_t 0.768 (0.766)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 19:37:54,463 - Train: 13.83% [683500/4942000] [138.3/1000.0] [batch_t 0.750 (0.766)] [data_t 0.002] [optim_t 0.748] [lr 0.005000] 2024-04-06 19:39:10,956 - Train: 13.83% [683600/4942000] [138.3/1000.0] [batch_t 0.767 (0.765)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-06 19:40:27,613 - Train: 13.83% [683700/4942000] [138.3/1000.0] [batch_t 0.759 (0.766)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-06 19:41:43,993 - Train: 13.84% [683800/4942000] [138.4/1000.0] [batch_t 0.769 (0.764)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-06 19:43:00,506 - Train: 13.84% [683900/4942000] [138.4/1000.0] [batch_t 0.752 (0.765)] [data_t 0.003] [optim_t 0.749] [lr 0.005000] 2024-04-06 19:44:17,033 - Train: 13.84% [684000/4942000] [138.4/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 19:45:33,505 - Train: 13.84% [684100/4942000] [138.4/1000.0] [batch_t 0.771 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 19:46:51,414 - Train: 13.84% [684200/4942000] [138.4/1000.0] [batch_t 0.769 (0.779)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 19:48:07,859 - Train: 13.85% [684300/4942000] [138.5/1000.0] [batch_t 0.757 (0.764)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-06 19:49:24,397 - Train: 13.85% [684400/4942000] [138.5/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 19:50:40,846 - Train: 13.85% [684500/4942000] [138.5/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 19:51:57,162 - Train: 13.85% [684600/4942000] [138.5/1000.0] [batch_t 0.763 (0.763)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-06 19:53:13,624 - Train: 13.85% [684700/4942000] [138.5/1000.0] [batch_t 0.778 (0.765)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 19:54:30,147 - Train: 13.86% [684800/4942000] [138.6/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 19:55:46,588 - Train: 13.86% [684900/4942000] [138.6/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 19:57:03,033 - Train: 13.86% [685000/4942000] [138.6/1000.0] [batch_t 0.753 (0.764)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-06 19:58:19,604 - Train: 13.86% [685100/4942000] [138.6/1000.0] [batch_t 0.759 (0.766)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-06 19:59:36,055 - Train: 13.86% [685200/4942000] [138.6/1000.0] [batch_t 0.753 (0.764)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-06 20:00:52,609 - Train: 13.87% [685300/4942000] [138.7/1000.0] [batch_t 0.755 (0.765)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-06 20:02:09,182 - Train: 13.87% [685400/4942000] [138.7/1000.0] [batch_t 0.782 (0.766)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-06 20:03:25,574 - Train: 13.87% [685500/4942000] [138.7/1000.0] [batch_t 0.771 (0.764)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 20:04:42,123 - Train: 13.87% [685600/4942000] [138.7/1000.0] [batch_t 0.771 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 20:05:58,625 - Train: 13.87% [685700/4942000] [138.7/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 20:07:15,063 - Train: 13.88% [685800/4942000] [138.8/1000.0] [batch_t 0.768 (0.764)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 20:08:31,479 - Train: 13.88% [685900/4942000] [138.8/1000.0] [batch_t 0.757 (0.764)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-06 20:09:47,960 - Train: 13.88% [686000/4942000] [138.8/1000.0] [batch_t 0.762 (0.765)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-06 20:11:04,563 - Train: 13.88% [686100/4942000] [138.8/1000.0] [batch_t 0.770 (0.766)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-06 20:12:20,931 - Train: 13.89% [686200/4942000] [138.9/1000.0] [batch_t 0.771 (0.764)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-06 20:13:37,408 - Train: 13.89% [686300/4942000] [138.9/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-06 20:14:53,836 - Train: 13.89% [686400/4942000] [138.9/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 20:16:10,219 - Train: 13.89% [686500/4942000] [138.9/1000.0] [batch_t 0.749 (0.764)] [data_t 0.003] [optim_t 0.746] [lr 0.005000] 2024-04-06 20:17:26,725 - Train: 13.89% [686600/4942000] [138.9/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 20:18:43,233 - Train: 13.90% [686700/4942000] [139.0/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 20:19:59,684 - Train: 13.90% [686800/4942000] [139.0/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 20:21:16,183 - Train: 13.90% [686900/4942000] [139.0/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 20:21:45,232 - ==> Total time: 4 days, 2:24:24 Eta: 25 days, 9:33:20 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-06 20:22:34,938 - Train: 13.90% [687000/4942000] [139.0/1000.0] [batch_t 0.771 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 20:23:51,394 - Train: 13.90% [687100/4942000] [139.0/1000.0] [batch_t 0.757 (0.764)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-06 20:25:07,917 - Train: 13.91% [687200/4942000] [139.1/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 20:26:24,377 - Train: 13.91% [687300/4942000] [139.1/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 20:27:40,716 - Train: 13.91% [687400/4942000] [139.1/1000.0] [batch_t 0.766 (0.763)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-06 20:28:57,143 - Train: 13.91% [687500/4942000] [139.1/1000.0] [batch_t 0.762 (0.764)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-06 20:30:13,587 - Train: 13.91% [687600/4942000] [139.1/1000.0] [batch_t 0.768 (0.764)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 20:31:30,088 - Train: 13.92% [687700/4942000] [139.2/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 20:32:46,536 - Train: 13.92% [687800/4942000] [139.2/1000.0] [batch_t 0.771 (0.764)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 20:34:03,084 - Train: 13.92% [687900/4942000] [139.2/1000.0] [batch_t 0.766 (0.765)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-06 20:35:19,490 - Train: 13.92% [688000/4942000] [139.2/1000.0] [batch_t 0.767 (0.764)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 20:36:35,992 - Train: 13.92% [688100/4942000] [139.2/1000.0] [batch_t 0.764 (0.765)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-06 20:37:52,550 - Train: 13.93% [688200/4942000] [139.3/1000.0] [batch_t 0.767 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 20:39:09,032 - Train: 13.93% [688300/4942000] [139.3/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 20:40:25,490 - Train: 13.93% [688400/4942000] [139.3/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 20:41:41,932 - Train: 13.93% [688500/4942000] [139.3/1000.0] [batch_t 0.761 (0.764)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-06 20:42:58,378 - Train: 13.93% [688600/4942000] [139.3/1000.0] [batch_t 0.771 (0.764)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-06 20:44:14,898 - Train: 13.94% [688700/4942000] [139.4/1000.0] [batch_t 0.755 (0.765)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-06 20:45:31,430 - Train: 13.94% [688800/4942000] [139.4/1000.0] [batch_t 0.771 (0.765)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-06 20:46:47,807 - Train: 13.94% [688900/4942000] [139.4/1000.0] [batch_t 0.776 (0.764)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-06 20:48:04,240 - Train: 13.94% [689000/4942000] [139.4/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-06 20:49:20,696 - Train: 13.94% [689100/4942000] [139.4/1000.0] [batch_t 0.772 (0.764)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-06 20:50:37,275 - Train: 13.95% [689200/4942000] [139.5/1000.0] [batch_t 0.783 (0.766)] [data_t 0.002] [optim_t 0.781] [lr 0.005000] 2024-04-06 20:51:53,703 - Train: 13.95% [689300/4942000] [139.5/1000.0] [batch_t 0.766 (0.764)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-06 20:53:10,049 - Train: 13.95% [689400/4942000] [139.5/1000.0] [batch_t 0.779 (0.763)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-06 20:54:26,442 - Train: 13.95% [689500/4942000] [139.5/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-06 20:55:42,822 - Train: 13.95% [689600/4942000] [139.5/1000.0] [batch_t 0.764 (0.764)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 20:56:59,367 - Train: 13.96% [689700/4942000] [139.6/1000.0] [batch_t 0.773 (0.765)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-06 20:58:15,798 - Train: 13.96% [689800/4942000] [139.6/1000.0] [batch_t 0.759 (0.764)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-06 20:59:32,078 - Train: 13.96% [689900/4942000] [139.6/1000.0] [batch_t 0.749 (0.763)] [data_t 0.003] [optim_t 0.746] [lr 0.005000] 2024-04-06 21:00:48,579 - Train: 13.96% [690000/4942000] [139.6/1000.0] [batch_t 0.758 (0.765)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-06 21:02:04,889 - Train: 13.96% [690100/4942000] [139.6/1000.0] [batch_t 0.749 (0.763)] [data_t 0.002] [optim_t 0.747] [lr 0.005000] 2024-04-06 21:03:21,395 - Train: 13.97% [690200/4942000] [139.7/1000.0] [batch_t 0.762 (0.765)] [data_t 0.004] [optim_t 0.759] [lr 0.005000] 2024-04-06 21:04:37,878 - Train: 13.97% [690300/4942000] [139.7/1000.0] [batch_t 0.757 (0.765)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-06 21:05:54,508 - Train: 13.97% [690400/4942000] [139.7/1000.0] [batch_t 0.776 (0.766)] [data_t 0.002] [optim_t 0.773] [lr 0.005000] 2024-04-06 21:07:11,047 - Train: 13.97% [690500/4942000] [139.7/1000.0] [batch_t 0.773 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-06 21:08:27,559 - Train: 13.97% [690600/4942000] [139.7/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 21:09:44,026 - Train: 13.98% [690700/4942000] [139.8/1000.0] [batch_t 0.759 (0.765)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-06 21:11:00,525 - Train: 13.98% [690800/4942000] [139.8/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 21:12:17,066 - Train: 13.98% [690900/4942000] [139.8/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 21:13:33,408 - Train: 13.98% [691000/4942000] [139.8/1000.0] [batch_t 0.761 (0.763)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-06 21:14:49,809 - Train: 13.98% [691100/4942000] [139.8/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 21:16:06,307 - Train: 13.99% [691200/4942000] [139.9/1000.0] [batch_t 0.753 (0.765)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-06 21:17:22,734 - Train: 13.99% [691300/4942000] [139.9/1000.0] [batch_t 0.783 (0.764)] [data_t 0.003] [optim_t 0.780] [lr 0.005000] 2024-04-06 21:18:39,098 - Train: 13.99% [691400/4942000] [139.9/1000.0] [batch_t 0.776 (0.764)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-06 21:19:55,515 - Train: 13.99% [691500/4942000] [139.9/1000.0] [batch_t 0.770 (0.764)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-06 21:21:12,024 - Train: 13.99% [691600/4942000] [139.9/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 21:22:28,434 - Train: 14.00% [691700/4942000] [140.0/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 21:23:44,900 - Train: 14.00% [691800/4942000] [140.0/1000.0] [batch_t 0.765 (0.765)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-06 21:24:46,112 - ==> Total time: 4 days, 3:27:25 Eta: 25 days, 10:57:01 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-06 21:25:03,519 - Train: 14.00% [691900/4942000] [140.0/1000.0] [batch_t 0.765 (0.765)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-06 21:26:19,918 - Train: 14.00% [692000/4942000] [140.0/1000.0] [batch_t 0.768 (0.764)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 21:27:36,385 - Train: 14.00% [692100/4942000] [140.0/1000.0] [batch_t 0.771 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 21:28:52,892 - Train: 14.01% [692200/4942000] [140.1/1000.0] [batch_t 0.782 (0.765)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-06 21:30:09,400 - Train: 14.01% [692300/4942000] [140.1/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 21:31:25,795 - Train: 14.01% [692400/4942000] [140.1/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 21:32:42,334 - Train: 14.01% [692500/4942000] [140.1/1000.0] [batch_t 0.766 (0.765)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-06 21:33:58,720 - Train: 14.01% [692600/4942000] [140.1/1000.0] [batch_t 0.762 (0.764)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-06 21:35:15,126 - Train: 14.02% [692700/4942000] [140.2/1000.0] [batch_t 0.766 (0.764)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-06 21:36:31,625 - Train: 14.02% [692800/4942000] [140.2/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 21:37:48,186 - Train: 14.02% [692900/4942000] [140.2/1000.0] [batch_t 0.755 (0.766)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-06 21:39:04,675 - Train: 14.02% [693000/4942000] [140.2/1000.0] [batch_t 0.749 (0.765)] [data_t 0.002] [optim_t 0.747] [lr 0.005000] 2024-04-06 21:40:21,175 - Train: 14.02% [693100/4942000] [140.2/1000.0] [batch_t 0.758 (0.765)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-06 21:41:37,607 - Train: 14.03% [693200/4942000] [140.3/1000.0] [batch_t 0.765 (0.764)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-06 21:42:54,161 - Train: 14.03% [693300/4942000] [140.3/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-06 21:44:10,462 - Train: 14.03% [693400/4942000] [140.3/1000.0] [batch_t 0.754 (0.763)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-06 21:45:27,030 - Train: 14.03% [693500/4942000] [140.3/1000.0] [batch_t 0.758 (0.766)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-06 21:46:43,430 - Train: 14.03% [693600/4942000] [140.3/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 21:47:59,913 - Train: 14.04% [693700/4942000] [140.4/1000.0] [batch_t 0.771 (0.765)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-06 21:49:16,367 - Train: 14.04% [693800/4942000] [140.4/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 21:50:32,805 - Train: 14.04% [693900/4942000] [140.4/1000.0] [batch_t 0.769 (0.764)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 21:51:49,242 - Train: 14.04% [694000/4942000] [140.4/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 21:53:05,701 - Train: 14.04% [694100/4942000] [140.4/1000.0] [batch_t 0.769 (0.764)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-06 21:54:22,214 - Train: 14.05% [694200/4942000] [140.5/1000.0] [batch_t 0.771 (0.765)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-06 21:55:38,636 - Train: 14.05% [694300/4942000] [140.5/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 21:56:54,924 - Train: 14.05% [694400/4942000] [140.5/1000.0] [batch_t 0.759 (0.763)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-06 21:58:11,384 - Train: 14.05% [694500/4942000] [140.5/1000.0] [batch_t 0.750 (0.765)] [data_t 0.002] [optim_t 0.747] [lr 0.005000] 2024-04-06 21:59:27,907 - Train: 14.06% [694600/4942000] [140.6/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-06 22:00:44,484 - Train: 14.06% [694700/4942000] [140.6/1000.0] [batch_t 0.771 (0.766)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 22:02:00,894 - Train: 14.06% [694800/4942000] [140.6/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 22:03:17,213 - Train: 14.06% [694900/4942000] [140.6/1000.0] [batch_t 0.753 (0.763)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-06 22:04:33,632 - Train: 14.06% [695000/4942000] [140.6/1000.0] [batch_t 0.758 (0.764)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-06 22:05:50,102 - Train: 14.07% [695100/4942000] [140.7/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 22:07:06,541 - Train: 14.07% [695200/4942000] [140.7/1000.0] [batch_t 0.774 (0.764)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-06 22:08:22,950 - Train: 14.07% [695300/4942000] [140.7/1000.0] [batch_t 0.778 (0.764)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 22:09:39,642 - Train: 14.07% [695400/4942000] [140.7/1000.0] [batch_t 0.753 (0.767)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-06 22:10:56,070 - Train: 14.07% [695500/4942000] [140.7/1000.0] [batch_t 0.763 (0.764)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-06 22:12:12,473 - Train: 14.08% [695600/4942000] [140.8/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-06 22:13:28,938 - Train: 14.08% [695700/4942000] [140.8/1000.0] [batch_t 0.764 (0.765)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 22:14:45,198 - Train: 14.08% [695800/4942000] [140.8/1000.0] [batch_t 0.761 (0.763)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-06 22:16:01,599 - Train: 14.08% [695900/4942000] [140.8/1000.0] [batch_t 0.777 (0.764)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-06 22:17:18,089 - Train: 14.08% [696000/4942000] [140.8/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 22:18:34,599 - Train: 14.09% [696100/4942000] [140.9/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 22:19:51,176 - Train: 14.09% [696200/4942000] [140.9/1000.0] [batch_t 0.772 (0.766)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 22:21:07,631 - Train: 14.09% [696300/4942000] [140.9/1000.0] [batch_t 0.770 (0.764)] [data_t 0.004] [optim_t 0.766] [lr 0.005000] 2024-04-06 22:22:24,124 - Train: 14.09% [696400/4942000] [140.9/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 22:23:40,594 - Train: 14.09% [696500/4942000] [140.9/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-06 22:24:57,150 - Train: 14.10% [696600/4942000] [141.0/1000.0] [batch_t 0.762 (0.765)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-06 22:26:13,516 - Train: 14.10% [696700/4942000] [141.0/1000.0] [batch_t 0.759 (0.764)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-06 22:27:29,985 - Train: 14.10% [696800/4942000] [141.0/1000.0] [batch_t 0.759 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-06 22:27:46,788 - ==> Total time: 4 days, 4:30:25 Eta: 25 days, 12:18:35 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-06 22:28:48,276 - Train: 14.10% [696900/4942000] [141.0/1000.0] [batch_t 0.759 (0.763)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-06 22:30:04,702 - Train: 14.10% [697000/4942000] [141.0/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 22:31:21,066 - Train: 14.11% [697100/4942000] [141.1/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 22:32:37,551 - Train: 14.11% [697200/4942000] [141.1/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 22:33:54,014 - Train: 14.11% [697300/4942000] [141.1/1000.0] [batch_t 0.767 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 22:35:10,408 - Train: 14.11% [697400/4942000] [141.1/1000.0] [batch_t 0.773 (0.764)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-06 22:36:26,772 - Train: 14.11% [697500/4942000] [141.1/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-06 22:37:43,106 - Train: 14.12% [697600/4942000] [141.2/1000.0] [batch_t 0.763 (0.763)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 22:38:59,554 - Train: 14.12% [697700/4942000] [141.2/1000.0] [batch_t 0.767 (0.764)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-06 22:40:15,979 - Train: 14.12% [697800/4942000] [141.2/1000.0] [batch_t 0.773 (0.764)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 22:41:32,480 - Train: 14.12% [697900/4942000] [141.2/1000.0] [batch_t 0.764 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 22:42:48,868 - Train: 14.12% [698000/4942000] [141.2/1000.0] [batch_t 0.767 (0.763)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 22:44:05,259 - Train: 14.13% [698100/4942000] [141.3/1000.0] [batch_t 0.773 (0.764)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 22:45:21,717 - Train: 14.13% [698200/4942000] [141.3/1000.0] [batch_t 0.766 (0.764)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-06 22:46:38,155 - Train: 14.13% [698300/4942000] [141.3/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-06 22:47:54,593 - Train: 14.13% [698400/4942000] [141.3/1000.0] [batch_t 0.769 (0.764)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 22:49:11,152 - Train: 14.13% [698500/4942000] [141.3/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 22:50:27,517 - Train: 14.14% [698600/4942000] [141.4/1000.0] [batch_t 0.753 (0.764)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-06 22:51:43,973 - Train: 14.14% [698700/4942000] [141.4/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-06 22:53:00,211 - Train: 14.14% [698800/4942000] [141.4/1000.0] [batch_t 0.767 (0.762)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 22:54:16,613 - Train: 14.14% [698900/4942000] [141.4/1000.0] [batch_t 0.757 (0.764)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-06 22:55:33,072 - Train: 14.14% [699000/4942000] [141.4/1000.0] [batch_t 0.769 (0.764)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 22:56:49,529 - Train: 14.15% [699100/4942000] [141.5/1000.0] [batch_t 0.756 (0.764)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-06 22:58:06,143 - Train: 14.15% [699200/4942000] [141.5/1000.0] [batch_t 0.766 (0.766)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 22:59:22,548 - Train: 14.15% [699300/4942000] [141.5/1000.0] [batch_t 0.760 (0.764)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-06 23:00:38,977 - Train: 14.15% [699400/4942000] [141.5/1000.0] [batch_t 0.773 (0.764)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 23:01:55,571 - Train: 14.15% [699500/4942000] [141.5/1000.0] [batch_t 0.745 (0.766)] [data_t 0.003] [optim_t 0.742] [lr 0.005000] 2024-04-06 23:03:12,024 - Train: 14.16% [699600/4942000] [141.6/1000.0] [batch_t 0.785 (0.764)] [data_t 0.003] [optim_t 0.783] [lr 0.005000] 2024-04-06 23:04:28,673 - Train: 14.16% [699700/4942000] [141.6/1000.0] [batch_t 0.776 (0.766)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 23:05:45,116 - Train: 14.16% [699800/4942000] [141.6/1000.0] [batch_t 0.754 (0.764)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-06 23:07:01,451 - Train: 14.16% [699900/4942000] [141.6/1000.0] [batch_t 0.754 (0.763)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-06 23:08:17,973 - Train: 14.16% [700000/4942000] [141.6/1000.0] [batch_t 0.754 (0.765)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-06 23:09:34,495 - Train: 14.17% [700100/4942000] [141.7/1000.0] [batch_t 0.767 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 23:10:51,019 - Train: 14.17% [700200/4942000] [141.7/1000.0] [batch_t 0.758 (0.765)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-06 23:12:07,610 - Train: 14.17% [700300/4942000] [141.7/1000.0] [batch_t 0.746 (0.766)] [data_t 0.003] [optim_t 0.743] [lr 0.005000] 2024-04-06 23:13:24,057 - Train: 14.17% [700400/4942000] [141.7/1000.0] [batch_t 0.767 (0.764)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 23:14:40,646 - Train: 14.17% [700500/4942000] [141.7/1000.0] [batch_t 0.768 (0.766)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 23:15:57,172 - Train: 14.18% [700600/4942000] [141.8/1000.0] [batch_t 0.753 (0.765)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-06 23:17:13,722 - Train: 14.18% [700700/4942000] [141.8/1000.0] [batch_t 0.782 (0.765)] [data_t 0.002] [optim_t 0.779] [lr 0.005000] 2024-04-06 23:18:30,260 - Train: 14.18% [700800/4942000] [141.8/1000.0] [batch_t 0.764 (0.765)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-06 23:19:46,751 - Train: 14.18% [700900/4942000] [141.8/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 23:21:03,260 - Train: 14.18% [701000/4942000] [141.8/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 23:22:19,720 - Train: 14.19% [701100/4942000] [141.9/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 23:23:36,201 - Train: 14.19% [701200/4942000] [141.9/1000.0] [batch_t 0.759 (0.765)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-06 23:24:52,507 - Train: 14.19% [701300/4942000] [141.9/1000.0] [batch_t 0.763 (0.763)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-06 23:26:08,906 - Train: 14.19% [701400/4942000] [141.9/1000.0] [batch_t 0.751 (0.764)] [data_t 0.002] [optim_t 0.749] [lr 0.005000] 2024-04-06 23:27:25,428 - Train: 14.19% [701500/4942000] [141.9/1000.0] [batch_t 0.761 (0.765)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-06 23:28:41,906 - Train: 14.20% [701600/4942000] [142.0/1000.0] [batch_t 0.764 (0.765)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-06 23:29:58,415 - Train: 14.20% [701700/4942000] [142.0/1000.0] [batch_t 0.767 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 23:30:47,398 - ==> Total time: 4 days, 5:33:26 Eta: 25 days, 13:38:07 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-06 23:31:16,743 - Train: 14.20% [701800/4942000] [142.0/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 23:32:33,193 - Train: 14.20% [701900/4942000] [142.0/1000.0] [batch_t 0.757 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-06 23:33:49,752 - Train: 14.20% [702000/4942000] [142.0/1000.0] [batch_t 0.765 (0.766)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-06 23:35:06,125 - Train: 14.21% [702100/4942000] [142.1/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-06 23:36:22,693 - Train: 14.21% [702200/4942000] [142.1/1000.0] [batch_t 0.744 (0.766)] [data_t 0.002] [optim_t 0.742] [lr 0.005000] 2024-04-06 23:37:39,033 - Train: 14.21% [702300/4942000] [142.1/1000.0] [batch_t 0.767 (0.763)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 23:38:55,497 - Train: 14.21% [702400/4942000] [142.1/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-06 23:40:12,004 - Train: 14.21% [702500/4942000] [142.1/1000.0] [batch_t 0.765 (0.765)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-06 23:41:28,338 - Train: 14.22% [702600/4942000] [142.2/1000.0] [batch_t 0.773 (0.763)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 23:42:44,821 - Train: 14.22% [702700/4942000] [142.2/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 23:44:01,324 - Train: 14.22% [702800/4942000] [142.2/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 23:45:17,677 - Train: 14.22% [702900/4942000] [142.2/1000.0] [batch_t 0.771 (0.763)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-06 23:46:34,036 - Train: 14.23% [703000/4942000] [142.3/1000.0] [batch_t 0.773 (0.764)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-06 23:47:50,451 - Train: 14.23% [703100/4942000] [142.3/1000.0] [batch_t 0.754 (0.764)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-06 23:49:06,960 - Train: 14.23% [703200/4942000] [142.3/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 23:50:23,484 - Train: 14.23% [703300/4942000] [142.3/1000.0] [batch_t 0.759 (0.765)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-06 23:51:39,935 - Train: 14.23% [703400/4942000] [142.3/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 23:52:56,447 - Train: 14.24% [703500/4942000] [142.4/1000.0] [batch_t 0.773 (0.765)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-06 23:54:12,836 - Train: 14.24% [703600/4942000] [142.4/1000.0] [batch_t 0.769 (0.764)] [data_t 0.004] [optim_t 0.765] [lr 0.005000] 2024-04-06 23:55:29,329 - Train: 14.24% [703700/4942000] [142.4/1000.0] [batch_t 0.767 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 23:56:45,668 - Train: 14.24% [703800/4942000] [142.4/1000.0] [batch_t 0.758 (0.763)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-06 23:58:02,165 - Train: 14.24% [703900/4942000] [142.4/1000.0] [batch_t 0.754 (0.765)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-06 23:59:18,647 - Train: 14.25% [704000/4942000] [142.5/1000.0] [batch_t 0.767 (0.765)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-07 00:00:35,195 - Train: 14.25% [704100/4942000] [142.5/1000.0] [batch_t 0.767 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-07 00:01:51,683 - Train: 14.25% [704200/4942000] [142.5/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-07 00:03:08,234 - Train: 14.25% [704300/4942000] [142.5/1000.0] [batch_t 0.771 (0.765)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-07 00:04:24,670 - Train: 14.25% [704400/4942000] [142.5/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-07 00:05:41,179 - Train: 14.26% [704500/4942000] [142.6/1000.0] [batch_t 0.759 (0.765)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-07 00:06:57,698 - Train: 14.26% [704600/4942000] [142.6/1000.0] [batch_t 0.759 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 00:08:14,173 - Train: 14.26% [704700/4942000] [142.6/1000.0] [batch_t 0.771 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-07 00:09:30,847 - Train: 14.26% [704800/4942000] [142.6/1000.0] [batch_t 0.771 (0.767)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-07 00:10:47,373 - Train: 14.26% [704900/4942000] [142.6/1000.0] [batch_t 0.759 (0.765)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-07 00:12:03,714 - Train: 14.27% [705000/4942000] [142.7/1000.0] [batch_t 0.769 (0.763)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-07 00:13:20,231 - Train: 14.27% [705100/4942000] [142.7/1000.0] [batch_t 0.771 (0.765)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-07 00:14:36,727 - Train: 14.27% [705200/4942000] [142.7/1000.0] [batch_t 0.764 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-07 00:15:53,130 - Train: 14.27% [705300/4942000] [142.7/1000.0] [batch_t 0.754 (0.764)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-07 00:17:09,602 - Train: 14.27% [705400/4942000] [142.7/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-07 00:18:25,941 - Train: 14.28% [705500/4942000] [142.8/1000.0] [batch_t 0.772 (0.763)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-07 00:19:42,392 - Train: 14.28% [705600/4942000] [142.8/1000.0] [batch_t 0.752 (0.764)] [data_t 0.003] [optim_t 0.749] [lr 0.005000] 2024-04-07 00:20:58,795 - Train: 14.28% [705700/4942000] [142.8/1000.0] [batch_t 0.769 (0.764)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-07 00:22:15,347 - Train: 14.28% [705800/4942000] [142.8/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-07 00:23:31,900 - Train: 14.28% [705900/4942000] [142.8/1000.0] [batch_t 0.769 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-07 00:24:48,276 - Train: 14.29% [706000/4942000] [142.9/1000.0] [batch_t 0.755 (0.764)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-07 00:26:04,770 - Train: 14.29% [706100/4942000] [142.9/1000.0] [batch_t 0.764 (0.765)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-07 00:27:21,238 - Train: 14.29% [706200/4942000] [142.9/1000.0] [batch_t 0.755 (0.764)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-07 00:28:37,848 - Train: 14.29% [706300/4942000] [142.9/1000.0] [batch_t 0.752 (0.766)] [data_t 0.003] [optim_t 0.749] [lr 0.005000] 2024-04-07 00:29:54,371 - Train: 14.29% [706400/4942000] [142.9/1000.0] [batch_t 0.757 (0.765)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-07 00:31:10,838 - Train: 14.30% [706500/4942000] [143.0/1000.0] [batch_t 0.770 (0.765)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-07 00:32:27,273 - Train: 14.30% [706600/4942000] [143.0/1000.0] [batch_t 0.765 (0.764)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-07 00:33:43,746 - Train: 14.30% [706700/4942000] [143.0/1000.0] [batch_t 0.766 (0.765)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-07 00:33:48,360 - ==> Total time: 4 days, 6:36:27 Eta: 25 days, 14:55:42 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-07 00:35:02,451 - Train: 14.30% [706800/4942000] [143.0/1000.0] [batch_t 0.769 (0.766)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-07 00:36:18,943 - Train: 14.30% [706900/4942000] [143.0/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 00:37:35,430 - Train: 14.31% [707000/4942000] [143.1/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 00:38:52,015 - Train: 14.31% [707100/4942000] [143.1/1000.0] [batch_t 0.763 (0.766)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 00:40:08,429 - Train: 14.31% [707200/4942000] [143.1/1000.0] [batch_t 0.769 (0.764)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-07 00:41:24,876 - Train: 14.31% [707300/4942000] [143.1/1000.0] [batch_t 0.778 (0.764)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-07 00:42:41,381 - Train: 14.31% [707400/4942000] [143.1/1000.0] [batch_t 0.766 (0.765)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-07 00:43:57,885 - Train: 14.32% [707500/4942000] [143.2/1000.0] [batch_t 0.773 (0.765)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-07 00:45:14,398 - Train: 14.32% [707600/4942000] [143.2/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-07 00:46:30,861 - Train: 14.32% [707700/4942000] [143.2/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-07 00:47:47,257 - Train: 14.32% [707800/4942000] [143.2/1000.0] [batch_t 0.768 (0.764)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-07 00:49:03,891 - Train: 14.32% [707900/4942000] [143.2/1000.0] [batch_t 0.774 (0.766)] [data_t 0.002] [optim_t 0.772] [lr 0.005000] 2024-04-07 00:50:20,361 - Train: 14.33% [708000/4942000] [143.3/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 00:51:36,964 - Train: 14.33% [708100/4942000] [143.3/1000.0] [batch_t 0.759 (0.766)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 00:52:53,240 - Train: 14.33% [708200/4942000] [143.3/1000.0] [batch_t 0.759 (0.763)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 00:54:09,754 - Train: 14.33% [708300/4942000] [143.3/1000.0] [batch_t 0.773 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-07 00:55:26,277 - Train: 14.33% [708400/4942000] [143.3/1000.0] [batch_t 0.754 (0.765)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-07 00:56:42,901 - Train: 14.34% [708500/4942000] [143.4/1000.0] [batch_t 0.758 (0.766)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-07 00:57:59,351 - Train: 14.34% [708600/4942000] [143.4/1000.0] [batch_t 0.769 (0.764)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-07 00:59:15,802 - Train: 14.34% [708700/4942000] [143.4/1000.0] [batch_t 0.754 (0.764)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-07 01:00:32,192 - Train: 14.34% [708800/4942000] [143.4/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 01:01:48,563 - Train: 14.34% [708900/4942000] [143.4/1000.0] [batch_t 0.771 (0.764)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-07 01:03:05,024 - Train: 14.35% [709000/4942000] [143.5/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-07 01:04:21,569 - Train: 14.35% [709100/4942000] [143.5/1000.0] [batch_t 0.786 (0.765)] [data_t 0.002] [optim_t 0.784] [lr 0.005000] 2024-04-07 01:05:37,957 - Train: 14.35% [709200/4942000] [143.5/1000.0] [batch_t 0.771 (0.764)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-07 01:06:54,352 - Train: 14.35% [709300/4942000] [143.5/1000.0] [batch_t 0.753 (0.764)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-07 01:08:10,788 - Train: 14.35% [709400/4942000] [143.5/1000.0] [batch_t 0.757 (0.764)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-07 01:09:27,240 - Train: 14.36% [709500/4942000] [143.6/1000.0] [batch_t 0.754 (0.764)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-07 01:10:43,674 - Train: 14.36% [709600/4942000] [143.6/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 01:12:00,138 - Train: 14.36% [709700/4942000] [143.6/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-07 01:13:16,617 - Train: 14.36% [709800/4942000] [143.6/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-07 01:14:33,195 - Train: 14.36% [709900/4942000] [143.6/1000.0] [batch_t 0.757 (0.766)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-07 01:15:49,663 - Train: 14.37% [710000/4942000] [143.7/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-07 01:17:06,257 - Train: 14.37% [710100/4942000] [143.7/1000.0] [batch_t 0.763 (0.766)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 01:18:22,578 - Train: 14.37% [710200/4942000] [143.7/1000.0] [batch_t 0.755 (0.763)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-07 01:19:38,949 - Train: 14.37% [710300/4942000] [143.7/1000.0] [batch_t 0.752 (0.764)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-07 01:20:55,360 - Train: 14.37% [710400/4942000] [143.7/1000.0] [batch_t 0.761 (0.764)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-07 01:22:11,824 - Train: 14.38% [710500/4942000] [143.8/1000.0] [batch_t 0.767 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-07 01:23:28,284 - Train: 14.38% [710600/4942000] [143.8/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 01:24:44,778 - Train: 14.38% [710700/4942000] [143.8/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-07 01:26:01,233 - Train: 14.38% [710800/4942000] [143.8/1000.0] [batch_t 0.752 (0.764)] [data_t 0.003] [optim_t 0.749] [lr 0.005000] 2024-04-07 01:27:17,735 - Train: 14.38% [710900/4942000] [143.8/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-07 01:28:34,214 - Train: 14.39% [711000/4942000] [143.9/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-07 01:29:50,652 - Train: 14.39% [711100/4942000] [143.9/1000.0] [batch_t 0.771 (0.764)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-07 01:31:07,048 - Train: 14.39% [711200/4942000] [143.9/1000.0] [batch_t 0.777 (0.764)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-07 01:32:23,546 - Train: 14.39% [711300/4942000] [143.9/1000.0] [batch_t 0.782 (0.765)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-07 01:33:40,031 - Train: 14.39% [711400/4942000] [143.9/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-07 01:34:56,527 - Train: 14.40% [711500/4942000] [144.0/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-07 01:36:12,842 - Train: 14.40% [711600/4942000] [144.0/1000.0] [batch_t 0.768 (0.763)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 01:36:49,497 - ==> Total time: 4 days, 7:39:28 Eta: 25 days, 16:11:20 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-07 01:37:31,411 - Train: 14.40% [711700/4942000] [144.0/1000.0] [batch_t 0.777 (0.764)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-07 01:38:47,755 - Train: 14.40% [711800/4942000] [144.0/1000.0] [batch_t 0.748 (0.763)] [data_t 0.003] [optim_t 0.745] [lr 0.005000] 2024-04-07 01:40:04,199 - Train: 14.41% [711900/4942000] [144.1/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-07 01:41:20,725 - Train: 14.41% [712000/4942000] [144.1/1000.0] [batch_t 0.775 (0.765)] [data_t 0.002] [optim_t 0.773] [lr 0.005000] 2024-04-07 01:42:37,223 - Train: 14.41% [712100/4942000] [144.1/1000.0] [batch_t 0.754 (0.765)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-07 01:43:53,598 - Train: 14.41% [712200/4942000] [144.1/1000.0] [batch_t 0.772 (0.764)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-07 01:45:10,071 - Train: 14.41% [712300/4942000] [144.1/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 01:46:26,531 - Train: 14.42% [712400/4942000] [144.2/1000.0] [batch_t 0.758 (0.765)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-07 01:47:42,897 - Train: 14.42% [712500/4942000] [144.2/1000.0] [batch_t 0.766 (0.764)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-07 01:48:59,426 - Train: 14.42% [712600/4942000] [144.2/1000.0] [batch_t 0.758 (0.765)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-07 01:50:15,808 - Train: 14.42% [712700/4942000] [144.2/1000.0] [batch_t 0.760 (0.764)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-07 01:51:32,270 - Train: 14.42% [712800/4942000] [144.2/1000.0] [batch_t 0.769 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-07 01:52:48,736 - Train: 14.43% [712900/4942000] [144.3/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-07 01:54:05,180 - Train: 14.43% [713000/4942000] [144.3/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 01:55:21,712 - Train: 14.43% [713100/4942000] [144.3/1000.0] [batch_t 0.760 (0.765)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-07 01:56:38,183 - Train: 14.43% [713200/4942000] [144.3/1000.0] [batch_t 0.773 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-07 01:57:54,676 - Train: 14.43% [713300/4942000] [144.3/1000.0] [batch_t 0.761 (0.765)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-07 01:59:11,126 - Train: 14.44% [713400/4942000] [144.4/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 02:00:27,627 - Train: 14.44% [713500/4942000] [144.4/1000.0] [batch_t 0.770 (0.765)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-07 02:01:44,057 - Train: 14.44% [713600/4942000] [144.4/1000.0] [batch_t 0.781 (0.764)] [data_t 0.002] [optim_t 0.778] [lr 0.005000] 2024-04-07 02:03:00,440 - Train: 14.44% [713700/4942000] [144.4/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 02:04:16,807 - Train: 14.44% [713800/4942000] [144.4/1000.0] [batch_t 0.772 (0.764)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-07 02:05:33,305 - Train: 14.45% [713900/4942000] [144.5/1000.0] [batch_t 0.771 (0.765)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-07 02:06:49,792 - Train: 14.45% [714000/4942000] [144.5/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-07 02:08:06,213 - Train: 14.45% [714100/4942000] [144.5/1000.0] [batch_t 0.784 (0.764)] [data_t 0.003] [optim_t 0.782] [lr 0.005000] 2024-04-07 02:09:22,747 - Train: 14.45% [714200/4942000] [144.5/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 02:10:39,152 - Train: 14.45% [714300/4942000] [144.5/1000.0] [batch_t 0.772 (0.764)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-07 02:11:55,567 - Train: 14.46% [714400/4942000] [144.6/1000.0] [batch_t 0.770 (0.764)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-07 02:13:11,990 - Train: 14.46% [714500/4942000] [144.6/1000.0] [batch_t 0.759 (0.764)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-07 02:14:28,465 - Train: 14.46% [714600/4942000] [144.6/1000.0] [batch_t 0.759 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 02:15:44,969 - Train: 14.46% [714700/4942000] [144.6/1000.0] [batch_t 0.778 (0.765)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-07 02:17:01,460 - Train: 14.46% [714800/4942000] [144.6/1000.0] [batch_t 0.755 (0.765)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-07 02:18:18,061 - Train: 14.47% [714900/4942000] [144.7/1000.0] [batch_t 0.752 (0.766)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-07 02:19:34,549 - Train: 14.47% [715000/4942000] [144.7/1000.0] [batch_t 0.769 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-07 02:20:50,984 - Train: 14.47% [715100/4942000] [144.7/1000.0] [batch_t 0.769 (0.764)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-07 02:22:07,469 - Train: 14.47% [715200/4942000] [144.7/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 02:23:23,993 - Train: 14.47% [715300/4942000] [144.7/1000.0] [batch_t 0.769 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-07 02:24:40,508 - Train: 14.48% [715400/4942000] [144.8/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-07 02:25:57,027 - Train: 14.48% [715500/4942000] [144.8/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-07 02:27:13,371 - Train: 14.48% [715600/4942000] [144.8/1000.0] [batch_t 0.759 (0.763)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 02:28:29,759 - Train: 14.48% [715700/4942000] [144.8/1000.0] [batch_t 0.761 (0.764)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-07 02:29:46,270 - Train: 14.48% [715800/4942000] [144.8/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-07 02:31:02,793 - Train: 14.49% [715900/4942000] [144.9/1000.0] [batch_t 0.767 (0.765)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-07 02:32:19,157 - Train: 14.49% [716000/4942000] [144.9/1000.0] [batch_t 0.757 (0.764)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-07 02:33:35,518 - Train: 14.49% [716100/4942000] [144.9/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 02:34:51,818 - Train: 14.49% [716200/4942000] [144.9/1000.0] [batch_t 0.773 (0.763)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-07 02:36:08,163 - Train: 14.49% [716300/4942000] [144.9/1000.0] [batch_t 0.753 (0.763)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-07 02:37:24,708 - Train: 14.50% [716400/4942000] [145.0/1000.0] [batch_t 0.778 (0.765)] [data_t 0.002] [optim_t 0.776] [lr 0.005000] 2024-04-07 02:38:41,054 - Train: 14.50% [716500/4942000] [145.0/1000.0] [batch_t 0.776 (0.763)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-07 02:39:49,817 - ==> Total time: 4 days, 8:42:29 Eta: 25 days, 17:24:59 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-07 02:39:59,545 - Train: 14.50% [716600/4942000] [145.0/1000.0] [batch_t 0.768 (0.767)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-07 02:41:16,032 - Train: 14.50% [716700/4942000] [145.0/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-07 02:42:32,450 - Train: 14.50% [716800/4942000] [145.0/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-07 02:43:48,941 - Train: 14.51% [716900/4942000] [145.1/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 02:45:05,372 - Train: 14.51% [717000/4942000] [145.1/1000.0] [batch_t 0.771 (0.764)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-07 02:46:21,877 - Train: 14.51% [717100/4942000] [145.1/1000.0] [batch_t 0.759 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 02:47:38,254 - Train: 14.51% [717200/4942000] [145.1/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 02:48:54,764 - Train: 14.51% [717300/4942000] [145.1/1000.0] [batch_t 0.771 (0.765)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-07 02:50:11,287 - Train: 14.52% [717400/4942000] [145.2/1000.0] [batch_t 0.753 (0.765)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-07 02:51:27,659 - Train: 14.52% [717500/4942000] [145.2/1000.0] [batch_t 0.771 (0.764)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-07 02:52:44,159 - Train: 14.52% [717600/4942000] [145.2/1000.0] [batch_t 0.774 (0.765)] [data_t 0.002] [optim_t 0.772] [lr 0.005000] 2024-04-07 02:54:00,494 - Train: 14.52% [717700/4942000] [145.2/1000.0] [batch_t 0.762 (0.763)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-07 02:55:17,030 - Train: 14.52% [717800/4942000] [145.2/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-07 02:56:33,473 - Train: 14.53% [717900/4942000] [145.3/1000.0] [batch_t 0.772 (0.764)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-07 02:57:49,845 - Train: 14.53% [718000/4942000] [145.3/1000.0] [batch_t 0.754 (0.764)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-07 02:59:06,323 - Train: 14.53% [718100/4942000] [145.3/1000.0] [batch_t 0.775 (0.765)] [data_t 0.002] [optim_t 0.773] [lr 0.005000] 2024-04-07 03:00:22,723 - Train: 14.53% [718200/4942000] [145.3/1000.0] [batch_t 0.749 (0.764)] [data_t 0.003] [optim_t 0.746] [lr 0.005000] 2024-04-07 03:01:39,291 - Train: 14.53% [718300/4942000] [145.3/1000.0] [batch_t 0.772 (0.766)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-07 03:02:55,795 - Train: 14.54% [718400/4942000] [145.4/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 03:04:12,322 - Train: 14.54% [718500/4942000] [145.4/1000.0] [batch_t 0.757 (0.765)] [data_t 0.002] [optim_t 0.754] [lr 0.005000] 2024-04-07 03:05:28,774 - Train: 14.54% [718600/4942000] [145.4/1000.0] [batch_t 0.769 (0.764)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-07 03:06:45,217 - Train: 14.54% [718700/4942000] [145.4/1000.0] [batch_t 0.776 (0.764)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-07 03:08:01,677 - Train: 14.54% [718800/4942000] [145.4/1000.0] [batch_t 0.753 (0.765)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-07 03:09:18,129 - Train: 14.55% [718900/4942000] [145.5/1000.0] [batch_t 0.773 (0.764)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-07 03:10:34,697 - Train: 14.55% [719000/4942000] [145.5/1000.0] [batch_t 0.773 (0.766)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-07 03:11:51,152 - Train: 14.55% [719100/4942000] [145.5/1000.0] [batch_t 0.771 (0.764)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-07 03:13:07,610 - Train: 14.55% [719200/4942000] [145.5/1000.0] [batch_t 0.783 (0.764)] [data_t 0.002] [optim_t 0.781] [lr 0.005000] 2024-04-07 03:14:24,116 - Train: 14.55% [719300/4942000] [145.5/1000.0] [batch_t 0.766 (0.765)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-07 03:15:40,574 - Train: 14.56% [719400/4942000] [145.6/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 03:16:57,040 - Train: 14.56% [719500/4942000] [145.6/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-07 03:18:13,573 - Train: 14.56% [719600/4942000] [145.6/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-07 03:19:30,142 - Train: 14.56% [719700/4942000] [145.6/1000.0] [batch_t 0.772 (0.766)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-07 03:20:46,589 - Train: 14.56% [719800/4942000] [145.6/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 03:22:03,114 - Train: 14.57% [719900/4942000] [145.7/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 03:23:19,560 - Train: 14.57% [720000/4942000] [145.7/1000.0] [batch_t 0.770 (0.764)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-07 03:24:36,057 - Train: 14.57% [720100/4942000] [145.7/1000.0] [batch_t 0.758 (0.765)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-07 03:25:52,569 - Train: 14.57% [720200/4942000] [145.7/1000.0] [batch_t 0.777 (0.765)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-07 03:27:09,068 - Train: 14.58% [720300/4942000] [145.8/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-07 03:28:25,515 - Train: 14.58% [720400/4942000] [145.8/1000.0] [batch_t 0.759 (0.764)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-07 03:29:41,943 - Train: 14.58% [720500/4942000] [145.8/1000.0] [batch_t 0.773 (0.764)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-07 03:30:58,297 - Train: 14.58% [720600/4942000] [145.8/1000.0] [batch_t 0.768 (0.763)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 03:32:14,715 - Train: 14.58% [720700/4942000] [145.8/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 03:33:31,095 - Train: 14.59% [720800/4942000] [145.9/1000.0] [batch_t 0.771 (0.764)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-07 03:34:47,560 - Train: 14.59% [720900/4942000] [145.9/1000.0] [batch_t 0.777 (0.765)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-07 03:36:04,068 - Train: 14.59% [721000/4942000] [145.9/1000.0] [batch_t 0.774 (0.765)] [data_t 0.002] [optim_t 0.772] [lr 0.005000] 2024-04-07 03:37:20,457 - Train: 14.59% [721100/4942000] [145.9/1000.0] [batch_t 0.743 (0.764)] [data_t 0.003] [optim_t 0.740] [lr 0.005000] 2024-04-07 03:38:36,857 - Train: 14.59% [721200/4942000] [145.9/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 03:39:53,462 - Train: 14.60% [721300/4942000] [146.0/1000.0] [batch_t 0.767 (0.766)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 03:41:09,990 - Train: 14.60% [721400/4942000] [146.0/1000.0] [batch_t 0.767 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-07 03:42:26,448 - Train: 14.60% [721500/4942000] [146.0/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 03:42:50,891 - ==> Total time: 4 days, 9:45:30 Eta: 25 days, 18:36:50 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-07 03:43:44,782 - Train: 14.60% [721600/4942000] [146.0/1000.0] [batch_t 0.767 (0.763)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 03:45:01,244 - Train: 14.60% [721700/4942000] [146.0/1000.0] [batch_t 0.774 (0.765)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-07 03:46:17,623 - Train: 14.61% [721800/4942000] [146.1/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-07 03:47:34,059 - Train: 14.61% [721900/4942000] [146.1/1000.0] [batch_t 0.772 (0.764)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-07 03:48:50,403 - Train: 14.61% [722000/4942000] [146.1/1000.0] [batch_t 0.763 (0.763)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-07 03:50:06,952 - Train: 14.61% [722100/4942000] [146.1/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-07 03:51:23,414 - Train: 14.61% [722200/4942000] [146.1/1000.0] [batch_t 0.752 (0.765)] [data_t 0.003] [optim_t 0.749] [lr 0.005000] 2024-04-07 03:52:39,825 - Train: 14.62% [722300/4942000] [146.2/1000.0] [batch_t 0.748 (0.764)] [data_t 0.004] [optim_t 0.744] [lr 0.005000] 2024-04-07 03:53:56,349 - Train: 14.62% [722400/4942000] [146.2/1000.0] [batch_t 0.771 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-07 03:55:12,791 - Train: 14.62% [722500/4942000] [146.2/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 03:56:29,316 - Train: 14.62% [722600/4942000] [146.2/1000.0] [batch_t 0.776 (0.765)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-07 03:57:45,781 - Train: 14.62% [722700/4942000] [146.2/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 03:59:02,166 - Train: 14.63% [722800/4942000] [146.3/1000.0] [batch_t 0.743 (0.764)] [data_t 0.003] [optim_t 0.740] [lr 0.005000] 2024-04-07 04:00:18,744 - Train: 14.63% [722900/4942000] [146.3/1000.0] [batch_t 0.769 (0.766)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-07 04:01:35,282 - Train: 14.63% [723000/4942000] [146.3/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-07 04:02:51,855 - Train: 14.63% [723100/4942000] [146.3/1000.0] [batch_t 0.758 (0.766)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-07 04:04:08,218 - Train: 14.63% [723200/4942000] [146.3/1000.0] [batch_t 0.759 (0.764)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-07 04:05:24,622 - Train: 14.64% [723300/4942000] [146.4/1000.0] [batch_t 0.771 (0.764)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-07 04:06:41,020 - Train: 14.64% [723400/4942000] [146.4/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-07 04:07:57,522 - Train: 14.64% [723500/4942000] [146.4/1000.0] [batch_t 0.764 (0.765)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-07 04:09:14,041 - Train: 14.64% [723600/4942000] [146.4/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-07 04:10:30,554 - Train: 14.64% [723700/4942000] [146.4/1000.0] [batch_t 0.764 (0.765)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-07 04:11:47,103 - Train: 14.65% [723800/4942000] [146.5/1000.0] [batch_t 0.745 (0.765)] [data_t 0.003] [optim_t 0.742] [lr 0.005000] 2024-04-07 04:13:03,547 - Train: 14.65% [723900/4942000] [146.5/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 04:14:19,920 - Train: 14.65% [724000/4942000] [146.5/1000.0] [batch_t 0.752 (0.764)] [data_t 0.002] [optim_t 0.749] [lr 0.005000] 2024-04-07 04:15:36,555 - Train: 14.65% [724100/4942000] [146.5/1000.0] [batch_t 0.763 (0.766)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 04:16:53,130 - Train: 14.65% [724200/4942000] [146.5/1000.0] [batch_t 0.768 (0.766)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 04:18:09,694 - Train: 14.66% [724300/4942000] [146.6/1000.0] [batch_t 0.759 (0.766)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 04:19:26,238 - Train: 14.66% [724400/4942000] [146.6/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-07 04:20:42,716 - Train: 14.66% [724500/4942000] [146.6/1000.0] [batch_t 0.765 (0.765)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-07 04:21:59,181 - Train: 14.66% [724600/4942000] [146.6/1000.0] [batch_t 0.764 (0.765)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-07 04:23:15,601 - Train: 14.66% [724700/4942000] [146.6/1000.0] [batch_t 0.776 (0.764)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-07 04:24:31,993 - Train: 14.67% [724800/4942000] [146.7/1000.0] [batch_t 0.753 (0.764)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-07 04:25:48,397 - Train: 14.67% [724900/4942000] [146.7/1000.0] [batch_t 0.769 (0.764)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-07 04:27:04,969 - Train: 14.67% [725000/4942000] [146.7/1000.0] [batch_t 0.771 (0.766)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-07 04:28:21,403 - Train: 14.67% [725100/4942000] [146.7/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 04:29:37,842 - Train: 14.67% [725200/4942000] [146.7/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-07 04:30:54,280 - Train: 14.68% [725300/4942000] [146.8/1000.0] [batch_t 0.749 (0.764)] [data_t 0.003] [optim_t 0.746] [lr 0.005000] 2024-04-07 04:32:10,727 - Train: 14.68% [725400/4942000] [146.8/1000.0] [batch_t 0.753 (0.764)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-07 04:33:27,193 - Train: 14.68% [725500/4942000] [146.8/1000.0] [batch_t 0.754 (0.765)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-07 04:34:43,686 - Train: 14.68% [725600/4942000] [146.8/1000.0] [batch_t 0.769 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-07 04:36:00,110 - Train: 14.68% [725700/4942000] [146.8/1000.0] [batch_t 0.776 (0.764)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-07 04:37:16,672 - Train: 14.69% [725800/4942000] [146.9/1000.0] [batch_t 0.769 (0.766)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-07 04:38:33,188 - Train: 14.69% [725900/4942000] [146.9/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 04:39:49,587 - Train: 14.69% [726000/4942000] [146.9/1000.0] [batch_t 0.773 (0.764)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-07 04:41:06,066 - Train: 14.69% [726100/4942000] [146.9/1000.0] [batch_t 0.769 (0.765)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-07 04:42:22,597 - Train: 14.69% [726200/4942000] [146.9/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-07 04:43:39,099 - Train: 14.70% [726300/4942000] [147.0/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 04:44:55,651 - Train: 14.70% [726400/4942000] [147.0/1000.0] [batch_t 0.752 (0.765)] [data_t 0.003] [optim_t 0.749] [lr 0.005000] 2024-04-07 04:45:52,096 - ==> Total time: 4 days, 10:48:31 Eta: 25 days, 19:46:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-07 04:46:14,111 - Train: 14.70% [726500/4942000] [147.0/1000.0] [batch_t 0.753 (0.764)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-07 04:47:30,611 - Train: 14.70% [726600/4942000] [147.0/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 04:48:47,065 - Train: 14.70% [726700/4942000] [147.0/1000.0] [batch_t 0.763 (0.764)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-07 04:50:03,735 - Train: 14.71% [726800/4942000] [147.1/1000.0] [batch_t 0.773 (0.767)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-07 04:51:20,067 - Train: 14.71% [726900/4942000] [147.1/1000.0] [batch_t 0.768 (0.763)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 04:52:36,422 - Train: 14.71% [727000/4942000] [147.1/1000.0] [batch_t 0.758 (0.763)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-07 04:53:52,992 - Train: 14.71% [727100/4942000] [147.1/1000.0] [batch_t 0.757 (0.766)] [data_t 0.002] [optim_t 0.754] [lr 0.005000] 2024-04-07 04:55:09,474 - Train: 14.71% [727200/4942000] [147.1/1000.0] [batch_t 0.777 (0.765)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-07 04:56:25,917 - Train: 14.72% [727300/4942000] [147.2/1000.0] [batch_t 0.754 (0.764)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-07 04:57:42,382 - Train: 14.72% [727400/4942000] [147.2/1000.0] [batch_t 0.771 (0.765)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-07 04:58:58,774 - Train: 14.72% [727500/4942000] [147.2/1000.0] [batch_t 0.761 (0.764)] [data_t 0.002] [optim_t 0.758] [lr 0.005000] 2024-04-07 05:00:15,287 - Train: 14.72% [727600/4942000] [147.2/1000.0] [batch_t 0.757 (0.765)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-07 05:01:31,794 - Train: 14.72% [727700/4942000] [147.2/1000.0] [batch_t 0.759 (0.765)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-07 05:02:48,226 - Train: 14.73% [727800/4942000] [147.3/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 05:04:04,679 - Train: 14.73% [727900/4942000] [147.3/1000.0] [batch_t 0.764 (0.764)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-07 05:05:21,290 - Train: 14.73% [728000/4942000] [147.3/1000.0] [batch_t 0.769 (0.766)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-07 05:06:37,734 - Train: 14.73% [728100/4942000] [147.3/1000.0] [batch_t 0.771 (0.764)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-07 05:07:54,113 - Train: 14.73% [728200/4942000] [147.3/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-07 05:09:10,672 - Train: 14.74% [728300/4942000] [147.4/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-07 05:10:27,239 - Train: 14.74% [728400/4942000] [147.4/1000.0] [batch_t 0.772 (0.766)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-07 05:11:43,789 - Train: 14.74% [728500/4942000] [147.4/1000.0] [batch_t 0.766 (0.765)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-07 05:13:00,203 - Train: 14.74% [728600/4942000] [147.4/1000.0] [batch_t 0.744 (0.764)] [data_t 0.003] [optim_t 0.741] [lr 0.005000] 2024-04-07 05:14:16,722 - Train: 14.75% [728700/4942000] [147.5/1000.0] [batch_t 0.760 (0.765)] [data_t 0.002] [optim_t 0.758] [lr 0.005000] 2024-04-07 05:15:33,255 - Train: 14.75% [728800/4942000] [147.5/1000.0] [batch_t 0.777 (0.765)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-07 05:16:49,833 - Train: 14.75% [728900/4942000] [147.5/1000.0] [batch_t 0.758 (0.766)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-07 05:18:06,253 - Train: 14.75% [729000/4942000] [147.5/1000.0] [batch_t 0.775 (0.764)] [data_t 0.003] [optim_t 0.772] [lr 0.005000] 2024-04-07 05:19:22,647 - Train: 14.75% [729100/4942000] [147.5/1000.0] [batch_t 0.773 (0.764)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-07 05:20:39,066 - Train: 14.76% [729200/4942000] [147.6/1000.0] [batch_t 0.764 (0.764)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-07 05:21:55,549 - Train: 14.76% [729300/4942000] [147.6/1000.0] [batch_t 0.774 (0.765)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-07 05:23:12,067 - Train: 14.76% [729400/4942000] [147.6/1000.0] [batch_t 0.767 (0.765)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-07 05:24:28,659 - Train: 14.76% [729500/4942000] [147.6/1000.0] [batch_t 0.763 (0.766)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 05:25:45,163 - Train: 14.76% [729600/4942000] [147.6/1000.0] [batch_t 0.769 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-07 05:27:01,568 - Train: 14.77% [729700/4942000] [147.7/1000.0] [batch_t 0.749 (0.764)] [data_t 0.003] [optim_t 0.747] [lr 0.005000] 2024-04-07 05:28:18,006 - Train: 14.77% [729800/4942000] [147.7/1000.0] [batch_t 0.760 (0.764)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-07 05:29:34,391 - Train: 14.77% [729900/4942000] [147.7/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-07 05:30:50,834 - Train: 14.77% [730000/4942000] [147.7/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-07 05:32:07,221 - Train: 14.77% [730100/4942000] [147.7/1000.0] [batch_t 0.779 (0.764)] [data_t 0.002] [optim_t 0.776] [lr 0.005000] 2024-04-07 05:33:23,778 - Train: 14.78% [730200/4942000] [147.8/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 05:34:40,348 - Train: 14.78% [730300/4942000] [147.8/1000.0] [batch_t 0.766 (0.766)] [data_t 0.002] [optim_t 0.763] [lr 0.005000] 2024-04-07 05:35:56,783 - Train: 14.78% [730400/4942000] [147.8/1000.0] [batch_t 0.771 (0.764)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-07 05:37:13,354 - Train: 14.78% [730500/4942000] [147.8/1000.0] [batch_t 0.767 (0.766)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 05:38:29,820 - Train: 14.78% [730600/4942000] [147.8/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-07 05:39:46,350 - Train: 14.79% [730700/4942000] [147.9/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-07 05:41:02,780 - Train: 14.79% [730800/4942000] [147.9/1000.0] [batch_t 0.766 (0.764)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-07 05:42:19,341 - Train: 14.79% [730900/4942000] [147.9/1000.0] [batch_t 0.767 (0.766)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 05:43:35,736 - Train: 14.79% [731000/4942000] [147.9/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-07 05:44:52,222 - Train: 14.79% [731100/4942000] [147.9/1000.0] [batch_t 0.766 (0.765)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-07 05:46:08,823 - Train: 14.80% [731200/4942000] [148.0/1000.0] [batch_t 0.763 (0.766)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-07 05:47:25,309 - Train: 14.80% [731300/4942000] [148.0/1000.0] [batch_t 0.759 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 05:48:41,715 - Train: 14.80% [731400/4942000] [148.0/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 05:48:53,945 - ==> Total time: 4 days, 11:51:33 Eta: 25 days, 20:55:09 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-07 05:49:59,928 - Train: 14.80% [731500/4942000] [148.0/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 05:51:16,404 - Train: 14.80% [731600/4942000] [148.0/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-07 05:52:32,925 - Train: 14.81% [731700/4942000] [148.1/1000.0] [batch_t 0.773 (0.765)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-07 05:53:49,235 - Train: 14.81% [731800/4942000] [148.1/1000.0] [batch_t 0.761 (0.763)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-07 05:55:05,704 - Train: 14.81% [731900/4942000] [148.1/1000.0] [batch_t 0.758 (0.765)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-07 05:56:22,187 - Train: 14.81% [732000/4942000] [148.1/1000.0] [batch_t 0.770 (0.765)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-07 05:57:38,697 - Train: 14.81% [732100/4942000] [148.1/1000.0] [batch_t 0.764 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-07 05:58:55,090 - Train: 14.82% [732200/4942000] [148.2/1000.0] [batch_t 0.757 (0.764)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-07 06:00:11,283 - Train: 14.82% [732300/4942000] [148.2/1000.0] [batch_t 0.739 (0.762)] [data_t 0.003] [optim_t 0.736] [lr 0.005000] 2024-04-07 06:01:27,691 - Train: 14.82% [732400/4942000] [148.2/1000.0] [batch_t 0.763 (0.764)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-07 06:02:44,092 - Train: 14.82% [732500/4942000] [148.2/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 06:04:00,548 - Train: 14.82% [732600/4942000] [148.2/1000.0] [batch_t 0.769 (0.764)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-07 06:05:17,160 - Train: 14.83% [732700/4942000] [148.3/1000.0] [batch_t 0.767 (0.766)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 06:06:33,653 - Train: 14.83% [732800/4942000] [148.3/1000.0] [batch_t 0.771 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-07 06:07:50,183 - Train: 14.83% [732900/4942000] [148.3/1000.0] [batch_t 0.778 (0.765)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-07 06:09:06,632 - Train: 14.83% [733000/4942000] [148.3/1000.0] [batch_t 0.769 (0.764)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-07 06:10:23,074 - Train: 14.83% [733100/4942000] [148.3/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-07 06:11:39,561 - Train: 14.84% [733200/4942000] [148.4/1000.0] [batch_t 0.767 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-07 06:12:56,057 - Train: 14.84% [733300/4942000] [148.4/1000.0] [batch_t 0.778 (0.765)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-07 06:14:12,418 - Train: 14.84% [733400/4942000] [148.4/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 06:15:29,007 - Train: 14.84% [733500/4942000] [148.4/1000.0] [batch_t 0.764 (0.766)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-07 06:16:45,434 - Train: 14.84% [733600/4942000] [148.4/1000.0] [batch_t 0.772 (0.764)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-07 06:18:01,948 - Train: 14.85% [733700/4942000] [148.5/1000.0] [batch_t 0.776 (0.765)] [data_t 0.002] [optim_t 0.773] [lr 0.005000] 2024-04-07 06:19:18,405 - Train: 14.85% [733800/4942000] [148.5/1000.0] [batch_t 0.772 (0.764)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-07 06:20:34,762 - Train: 14.85% [733900/4942000] [148.5/1000.0] [batch_t 0.767 (0.763)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-07 06:21:51,245 - Train: 14.85% [734000/4942000] [148.5/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-07 06:23:07,661 - Train: 14.85% [734100/4942000] [148.5/1000.0] [batch_t 0.752 (0.764)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-07 06:24:24,135 - Train: 14.86% [734200/4942000] [148.6/1000.0] [batch_t 0.764 (0.765)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-07 06:25:40,500 - Train: 14.86% [734300/4942000] [148.6/1000.0] [batch_t 0.753 (0.764)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-07 06:26:56,970 - Train: 14.86% [734400/4942000] [148.6/1000.0] [batch_t 0.753 (0.765)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-07 06:28:13,428 - Train: 14.86% [734500/4942000] [148.6/1000.0] [batch_t 0.766 (0.764)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-07 06:29:29,951 - Train: 14.86% [734600/4942000] [148.6/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 06:30:46,377 - Train: 14.87% [734700/4942000] [148.7/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 06:32:02,845 - Train: 14.87% [734800/4942000] [148.7/1000.0] [batch_t 0.769 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-07 06:33:19,232 - Train: 14.87% [734900/4942000] [148.7/1000.0] [batch_t 0.753 (0.764)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-07 06:34:35,721 - Train: 14.87% [735000/4942000] [148.7/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-07 06:35:52,199 - Train: 14.87% [735100/4942000] [148.7/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 06:37:08,667 - Train: 14.88% [735200/4942000] [148.8/1000.0] [batch_t 0.771 (0.765)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-07 06:38:25,099 - Train: 14.88% [735300/4942000] [148.8/1000.0] [batch_t 0.773 (0.764)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-07 06:39:41,510 - Train: 14.88% [735400/4942000] [148.8/1000.0] [batch_t 0.759 (0.764)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-07 06:40:58,031 - Train: 14.88% [735500/4942000] [148.8/1000.0] [batch_t 0.764 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-07 06:42:14,631 - Train: 14.88% [735600/4942000] [148.8/1000.0] [batch_t 0.768 (0.766)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 06:43:31,177 - Train: 14.89% [735700/4942000] [148.9/1000.0] [batch_t 0.774 (0.765)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-07 06:44:47,612 - Train: 14.89% [735800/4942000] [148.9/1000.0] [batch_t 0.775 (0.764)] [data_t 0.002] [optim_t 0.773] [lr 0.005000] 2024-04-07 06:46:04,104 - Train: 14.89% [735900/4942000] [148.9/1000.0] [batch_t 0.747 (0.765)] [data_t 0.003] [optim_t 0.744] [lr 0.005000] 2024-04-07 06:47:20,545 - Train: 14.89% [736000/4942000] [148.9/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 06:48:37,015 - Train: 14.89% [736100/4942000] [148.9/1000.0] [batch_t 0.769 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-07 06:49:53,534 - Train: 14.90% [736200/4942000] [149.0/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-07 06:51:10,065 - Train: 14.90% [736300/4942000] [149.0/1000.0] [batch_t 0.771 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-07 06:51:54,346 - ==> Total time: 4 days, 12:54:33 Eta: 25 days, 22:01:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-07 06:52:28,818 - Train: 14.90% [736400/4942000] [149.0/1000.0] [batch_t 0.757 (0.763)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-07 06:53:45,230 - Train: 14.90% [736500/4942000] [149.0/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 06:55:01,727 - Train: 14.90% [736600/4942000] [149.0/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 06:56:18,228 - Train: 14.91% [736700/4942000] [149.1/1000.0] [batch_t 0.774 (0.765)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-07 06:57:34,554 - Train: 14.91% [736800/4942000] [149.1/1000.0] [batch_t 0.760 (0.763)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-07 06:58:51,000 - Train: 14.91% [736900/4942000] [149.1/1000.0] [batch_t 0.761 (0.764)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-07 07:00:07,505 - Train: 14.91% [737000/4942000] [149.1/1000.0] [batch_t 0.744 (0.765)] [data_t 0.003] [optim_t 0.741] [lr 0.005000] 2024-04-07 07:01:24,092 - Train: 14.92% [737100/4942000] [149.2/1000.0] [batch_t 0.767 (0.766)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 07:02:40,464 - Train: 14.92% [737200/4942000] [149.2/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 07:03:56,816 - Train: 14.92% [737300/4942000] [149.2/1000.0] [batch_t 0.769 (0.763)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-07 07:05:13,307 - Train: 14.92% [737400/4942000] [149.2/1000.0] [batch_t 0.764 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-07 07:06:29,713 - Train: 14.92% [737500/4942000] [149.2/1000.0] [batch_t 0.767 (0.764)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-07 07:07:46,184 - Train: 14.93% [737600/4942000] [149.3/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-07 07:09:02,694 - Train: 14.93% [737700/4942000] [149.3/1000.0] [batch_t 0.767 (0.765)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-07 07:10:19,200 - Train: 14.93% [737800/4942000] [149.3/1000.0] [batch_t 0.782 (0.765)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-07 07:11:35,625 - Train: 14.93% [737900/4942000] [149.3/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 07:12:52,191 - Train: 14.93% [738000/4942000] [149.3/1000.0] [batch_t 0.776 (0.766)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-07 07:14:08,706 - Train: 14.94% [738100/4942000] [149.4/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-07 07:15:25,068 - Train: 14.94% [738200/4942000] [149.4/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-07 07:16:41,487 - Train: 14.94% [738300/4942000] [149.4/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-07 07:17:57,964 - Train: 14.94% [738400/4942000] [149.4/1000.0] [batch_t 0.771 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-07 07:19:14,450 - Train: 14.94% [738500/4942000] [149.4/1000.0] [batch_t 0.747 (0.765)] [data_t 0.002] [optim_t 0.745] [lr 0.005000] 2024-04-07 07:20:30,855 - Train: 14.95% [738600/4942000] [149.5/1000.0] [batch_t 0.773 (0.764)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-07 07:21:47,319 - Train: 14.95% [738700/4942000] [149.5/1000.0] [batch_t 0.764 (0.765)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-07 07:23:03,783 - Train: 14.95% [738800/4942000] [149.5/1000.0] [batch_t 0.762 (0.765)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-07 07:24:20,315 - Train: 14.95% [738900/4942000] [149.5/1000.0] [batch_t 0.758 (0.765)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-07 07:25:36,707 - Train: 14.95% [739000/4942000] [149.5/1000.0] [batch_t 0.768 (0.764)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-07 07:26:53,284 - Train: 14.96% [739100/4942000] [149.6/1000.0] [batch_t 0.772 (0.766)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-07 07:28:09,767 - Train: 14.96% [739200/4942000] [149.6/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-07 07:29:26,271 - Train: 14.96% [739300/4942000] [149.6/1000.0] [batch_t 0.773 (0.765)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-07 07:30:42,773 - Train: 14.96% [739400/4942000] [149.6/1000.0] [batch_t 0.757 (0.765)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-07 07:31:59,220 - Train: 14.96% [739500/4942000] [149.6/1000.0] [batch_t 0.777 (0.764)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-07 07:33:15,677 - Train: 14.97% [739600/4942000] [149.7/1000.0] [batch_t 0.772 (0.764)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-07 07:34:32,232 - Train: 14.97% [739700/4942000] [149.7/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-07 07:35:48,717 - Train: 14.97% [739800/4942000] [149.7/1000.0] [batch_t 0.759 (0.765)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-07 07:37:05,107 - Train: 14.97% [739900/4942000] [149.7/1000.0] [batch_t 0.765 (0.764)] [data_t 0.002] [optim_t 0.763] [lr 0.005000] 2024-04-07 07:38:21,628 - Train: 14.97% [740000/4942000] [149.7/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 07:39:37,993 - Train: 14.98% [740100/4942000] [149.8/1000.0] [batch_t 0.766 (0.764)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-07 07:40:54,358 - Train: 14.98% [740200/4942000] [149.8/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-07 07:42:10,781 - Train: 14.98% [740300/4942000] [149.8/1000.0] [batch_t 0.773 (0.764)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-07 07:43:27,291 - Train: 14.98% [740400/4942000] [149.8/1000.0] [batch_t 0.772 (0.765)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-07 07:44:43,735 - Train: 14.98% [740500/4942000] [149.8/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-07 07:46:00,091 - Train: 14.99% [740600/4942000] [149.9/1000.0] [batch_t 0.758 (0.763)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 07:47:16,582 - Train: 14.99% [740700/4942000] [149.9/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-07 07:48:33,065 - Train: 14.99% [740800/4942000] [149.9/1000.0] [batch_t 0.755 (0.765)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-07 07:49:49,512 - Train: 14.99% [740900/4942000] [149.9/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-07 07:51:05,963 - Train: 14.99% [741000/4942000] [149.9/1000.0] [batch_t 0.758 (0.764)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-07 07:52:22,498 - Train: 15.00% [741100/4942000] [150.0/1000.0] [batch_t 0.752 (0.765)] [data_t 0.003] [optim_t 0.749] [lr 0.005000] 2024-04-07 07:53:39,026 - Train: 15.00% [741200/4942000] [150.0/1000.0] [batch_t 0.762 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 07:54:55,522 - Train: 15.00% [741300/4942000] [150.0/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 07:55:10,829 - Test: 16.13% [50/310] [batch_t 0.280 (0.288)] 2024-04-07 07:55:24,917 - Test: 32.26% [100/310] [batch_t 0.283 (0.285)] 2024-04-07 07:55:39,007 - Test: 48.39% [150/310] [batch_t 0.279 (0.284)] 2024-04-07 07:55:53,112 - Test: 64.52% [200/310] [batch_t 0.287 (0.283)] 2024-04-07 07:56:07,300 - Test: 80.65% [250/310] [batch_t 0.287 (0.283)] 2024-04-07 07:56:21,319 - Test: 96.77% [300/310] [batch_t 0.281 (0.283)] 2024-04-07 07:56:23,994 - Test: 100.00% [310/310] [batch_t 0.152 (0.282)] 2024-04-07 08:22:57,092 - ==> Metric Time for coco : 0.004 (mAUROC_sp_max) 0.002 (mAP_sp_max) 0.001 (mF1_max_sp_max) 358.078 (mAUROC_px) 283.320 (mAP_px) 33.950 (mF1_max_px) 838.758 (mAUPRO_px) 11.811 (mF1_px_0.2_0.8_0.1) 11.831 (mAcc_px_0.2_0.8_0.1) 11.781 (mIoU_px_0.2_0.8_0.1) 34.153 (mIoU_max_px) 2024-04-07 08:22:57,596 - | Name | mAUROC_sp_max | mAUROC_sp_max (Max) | mAP_sp_max | mAP_sp_max (Max) | mF1_max_sp_max | mF1_max_sp_max (Max) | mAUROC_px | mAUROC_px (Max) | mAP_px | mAP_px (Max) | mF1_max_px | mF1_max_px (Max) | mAUPRO_px | mAUPRO_px (Max) | mF1_px_0.2_0.8_0.1 | mF1_px_0.2_0.8_0.1 (Max) | mAcc_px_0.2_0.8_0.1 | mAcc_px_0.2_0.8_0.1 (Max) | mIoU_px_0.2_0.8_0.1 | mIoU_px_0.2_0.8_0.1 (Max) | mIoU_max_px | mIoU_max_px (Max) | |:------:|:---------------:|:---------------------:|:------------:|:------------------:|:----------------:|:----------------------:|:-----------:|:------------------:|:--------:|:------------------:|:------------:|:------------------:|:-----------:|:------------------:|:--------------------:|:--------------------------:|:---------------------:|:---------------------------:|:---------------------:|:---------------------------:|:-------------:|:-------------------:| | coco | 65.506 | 66.882 (50 epoch) | 45.534 | 46.681 (100 epoch) | 53.532 | 54.576 (50 epoch) | 71.808 | 71.808 (150 epoch) | 14.562 | 14.562 (150 epoch) | 22.065 | 22.065 (150 epoch) | 44.226 | 44.441 (50 epoch) | 10.780 | 11.792 (50 epoch) | 39.538 | 44.665 (50 epoch) | 5.888 | 6.432 (100 epoch) | 12.400 | 12.400 (150 epoch) | 2024-04-07 08:22:58,188 - ==> Total time: 4 days, 14:25:37 Eta: 26 days, 1:45:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-07 08:24:17,232 - Train: 15.00% [741400/4942000] [150.0/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-07 08:25:33,585 - Train: 15.00% [741500/4942000] [150.0/1000.0] [batch_t 0.759 (0.763)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 08:26:50,023 - Train: 15.01% [741600/4942000] [150.1/1000.0] [batch_t 0.773 (0.764)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-07 08:28:06,224 - Train: 15.01% [741700/4942000] [150.1/1000.0] [batch_t 0.759 (0.762)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 08:29:22,630 - Train: 15.01% [741800/4942000] [150.1/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 08:30:38,966 - Train: 15.01% [741900/4942000] [150.1/1000.0] [batch_t 0.755 (0.763)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-07 08:31:55,221 - Train: 15.01% [742000/4942000] [150.1/1000.0] [batch_t 0.759 (0.762)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 08:33:11,505 - Train: 15.02% [742100/4942000] [150.2/1000.0] [batch_t 0.769 (0.763)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-07 08:34:27,856 - Train: 15.02% [742200/4942000] [150.2/1000.0] [batch_t 0.749 (0.763)] [data_t 0.002] [optim_t 0.746] [lr 0.005000] 2024-04-07 08:35:44,200 - Train: 15.02% [742300/4942000] [150.2/1000.0] [batch_t 0.776 (0.763)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-07 08:37:00,592 - Train: 15.02% [742400/4942000] [150.2/1000.0] [batch_t 0.759 (0.764)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-07 08:38:16,931 - Train: 15.02% [742500/4942000] [150.2/1000.0] [batch_t 0.757 (0.763)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-07 08:39:33,321 - Train: 15.03% [742600/4942000] [150.3/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-07 08:40:49,469 - Train: 15.03% [742700/4942000] [150.3/1000.0] [batch_t 0.773 (0.761)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-07 08:42:05,779 - Train: 15.03% [742800/4942000] [150.3/1000.0] [batch_t 0.767 (0.763)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 08:43:22,120 - Train: 15.03% [742900/4942000] [150.3/1000.0] [batch_t 0.769 (0.763)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-07 08:44:38,428 - Train: 15.03% [743000/4942000] [150.3/1000.0] [batch_t 0.763 (0.763)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 08:45:54,788 - Train: 15.04% [743100/4942000] [150.4/1000.0] [batch_t 0.766 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 08:47:11,148 - Train: 15.04% [743200/4942000] [150.4/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-07 08:48:27,457 - Train: 15.04% [743300/4942000] [150.4/1000.0] [batch_t 0.777 (0.763)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-07 08:49:43,828 - Train: 15.04% [743400/4942000] [150.4/1000.0] [batch_t 0.767 (0.764)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-07 08:51:00,172 - Train: 15.04% [743500/4942000] [150.4/1000.0] [batch_t 0.753 (0.763)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-07 08:52:16,557 - Train: 15.05% [743600/4942000] [150.5/1000.0] [batch_t 0.759 (0.764)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-07 08:53:32,916 - Train: 15.05% [743700/4942000] [150.5/1000.0] [batch_t 0.748 (0.764)] [data_t 0.002] [optim_t 0.746] [lr 0.005000] 2024-04-07 08:54:49,216 - Train: 15.05% [743800/4942000] [150.5/1000.0] [batch_t 0.767 (0.763)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 08:56:05,731 - Train: 15.05% [743900/4942000] [150.5/1000.0] [batch_t 0.764 (0.765)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-07 08:57:22,045 - Train: 15.05% [744000/4942000] [150.5/1000.0] [batch_t 0.768 (0.763)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-07 08:58:38,431 - Train: 15.06% [744100/4942000] [150.6/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 08:59:54,835 - Train: 15.06% [744200/4942000] [150.6/1000.0] [batch_t 0.741 (0.764)] [data_t 0.003] [optim_t 0.738] [lr 0.005000] 2024-04-07 09:01:11,215 - Train: 15.06% [744300/4942000] [150.6/1000.0] [batch_t 0.754 (0.764)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-07 09:02:27,588 - Train: 15.06% [744400/4942000] [150.6/1000.0] [batch_t 0.748 (0.764)] [data_t 0.002] [optim_t 0.746] [lr 0.005000] 2024-04-07 09:03:43,999 - Train: 15.06% [744500/4942000] [150.6/1000.0] [batch_t 0.769 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 09:05:00,287 - Train: 15.07% [744600/4942000] [150.7/1000.0] [batch_t 0.764 (0.763)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-07 09:06:16,571 - Train: 15.07% [744700/4942000] [150.7/1000.0] [batch_t 0.758 (0.763)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-07 09:07:33,112 - Train: 15.07% [744800/4942000] [150.7/1000.0] [batch_t 0.761 (0.765)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-07 09:08:49,467 - Train: 15.07% [744900/4942000] [150.7/1000.0] [batch_t 0.758 (0.763)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-07 09:10:05,665 - Train: 15.07% [745000/4942000] [150.7/1000.0] [batch_t 0.755 (0.762)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-07 09:11:22,085 - Train: 15.08% [745100/4942000] [150.8/1000.0] [batch_t 0.764 (0.764)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-07 09:12:38,375 - Train: 15.08% [745200/4942000] [150.8/1000.0] [batch_t 0.761 (0.763)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-07 09:13:54,705 - Train: 15.08% [745300/4942000] [150.8/1000.0] [batch_t 0.763 (0.763)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 09:15:10,968 - Train: 15.08% [745400/4942000] [150.8/1000.0] [batch_t 0.757 (0.763)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-07 09:16:27,331 - Train: 15.08% [745500/4942000] [150.8/1000.0] [batch_t 0.773 (0.764)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-07 09:17:43,637 - Train: 15.09% [745600/4942000] [150.9/1000.0] [batch_t 0.772 (0.763)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-07 09:19:00,059 - Train: 15.09% [745700/4942000] [150.9/1000.0] [batch_t 0.770 (0.764)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-07 09:20:16,344 - Train: 15.09% [745800/4942000] [150.9/1000.0] [batch_t 0.762 (0.763)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 09:21:32,651 - Train: 15.09% [745900/4942000] [150.9/1000.0] [batch_t 0.759 (0.763)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 09:22:49,015 - Train: 15.10% [746000/4942000] [151.0/1000.0] [batch_t 0.757 (0.764)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-07 09:24:05,319 - Train: 15.10% [746100/4942000] [151.0/1000.0] [batch_t 0.773 (0.763)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-07 09:25:21,741 - Train: 15.10% [746200/4942000] [151.0/1000.0] [batch_t 0.759 (0.764)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-07 09:25:53,786 - ==> Total time: 4 days, 15:28:32 Eta: 26 days, 2:46:28 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-07 09:26:40,025 - Train: 15.10% [746300/4942000] [151.0/1000.0] [batch_t 0.748 (0.763)] [data_t 0.002] [optim_t 0.746] [lr 0.005000] 2024-04-07 09:27:56,371 - Train: 15.10% [746400/4942000] [151.0/1000.0] [batch_t 0.762 (0.763)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-07 09:29:12,632 - Train: 15.11% [746500/4942000] [151.1/1000.0] [batch_t 0.769 (0.763)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-07 09:30:29,023 - Train: 15.11% [746600/4942000] [151.1/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 09:31:45,303 - Train: 15.11% [746700/4942000] [151.1/1000.0] [batch_t 0.772 (0.763)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-07 09:33:01,535 - Train: 15.11% [746800/4942000] [151.1/1000.0] [batch_t 0.754 (0.762)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-07 09:34:17,647 - Train: 15.11% [746900/4942000] [151.1/1000.0] [batch_t 0.758 (0.761)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 09:35:33,948 - Train: 15.12% [747000/4942000] [151.2/1000.0] [batch_t 0.766 (0.763)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 09:36:50,287 - Train: 15.12% [747100/4942000] [151.2/1000.0] [batch_t 0.772 (0.763)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-07 09:38:06,506 - Train: 15.12% [747200/4942000] [151.2/1000.0] [batch_t 0.753 (0.762)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-07 09:39:22,828 - Train: 15.12% [747300/4942000] [151.2/1000.0] [batch_t 0.774 (0.763)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-07 09:40:39,161 - Train: 15.12% [747400/4942000] [151.2/1000.0] [batch_t 0.762 (0.763)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 09:41:55,443 - Train: 15.13% [747500/4942000] [151.3/1000.0] [batch_t 0.762 (0.763)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-07 09:43:11,880 - Train: 15.13% [747600/4942000] [151.3/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 09:44:28,223 - Train: 15.13% [747700/4942000] [151.3/1000.0] [batch_t 0.767 (0.763)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-07 09:45:44,449 - Train: 15.13% [747800/4942000] [151.3/1000.0] [batch_t 0.757 (0.762)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-07 09:47:00,734 - Train: 15.13% [747900/4942000] [151.3/1000.0] [batch_t 0.762 (0.763)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-07 09:48:16,966 - Train: 15.14% [748000/4942000] [151.4/1000.0] [batch_t 0.758 (0.762)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-07 09:49:33,155 - Train: 15.14% [748100/4942000] [151.4/1000.0] [batch_t 0.767 (0.762)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-07 09:50:49,506 - Train: 15.14% [748200/4942000] [151.4/1000.0] [batch_t 0.754 (0.763)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-07 09:52:05,894 - Train: 15.14% [748300/4942000] [151.4/1000.0] [batch_t 0.754 (0.764)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-07 09:53:22,181 - Train: 15.14% [748400/4942000] [151.4/1000.0] [batch_t 0.762 (0.763)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-07 09:54:38,502 - Train: 15.15% [748500/4942000] [151.5/1000.0] [batch_t 0.775 (0.763)] [data_t 0.002] [optim_t 0.772] [lr 0.005000] 2024-04-07 09:55:54,749 - Train: 15.15% [748600/4942000] [151.5/1000.0] [batch_t 0.762 (0.762)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 09:57:10,987 - Train: 15.15% [748700/4942000] [151.5/1000.0] [batch_t 0.768 (0.762)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 09:58:27,387 - Train: 15.15% [748800/4942000] [151.5/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 09:59:43,801 - Train: 15.15% [748900/4942000] [151.5/1000.0] [batch_t 0.768 (0.764)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-07 10:01:00,103 - Train: 15.16% [749000/4942000] [151.6/1000.0] [batch_t 0.758 (0.763)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-07 10:02:16,273 - Train: 15.16% [749100/4942000] [151.6/1000.0] [batch_t 0.763 (0.762)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 10:03:32,719 - Train: 15.16% [749200/4942000] [151.6/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 10:04:49,137 - Train: 15.16% [749300/4942000] [151.6/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-07 10:06:05,329 - Train: 15.16% [749400/4942000] [151.6/1000.0] [batch_t 0.758 (0.762)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-07 10:07:21,629 - Train: 15.17% [749500/4942000] [151.7/1000.0] [batch_t 0.776 (0.763)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-07 10:08:38,126 - Train: 15.17% [749600/4942000] [151.7/1000.0] [batch_t 0.768 (0.765)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-07 10:09:54,458 - Train: 15.17% [749700/4942000] [151.7/1000.0] [batch_t 0.759 (0.763)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 10:11:10,815 - Train: 15.17% [749800/4942000] [151.7/1000.0] [batch_t 0.760 (0.763)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-07 10:12:27,214 - Train: 15.17% [749900/4942000] [151.7/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-07 10:13:43,441 - Train: 15.18% [750000/4942000] [151.8/1000.0] [batch_t 0.747 (0.762)] [data_t 0.003] [optim_t 0.744] [lr 0.005000] 2024-04-07 10:14:59,834 - Train: 15.18% [750100/4942000] [151.8/1000.0] [batch_t 0.754 (0.764)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-07 10:16:16,282 - Train: 15.18% [750200/4942000] [151.8/1000.0] [batch_t 0.772 (0.764)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-07 10:17:32,630 - Train: 15.18% [750300/4942000] [151.8/1000.0] [batch_t 0.767 (0.763)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 10:18:48,993 - Train: 15.18% [750400/4942000] [151.8/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 10:20:05,279 - Train: 15.19% [750500/4942000] [151.9/1000.0] [batch_t 0.756 (0.763)] [data_t 0.002] [optim_t 0.754] [lr 0.005000] 2024-04-07 10:21:21,651 - Train: 15.19% [750600/4942000] [151.9/1000.0] [batch_t 0.763 (0.764)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-07 10:22:38,014 - Train: 15.19% [750700/4942000] [151.9/1000.0] [batch_t 0.744 (0.764)] [data_t 0.003] [optim_t 0.741] [lr 0.005000] 2024-04-07 10:23:54,445 - Train: 15.19% [750800/4942000] [151.9/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 10:26:33,107 - Train: 15.19% [750900/4942000] [151.9/1000.0] [batch_t 0.788 (1.586)] [data_t 0.003] [optim_t 0.785] [lr 0.005000] 2024-04-07 10:34:43,004 - Train: 15.20% [751000/4942000] [152.0/1000.0] [batch_t 0.771 (4.899)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-07 10:52:42,370 - Train: 15.20% [751100/4942000] [152.0/1000.0] [batch_t 0.784 (10.794)] [data_t 0.003] [optim_t 0.781] [lr 0.005000] 2024-04-07 10:55:11,389 - ==> Total time: 4 days, 16:57:50 Eta: 26 days, 6:13:13 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-07 10:55:29,969 - Train: 15.20% [751200/4942000] [152.0/1000.0] [batch_t 0.772 (0.988)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-07 10:56:46,529 - Train: 15.20% [751300/4942000] [152.0/1000.0] [batch_t 0.751 (0.766)] [data_t 0.003] [optim_t 0.748] [lr 0.005000] 2024-04-07 10:58:02,688 - Train: 15.20% [751400/4942000] [152.0/1000.0] [batch_t 0.766 (0.761)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-07 10:59:18,992 - Train: 15.21% [751500/4942000] [152.1/1000.0] [batch_t 0.761 (0.763)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-07 11:00:35,368 - Train: 15.21% [751600/4942000] [152.1/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 11:01:51,600 - Train: 15.21% [751700/4942000] [152.1/1000.0] [batch_t 0.764 (0.762)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-07 11:03:07,975 - Train: 15.21% [751800/4942000] [152.1/1000.0] [batch_t 0.753 (0.764)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-07 11:04:24,239 - Train: 15.21% [751900/4942000] [152.1/1000.0] [batch_t 0.768 (0.763)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 11:05:40,562 - Train: 15.22% [752000/4942000] [152.2/1000.0] [batch_t 0.759 (0.763)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 11:06:56,860 - Train: 15.22% [752100/4942000] [152.2/1000.0] [batch_t 0.764 (0.763)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-07 11:08:13,132 - Train: 15.22% [752200/4942000] [152.2/1000.0] [batch_t 0.767 (0.763)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-07 11:09:29,599 - Train: 15.22% [752300/4942000] [152.2/1000.0] [batch_t 0.756 (0.765)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-07 11:10:47,593 - Train: 15.22% [752400/4942000] [152.2/1000.0] [batch_t 0.761 (0.780)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-07 11:12:04,106 - Train: 15.23% [752500/4942000] [152.3/1000.0] [batch_t 0.758 (0.765)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 11:13:20,406 - Train: 15.23% [752600/4942000] [152.3/1000.0] [batch_t 0.758 (0.763)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-07 11:14:36,722 - Train: 15.23% [752700/4942000] [152.3/1000.0] [batch_t 0.772 (0.763)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-07 11:15:53,088 - Train: 15.23% [752800/4942000] [152.3/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 11:17:09,443 - Train: 15.23% [752900/4942000] [152.3/1000.0] [batch_t 0.766 (0.763)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 11:18:25,662 - Train: 15.24% [753000/4942000] [152.4/1000.0] [batch_t 0.755 (0.762)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-07 11:19:42,036 - Train: 15.24% [753100/4942000] [152.4/1000.0] [batch_t 0.763 (0.764)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-07 11:20:58,428 - Train: 15.24% [753200/4942000] [152.4/1000.0] [batch_t 0.755 (0.764)] [data_t 0.003] [optim_t 0.753] [lr 0.005000] 2024-04-07 11:22:14,727 - Train: 15.24% [753300/4942000] [152.4/1000.0] [batch_t 0.753 (0.763)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-07 11:23:31,098 - Train: 15.24% [753400/4942000] [152.4/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 11:24:47,351 - Train: 15.25% [753500/4942000] [152.5/1000.0] [batch_t 0.774 (0.762)] [data_t 0.002] [optim_t 0.772] [lr 0.005000] 2024-04-07 11:26:03,748 - Train: 15.25% [753600/4942000] [152.5/1000.0] [batch_t 0.779 (0.764)] [data_t 0.002] [optim_t 0.776] [lr 0.005000] 2024-04-07 11:27:20,023 - Train: 15.25% [753700/4942000] [152.5/1000.0] [batch_t 0.762 (0.763)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 11:28:36,355 - Train: 15.25% [753800/4942000] [152.5/1000.0] [batch_t 0.763 (0.763)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-07 11:29:52,750 - Train: 15.25% [753900/4942000] [152.5/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 11:31:09,216 - Train: 15.26% [754000/4942000] [152.6/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 11:32:25,515 - Train: 15.26% [754100/4942000] [152.6/1000.0] [batch_t 0.763 (0.763)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 11:33:41,854 - Train: 15.26% [754200/4942000] [152.6/1000.0] [batch_t 0.763 (0.763)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 11:34:58,103 - Train: 15.26% [754300/4942000] [152.6/1000.0] [batch_t 0.769 (0.762)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-07 11:36:14,356 - Train: 15.27% [754400/4942000] [152.7/1000.0] [batch_t 0.752 (0.762)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-07 11:37:30,687 - Train: 15.27% [754500/4942000] [152.7/1000.0] [batch_t 0.753 (0.763)] [data_t 0.003] [optim_t 0.749] [lr 0.005000] 2024-04-07 11:38:48,383 - Train: 15.27% [754600/4942000] [152.7/1000.0] [batch_t 0.759 (0.777)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 11:40:04,583 - Train: 15.27% [754700/4942000] [152.7/1000.0] [batch_t 0.755 (0.762)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-07 11:42:26,520 - Train: 15.27% [754800/4942000] [152.7/1000.0] [batch_t 3.293 (1.419)] [data_t 2.513] [optim_t 0.780] [lr 0.005000] 2024-04-07 11:50:30,761 - Train: 15.28% [754900/4942000] [152.8/1000.0] [batch_t 0.749 (4.842)] [data_t 0.002] [optim_t 0.746] [lr 0.005000] 2024-04-07 11:51:56,156 - Train: 15.28% [755000/4942000] [152.8/1000.0] [batch_t 0.771 (0.854)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-07 11:54:56,244 - Train: 15.28% [755100/4942000] [152.8/1000.0] [batch_t 6.874 (1.801)] [data_t 6.098] [optim_t 0.776] [lr 0.005000] 2024-04-07 11:58:38,208 - Train: 15.28% [755200/4942000] [152.8/1000.0] [batch_t 9.125 (2.220)] [data_t 8.354] [optim_t 0.770] [lr 0.005000] 2024-04-07 12:03:33,620 - Train: 15.28% [755300/4942000] [152.8/1000.0] [batch_t 6.060 (2.954)] [data_t 5.293] [optim_t 0.767] [lr 0.005000] 2024-04-07 12:08:08,216 - Train: 15.29% [755400/4942000] [152.9/1000.0] [batch_t 1.778 (2.746)] [data_t 0.998] [optim_t 0.780] [lr 0.005000] 2024-04-07 12:15:39,720 - Train: 15.29% [755500/4942000] [152.9/1000.0] [batch_t 5.063 (4.515)] [data_t 4.307] [optim_t 0.755] [lr 0.005000] 2024-04-07 12:18:35,873 - Train: 15.29% [755600/4942000] [152.9/1000.0] [batch_t 1.477 (1.761)] [data_t 0.712] [optim_t 0.764] [lr 0.005000] 2024-04-07 12:20:08,505 - Train: 15.29% [755700/4942000] [152.9/1000.0] [batch_t 0.772 (0.926)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-07 12:21:24,839 - Train: 15.29% [755800/4942000] [152.9/1000.0] [batch_t 0.772 (0.763)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-07 12:22:41,064 - Train: 15.30% [755900/4942000] [153.0/1000.0] [batch_t 0.759 (0.762)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-07 12:23:57,240 - Train: 15.30% [756000/4942000] [153.0/1000.0] [batch_t 0.755 (0.762)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-07 12:25:13,519 - Train: 15.30% [756100/4942000] [153.0/1000.0] [batch_t 0.763 (0.763)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 12:25:33,334 - ==> Total time: 4 days, 18:28:12 Eta: 26 days, 9:42:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-07 12:26:32,059 - Train: 15.30% [756200/4942000] [153.0/1000.0] [batch_t 0.753 (0.763)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-07 12:27:48,418 - Train: 15.30% [756300/4942000] [153.0/1000.0] [batch_t 0.770 (0.763)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-07 12:29:04,785 - Train: 15.31% [756400/4942000] [153.1/1000.0] [batch_t 0.765 (0.764)] [data_t 0.002] [optim_t 0.763] [lr 0.005000] 2024-04-07 12:30:21,060 - Train: 15.31% [756500/4942000] [153.1/1000.0] [batch_t 0.773 (0.763)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-07 12:31:37,401 - Train: 15.31% [756600/4942000] [153.1/1000.0] [batch_t 0.763 (0.763)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-07 12:32:33,820 - Train: 15.31% [756700/4942000] [153.1/1000.0] [batch_t 0.328 (0.564)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 12:33:06,890 - Train: 15.31% [756800/4942000] [153.1/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 12:33:39,599 - Train: 15.32% [756900/4942000] [153.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 12:34:12,843 - Train: 15.32% [757000/4942000] [153.2/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 12:34:45,576 - Train: 15.32% [757100/4942000] [153.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 12:35:18,337 - Train: 15.32% [757200/4942000] [153.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 12:35:51,087 - Train: 15.32% [757300/4942000] [153.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 12:36:23,855 - Train: 15.33% [757400/4942000] [153.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 12:36:56,539 - Train: 15.33% [757500/4942000] [153.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 12:37:29,255 - Train: 15.33% [757600/4942000] [153.3/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 12:38:01,952 - Train: 15.33% [757700/4942000] [153.3/1000.0] [batch_t 0.320 (0.327)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-07 12:38:34,668 - Train: 15.33% [757800/4942000] [153.3/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 12:39:07,399 - Train: 15.34% [757900/4942000] [153.4/1000.0] [batch_t 0.337 (0.327)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-07 12:39:40,201 - Train: 15.34% [758000/4942000] [153.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 12:40:13,430 - Train: 15.34% [758100/4942000] [153.4/1000.0] [batch_t 0.324 (0.332)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 12:40:46,089 - Train: 15.34% [758200/4942000] [153.4/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 12:41:18,967 - Train: 15.34% [758300/4942000] [153.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 12:41:51,662 - Train: 15.35% [758400/4942000] [153.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 12:42:24,337 - Train: 15.35% [758500/4942000] [153.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 12:42:56,977 - Train: 15.35% [758600/4942000] [153.5/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 12:43:29,783 - Train: 15.35% [758700/4942000] [153.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 12:44:02,467 - Train: 15.35% [758800/4942000] [153.5/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-07 12:44:36,457 - Train: 15.36% [758900/4942000] [153.6/1000.0] [batch_t 0.325 (0.340)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 12:45:09,538 - Train: 15.36% [759000/4942000] [153.6/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 12:45:42,215 - Train: 15.36% [759100/4942000] [153.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 12:46:14,940 - Train: 15.36% [759200/4942000] [153.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 12:46:47,620 - Train: 15.36% [759300/4942000] [153.6/1000.0] [batch_t 0.320 (0.327)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-07 12:47:20,429 - Train: 15.37% [759400/4942000] [153.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 12:47:53,183 - Train: 15.37% [759500/4942000] [153.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 12:48:26,651 - Train: 15.37% [759600/4942000] [153.7/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 12:48:59,354 - Train: 15.37% [759700/4942000] [153.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 12:49:32,077 - Train: 15.37% [759800/4942000] [153.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 12:50:05,192 - Train: 15.38% [759900/4942000] [153.8/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 12:50:37,904 - Train: 15.38% [760000/4942000] [153.8/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 12:51:11,393 - Train: 15.38% [760100/4942000] [153.8/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 12:51:44,141 - Train: 15.38% [760200/4942000] [153.8/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 12:52:16,828 - Train: 15.38% [760300/4942000] [153.8/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 12:52:49,467 - Train: 15.39% [760400/4942000] [153.9/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 12:53:22,167 - Train: 15.39% [760500/4942000] [153.9/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 12:53:54,845 - Train: 15.39% [760600/4942000] [153.9/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 12:54:27,528 - Train: 15.39% [760700/4942000] [153.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 12:55:00,275 - Train: 15.39% [760800/4942000] [153.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 12:55:33,081 - Train: 15.40% [760900/4942000] [154.0/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-07 12:56:07,590 - Train: 15.40% [761000/4942000] [154.0/1000.0] [batch_t 0.327 (0.345)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 12:56:29,792 - ==> Total time: 4 days, 18:59:08 Eta: 26 days, 7:40:31 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-07 12:56:42,292 - Train: 15.40% [761100/4942000] [154.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 12:57:14,960 - Train: 15.40% [761200/4942000] [154.0/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 12:57:47,649 - Train: 15.40% [761300/4942000] [154.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 12:58:21,111 - Train: 15.41% [761400/4942000] [154.1/1000.0] [batch_t 0.324 (0.335)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 12:58:53,757 - Train: 15.41% [761500/4942000] [154.1/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 12:59:27,543 - Train: 15.41% [761600/4942000] [154.1/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:00:00,269 - Train: 15.41% [761700/4942000] [154.1/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 13:00:34,441 - Train: 15.41% [761800/4942000] [154.1/1000.0] [batch_t 0.328 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:01:07,283 - Train: 15.42% [761900/4942000] [154.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:01:39,988 - Train: 15.42% [762000/4942000] [154.2/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 13:02:12,759 - Train: 15.42% [762100/4942000] [154.2/1000.0] [batch_t 0.320 (0.328)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-07 13:02:45,556 - Train: 15.42% [762200/4942000] [154.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 13:03:18,341 - Train: 15.42% [762300/4942000] [154.2/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 13:03:51,010 - Train: 15.43% [762400/4942000] [154.3/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 13:04:23,710 - Train: 15.43% [762500/4942000] [154.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:04:56,402 - Train: 15.43% [762600/4942000] [154.3/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 13:05:29,102 - Train: 15.43% [762700/4942000] [154.3/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 13:06:01,792 - Train: 15.44% [762800/4942000] [154.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:06:34,436 - Train: 15.44% [762900/4942000] [154.4/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:07:07,223 - Train: 15.44% [763000/4942000] [154.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:07:39,870 - Train: 15.44% [763100/4942000] [154.4/1000.0] [batch_t 0.322 (0.326)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-07 13:08:12,566 - Train: 15.44% [763200/4942000] [154.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:08:45,243 - Train: 15.45% [763300/4942000] [154.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 13:09:17,967 - Train: 15.45% [763400/4942000] [154.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:09:50,616 - Train: 15.45% [763500/4942000] [154.5/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 13:10:23,350 - Train: 15.45% [763600/4942000] [154.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:10:55,999 - Train: 15.45% [763700/4942000] [154.5/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 13:11:28,669 - Train: 15.46% [763800/4942000] [154.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:12:01,341 - Train: 15.46% [763900/4942000] [154.6/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 13:12:33,996 - Train: 15.46% [764000/4942000] [154.6/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:13:06,643 - Train: 15.46% [764100/4942000] [154.6/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 13:13:39,355 - Train: 15.46% [764200/4942000] [154.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 13:14:12,053 - Train: 15.47% [764300/4942000] [154.7/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 13:14:44,726 - Train: 15.47% [764400/4942000] [154.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 13:15:17,448 - Train: 15.47% [764500/4942000] [154.7/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 13:15:50,103 - Train: 15.47% [764600/4942000] [154.7/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:16:22,786 - Train: 15.47% [764700/4942000] [154.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 13:16:55,442 - Train: 15.48% [764800/4942000] [154.8/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 13:17:28,071 - Train: 15.48% [764900/4942000] [154.8/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:18:00,886 - Train: 15.48% [765000/4942000] [154.8/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 13:18:33,647 - Train: 15.48% [765100/4942000] [154.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 13:19:06,307 - Train: 15.48% [765200/4942000] [154.8/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 13:19:42,147 - Train: 15.49% [765300/4942000] [154.9/1000.0] [batch_t 0.327 (0.358)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 13:20:14,821 - Train: 15.49% [765400/4942000] [154.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:20:47,636 - Train: 15.49% [765500/4942000] [154.9/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 13:21:20,336 - Train: 15.49% [765600/4942000] [154.9/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 13:21:52,979 - Train: 15.49% [765700/4942000] [154.9/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:22:25,637 - Train: 15.50% [765800/4942000] [155.0/1000.0] [batch_t 0.321 (0.326)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-07 13:22:58,282 - Train: 15.50% [765900/4942000] [155.0/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 13:23:31,011 - Train: 15.50% [766000/4942000] [155.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:23:34,287 - ==> Total time: 4 days, 19:26:13 Eta: 26 days, 5:19:05 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-07 13:24:05,801 - Train: 15.50% [766100/4942000] [155.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:24:38,398 - Train: 15.50% [766200/4942000] [155.0/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 13:25:11,218 - Train: 15.51% [766300/4942000] [155.1/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 13:25:43,869 - Train: 15.51% [766400/4942000] [155.1/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 13:26:18,036 - Train: 15.51% [766500/4942000] [155.1/1000.0] [batch_t 0.325 (0.342)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 13:26:50,700 - Train: 15.51% [766600/4942000] [155.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:27:23,359 - Train: 15.51% [766700/4942000] [155.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:27:56,000 - Train: 15.52% [766800/4942000] [155.2/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:28:28,697 - Train: 15.52% [766900/4942000] [155.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:29:01,436 - Train: 15.52% [767000/4942000] [155.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 13:29:34,114 - Train: 15.52% [767100/4942000] [155.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 13:30:06,773 - Train: 15.52% [767200/4942000] [155.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:30:39,483 - Train: 15.53% [767300/4942000] [155.3/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 13:31:12,127 - Train: 15.53% [767400/4942000] [155.3/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 13:31:44,808 - Train: 15.53% [767500/4942000] [155.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 13:32:17,499 - Train: 15.53% [767600/4942000] [155.3/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 13:32:50,218 - Train: 15.53% [767700/4942000] [155.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 13:33:22,918 - Train: 15.54% [767800/4942000] [155.4/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 13:33:55,611 - Train: 15.54% [767900/4942000] [155.4/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 13:34:28,358 - Train: 15.54% [768000/4942000] [155.4/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 13:35:01,020 - Train: 15.54% [768100/4942000] [155.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:35:33,684 - Train: 15.54% [768200/4942000] [155.4/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 13:36:06,329 - Train: 15.55% [768300/4942000] [155.5/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:36:39,043 - Train: 15.55% [768400/4942000] [155.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:37:11,731 - Train: 15.55% [768500/4942000] [155.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:37:44,394 - Train: 15.55% [768600/4942000] [155.5/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 13:38:17,130 - Train: 15.55% [768700/4942000] [155.5/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 13:38:49,875 - Train: 15.56% [768800/4942000] [155.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 13:39:22,601 - Train: 15.56% [768900/4942000] [155.6/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 13:39:55,309 - Train: 15.56% [769000/4942000] [155.6/1000.0] [batch_t 0.320 (0.327)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-07 13:40:27,968 - Train: 15.56% [769100/4942000] [155.6/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 13:41:00,854 - Train: 15.56% [769200/4942000] [155.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 13:41:33,611 - Train: 15.57% [769300/4942000] [155.7/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-07 13:42:06,403 - Train: 15.57% [769400/4942000] [155.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 13:42:39,108 - Train: 15.57% [769500/4942000] [155.7/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 13:43:11,850 - Train: 15.57% [769600/4942000] [155.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:43:44,621 - Train: 15.57% [769700/4942000] [155.7/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 13:44:17,379 - Train: 15.58% [769800/4942000] [155.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 13:44:50,061 - Train: 15.58% [769900/4942000] [155.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 13:45:22,763 - Train: 15.58% [770000/4942000] [155.8/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 13:45:55,498 - Train: 15.58% [770100/4942000] [155.8/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 13:46:28,258 - Train: 15.58% [770200/4942000] [155.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 13:47:00,936 - Train: 15.59% [770300/4942000] [155.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:47:33,659 - Train: 15.59% [770400/4942000] [155.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 13:48:06,447 - Train: 15.59% [770500/4942000] [155.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:48:39,244 - Train: 15.59% [770600/4942000] [155.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 13:49:12,115 - Train: 15.59% [770700/4942000] [155.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:49:44,857 - Train: 15.60% [770800/4942000] [156.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:50:17,602 - Train: 15.60% [770900/4942000] [156.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:50:34,644 - ==> Total time: 4 days, 19:53:13 Eta: 26 days, 2:58:45 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-07 13:50:52,298 - Train: 15.60% [771000/4942000] [156.0/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 13:51:25,059 - Train: 15.60% [771100/4942000] [156.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 13:51:57,706 - Train: 15.61% [771200/4942000] [156.1/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 13:52:30,430 - Train: 15.61% [771300/4942000] [156.1/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 13:53:03,208 - Train: 15.61% [771400/4942000] [156.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:53:35,920 - Train: 15.61% [771500/4942000] [156.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:54:08,669 - Train: 15.61% [771600/4942000] [156.1/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 13:54:41,388 - Train: 15.62% [771700/4942000] [156.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 13:55:14,096 - Train: 15.62% [771800/4942000] [156.2/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 13:55:46,746 - Train: 15.62% [771900/4942000] [156.2/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:56:19,558 - Train: 15.62% [772000/4942000] [156.2/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 13:56:52,247 - Train: 15.62% [772100/4942000] [156.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 13:57:24,972 - Train: 15.63% [772200/4942000] [156.3/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 13:57:57,701 - Train: 15.63% [772300/4942000] [156.3/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 13:58:30,333 - Train: 15.63% [772400/4942000] [156.3/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 13:59:02,938 - Train: 15.63% [772500/4942000] [156.3/1000.0] [batch_t 0.322 (0.326)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-07 13:59:35,622 - Train: 15.63% [772600/4942000] [156.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 14:00:08,277 - Train: 15.64% [772700/4942000] [156.4/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 14:00:40,912 - Train: 15.64% [772800/4942000] [156.4/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 14:01:15,884 - Train: 15.64% [772900/4942000] [156.4/1000.0] [batch_t 0.333 (0.350)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 14:01:48,706 - Train: 15.64% [773000/4942000] [156.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 14:02:23,121 - Train: 15.64% [773100/4942000] [156.4/1000.0] [batch_t 0.323 (0.344)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 14:02:55,771 - Train: 15.65% [773200/4942000] [156.5/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 14:03:40,282 - Train: 15.65% [773300/4942000] [156.5/1000.0] [batch_t 0.328 (0.445)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 14:04:19,165 - Train: 15.65% [773400/4942000] [156.5/1000.0] [batch_t 0.329 (0.389)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 14:04:51,900 - Train: 15.65% [773500/4942000] [156.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 14:05:24,549 - Train: 15.65% [773600/4942000] [156.5/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 14:05:58,260 - Train: 15.66% [773700/4942000] [156.6/1000.0] [batch_t 0.324 (0.337)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 14:07:19,425 - Train: 15.66% [773800/4942000] [156.6/1000.0] [batch_t 0.323 (0.812)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 14:07:52,141 - Train: 15.66% [773900/4942000] [156.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 14:08:24,837 - Train: 15.66% [774000/4942000] [156.6/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 14:08:57,484 - Train: 15.66% [774100/4942000] [156.6/1000.0] [batch_t 0.333 (0.326)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 14:09:30,207 - Train: 15.67% [774200/4942000] [156.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 14:10:02,906 - Train: 15.67% [774300/4942000] [156.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 14:10:35,616 - Train: 15.67% [774400/4942000] [156.7/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 14:18:20,106 - Train: 15.67% [774500/4942000] [156.7/1000.0] [batch_t 5.471 (4.645)] [data_t 4.716] [optim_t 0.755] [lr 0.005000] 2024-04-07 14:27:38,033 - Train: 15.67% [774600/4942000] [156.7/1000.0] [batch_t 0.756 (5.579)] [data_t 0.003] [optim_t 0.753] [lr 0.005000] 2024-04-07 14:29:12,648 - Train: 15.68% [774700/4942000] [156.8/1000.0] [batch_t 0.762 (0.946)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 14:30:29,117 - Train: 15.68% [774800/4942000] [156.8/1000.0] [batch_t 0.772 (0.765)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-07 14:31:45,486 - Train: 15.68% [774900/4942000] [156.8/1000.0] [batch_t 0.759 (0.764)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-07 14:33:06,729 - Train: 15.68% [775000/4942000] [156.8/1000.0] [batch_t 0.765 (0.812)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-07 14:34:30,383 - Train: 15.68% [775100/4942000] [156.8/1000.0] [batch_t 0.764 (0.836)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-07 14:35:51,360 - Train: 15.69% [775200/4942000] [156.9/1000.0] [batch_t 0.773 (0.810)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-07 14:37:07,609 - Train: 15.69% [775300/4942000] [156.9/1000.0] [batch_t 0.762 (0.762)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-07 14:38:23,808 - Train: 15.69% [775400/4942000] [156.9/1000.0] [batch_t 0.758 (0.762)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 14:39:39,998 - Train: 15.69% [775500/4942000] [156.9/1000.0] [batch_t 0.760 (0.762)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-07 14:40:56,351 - Train: 15.69% [775600/4942000] [156.9/1000.0] [batch_t 0.757 (0.763)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-07 14:42:12,729 - Train: 15.70% [775700/4942000] [157.0/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-07 14:43:29,083 - Train: 15.70% [775800/4942000] [157.0/1000.0] [batch_t 0.766 (0.763)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-07 14:44:40,788 - ==> Total time: 4 days, 20:47:19 Eta: 26 days, 3:05:21 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-07 14:44:47,998 - Train: 15.70% [775900/4942000] [157.0/1000.0] [batch_t 0.762 (0.811)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 14:46:06,915 - Train: 15.70% [776000/4942000] [157.0/1000.0] [batch_t 0.829 (0.789)] [data_t 0.065] [optim_t 0.763] [lr 0.005000] 2024-04-07 14:47:23,668 - Train: 15.70% [776100/4942000] [157.0/1000.0] [batch_t 0.752 (0.767)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-07 14:48:40,035 - Train: 15.71% [776200/4942000] [157.1/1000.0] [batch_t 0.754 (0.764)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-07 14:49:56,380 - Train: 15.71% [776300/4942000] [157.1/1000.0] [batch_t 0.770 (0.763)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-07 14:51:12,731 - Train: 15.71% [776400/4942000] [157.1/1000.0] [batch_t 0.769 (0.763)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-07 14:52:29,055 - Train: 15.71% [776500/4942000] [157.1/1000.0] [batch_t 0.772 (0.763)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-07 14:53:45,422 - Train: 15.71% [776600/4942000] [157.1/1000.0] [batch_t 0.769 (0.764)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-07 14:55:01,704 - Train: 15.72% [776700/4942000] [157.2/1000.0] [batch_t 0.755 (0.763)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-07 14:56:17,960 - Train: 15.72% [776800/4942000] [157.2/1000.0] [batch_t 0.767 (0.762)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 14:57:34,287 - Train: 15.72% [776900/4942000] [157.2/1000.0] [batch_t 0.772 (0.763)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-07 14:58:50,597 - Train: 15.72% [777000/4942000] [157.2/1000.0] [batch_t 0.752 (0.763)] [data_t 0.003] [optim_t 0.749] [lr 0.005000] 2024-04-07 15:00:06,974 - Train: 15.72% [777100/4942000] [157.2/1000.0] [batch_t 0.758 (0.764)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-07 15:01:23,131 - Train: 15.73% [777200/4942000] [157.3/1000.0] [batch_t 0.772 (0.761)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-07 15:02:39,502 - Train: 15.73% [777300/4942000] [157.3/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-07 15:03:55,832 - Train: 15.73% [777400/4942000] [157.3/1000.0] [batch_t 0.761 (0.763)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-07 15:05:12,067 - Train: 15.73% [777500/4942000] [157.3/1000.0] [batch_t 0.773 (0.762)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-07 15:06:28,373 - Train: 15.73% [777600/4942000] [157.3/1000.0] [batch_t 0.770 (0.763)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-07 15:07:44,774 - Train: 15.74% [777700/4942000] [157.4/1000.0] [batch_t 0.769 (0.764)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-07 15:09:01,083 - Train: 15.74% [777800/4942000] [157.4/1000.0] [batch_t 0.773 (0.763)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-07 15:10:17,432 - Train: 15.74% [777900/4942000] [157.4/1000.0] [batch_t 0.758 (0.763)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-07 15:11:33,686 - Train: 15.74% [778000/4942000] [157.4/1000.0] [batch_t 0.758 (0.762)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-07 15:12:49,904 - Train: 15.74% [778100/4942000] [157.4/1000.0] [batch_t 0.763 (0.762)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 15:14:06,020 - Train: 15.75% [778200/4942000] [157.5/1000.0] [batch_t 0.758 (0.761)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 15:15:22,330 - Train: 15.75% [778300/4942000] [157.5/1000.0] [batch_t 0.759 (0.763)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-07 15:16:38,555 - Train: 15.75% [778400/4942000] [157.5/1000.0] [batch_t 0.773 (0.762)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-07 15:17:54,665 - Train: 15.75% [778500/4942000] [157.5/1000.0] [batch_t 0.753 (0.761)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-07 15:19:10,921 - Train: 15.75% [778600/4942000] [157.5/1000.0] [batch_t 0.775 (0.762)] [data_t 0.002] [optim_t 0.772] [lr 0.005000] 2024-04-07 15:20:27,310 - Train: 15.76% [778700/4942000] [157.6/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 15:21:43,585 - Train: 15.76% [778800/4942000] [157.6/1000.0] [batch_t 0.762 (0.763)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-07 15:22:59,972 - Train: 15.76% [778900/4942000] [157.6/1000.0] [batch_t 0.758 (0.764)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-07 15:24:16,283 - Train: 15.76% [779000/4942000] [157.6/1000.0] [batch_t 0.762 (0.763)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-07 15:25:32,597 - Train: 15.76% [779100/4942000] [157.6/1000.0] [batch_t 0.767 (0.763)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-07 15:26:49,039 - Train: 15.77% [779200/4942000] [157.7/1000.0] [batch_t 0.773 (0.764)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-07 15:28:05,349 - Train: 15.77% [779300/4942000] [157.7/1000.0] [batch_t 0.758 (0.763)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-07 15:29:21,686 - Train: 15.77% [779400/4942000] [157.7/1000.0] [batch_t 0.756 (0.763)] [data_t 0.002] [optim_t 0.754] [lr 0.005000] 2024-04-07 15:30:38,010 - Train: 15.77% [779500/4942000] [157.7/1000.0] [batch_t 0.766 (0.763)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-07 15:31:54,483 - Train: 15.77% [779600/4942000] [157.7/1000.0] [batch_t 0.778 (0.765)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-07 15:32:43,524 - Train: 15.78% [779700/4942000] [157.8/1000.0] [batch_t 0.326 (0.490)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 15:33:16,264 - Train: 15.78% [779800/4942000] [157.8/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 15:33:48,979 - Train: 15.78% [779900/4942000] [157.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 15:34:21,662 - Train: 15.78% [780000/4942000] [157.8/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 15:34:54,319 - Train: 15.79% [780100/4942000] [157.9/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 15:35:27,099 - Train: 15.79% [780200/4942000] [157.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 15:35:59,817 - Train: 15.79% [780300/4942000] [157.9/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 15:36:33,532 - Train: 15.79% [780400/4942000] [157.9/1000.0] [batch_t 0.326 (0.337)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 15:37:11,853 - Train: 15.79% [780500/4942000] [157.9/1000.0] [batch_t 0.326 (0.383)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 15:37:44,595 - Train: 15.80% [780600/4942000] [158.0/1000.0] [batch_t 0.339 (0.327)] [data_t 0.002] [optim_t 0.337] [lr 0.005000] 2024-04-07 15:38:27,313 - Train: 15.80% [780700/4942000] [158.0/1000.0] [batch_t 0.327 (0.427)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 15:39:54,918 - Train: 15.80% [780800/4942000] [158.0/1000.0] [batch_t 3.198 (0.876)] [data_t 2.872] [optim_t 0.326] [lr 0.005000] 2024-04-07 15:40:25,186 - ==> Total time: 4 days, 21:43:04 Eta: 26 days, 3:19:55 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-07 15:41:42,643 - Train: 15.80% [780900/4942000] [158.0/1000.0] [batch_t 0.325 (1.161)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 15:43:39,747 - Train: 15.80% [781000/4942000] [158.0/1000.0] [batch_t 2.555 (1.171)] [data_t 1.795] [optim_t 0.760] [lr 0.005000] 2024-04-07 15:46:08,929 - Train: 15.81% [781100/4942000] [158.1/1000.0] [batch_t 7.390 (1.492)] [data_t 6.618] [optim_t 0.772] [lr 0.005000] 2024-04-07 15:47:41,472 - Train: 15.81% [781200/4942000] [158.1/1000.0] [batch_t 0.768 (0.925)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-07 15:49:20,221 - Train: 15.81% [781300/4942000] [158.1/1000.0] [batch_t 0.772 (0.987)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-07 15:51:14,719 - Train: 15.81% [781400/4942000] [158.1/1000.0] [batch_t 0.758 (1.145)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-07 15:52:35,394 - Train: 15.81% [781500/4942000] [158.1/1000.0] [batch_t 0.763 (0.807)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-07 15:53:51,617 - Train: 15.82% [781600/4942000] [158.2/1000.0] [batch_t 0.759 (0.762)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-07 15:55:07,931 - Train: 15.82% [781700/4942000] [158.2/1000.0] [batch_t 0.766 (0.763)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-07 15:56:24,247 - Train: 15.82% [781800/4942000] [158.2/1000.0] [batch_t 0.769 (0.763)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-07 15:57:40,603 - Train: 15.82% [781900/4942000] [158.2/1000.0] [batch_t 0.764 (0.763)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-07 15:58:56,857 - Train: 15.82% [782000/4942000] [158.2/1000.0] [batch_t 0.761 (0.762)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-07 16:00:13,064 - Train: 15.83% [782100/4942000] [158.3/1000.0] [batch_t 0.771 (0.762)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-07 16:01:29,303 - Train: 15.83% [782200/4942000] [158.3/1000.0] [batch_t 0.759 (0.762)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-07 16:02:45,589 - Train: 15.83% [782300/4942000] [158.3/1000.0] [batch_t 0.758 (0.763)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-07 16:04:01,893 - Train: 15.83% [782400/4942000] [158.3/1000.0] [batch_t 0.767 (0.763)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 16:05:18,281 - Train: 15.83% [782500/4942000] [158.3/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-07 16:06:34,748 - Train: 15.84% [782600/4942000] [158.4/1000.0] [batch_t 0.777 (0.765)] [data_t 0.002] [optim_t 0.775] [lr 0.005000] 2024-04-07 16:07:51,100 - Train: 15.84% [782700/4942000] [158.4/1000.0] [batch_t 0.764 (0.763)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 16:09:07,433 - Train: 15.84% [782800/4942000] [158.4/1000.0] [batch_t 0.772 (0.763)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-07 16:10:23,628 - Train: 15.84% [782900/4942000] [158.4/1000.0] [batch_t 0.743 (0.762)] [data_t 0.003] [optim_t 0.740] [lr 0.005000] 2024-04-07 16:11:39,927 - Train: 15.84% [783000/4942000] [158.4/1000.0] [batch_t 0.769 (0.763)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-07 16:12:56,370 - Train: 15.85% [783100/4942000] [158.5/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 16:14:12,613 - Train: 15.85% [783200/4942000] [158.5/1000.0] [batch_t 0.769 (0.762)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-07 16:15:29,149 - Train: 15.85% [783300/4942000] [158.5/1000.0] [batch_t 0.753 (0.765)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-07 16:16:45,611 - Train: 15.85% [783400/4942000] [158.5/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 16:18:02,020 - Train: 15.85% [783500/4942000] [158.5/1000.0] [batch_t 0.772 (0.764)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-07 16:19:18,400 - Train: 15.86% [783600/4942000] [158.6/1000.0] [batch_t 0.778 (0.764)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-07 16:20:34,680 - Train: 15.86% [783700/4942000] [158.6/1000.0] [batch_t 0.744 (0.763)] [data_t 0.003] [optim_t 0.741] [lr 0.005000] 2024-04-07 16:21:50,989 - Train: 15.86% [783800/4942000] [158.6/1000.0] [batch_t 0.770 (0.763)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-07 16:23:07,418 - Train: 15.86% [783900/4942000] [158.6/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-07 16:24:23,803 - Train: 15.86% [784000/4942000] [158.6/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 16:25:40,201 - Train: 15.87% [784100/4942000] [158.7/1000.0] [batch_t 0.768 (0.764)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-07 16:26:56,640 - Train: 15.87% [784200/4942000] [158.7/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 16:28:12,908 - Train: 15.87% [784300/4942000] [158.7/1000.0] [batch_t 0.763 (0.763)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-07 16:29:29,204 - Train: 15.87% [784400/4942000] [158.7/1000.0] [batch_t 0.755 (0.763)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-07 16:30:45,636 - Train: 15.87% [784500/4942000] [158.7/1000.0] [batch_t 0.769 (0.764)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-07 16:32:01,960 - Train: 15.88% [784600/4942000] [158.8/1000.0] [batch_t 0.757 (0.763)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-07 16:33:18,208 - Train: 15.88% [784700/4942000] [158.8/1000.0] [batch_t 0.771 (0.762)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-07 16:34:34,512 - Train: 15.88% [784800/4942000] [158.8/1000.0] [batch_t 0.772 (0.763)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-07 16:35:50,651 - Train: 15.88% [784900/4942000] [158.8/1000.0] [batch_t 0.753 (0.761)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-07 16:37:06,806 - Train: 15.88% [785000/4942000] [158.8/1000.0] [batch_t 0.755 (0.761)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-07 16:38:23,298 - Train: 15.89% [785100/4942000] [158.9/1000.0] [batch_t 0.754 (0.765)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-07 16:39:39,605 - Train: 15.89% [785200/4942000] [158.9/1000.0] [batch_t 0.764 (0.763)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-07 16:40:55,875 - Train: 15.89% [785300/4942000] [158.9/1000.0] [batch_t 0.768 (0.763)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-07 16:42:12,341 - Train: 15.89% [785400/4942000] [158.9/1000.0] [batch_t 0.761 (0.765)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-07 16:43:28,618 - Train: 15.89% [785500/4942000] [158.9/1000.0] [batch_t 0.763 (0.763)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-07 16:44:44,984 - Train: 15.90% [785600/4942000] [159.0/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-07 16:46:01,275 - Train: 15.90% [785700/4942000] [159.0/1000.0] [batch_t 0.772 (0.763)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-07 16:47:00,541 - ==> Total time: 4 days, 22:49:39 Eta: 26 days, 4:30:58 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-07 16:47:22,146 - Train: 15.90% [785800/4942000] [159.0/1000.0] [batch_t 0.759 (0.886)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-07 16:48:38,390 - Train: 15.90% [785900/4942000] [159.0/1000.0] [batch_t 0.767 (0.762)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 16:49:54,734 - Train: 15.90% [786000/4942000] [159.0/1000.0] [batch_t 0.759 (0.763)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 16:51:10,997 - Train: 15.91% [786100/4942000] [159.1/1000.0] [batch_t 0.758 (0.763)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-07 16:52:27,123 - Train: 15.91% [786200/4942000] [159.1/1000.0] [batch_t 0.752 (0.761)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-07 16:53:43,404 - Train: 15.91% [786300/4942000] [159.1/1000.0] [batch_t 0.768 (0.763)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-07 16:54:59,485 - Train: 15.91% [786400/4942000] [159.1/1000.0] [batch_t 0.762 (0.761)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-07 16:56:15,698 - Train: 15.91% [786500/4942000] [159.1/1000.0] [batch_t 0.763 (0.762)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 16:57:31,811 - Train: 15.92% [786600/4942000] [159.2/1000.0] [batch_t 0.757 (0.761)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-07 16:58:47,960 - Train: 15.92% [786700/4942000] [159.2/1000.0] [batch_t 0.767 (0.761)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 17:00:04,273 - Train: 15.92% [786800/4942000] [159.2/1000.0] [batch_t 0.772 (0.763)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-07 17:01:20,438 - Train: 15.92% [786900/4942000] [159.2/1000.0] [batch_t 0.767 (0.762)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-07 17:02:36,594 - Train: 15.92% [787000/4942000] [159.2/1000.0] [batch_t 0.774 (0.761)] [data_t 0.003] [optim_t 0.772] [lr 0.005000] 2024-04-07 17:03:52,839 - Train: 15.93% [787100/4942000] [159.3/1000.0] [batch_t 0.769 (0.762)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-07 17:05:08,971 - Train: 15.93% [787200/4942000] [159.3/1000.0] [batch_t 0.754 (0.761)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-07 17:06:25,051 - Train: 15.93% [787300/4942000] [159.3/1000.0] [batch_t 0.764 (0.761)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-07 17:07:41,160 - Train: 15.93% [787400/4942000] [159.3/1000.0] [batch_t 0.763 (0.761)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 17:08:57,354 - Train: 15.93% [787500/4942000] [159.3/1000.0] [batch_t 0.767 (0.762)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-07 17:10:13,483 - Train: 15.94% [787600/4942000] [159.4/1000.0] [batch_t 0.763 (0.761)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-07 17:11:29,675 - Train: 15.94% [787700/4942000] [159.4/1000.0] [batch_t 0.765 (0.762)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-07 17:12:45,968 - Train: 15.94% [787800/4942000] [159.4/1000.0] [batch_t 0.744 (0.763)] [data_t 0.003] [optim_t 0.741] [lr 0.005000] 2024-04-07 17:14:02,270 - Train: 15.94% [787900/4942000] [159.4/1000.0] [batch_t 0.767 (0.763)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 17:15:18,450 - Train: 15.94% [788000/4942000] [159.4/1000.0] [batch_t 0.753 (0.762)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-07 17:16:34,838 - Train: 15.95% [788100/4942000] [159.5/1000.0] [batch_t 0.758 (0.764)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-07 17:17:51,148 - Train: 15.95% [788200/4942000] [159.5/1000.0] [batch_t 0.768 (0.763)] [data_t 0.004] [optim_t 0.765] [lr 0.005000] 2024-04-07 17:19:07,558 - Train: 15.95% [788300/4942000] [159.5/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 17:20:23,934 - Train: 15.95% [788400/4942000] [159.5/1000.0] [batch_t 0.768 (0.764)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-07 17:21:40,356 - Train: 15.96% [788500/4942000] [159.6/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 17:22:56,757 - Train: 15.96% [788600/4942000] [159.6/1000.0] [batch_t 0.763 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 17:24:13,112 - Train: 15.96% [788700/4942000] [159.6/1000.0] [batch_t 0.759 (0.763)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 17:25:29,426 - Train: 15.96% [788800/4942000] [159.6/1000.0] [batch_t 0.763 (0.763)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-07 17:26:45,835 - Train: 15.96% [788900/4942000] [159.6/1000.0] [batch_t 0.770 (0.764)] [data_t 0.004] [optim_t 0.766] [lr 0.005000] 2024-04-07 17:28:02,094 - Train: 15.97% [789000/4942000] [159.7/1000.0] [batch_t 0.762 (0.762)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-07 17:29:18,333 - Train: 15.97% [789100/4942000] [159.7/1000.0] [batch_t 0.764 (0.762)] [data_t 0.004] [optim_t 0.760] [lr 0.005000] 2024-04-07 17:30:34,881 - Train: 15.97% [789200/4942000] [159.7/1000.0] [batch_t 0.764 (0.765)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-07 17:31:51,287 - Train: 15.97% [789300/4942000] [159.7/1000.0] [batch_t 0.771 (0.764)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-07 17:33:07,636 - Train: 15.97% [789400/4942000] [159.7/1000.0] [batch_t 0.757 (0.763)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-07 17:34:24,195 - Train: 15.98% [789500/4942000] [159.8/1000.0] [batch_t 0.763 (0.765)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 17:35:40,404 - Train: 15.98% [789600/4942000] [159.8/1000.0] [batch_t 0.766 (0.762)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-07 17:36:56,860 - Train: 15.98% [789700/4942000] [159.8/1000.0] [batch_t 0.778 (0.764)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-07 17:38:13,252 - Train: 15.98% [789800/4942000] [159.8/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-07 17:39:29,688 - Train: 15.98% [789900/4942000] [159.8/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-07 17:40:46,173 - Train: 15.99% [790000/4942000] [159.9/1000.0] [batch_t 0.754 (0.765)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-07 17:42:02,661 - Train: 15.99% [790100/4942000] [159.9/1000.0] [batch_t 0.742 (0.765)] [data_t 0.003] [optim_t 0.739] [lr 0.005000] 2024-04-07 17:43:19,128 - Train: 15.99% [790200/4942000] [159.9/1000.0] [batch_t 0.750 (0.765)] [data_t 0.003] [optim_t 0.747] [lr 0.005000] 2024-04-07 17:44:35,570 - Train: 15.99% [790300/4942000] [159.9/1000.0] [batch_t 0.766 (0.764)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-07 17:45:51,942 - Train: 15.99% [790400/4942000] [159.9/1000.0] [batch_t 0.768 (0.764)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-07 17:47:08,287 - Train: 16.00% [790500/4942000] [160.0/1000.0] [batch_t 0.769 (0.763)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-07 17:48:24,653 - Train: 16.00% [790600/4942000] [160.0/1000.0] [batch_t 0.774 (0.764)] [data_t 0.003] [optim_t 0.772] [lr 0.005000] 2024-04-07 17:49:41,060 - Train: 16.00% [790700/4942000] [160.0/1000.0] [batch_t 0.769 (0.764)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-07 17:49:56,328 - ==> Total time: 4 days, 23:52:35 Eta: 26 days, 5:21:06 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-07 17:50:59,358 - Train: 16.00% [790800/4942000] [160.0/1000.0] [batch_t 0.758 (0.763)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-07 17:52:15,649 - Train: 16.00% [790900/4942000] [160.0/1000.0] [batch_t 0.762 (0.763)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-07 17:53:31,974 - Train: 16.01% [791000/4942000] [160.1/1000.0] [batch_t 0.768 (0.763)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 17:54:48,307 - Train: 16.01% [791100/4942000] [160.1/1000.0] [batch_t 0.766 (0.763)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-07 17:56:04,750 - Train: 16.01% [791200/4942000] [160.1/1000.0] [batch_t 0.762 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 17:57:21,089 - Train: 16.01% [791300/4942000] [160.1/1000.0] [batch_t 0.748 (0.763)] [data_t 0.003] [optim_t 0.745] [lr 0.005000] 2024-04-07 17:58:37,335 - Train: 16.01% [791400/4942000] [160.1/1000.0] [batch_t 0.762 (0.762)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-07 17:59:53,582 - Train: 16.02% [791500/4942000] [160.2/1000.0] [batch_t 0.758 (0.762)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-07 18:01:09,824 - Train: 16.02% [791600/4942000] [160.2/1000.0] [batch_t 0.753 (0.762)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-07 18:02:26,287 - Train: 16.02% [791700/4942000] [160.2/1000.0] [batch_t 0.752 (0.765)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-07 18:03:42,517 - Train: 16.02% [791800/4942000] [160.2/1000.0] [batch_t 0.767 (0.762)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 18:04:58,797 - Train: 16.02% [791900/4942000] [160.2/1000.0] [batch_t 0.764 (0.763)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-07 18:06:15,269 - Train: 16.03% [792000/4942000] [160.3/1000.0] [batch_t 0.763 (0.765)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-07 18:07:31,763 - Train: 16.03% [792100/4942000] [160.3/1000.0] [batch_t 0.758 (0.765)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-07 18:08:48,040 - Train: 16.03% [792200/4942000] [160.3/1000.0] [batch_t 0.764 (0.763)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-07 18:10:04,356 - Train: 16.03% [792300/4942000] [160.3/1000.0] [batch_t 0.770 (0.763)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-07 18:11:20,671 - Train: 16.03% [792400/4942000] [160.3/1000.0] [batch_t 0.750 (0.763)] [data_t 0.003] [optim_t 0.747] [lr 0.005000] 2024-04-07 18:12:37,092 - Train: 16.04% [792500/4942000] [160.4/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 18:13:53,277 - Train: 16.04% [792600/4942000] [160.4/1000.0] [batch_t 0.767 (0.762)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 18:15:09,531 - Train: 16.04% [792700/4942000] [160.4/1000.0] [batch_t 0.767 (0.762)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 18:16:25,771 - Train: 16.04% [792800/4942000] [160.4/1000.0] [batch_t 0.760 (0.762)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-07 18:17:42,030 - Train: 16.04% [792900/4942000] [160.4/1000.0] [batch_t 0.759 (0.763)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-07 18:18:58,269 - Train: 16.05% [793000/4942000] [160.5/1000.0] [batch_t 0.756 (0.762)] [data_t 0.003] [optim_t 0.753] [lr 0.005000] 2024-04-07 18:20:14,454 - Train: 16.05% [793100/4942000] [160.5/1000.0] [batch_t 0.750 (0.762)] [data_t 0.003] [optim_t 0.747] [lr 0.005000] 2024-04-07 18:21:30,746 - Train: 16.05% [793200/4942000] [160.5/1000.0] [batch_t 0.772 (0.763)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-07 18:22:47,175 - Train: 16.05% [793300/4942000] [160.5/1000.0] [batch_t 0.760 (0.764)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-07 18:24:03,445 - Train: 16.05% [793400/4942000] [160.5/1000.0] [batch_t 0.762 (0.763)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-07 18:25:19,755 - Train: 16.06% [793500/4942000] [160.6/1000.0] [batch_t 0.754 (0.763)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-07 18:26:36,199 - Train: 16.06% [793600/4942000] [160.6/1000.0] [batch_t 0.760 (0.764)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-07 18:27:52,622 - Train: 16.06% [793700/4942000] [160.6/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-07 18:29:08,822 - Train: 16.06% [793800/4942000] [160.6/1000.0] [batch_t 0.754 (0.762)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-07 18:30:25,120 - Train: 16.06% [793900/4942000] [160.6/1000.0] [batch_t 0.763 (0.763)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-07 18:31:41,303 - Train: 16.07% [794000/4942000] [160.7/1000.0] [batch_t 0.753 (0.762)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-07 18:32:39,773 - Train: 16.07% [794100/4942000] [160.7/1000.0] [batch_t 0.321 (0.585)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-07 18:33:12,530 - Train: 16.07% [794200/4942000] [160.7/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 18:33:45,232 - Train: 16.07% [794300/4942000] [160.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 18:34:17,865 - Train: 16.07% [794400/4942000] [160.7/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 18:34:50,520 - Train: 16.08% [794500/4942000] [160.8/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 18:35:23,411 - Train: 16.08% [794600/4942000] [160.8/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 18:35:56,108 - Train: 16.08% [794700/4942000] [160.8/1000.0] [batch_t 0.320 (0.327)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-07 18:36:28,850 - Train: 16.08% [794800/4942000] [160.8/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 18:37:01,567 - Train: 16.08% [794900/4942000] [160.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 18:37:34,272 - Train: 16.09% [795000/4942000] [160.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 18:38:07,149 - Train: 16.09% [795100/4942000] [160.9/1000.0] [batch_t 0.321 (0.329)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-07 18:38:39,799 - Train: 16.09% [795200/4942000] [160.9/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 18:39:15,095 - Train: 16.09% [795300/4942000] [160.9/1000.0] [batch_t 0.326 (0.353)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 18:39:47,769 - Train: 16.09% [795400/4942000] [160.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 18:40:20,424 - Train: 16.10% [795500/4942000] [161.0/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 18:40:53,173 - Train: 16.10% [795600/4942000] [161.0/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 18:41:13,600 - ==> Total time: 5 days, 0:43:52 Eta: 26 days, 5:09:09 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-07 18:41:28,183 - Train: 16.10% [795700/4942000] [161.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 18:42:00,858 - Train: 16.10% [795800/4942000] [161.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 18:42:33,635 - Train: 16.10% [795900/4942000] [161.0/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 18:43:06,373 - Train: 16.11% [796000/4942000] [161.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 18:43:38,959 - Train: 16.11% [796100/4942000] [161.1/1000.0] [batch_t 0.321 (0.326)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-07 18:44:11,610 - Train: 16.11% [796200/4942000] [161.1/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 18:44:44,255 - Train: 16.11% [796300/4942000] [161.1/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 18:45:16,880 - Train: 16.11% [796400/4942000] [161.1/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 18:45:49,490 - Train: 16.12% [796500/4942000] [161.2/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 18:46:22,127 - Train: 16.12% [796600/4942000] [161.2/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 18:46:54,781 - Train: 16.12% [796700/4942000] [161.2/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 18:47:27,467 - Train: 16.12% [796800/4942000] [161.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 18:48:00,240 - Train: 16.13% [796900/4942000] [161.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 18:48:32,976 - Train: 16.13% [797000/4942000] [161.3/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 18:49:05,649 - Train: 16.13% [797100/4942000] [161.3/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 18:49:38,316 - Train: 16.13% [797200/4942000] [161.3/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 18:50:11,176 - Train: 16.13% [797300/4942000] [161.3/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 18:50:43,890 - Train: 16.14% [797400/4942000] [161.4/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 18:51:16,565 - Train: 16.14% [797500/4942000] [161.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 18:51:49,234 - Train: 16.14% [797600/4942000] [161.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 18:52:21,959 - Train: 16.14% [797700/4942000] [161.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 18:52:54,653 - Train: 16.14% [797800/4942000] [161.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 18:53:27,385 - Train: 16.15% [797900/4942000] [161.5/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-07 18:54:00,085 - Train: 16.15% [798000/4942000] [161.5/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 18:54:32,788 - Train: 16.15% [798100/4942000] [161.5/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 18:55:05,524 - Train: 16.15% [798200/4942000] [161.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 18:55:38,357 - Train: 16.15% [798300/4942000] [161.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 18:56:11,087 - Train: 16.16% [798400/4942000] [161.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 18:56:43,737 - Train: 16.16% [798500/4942000] [161.6/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 18:57:16,493 - Train: 16.16% [798600/4942000] [161.6/1000.0] [batch_t 0.321 (0.327)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-07 18:57:49,414 - Train: 16.16% [798700/4942000] [161.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 18:58:22,050 - Train: 16.16% [798800/4942000] [161.6/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 18:58:54,654 - Train: 16.17% [798900/4942000] [161.7/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 18:59:27,296 - Train: 16.17% [799000/4942000] [161.7/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 18:59:59,950 - Train: 16.17% [799100/4942000] [161.7/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 19:00:32,552 - Train: 16.17% [799200/4942000] [161.7/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 19:01:05,202 - Train: 16.17% [799300/4942000] [161.7/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:01:37,914 - Train: 16.18% [799400/4942000] [161.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:02:10,608 - Train: 16.18% [799500/4942000] [161.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:02:43,301 - Train: 16.18% [799600/4942000] [161.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 19:03:16,028 - Train: 16.18% [799700/4942000] [161.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:03:48,744 - Train: 16.18% [799800/4942000] [161.8/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 19:04:21,470 - Train: 16.19% [799900/4942000] [161.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:04:54,191 - Train: 16.19% [800000/4942000] [161.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:05:26,898 - Train: 16.19% [800100/4942000] [161.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 19:05:59,792 - Train: 16.19% [800200/4942000] [161.9/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 19:06:32,507 - Train: 16.19% [800300/4942000] [161.9/1000.0] [batch_t 0.319 (0.327)] [data_t 0.002] [optim_t 0.317] [lr 0.005000] 2024-04-07 19:07:05,205 - Train: 16.20% [800400/4942000] [162.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 19:07:37,929 - Train: 16.20% [800500/4942000] [162.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:08:10,754 - Train: 16.20% [800600/4942000] [162.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:08:12,062 - ==> Total time: 5 days, 1:10:51 Eta: 26 days, 2:50:57 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-07 19:08:45,447 - Train: 16.20% [800700/4942000] [162.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:09:18,140 - Train: 16.20% [800800/4942000] [162.0/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 19:09:50,813 - Train: 16.21% [800900/4942000] [162.1/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 19:10:23,470 - Train: 16.21% [801000/4942000] [162.1/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 19:10:56,140 - Train: 16.21% [801100/4942000] [162.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 19:11:28,806 - Train: 16.21% [801200/4942000] [162.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:12:01,502 - Train: 16.21% [801300/4942000] [162.1/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 19:12:34,216 - Train: 16.22% [801400/4942000] [162.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 19:13:07,077 - Train: 16.22% [801500/4942000] [162.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:13:39,785 - Train: 16.22% [801600/4942000] [162.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:14:12,483 - Train: 16.22% [801700/4942000] [162.2/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 19:14:45,171 - Train: 16.22% [801800/4942000] [162.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 19:15:17,949 - Train: 16.23% [801900/4942000] [162.3/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 19:15:50,635 - Train: 16.23% [802000/4942000] [162.3/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 19:16:23,337 - Train: 16.23% [802100/4942000] [162.3/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 19:16:56,011 - Train: 16.23% [802200/4942000] [162.3/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 19:17:28,663 - Train: 16.23% [802300/4942000] [162.3/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 19:18:01,376 - Train: 16.24% [802400/4942000] [162.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 19:18:34,031 - Train: 16.24% [802500/4942000] [162.4/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:19:06,684 - Train: 16.24% [802600/4942000] [162.4/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:19:39,404 - Train: 16.24% [802700/4942000] [162.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 19:20:12,031 - Train: 16.24% [802800/4942000] [162.4/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:20:44,780 - Train: 16.25% [802900/4942000] [162.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 19:21:17,439 - Train: 16.25% [803000/4942000] [162.5/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-07 19:21:50,200 - Train: 16.25% [803100/4942000] [162.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:22:22,969 - Train: 16.25% [803200/4942000] [162.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 19:22:55,812 - Train: 16.25% [803300/4942000] [162.5/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 19:23:29,212 - Train: 16.26% [803400/4942000] [162.6/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 19:24:01,942 - Train: 16.26% [803500/4942000] [162.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 19:24:35,423 - Train: 16.26% [803600/4942000] [162.6/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:25:08,269 - Train: 16.26% [803700/4942000] [162.6/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 19:25:41,013 - Train: 16.26% [803800/4942000] [162.6/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 19:26:13,807 - Train: 16.27% [803900/4942000] [162.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 19:26:46,591 - Train: 16.27% [804000/4942000] [162.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 19:27:19,339 - Train: 16.27% [804100/4942000] [162.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:27:52,130 - Train: 16.27% [804200/4942000] [162.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 19:28:26,392 - Train: 16.27% [804300/4942000] [162.7/1000.0] [batch_t 0.326 (0.343)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 19:28:59,146 - Train: 16.28% [804400/4942000] [162.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 19:29:31,909 - Train: 16.28% [804500/4942000] [162.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 19:30:04,607 - Train: 16.28% [804600/4942000] [162.8/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 19:30:37,270 - Train: 16.28% [804700/4942000] [162.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:31:10,021 - Train: 16.28% [804800/4942000] [162.8/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 19:31:42,699 - Train: 16.29% [804900/4942000] [162.9/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 19:32:17,056 - Train: 16.29% [805000/4942000] [162.9/1000.0] [batch_t 0.328 (0.343)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:32:49,781 - Train: 16.29% [805100/4942000] [162.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 19:33:24,014 - Train: 16.29% [805200/4942000] [162.9/1000.0] [batch_t 0.332 (0.342)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 19:33:56,841 - Train: 16.30% [805300/4942000] [163.0/1000.0] [batch_t 0.321 (0.328)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-07 19:34:30,490 - Train: 16.30% [805400/4942000] [163.0/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:35:03,255 - Train: 16.30% [805500/4942000] [163.0/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 19:35:18,297 - ==> Total time: 5 days, 1:37:57 Eta: 26 days, 0:34:47 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-07 19:35:38,192 - Train: 16.30% [805600/4942000] [163.0/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 19:36:12,762 - Train: 16.30% [805700/4942000] [163.0/1000.0] [batch_t 0.328 (0.346)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:36:45,410 - Train: 16.31% [805800/4942000] [163.1/1000.0] [batch_t 0.331 (0.326)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 19:37:19,327 - Train: 16.31% [805900/4942000] [163.1/1000.0] [batch_t 0.324 (0.339)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 19:37:51,955 - Train: 16.31% [806000/4942000] [163.1/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 19:38:24,633 - Train: 16.31% [806100/4942000] [163.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 19:38:57,318 - Train: 16.31% [806200/4942000] [163.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:39:29,931 - Train: 16.32% [806300/4942000] [163.2/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 19:40:02,539 - Train: 16.32% [806400/4942000] [163.2/1000.0] [batch_t 0.325 (0.326)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-07 19:40:35,183 - Train: 16.32% [806500/4942000] [163.2/1000.0] [batch_t 0.321 (0.326)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-07 19:41:07,877 - Train: 16.32% [806600/4942000] [163.2/1000.0] [batch_t 0.339 (0.327)] [data_t 0.002] [optim_t 0.337] [lr 0.005000] 2024-04-07 19:41:40,493 - Train: 16.32% [806700/4942000] [163.2/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 19:42:13,191 - Train: 16.33% [806800/4942000] [163.3/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 19:42:45,880 - Train: 16.33% [806900/4942000] [163.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:43:18,561 - Train: 16.33% [807000/4942000] [163.3/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 19:43:51,261 - Train: 16.33% [807100/4942000] [163.3/1000.0] [batch_t 0.324 (0.327)] [data_t 0.003] [optim_t 0.321] [lr 0.005000] 2024-04-07 19:44:23,955 - Train: 16.33% [807200/4942000] [163.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 19:44:56,669 - Train: 16.34% [807300/4942000] [163.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:45:29,393 - Train: 16.34% [807400/4942000] [163.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 19:46:02,162 - Train: 16.34% [807500/4942000] [163.4/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 19:46:34,886 - Train: 16.34% [807600/4942000] [163.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:47:07,612 - Train: 16.34% [807700/4942000] [163.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 19:47:40,325 - Train: 16.35% [807800/4942000] [163.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:48:13,042 - Train: 16.35% [807900/4942000] [163.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:48:45,745 - Train: 16.35% [808000/4942000] [163.5/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 19:49:18,362 - Train: 16.35% [808100/4942000] [163.5/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 19:49:51,034 - Train: 16.35% [808200/4942000] [163.5/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 19:50:23,939 - Train: 16.36% [808300/4942000] [163.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:50:56,570 - Train: 16.36% [808400/4942000] [163.6/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 19:51:29,448 - Train: 16.36% [808500/4942000] [163.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:52:02,207 - Train: 16.36% [808600/4942000] [163.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 19:52:36,880 - Train: 16.36% [808700/4942000] [163.6/1000.0] [batch_t 0.327 (0.347)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 19:53:11,280 - Train: 16.37% [808800/4942000] [163.7/1000.0] [batch_t 0.328 (0.344)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:53:44,007 - Train: 16.37% [808900/4942000] [163.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:54:16,908 - Train: 16.37% [809000/4942000] [163.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:54:49,661 - Train: 16.37% [809100/4942000] [163.7/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 19:55:22,877 - Train: 16.37% [809200/4942000] [163.7/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 19:55:55,552 - Train: 16.38% [809300/4942000] [163.8/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 19:56:30,637 - Train: 16.38% [809400/4942000] [163.8/1000.0] [batch_t 0.329 (0.351)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:57:03,390 - Train: 16.38% [809500/4942000] [163.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:57:37,046 - Train: 16.38% [809600/4942000] [163.8/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:58:12,905 - Train: 16.38% [809700/4942000] [163.8/1000.0] [batch_t 0.328 (0.359)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:58:45,545 - Train: 16.39% [809800/4942000] [163.9/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 19:59:20,518 - Train: 16.39% [809900/4942000] [163.9/1000.0] [batch_t 0.322 (0.350)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-07 19:59:56,995 - Train: 16.39% [810000/4942000] [163.9/1000.0] [batch_t 0.330 (0.365)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 20:00:31,655 - Train: 16.39% [810100/4942000] [163.9/1000.0] [batch_t 0.332 (0.347)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 20:01:04,358 - Train: 16.39% [810200/4942000] [163.9/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 20:01:40,549 - Train: 16.40% [810300/4942000] [164.0/1000.0] [batch_t 0.327 (0.362)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:02:14,259 - Train: 16.40% [810400/4942000] [164.0/1000.0] [batch_t 0.326 (0.337)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 20:02:43,065 - ==> Total time: 5 days, 2:05:22 Eta: 25 days, 22:21:31 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-07 20:02:49,143 - Train: 16.40% [810500/4942000] [164.0/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 20:03:24,174 - Train: 16.40% [810600/4942000] [164.0/1000.0] [batch_t 0.328 (0.350)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:04:00,252 - Train: 16.40% [810700/4942000] [164.0/1000.0] [batch_t 0.330 (0.361)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 20:04:34,408 - Train: 16.41% [810800/4942000] [164.1/1000.0] [batch_t 0.322 (0.341)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 20:05:07,053 - Train: 16.41% [810900/4942000] [164.1/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 20:05:39,738 - Train: 16.41% [811000/4942000] [164.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:06:14,368 - Train: 16.41% [811100/4942000] [164.1/1000.0] [batch_t 0.325 (0.346)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 20:06:47,076 - Train: 16.41% [811200/4942000] [164.1/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 20:07:28,754 - Train: 16.42% [811300/4942000] [164.2/1000.0] [batch_t 0.323 (0.417)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 20:08:01,438 - Train: 16.42% [811400/4942000] [164.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 20:08:57,387 - Train: 16.42% [811500/4942000] [164.2/1000.0] [batch_t 0.325 (0.559)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 20:09:51,772 - Train: 16.42% [811600/4942000] [164.2/1000.0] [batch_t 0.326 (0.544)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 20:10:27,309 - Train: 16.42% [811700/4942000] [164.2/1000.0] [batch_t 0.328 (0.355)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:10:59,987 - Train: 16.43% [811800/4942000] [164.3/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-07 20:11:32,832 - Train: 16.43% [811900/4942000] [164.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:12:05,454 - Train: 16.43% [812000/4942000] [164.3/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:12:38,117 - Train: 16.43% [812100/4942000] [164.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:13:13,015 - Train: 16.43% [812200/4942000] [164.3/1000.0] [batch_t 0.328 (0.349)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:13:45,682 - Train: 16.44% [812300/4942000] [164.4/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 20:14:18,834 - Train: 16.44% [812400/4942000] [164.4/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:14:51,524 - Train: 16.44% [812500/4942000] [164.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:15:26,123 - Train: 16.44% [812600/4942000] [164.4/1000.0] [batch_t 0.323 (0.346)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 20:15:58,848 - Train: 16.44% [812700/4942000] [164.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 20:16:33,258 - Train: 16.45% [812800/4942000] [164.5/1000.0] [batch_t 0.327 (0.344)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 20:17:07,891 - Train: 16.45% [812900/4942000] [164.5/1000.0] [batch_t 0.324 (0.346)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 20:17:40,577 - Train: 16.45% [813000/4942000] [164.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:18:13,264 - Train: 16.45% [813100/4942000] [164.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:18:45,888 - Train: 16.45% [813200/4942000] [164.5/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 20:19:18,722 - Train: 16.46% [813300/4942000] [164.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:19:51,470 - Train: 16.46% [813400/4942000] [164.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:20:26,487 - Train: 16.46% [813500/4942000] [164.6/1000.0] [batch_t 0.327 (0.350)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 20:20:59,227 - Train: 16.46% [813600/4942000] [164.6/1000.0] [batch_t 0.320 (0.327)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-07 20:21:33,773 - Train: 16.46% [813700/4942000] [164.6/1000.0] [batch_t 0.328 (0.345)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:22:09,116 - Train: 16.47% [813800/4942000] [164.7/1000.0] [batch_t 0.334 (0.353)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 20:22:41,910 - Train: 16.47% [813900/4942000] [164.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 20:23:17,475 - Train: 16.47% [814000/4942000] [164.7/1000.0] [batch_t 0.323 (0.356)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 20:23:50,207 - Train: 16.47% [814100/4942000] [164.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 20:24:23,338 - Train: 16.48% [814200/4942000] [164.8/1000.0] [batch_t 0.325 (0.331)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 20:24:56,085 - Train: 16.48% [814300/4942000] [164.8/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 20:25:31,852 - Train: 16.48% [814400/4942000] [164.8/1000.0] [batch_t 0.326 (0.358)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 20:26:07,430 - Train: 16.48% [814500/4942000] [164.8/1000.0] [batch_t 0.588 (0.356)] [data_t 0.260] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:26:41,060 - Train: 16.48% [814600/4942000] [164.8/1000.0] [batch_t 0.323 (0.336)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 20:27:15,114 - Train: 16.49% [814700/4942000] [164.9/1000.0] [batch_t 0.326 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 20:27:47,803 - Train: 16.49% [814800/4942000] [164.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:28:23,478 - Train: 16.49% [814900/4942000] [164.9/1000.0] [batch_t 0.328 (0.357)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:28:56,211 - Train: 16.49% [815000/4942000] [164.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:29:32,135 - Train: 16.49% [815100/4942000] [164.9/1000.0] [batch_t 0.325 (0.359)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 20:30:05,013 - Train: 16.50% [815200/4942000] [165.0/1000.0] [batch_t 0.540 (0.329)] [data_t 0.217] [optim_t 0.323] [lr 0.005000] 2024-04-07 20:30:39,089 - Train: 16.50% [815300/4942000] [165.0/1000.0] [batch_t 0.331 (0.341)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 20:31:12,686 - Train: 16.50% [815400/4942000] [165.0/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 20:31:22,503 - ==> Total time: 5 days, 2:34:01 Eta: 25 days, 20:15:50 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-07 20:31:47,554 - Train: 16.50% [815500/4942000] [165.0/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:32:21,721 - Train: 16.50% [815600/4942000] [165.0/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:32:57,778 - Train: 16.51% [815700/4942000] [165.1/1000.0] [batch_t 0.328 (0.360)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:33:31,726 - Train: 16.51% [815800/4942000] [165.1/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 20:34:05,803 - Train: 16.51% [815900/4942000] [165.1/1000.0] [batch_t 1.749 (0.341)] [data_t 1.422] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:34:38,555 - Train: 16.51% [816000/4942000] [165.1/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 20:35:11,363 - Train: 16.51% [816100/4942000] [165.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 20:35:44,073 - Train: 16.52% [816200/4942000] [165.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:36:16,790 - Train: 16.52% [816300/4942000] [165.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:36:49,530 - Train: 16.52% [816400/4942000] [165.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 20:37:22,546 - Train: 16.52% [816500/4942000] [165.2/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 20:37:55,296 - Train: 16.52% [816600/4942000] [165.2/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 20:38:28,163 - Train: 16.53% [816700/4942000] [165.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 20:39:00,912 - Train: 16.53% [816800/4942000] [165.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:39:34,727 - Train: 16.53% [816900/4942000] [165.3/1000.0] [batch_t 0.325 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 20:40:07,452 - Train: 16.53% [817000/4942000] [165.3/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 20:40:40,176 - Train: 16.53% [817100/4942000] [165.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:41:13,488 - Train: 16.54% [817200/4942000] [165.4/1000.0] [batch_t 0.326 (0.333)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 20:41:46,181 - Train: 16.54% [817300/4942000] [165.4/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 20:42:19,261 - Train: 16.54% [817400/4942000] [165.4/1000.0] [batch_t 0.324 (0.331)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 20:42:52,037 - Train: 16.54% [817500/4942000] [165.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 20:43:24,901 - Train: 16.54% [817600/4942000] [165.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:43:57,623 - Train: 16.55% [817700/4942000] [165.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:44:32,038 - Train: 16.55% [817800/4942000] [165.5/1000.0] [batch_t 0.324 (0.344)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 20:45:04,880 - Train: 16.55% [817900/4942000] [165.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:45:37,569 - Train: 16.55% [818000/4942000] [165.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:46:10,468 - Train: 16.55% [818100/4942000] [165.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:46:43,171 - Train: 16.56% [818200/4942000] [165.6/1000.0] [batch_t 0.337 (0.327)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-07 20:47:16,815 - Train: 16.56% [818300/4942000] [165.6/1000.0] [batch_t 0.325 (0.336)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 20:47:49,663 - Train: 16.56% [818400/4942000] [165.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:48:23,395 - Train: 16.56% [818500/4942000] [165.6/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 20:48:56,064 - Train: 16.56% [818600/4942000] [165.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:49:28,845 - Train: 16.57% [818700/4942000] [165.7/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 20:50:01,545 - Train: 16.57% [818800/4942000] [165.7/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 20:50:35,433 - Train: 16.57% [818900/4942000] [165.7/1000.0] [batch_t 0.325 (0.339)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 20:51:09,737 - Train: 16.57% [819000/4942000] [165.7/1000.0] [batch_t 0.322 (0.343)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-07 20:51:42,413 - Train: 16.57% [819100/4942000] [165.7/1000.0] [batch_t 0.321 (0.327)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-07 20:52:15,737 - Train: 16.58% [819200/4942000] [165.8/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 20:52:48,404 - Train: 16.58% [819300/4942000] [165.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 20:53:22,711 - Train: 16.58% [819400/4942000] [165.8/1000.0] [batch_t 0.328 (0.343)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:53:55,387 - Train: 16.58% [819500/4942000] [165.8/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 20:54:29,708 - Train: 16.58% [819600/4942000] [165.8/1000.0] [batch_t 0.329 (0.343)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:55:02,369 - Train: 16.59% [819700/4942000] [165.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 20:55:37,033 - Train: 16.59% [819800/4942000] [165.9/1000.0] [batch_t 0.327 (0.347)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 20:56:12,558 - Train: 16.59% [819900/4942000] [165.9/1000.0] [batch_t 0.328 (0.355)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:56:45,309 - Train: 16.59% [820000/4942000] [165.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 20:57:19,765 - Train: 16.59% [820100/4942000] [165.9/1000.0] [batch_t 0.323 (0.344)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 20:57:52,521 - Train: 16.60% [820200/4942000] [166.0/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 20:58:27,678 - Train: 16.60% [820300/4942000] [166.0/1000.0] [batch_t 0.320 (0.351)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-07 20:58:51,340 - ==> Total time: 5 days, 3:01:30 Eta: 25 days, 18:05:24 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-07 20:59:03,047 - Train: 16.60% [820400/4942000] [166.0/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:59:38,195 - Train: 16.60% [820500/4942000] [166.0/1000.0] [batch_t 0.326 (0.351)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 21:00:12,918 - Train: 16.60% [820600/4942000] [166.0/1000.0] [batch_t 0.324 (0.347)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 21:00:45,708 - Train: 16.61% [820700/4942000] [166.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 21:01:19,965 - Train: 16.61% [820800/4942000] [166.1/1000.0] [batch_t 0.326 (0.342)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 21:01:52,734 - Train: 16.61% [820900/4942000] [166.1/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 21:02:25,566 - Train: 16.61% [821000/4942000] [166.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 21:02:58,508 - Train: 16.61% [821100/4942000] [166.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 21:03:32,415 - Train: 16.62% [821200/4942000] [166.2/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 21:04:06,955 - Train: 16.62% [821300/4942000] [166.2/1000.0] [batch_t 2.128 (0.345)] [data_t 1.796] [optim_t 0.332] [lr 0.005000] 2024-04-07 21:04:39,949 - Train: 16.62% [821400/4942000] [166.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 21:05:14,269 - Train: 16.62% [821500/4942000] [166.2/1000.0] [batch_t 0.337 (0.343)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-07 21:05:47,234 - Train: 16.62% [821600/4942000] [166.2/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 21:06:21,725 - Train: 16.63% [821700/4942000] [166.3/1000.0] [batch_t 0.331 (0.345)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 21:06:54,665 - Train: 16.63% [821800/4942000] [166.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 21:07:28,775 - Train: 16.63% [821900/4942000] [166.3/1000.0] [batch_t 0.325 (0.341)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 21:08:01,599 - Train: 16.63% [822000/4942000] [166.3/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 21:08:36,805 - Train: 16.63% [822100/4942000] [166.3/1000.0] [batch_t 0.329 (0.352)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 21:09:09,748 - Train: 16.64% [822200/4942000] [166.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 21:09:42,687 - Train: 16.64% [822300/4942000] [166.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 21:10:17,227 - Train: 16.64% [822400/4942000] [166.4/1000.0] [batch_t 0.330 (0.345)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 21:10:50,117 - Train: 16.64% [822500/4942000] [166.4/1000.0] [batch_t 0.321 (0.329)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-07 21:11:25,526 - Train: 16.65% [822600/4942000] [166.5/1000.0] [batch_t 0.340 (0.354)] [data_t 0.002] [optim_t 0.338] [lr 0.005000] 2024-04-07 21:11:58,435 - Train: 16.65% [822700/4942000] [166.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 21:12:32,251 - Train: 16.65% [822800/4942000] [166.5/1000.0] [batch_t 0.325 (0.338)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 21:13:06,049 - Train: 16.65% [822900/4942000] [166.5/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 21:13:38,834 - Train: 16.65% [823000/4942000] [166.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 21:14:11,580 - Train: 16.66% [823100/4942000] [166.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 21:14:44,348 - Train: 16.66% [823200/4942000] [166.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 21:15:18,601 - Train: 16.66% [823300/4942000] [166.6/1000.0] [batch_t 0.332 (0.342)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 21:15:51,388 - Train: 16.66% [823400/4942000] [166.6/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 21:16:24,220 - Train: 16.66% [823500/4942000] [166.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 21:16:57,200 - Train: 16.67% [823600/4942000] [166.7/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 21:17:30,754 - Train: 16.67% [823700/4942000] [166.7/1000.0] [batch_t 0.325 (0.335)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 21:18:03,639 - Train: 16.67% [823800/4942000] [166.7/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 21:18:36,567 - Train: 16.67% [823900/4942000] [166.7/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 21:19:09,549 - Train: 16.67% [824000/4942000] [166.7/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 21:19:42,383 - Train: 16.68% [824100/4942000] [166.8/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 21:20:15,295 - Train: 16.68% [824200/4942000] [166.8/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 21:20:48,150 - Train: 16.68% [824300/4942000] [166.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 21:21:22,230 - Train: 16.68% [824400/4942000] [166.8/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 21:21:55,091 - Train: 16.68% [824500/4942000] [166.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 21:22:28,868 - Train: 16.69% [824600/4942000] [166.9/1000.0] [batch_t 0.339 (0.338)] [data_t 0.002] [optim_t 0.337] [lr 0.005000] 2024-04-07 21:23:01,816 - Train: 16.69% [824700/4942000] [166.9/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 21:23:34,780 - Train: 16.69% [824800/4942000] [166.9/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 21:24:07,558 - Train: 16.69% [824900/4942000] [166.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 21:24:40,337 - Train: 16.69% [825000/4942000] [166.9/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 21:25:13,657 - Train: 16.70% [825100/4942000] [167.0/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 21:25:46,465 - Train: 16.70% [825200/4942000] [167.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 21:26:19,277 - Train: 16.70% [825300/4942000] [167.0/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 21:26:23,853 - ==> Total time: 5 days, 3:29:03 Eta: 25 days, 15:56:31 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-07 21:26:54,187 - Train: 16.70% [825400/4942000] [167.0/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 21:27:26,941 - Train: 16.70% [825500/4942000] [167.0/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 21:27:59,624 - Train: 16.71% [825600/4942000] [167.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 21:28:32,312 - Train: 16.71% [825700/4942000] [167.1/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 21:29:06,312 - Train: 16.71% [825800/4942000] [167.1/1000.0] [batch_t 1.597 (0.340)] [data_t 1.270] [optim_t 0.327] [lr 0.005000] 2024-04-07 21:29:39,065 - Train: 16.71% [825900/4942000] [167.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 21:30:11,818 - Train: 16.71% [826000/4942000] [167.1/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 21:30:44,541 - Train: 16.72% [826100/4942000] [167.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 21:31:17,265 - Train: 16.72% [826200/4942000] [167.2/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 21:31:49,961 - Train: 16.72% [826300/4942000] [167.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 21:32:24,263 - Train: 16.72% [826400/4942000] [167.2/1000.0] [batch_t 0.329 (0.343)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 21:32:57,040 - Train: 16.72% [826500/4942000] [167.2/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 21:33:29,852 - Train: 16.73% [826600/4942000] [167.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 21:34:02,614 - Train: 16.73% [826700/4942000] [167.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 21:34:35,546 - Train: 16.73% [826800/4942000] [167.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 21:35:08,319 - Train: 16.73% [826900/4942000] [167.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 21:35:41,103 - Train: 16.73% [827000/4942000] [167.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 21:36:13,832 - Train: 16.74% [827100/4942000] [167.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 21:36:46,607 - Train: 16.74% [827200/4942000] [167.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 21:37:19,986 - Train: 16.74% [827300/4942000] [167.4/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 21:37:52,643 - Train: 16.74% [827400/4942000] [167.4/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 21:38:25,411 - Train: 16.74% [827500/4942000] [167.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 21:38:58,150 - Train: 16.75% [827600/4942000] [167.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 21:39:31,835 - Train: 16.75% [827700/4942000] [167.5/1000.0] [batch_t 0.326 (0.337)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 21:40:04,568 - Train: 16.75% [827800/4942000] [167.5/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 21:40:38,195 - Train: 16.75% [827900/4942000] [167.5/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 21:41:10,904 - Train: 16.75% [828000/4942000] [167.5/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 21:41:43,602 - Train: 16.76% [828100/4942000] [167.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 21:42:22,567 - Train: 16.76% [828200/4942000] [167.6/1000.0] [batch_t 0.327 (0.390)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 21:42:55,415 - Train: 16.76% [828300/4942000] [167.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 21:43:28,178 - Train: 16.76% [828400/4942000] [167.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 21:44:03,386 - Train: 16.76% [828500/4942000] [167.6/1000.0] [batch_t 0.323 (0.352)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 21:44:36,886 - Train: 16.77% [828600/4942000] [167.7/1000.0] [batch_t 0.317 (0.335)] [data_t 0.002] [optim_t 0.315] [lr 0.005000] 2024-04-07 21:45:09,636 - Train: 16.77% [828700/4942000] [167.7/1000.0] [batch_t 0.334 (0.327)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 21:45:42,424 - Train: 16.77% [828800/4942000] [167.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 21:46:15,182 - Train: 16.77% [828900/4942000] [167.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 21:46:47,941 - Train: 16.77% [829000/4942000] [167.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 21:47:24,928 - Train: 16.78% [829100/4942000] [167.8/1000.0] [batch_t 0.330 (0.370)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 21:47:57,680 - Train: 16.78% [829200/4942000] [167.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 21:48:30,512 - Train: 16.78% [829300/4942000] [167.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 21:49:03,303 - Train: 16.78% [829400/4942000] [167.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 21:49:36,053 - Train: 16.78% [829500/4942000] [167.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 21:50:08,853 - Train: 16.79% [829600/4942000] [167.9/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-07 21:50:41,906 - Train: 16.79% [829700/4942000] [167.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 21:51:14,780 - Train: 16.79% [829800/4942000] [167.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 21:51:47,526 - Train: 16.79% [829900/4942000] [167.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 21:52:20,308 - Train: 16.79% [830000/4942000] [167.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 21:52:53,120 - Train: 16.80% [830100/4942000] [168.0/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 21:53:25,859 - Train: 16.80% [830200/4942000] [168.0/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 21:53:44,254 - ==> Total time: 5 days, 3:56:23 Eta: 25 days, 13:47:50 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-07 21:54:00,993 - Train: 16.80% [830300/4942000] [168.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 21:54:33,778 - Train: 16.80% [830400/4942000] [168.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 21:55:06,534 - Train: 16.80% [830500/4942000] [168.0/1000.0] [batch_t 0.334 (0.327)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 21:55:39,318 - Train: 16.81% [830600/4942000] [168.1/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 21:56:12,064 - Train: 16.81% [830700/4942000] [168.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 21:56:44,752 - Train: 16.81% [830800/4942000] [168.1/1000.0] [batch_t 0.324 (0.327)] [data_t 0.003] [optim_t 0.321] [lr 0.005000] 2024-04-07 21:57:17,556 - Train: 16.81% [830900/4942000] [168.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 21:57:50,663 - Train: 16.82% [831000/4942000] [168.2/1000.0] [batch_t 0.336 (0.331)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-07 21:58:23,484 - Train: 16.82% [831100/4942000] [168.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 21:58:56,226 - Train: 16.82% [831200/4942000] [168.2/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 21:59:29,055 - Train: 16.82% [831300/4942000] [168.2/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 22:00:01,803 - Train: 16.82% [831400/4942000] [168.2/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 22:00:34,550 - Train: 16.83% [831500/4942000] [168.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 22:01:07,306 - Train: 16.83% [831600/4942000] [168.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:01:40,193 - Train: 16.83% [831700/4942000] [168.3/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 22:02:13,023 - Train: 16.83% [831800/4942000] [168.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 22:02:45,960 - Train: 16.83% [831900/4942000] [168.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:03:18,854 - Train: 16.84% [832000/4942000] [168.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 22:03:51,713 - Train: 16.84% [832100/4942000] [168.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:04:24,639 - Train: 16.84% [832200/4942000] [168.4/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 22:04:58,091 - Train: 16.84% [832300/4942000] [168.4/1000.0] [batch_t 0.340 (0.334)] [data_t 0.002] [optim_t 0.338] [lr 0.005000] 2024-04-07 22:05:30,909 - Train: 16.84% [832400/4942000] [168.4/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 22:06:03,707 - Train: 16.85% [832500/4942000] [168.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:06:36,434 - Train: 16.85% [832600/4942000] [168.5/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 22:07:09,246 - Train: 16.85% [832700/4942000] [168.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 22:07:41,979 - Train: 16.85% [832800/4942000] [168.5/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 22:08:14,666 - Train: 16.85% [832900/4942000] [168.5/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 22:08:47,502 - Train: 16.86% [833000/4942000] [168.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 22:09:20,226 - Train: 16.86% [833100/4942000] [168.6/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 22:09:53,011 - Train: 16.86% [833200/4942000] [168.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 22:10:25,713 - Train: 16.86% [833300/4942000] [168.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:10:58,490 - Train: 16.86% [833400/4942000] [168.6/1000.0] [batch_t 0.321 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-07 22:11:31,223 - Train: 16.87% [833500/4942000] [168.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:12:03,933 - Train: 16.87% [833600/4942000] [168.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:12:36,621 - Train: 16.87% [833700/4942000] [168.7/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 22:13:10,328 - Train: 16.87% [833800/4942000] [168.7/1000.0] [batch_t 0.325 (0.337)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 22:13:43,218 - Train: 16.87% [833900/4942000] [168.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 22:14:16,032 - Train: 16.88% [834000/4942000] [168.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 22:14:48,847 - Train: 16.88% [834100/4942000] [168.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:15:21,735 - Train: 16.88% [834200/4942000] [168.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 22:15:54,644 - Train: 16.88% [834300/4942000] [168.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:16:27,544 - Train: 16.88% [834400/4942000] [168.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:17:00,483 - Train: 16.89% [834500/4942000] [168.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 22:17:33,328 - Train: 16.89% [834600/4942000] [168.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 22:18:06,169 - Train: 16.89% [834700/4942000] [168.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:18:39,089 - Train: 16.89% [834800/4942000] [168.9/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 22:19:11,968 - Train: 16.89% [834900/4942000] [168.9/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 22:19:44,855 - Train: 16.90% [835000/4942000] [169.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 22:20:17,813 - Train: 16.90% [835100/4942000] [169.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 22:20:50,072 - ==> Total time: 5 days, 4:23:29 Eta: 25 days, 11:39:09 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-07 22:20:53,289 - Train: 16.90% [835200/4942000] [169.0/1000.0] [batch_t 0.341 (0.368)] [data_t 0.002] [optim_t 0.339] [lr 0.005000] 2024-04-07 22:21:26,169 - Train: 16.90% [835300/4942000] [169.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 22:21:58,988 - Train: 16.90% [835400/4942000] [169.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 22:22:31,812 - Train: 16.91% [835500/4942000] [169.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:23:04,612 - Train: 16.91% [835600/4942000] [169.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 22:23:37,413 - Train: 16.91% [835700/4942000] [169.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 22:24:10,238 - Train: 16.91% [835800/4942000] [169.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 22:24:43,054 - Train: 16.91% [835900/4942000] [169.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:25:15,878 - Train: 16.92% [836000/4942000] [169.2/1000.0] [batch_t 0.337 (0.328)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-07 22:25:48,693 - Train: 16.92% [836100/4942000] [169.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 22:26:21,529 - Train: 16.92% [836200/4942000] [169.2/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-07 22:26:54,337 - Train: 16.92% [836300/4942000] [169.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:27:27,091 - Train: 16.92% [836400/4942000] [169.2/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 22:27:59,852 - Train: 16.93% [836500/4942000] [169.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:28:32,719 - Train: 16.93% [836600/4942000] [169.3/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 22:29:05,698 - Train: 16.93% [836700/4942000] [169.3/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 22:29:38,461 - Train: 16.93% [836800/4942000] [169.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:30:11,266 - Train: 16.93% [836900/4942000] [169.3/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 22:30:44,009 - Train: 16.94% [837000/4942000] [169.4/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 22:31:16,877 - Train: 16.94% [837100/4942000] [169.4/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 22:31:49,674 - Train: 16.94% [837200/4942000] [169.4/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 22:32:22,432 - Train: 16.94% [837300/4942000] [169.4/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 22:32:55,260 - Train: 16.94% [837400/4942000] [169.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:33:28,047 - Train: 16.95% [837500/4942000] [169.5/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 22:34:00,771 - Train: 16.95% [837600/4942000] [169.5/1000.0] [batch_t 0.342 (0.327)] [data_t 0.002] [optim_t 0.340] [lr 0.005000] 2024-04-07 22:34:33,573 - Train: 16.95% [837700/4942000] [169.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:35:06,309 - Train: 16.95% [837800/4942000] [169.5/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 22:35:39,053 - Train: 16.95% [837900/4942000] [169.5/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 22:36:11,810 - Train: 16.96% [838000/4942000] [169.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:36:44,758 - Train: 16.96% [838100/4942000] [169.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:37:17,640 - Train: 16.96% [838200/4942000] [169.6/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 22:37:50,511 - Train: 16.96% [838300/4942000] [169.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:38:23,400 - Train: 16.96% [838400/4942000] [169.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:38:56,207 - Train: 16.97% [838500/4942000] [169.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:39:29,066 - Train: 16.97% [838600/4942000] [169.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:40:01,895 - Train: 16.97% [838700/4942000] [169.7/1000.0] [batch_t 0.336 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-07 22:40:34,673 - Train: 16.97% [838800/4942000] [169.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 22:41:07,476 - Train: 16.97% [838900/4942000] [169.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:41:40,281 - Train: 16.98% [839000/4942000] [169.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:42:13,061 - Train: 16.98% [839100/4942000] [169.8/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 22:42:45,873 - Train: 16.98% [839200/4942000] [169.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:43:18,715 - Train: 16.98% [839300/4942000] [169.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 22:43:51,472 - Train: 16.99% [839400/4942000] [169.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:44:24,408 - Train: 16.99% [839500/4942000] [169.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:44:57,309 - Train: 16.99% [839600/4942000] [169.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:45:30,241 - Train: 16.99% [839700/4942000] [169.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 22:46:03,081 - Train: 16.99% [839800/4942000] [169.9/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 22:46:35,882 - Train: 17.00% [839900/4942000] [170.0/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 22:47:08,752 - Train: 17.00% [840000/4942000] [170.0/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 22:47:41,664 - Train: 17.00% [840100/4942000] [170.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 22:47:54,780 - ==> Total time: 5 days, 4:50:33 Eta: 25 days, 9:31:35 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-07 22:48:16,739 - Train: 17.00% [840200/4942000] [170.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 22:48:49,592 - Train: 17.00% [840300/4942000] [170.0/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 22:49:22,450 - Train: 17.01% [840400/4942000] [170.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 22:49:55,224 - Train: 17.01% [840500/4942000] [170.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 22:50:27,985 - Train: 17.01% [840600/4942000] [170.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:51:00,721 - Train: 17.01% [840700/4942000] [170.1/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 22:51:33,602 - Train: 17.01% [840800/4942000] [170.1/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 22:52:06,593 - Train: 17.02% [840900/4942000] [170.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:52:39,412 - Train: 17.02% [841000/4942000] [170.2/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 22:53:12,266 - Train: 17.02% [841100/4942000] [170.2/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 22:53:45,133 - Train: 17.02% [841200/4942000] [170.2/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 22:54:17,960 - Train: 17.02% [841300/4942000] [170.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:54:50,738 - Train: 17.03% [841400/4942000] [170.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:55:23,627 - Train: 17.03% [841500/4942000] [170.3/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 22:55:56,411 - Train: 17.03% [841600/4942000] [170.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 22:56:29,237 - Train: 17.03% [841700/4942000] [170.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 22:57:02,053 - Train: 17.03% [841800/4942000] [170.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:57:34,840 - Train: 17.04% [841900/4942000] [170.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 22:58:07,735 - Train: 17.04% [842000/4942000] [170.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:58:40,569 - Train: 17.04% [842100/4942000] [170.4/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 22:59:13,333 - Train: 17.04% [842200/4942000] [170.4/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 22:59:46,233 - Train: 17.04% [842300/4942000] [170.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:00:18,988 - Train: 17.05% [842400/4942000] [170.5/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 23:00:51,802 - Train: 17.05% [842500/4942000] [170.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 23:01:24,618 - Train: 17.05% [842600/4942000] [170.5/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 23:01:57,395 - Train: 17.05% [842700/4942000] [170.5/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 23:02:30,322 - Train: 17.05% [842800/4942000] [170.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:03:03,126 - Train: 17.06% [842900/4942000] [170.6/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 23:03:35,931 - Train: 17.06% [843000/4942000] [170.6/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 23:04:08,736 - Train: 17.06% [843100/4942000] [170.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 23:04:41,525 - Train: 17.06% [843200/4942000] [170.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:05:14,324 - Train: 17.06% [843300/4942000] [170.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 23:05:47,102 - Train: 17.07% [843400/4942000] [170.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:06:20,012 - Train: 17.07% [843500/4942000] [170.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 23:06:52,887 - Train: 17.07% [843600/4942000] [170.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 23:07:25,893 - Train: 17.07% [843700/4942000] [170.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 23:07:58,625 - Train: 17.07% [843800/4942000] [170.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:08:31,381 - Train: 17.08% [843900/4942000] [170.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 23:09:04,160 - Train: 17.08% [844000/4942000] [170.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 23:09:37,005 - Train: 17.08% [844100/4942000] [170.8/1000.0] [batch_t 0.336 (0.328)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-07 23:10:09,791 - Train: 17.08% [844200/4942000] [170.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 23:10:42,578 - Train: 17.08% [844300/4942000] [170.8/1000.0] [batch_t 0.321 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-07 23:11:15,416 - Train: 17.09% [844400/4942000] [170.9/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 23:11:48,218 - Train: 17.09% [844500/4942000] [170.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 23:12:21,069 - Train: 17.09% [844600/4942000] [170.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:12:53,876 - Train: 17.09% [844700/4942000] [170.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 23:13:26,720 - Train: 17.09% [844800/4942000] [170.9/1000.0] [batch_t 0.341 (0.328)] [data_t 0.002] [optim_t 0.339] [lr 0.005000] 2024-04-07 23:13:59,502 - Train: 17.10% [844900/4942000] [171.0/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-07 23:14:32,277 - Train: 17.10% [845000/4942000] [171.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 23:14:59,166 - ==> Total time: 5 days, 5:17:38 Eta: 25 days, 7:25:09 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-07 23:15:07,503 - Train: 17.10% [845100/4942000] [171.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:15:40,263 - Train: 17.10% [845200/4942000] [171.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:16:13,635 - Train: 17.10% [845300/4942000] [171.0/1000.0] [batch_t 0.330 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 23:16:46,324 - Train: 17.11% [845400/4942000] [171.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 23:17:19,041 - Train: 17.11% [845500/4942000] [171.1/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 23:17:51,809 - Train: 17.11% [845600/4942000] [171.1/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 23:18:26,273 - Train: 17.11% [845700/4942000] [171.1/1000.0] [batch_t 0.327 (0.345)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 23:18:59,039 - Train: 17.11% [845800/4942000] [171.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:19:31,739 - Train: 17.12% [845900/4942000] [171.2/1000.0] [batch_t 0.335 (0.327)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-07 23:20:04,473 - Train: 17.12% [846000/4942000] [171.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 23:20:37,256 - Train: 17.12% [846100/4942000] [171.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:21:09,965 - Train: 17.12% [846200/4942000] [171.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 23:21:42,692 - Train: 17.12% [846300/4942000] [171.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 23:22:15,426 - Train: 17.13% [846400/4942000] [171.3/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 23:22:48,124 - Train: 17.13% [846500/4942000] [171.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 23:23:21,087 - Train: 17.13% [846600/4942000] [171.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:23:53,857 - Train: 17.13% [846700/4942000] [171.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:24:26,670 - Train: 17.13% [846800/4942000] [171.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 23:24:59,443 - Train: 17.14% [846900/4942000] [171.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:25:32,178 - Train: 17.14% [847000/4942000] [171.4/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 23:26:04,919 - Train: 17.14% [847100/4942000] [171.4/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 23:26:37,697 - Train: 17.14% [847200/4942000] [171.4/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 23:27:11,572 - Train: 17.14% [847300/4942000] [171.4/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:27:44,302 - Train: 17.15% [847400/4942000] [171.5/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 23:28:17,131 - Train: 17.15% [847500/4942000] [171.5/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 23:28:49,932 - Train: 17.15% [847600/4942000] [171.5/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-07 23:29:22,691 - Train: 17.15% [847700/4942000] [171.5/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 23:29:55,464 - Train: 17.15% [847800/4942000] [171.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 23:30:28,176 - Train: 17.16% [847900/4942000] [171.6/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 23:31:01,024 - Train: 17.16% [848000/4942000] [171.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 23:31:33,761 - Train: 17.16% [848100/4942000] [171.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 23:32:06,461 - Train: 17.16% [848200/4942000] [171.6/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 23:32:39,257 - Train: 17.17% [848300/4942000] [171.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 23:33:12,010 - Train: 17.17% [848400/4942000] [171.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:33:44,744 - Train: 17.17% [848500/4942000] [171.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 23:34:17,486 - Train: 17.17% [848600/4942000] [171.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:34:50,223 - Train: 17.17% [848700/4942000] [171.7/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 23:35:22,973 - Train: 17.18% [848800/4942000] [171.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 23:35:55,646 - Train: 17.18% [848900/4942000] [171.8/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 23:36:28,472 - Train: 17.18% [849000/4942000] [171.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 23:37:01,194 - Train: 17.18% [849100/4942000] [171.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:37:33,980 - Train: 17.18% [849200/4942000] [171.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:38:06,756 - Train: 17.19% [849300/4942000] [171.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 23:38:39,485 - Train: 17.19% [849400/4942000] [171.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 23:39:12,227 - Train: 17.19% [849500/4942000] [171.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 23:39:45,001 - Train: 17.19% [849600/4942000] [171.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 23:40:17,865 - Train: 17.19% [849700/4942000] [171.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 23:40:50,645 - Train: 17.20% [849800/4942000] [172.0/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 23:41:23,552 - Train: 17.20% [849900/4942000] [172.0/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 23:41:56,397 - Train: 17.20% [850000/4942000] [172.0/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 23:42:04,319 - ==> Total time: 5 days, 5:44:43 Eta: 25 days, 5:19:57 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-07 23:42:32,473 - Train: 17.20% [850100/4942000] [172.0/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:43:06,410 - Train: 17.20% [850200/4942000] [172.0/1000.0] [batch_t 0.333 (0.339)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 23:43:39,194 - Train: 17.21% [850300/4942000] [172.1/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 23:44:12,070 - Train: 17.21% [850400/4942000] [172.1/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 23:44:45,009 - Train: 17.21% [850500/4942000] [172.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 23:45:17,918 - Train: 17.21% [850600/4942000] [172.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 23:45:50,780 - Train: 17.21% [850700/4942000] [172.1/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 23:46:23,813 - Train: 17.22% [850800/4942000] [172.2/1000.0] [batch_t 0.337 (0.330)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-07 23:46:56,568 - Train: 17.22% [850900/4942000] [172.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:47:29,428 - Train: 17.22% [851000/4942000] [172.2/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 23:48:02,245 - Train: 17.22% [851100/4942000] [172.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:48:35,127 - Train: 17.22% [851200/4942000] [172.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 23:49:07,939 - Train: 17.23% [851300/4942000] [172.3/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 23:49:40,822 - Train: 17.23% [851400/4942000] [172.3/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 23:50:14,478 - Train: 17.23% [851500/4942000] [172.3/1000.0] [batch_t 0.334 (0.336)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 23:50:47,358 - Train: 17.23% [851600/4942000] [172.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:51:20,181 - Train: 17.23% [851700/4942000] [172.3/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 23:51:53,001 - Train: 17.24% [851800/4942000] [172.4/1000.0] [batch_t 0.336 (0.328)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-07 23:52:25,854 - Train: 17.24% [851900/4942000] [172.4/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 23:52:58,701 - Train: 17.24% [852000/4942000] [172.4/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 23:53:31,493 - Train: 17.24% [852100/4942000] [172.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 23:54:04,513 - Train: 17.24% [852200/4942000] [172.4/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 23:54:37,495 - Train: 17.25% [852300/4942000] [172.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 23:55:10,434 - Train: 17.25% [852400/4942000] [172.5/1000.0] [batch_t 0.326 (0.329)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 23:55:43,438 - Train: 17.25% [852500/4942000] [172.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:56:16,497 - Train: 17.25% [852600/4942000] [172.5/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 23:56:49,489 - Train: 17.25% [852700/4942000] [172.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 23:57:23,340 - Train: 17.26% [852800/4942000] [172.6/1000.0] [batch_t 0.331 (0.338)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 23:57:56,379 - Train: 17.26% [852900/4942000] [172.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 23:58:30,276 - Train: 17.26% [853000/4942000] [172.6/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 23:59:03,219 - Train: 17.26% [853100/4942000] [172.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 23:59:36,393 - Train: 17.26% [853200/4942000] [172.6/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 00:00:09,392 - Train: 17.27% [853300/4942000] [172.7/1000.0] [batch_t 0.321 (0.330)] [data_t 0.003] [optim_t 0.318] [lr 0.005000] 2024-04-08 00:00:42,429 - Train: 17.27% [853400/4942000] [172.7/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 00:01:15,391 - Train: 17.27% [853500/4942000] [172.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:01:48,248 - Train: 17.27% [853600/4942000] [172.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 00:02:21,298 - Train: 17.27% [853700/4942000] [172.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:02:54,352 - Train: 17.28% [853800/4942000] [172.8/1000.0] [batch_t 0.339 (0.330)] [data_t 0.003] [optim_t 0.336] [lr 0.005000] 2024-04-08 00:03:27,409 - Train: 17.28% [853900/4942000] [172.8/1000.0] [batch_t 0.335 (0.330)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 00:04:00,357 - Train: 17.28% [854000/4942000] [172.8/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 00:04:36,143 - Train: 17.28% [854100/4942000] [172.8/1000.0] [batch_t 0.328 (0.358)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 00:05:10,069 - Train: 17.28% [854200/4942000] [172.8/1000.0] [batch_t 0.330 (0.339)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:05:43,044 - Train: 17.29% [854300/4942000] [172.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 00:06:16,035 - Train: 17.29% [854400/4942000] [172.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 00:06:48,968 - Train: 17.29% [854500/4942000] [172.9/1000.0] [batch_t 0.324 (0.329)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-08 00:07:22,051 - Train: 17.29% [854600/4942000] [172.9/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 00:07:54,956 - Train: 17.29% [854700/4942000] [172.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:08:27,983 - Train: 17.30% [854800/4942000] [173.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 00:09:01,064 - Train: 17.30% [854900/4942000] [173.0/1000.0] [batch_t 0.324 (0.331)] [data_t 0.003] [optim_t 0.321] [lr 0.005000] 2024-04-08 00:09:22,838 - ==> Total time: 5 days, 6:12:02 Eta: 25 days, 3:16:56 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 00:09:36,574 - Train: 17.30% [855000/4942000] [173.0/1000.0] [batch_t 0.326 (0.332)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 00:10:09,518 - Train: 17.30% [855100/4942000] [173.0/1000.0] [batch_t 0.324 (0.329)] [data_t 0.003] [optim_t 0.321] [lr 0.005000] 2024-04-08 00:10:42,384 - Train: 17.30% [855200/4942000] [173.0/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 00:11:15,255 - Train: 17.31% [855300/4942000] [173.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 00:11:48,268 - Train: 17.31% [855400/4942000] [173.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:12:21,171 - Train: 17.31% [855500/4942000] [173.1/1000.0] [batch_t 0.324 (0.329)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-08 00:12:54,131 - Train: 17.31% [855600/4942000] [173.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 00:13:27,930 - Train: 17.31% [855700/4942000] [173.1/1000.0] [batch_t 0.326 (0.338)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 00:14:00,897 - Train: 17.32% [855800/4942000] [173.2/1000.0] [batch_t 0.333 (0.330)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 00:14:34,487 - Train: 17.32% [855900/4942000] [173.2/1000.0] [batch_t 0.334 (0.336)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 00:15:07,986 - Train: 17.32% [856000/4942000] [173.2/1000.0] [batch_t 0.322 (0.335)] [data_t 0.003] [optim_t 0.319] [lr 0.005000] 2024-04-08 00:15:40,910 - Train: 17.32% [856100/4942000] [173.2/1000.0] [batch_t 0.325 (0.329)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 00:16:13,829 - Train: 17.32% [856200/4942000] [173.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 00:16:46,771 - Train: 17.33% [856300/4942000] [173.3/1000.0] [batch_t 0.333 (0.329)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 00:17:20,649 - Train: 17.33% [856400/4942000] [173.3/1000.0] [batch_t 0.330 (0.339)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 00:17:53,839 - Train: 17.33% [856500/4942000] [173.3/1000.0] [batch_t 0.336 (0.332)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-08 00:18:27,422 - Train: 17.33% [856600/4942000] [173.3/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 00:19:00,355 - Train: 17.34% [856700/4942000] [173.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 00:19:33,183 - Train: 17.34% [856800/4942000] [173.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 00:20:06,103 - Train: 17.34% [856900/4942000] [173.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 00:20:38,948 - Train: 17.34% [857000/4942000] [173.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 00:21:11,884 - Train: 17.34% [857100/4942000] [173.4/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 00:21:44,770 - Train: 17.35% [857200/4942000] [173.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 00:22:17,696 - Train: 17.35% [857300/4942000] [173.5/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 00:22:50,599 - Train: 17.35% [857400/4942000] [173.5/1000.0] [batch_t 0.322 (0.329)] [data_t 0.003] [optim_t 0.319] [lr 0.005000] 2024-04-08 00:23:24,515 - Train: 17.35% [857500/4942000] [173.5/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 00:23:57,418 - Train: 17.35% [857600/4942000] [173.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 00:24:31,029 - Train: 17.36% [857700/4942000] [173.6/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 00:25:03,912 - Train: 17.36% [857800/4942000] [173.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 00:25:37,918 - Train: 17.36% [857900/4942000] [173.6/1000.0] [batch_t 0.323 (0.340)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 00:26:10,751 - Train: 17.36% [858000/4942000] [173.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 00:26:43,623 - Train: 17.36% [858100/4942000] [173.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 00:27:16,478 - Train: 17.37% [858200/4942000] [173.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 00:27:49,373 - Train: 17.37% [858300/4942000] [173.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 00:28:22,309 - Train: 17.37% [858400/4942000] [173.7/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 00:28:55,158 - Train: 17.37% [858500/4942000] [173.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 00:29:28,014 - Train: 17.37% [858600/4942000] [173.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 00:30:00,912 - Train: 17.38% [858700/4942000] [173.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 00:30:34,688 - Train: 17.38% [858800/4942000] [173.8/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 00:31:09,628 - Train: 17.38% [858900/4942000] [173.8/1000.0] [batch_t 0.339 (0.349)] [data_t 0.002] [optim_t 0.337] [lr 0.005000] 2024-04-08 00:31:42,550 - Train: 17.38% [859000/4942000] [173.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 00:32:16,467 - Train: 17.38% [859100/4942000] [173.8/1000.0] [batch_t 0.331 (0.339)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 00:32:49,413 - Train: 17.39% [859200/4942000] [173.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 00:33:22,908 - Train: 17.39% [859300/4942000] [173.9/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:33:55,946 - Train: 17.39% [859400/4942000] [173.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 00:34:29,341 - Train: 17.39% [859500/4942000] [173.9/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:35:02,191 - Train: 17.39% [859600/4942000] [173.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 00:35:35,349 - Train: 17.40% [859700/4942000] [174.0/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:36:09,372 - Train: 17.40% [859800/4942000] [174.0/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:36:42,197 - Train: 17.40% [859900/4942000] [174.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 00:36:44,826 - ==> Total time: 5 days, 6:39:24 Eta: 25 days, 1:15:18 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 00:37:17,866 - Train: 17.40% [860000/4942000] [174.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 00:37:50,756 - Train: 17.40% [860100/4942000] [174.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:38:24,222 - Train: 17.41% [860200/4942000] [174.1/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 00:38:57,002 - Train: 17.41% [860300/4942000] [174.1/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 00:39:30,922 - Train: 17.41% [860400/4942000] [174.1/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:40:03,814 - Train: 17.41% [860500/4942000] [174.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 00:40:37,346 - Train: 17.41% [860600/4942000] [174.1/1000.0] [batch_t 0.335 (0.335)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 00:41:11,360 - Train: 17.42% [860700/4942000] [174.2/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 00:41:44,115 - Train: 17.42% [860800/4942000] [174.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 00:42:16,873 - Train: 17.42% [860900/4942000] [174.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 00:42:49,685 - Train: 17.42% [861000/4942000] [174.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 00:43:22,972 - Train: 17.42% [861100/4942000] [174.2/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:43:55,822 - Train: 17.43% [861200/4942000] [174.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 00:44:29,429 - Train: 17.43% [861300/4942000] [174.3/1000.0] [batch_t 0.325 (0.336)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 00:45:02,277 - Train: 17.43% [861400/4942000] [174.3/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 00:45:35,111 - Train: 17.43% [861500/4942000] [174.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 00:46:09,062 - Train: 17.43% [861600/4942000] [174.3/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 00:46:41,878 - Train: 17.44% [861700/4942000] [174.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:47:15,052 - Train: 17.44% [861800/4942000] [174.4/1000.0] [batch_t 0.324 (0.332)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 00:47:47,947 - Train: 17.44% [861900/4942000] [174.4/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 00:48:20,885 - Train: 17.44% [862000/4942000] [174.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 00:48:53,886 - Train: 17.44% [862100/4942000] [174.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 00:49:27,669 - Train: 17.45% [862200/4942000] [174.5/1000.0] [batch_t 0.324 (0.338)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 00:50:00,558 - Train: 17.45% [862300/4942000] [174.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 00:50:33,723 - Train: 17.45% [862400/4942000] [174.5/1000.0] [batch_t 0.325 (0.332)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 00:51:07,014 - Train: 17.45% [862500/4942000] [174.5/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:51:39,809 - Train: 17.45% [862600/4942000] [174.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 00:52:12,616 - Train: 17.46% [862700/4942000] [174.6/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 00:52:45,527 - Train: 17.46% [862800/4942000] [174.6/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 00:53:18,418 - Train: 17.46% [862900/4942000] [174.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:53:51,327 - Train: 17.46% [863000/4942000] [174.6/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 00:54:25,042 - Train: 17.46% [863100/4942000] [174.6/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 00:54:57,893 - Train: 17.47% [863200/4942000] [174.7/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 00:55:30,762 - Train: 17.47% [863300/4942000] [174.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 00:56:03,604 - Train: 17.47% [863400/4942000] [174.7/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 00:56:37,024 - Train: 17.47% [863500/4942000] [174.7/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 00:57:09,954 - Train: 17.47% [863600/4942000] [174.7/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 00:57:42,737 - Train: 17.48% [863700/4942000] [174.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 00:58:15,645 - Train: 17.48% [863800/4942000] [174.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:58:48,462 - Train: 17.48% [863900/4942000] [174.8/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 00:59:21,310 - Train: 17.48% [864000/4942000] [174.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:59:54,182 - Train: 17.48% [864100/4942000] [174.8/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 01:00:27,101 - Train: 17.49% [864200/4942000] [174.9/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 01:00:59,948 - Train: 17.49% [864300/4942000] [174.9/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 01:01:32,829 - Train: 17.49% [864400/4942000] [174.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 01:02:05,655 - Train: 17.49% [864500/4942000] [174.9/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 01:02:38,652 - Train: 17.49% [864600/4942000] [174.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:03:11,525 - Train: 17.50% [864700/4942000] [175.0/1000.0] [batch_t 0.321 (0.329)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-08 01:03:44,377 - Train: 17.50% [864800/4942000] [175.0/1000.0] [batch_t 0.339 (0.328)] [data_t 0.002] [optim_t 0.337] [lr 0.005000] 2024-04-08 01:04:00,862 - ==> Total time: 5 days, 7:06:40 Eta: 24 days, 23:14:17 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 01:04:21,287 - Train: 17.50% [864900/4942000] [175.0/1000.0] [batch_t 0.325 (0.334)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 01:04:54,151 - Train: 17.50% [865000/4942000] [175.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:05:26,932 - Train: 17.51% [865100/4942000] [175.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 01:05:59,724 - Train: 17.51% [865200/4942000] [175.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 01:06:32,524 - Train: 17.51% [865300/4942000] [175.1/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 01:07:06,188 - Train: 17.51% [865400/4942000] [175.1/1000.0] [batch_t 0.333 (0.337)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 01:07:38,912 - Train: 17.51% [865500/4942000] [175.1/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 01:08:11,739 - Train: 17.52% [865600/4942000] [175.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 01:08:44,490 - Train: 17.52% [865700/4942000] [175.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 01:09:17,298 - Train: 17.52% [865800/4942000] [175.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 01:09:50,219 - Train: 17.52% [865900/4942000] [175.2/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 01:10:22,990 - Train: 17.52% [866000/4942000] [175.2/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 01:10:55,725 - Train: 17.53% [866100/4942000] [175.3/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 01:11:28,506 - Train: 17.53% [866200/4942000] [175.3/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 01:12:01,312 - Train: 17.53% [866300/4942000] [175.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:12:34,258 - Train: 17.53% [866400/4942000] [175.3/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 01:13:07,401 - Train: 17.53% [866500/4942000] [175.3/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 01:13:40,184 - Train: 17.54% [866600/4942000] [175.4/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 01:14:12,980 - Train: 17.54% [866700/4942000] [175.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 01:14:45,789 - Train: 17.54% [866800/4942000] [175.4/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 01:15:18,575 - Train: 17.54% [866900/4942000] [175.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:15:51,413 - Train: 17.54% [867000/4942000] [175.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 01:16:24,247 - Train: 17.55% [867100/4942000] [175.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 01:16:57,095 - Train: 17.55% [867200/4942000] [175.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:17:29,932 - Train: 17.55% [867300/4942000] [175.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:18:02,753 - Train: 17.55% [867400/4942000] [175.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 01:18:35,525 - Train: 17.55% [867500/4942000] [175.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 01:19:08,261 - Train: 17.56% [867600/4942000] [175.6/1000.0] [batch_t 0.335 (0.327)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 01:19:41,090 - Train: 17.56% [867700/4942000] [175.6/1000.0] [batch_t 0.337 (0.328)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-08 01:20:13,997 - Train: 17.56% [867800/4942000] [175.6/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 01:20:46,725 - Train: 17.56% [867900/4942000] [175.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 01:21:19,455 - Train: 17.56% [868000/4942000] [175.6/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 01:21:52,215 - Train: 17.57% [868100/4942000] [175.7/1000.0] [batch_t 0.336 (0.328)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-08 01:22:24,965 - Train: 17.57% [868200/4942000] [175.7/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 01:22:57,778 - Train: 17.57% [868300/4942000] [175.7/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 01:23:30,551 - Train: 17.57% [868400/4942000] [175.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 01:24:03,290 - Train: 17.57% [868500/4942000] [175.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 01:24:37,085 - Train: 17.58% [868600/4942000] [175.8/1000.0] [batch_t 0.324 (0.338)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 01:25:11,371 - Train: 17.58% [868700/4942000] [175.8/1000.0] [batch_t 0.335 (0.343)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 01:25:44,128 - Train: 17.58% [868800/4942000] [175.8/1000.0] [batch_t 0.334 (0.327)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 01:26:17,174 - Train: 17.58% [868900/4942000] [175.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:26:49,947 - Train: 17.58% [869000/4942000] [175.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:27:22,715 - Train: 17.59% [869100/4942000] [175.9/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 01:27:55,618 - Train: 17.59% [869200/4942000] [175.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:28:29,595 - Train: 17.59% [869300/4942000] [175.9/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 01:29:02,376 - Train: 17.59% [869400/4942000] [175.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 01:29:35,185 - Train: 17.59% [869500/4942000] [175.9/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 01:30:08,506 - Train: 17.60% [869600/4942000] [176.0/1000.0] [batch_t 0.334 (0.333)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 01:30:41,295 - Train: 17.60% [869700/4942000] [176.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 01:31:11,486 - ==> Total time: 5 days, 7:33:50 Eta: 24 days, 21:13:54 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 01:31:16,337 - Train: 17.60% [869800/4942000] [176.0/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 01:31:49,103 - Train: 17.60% [869900/4942000] [176.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 01:32:22,703 - Train: 17.60% [870000/4942000] [176.0/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 01:32:55,479 - Train: 17.61% [870100/4942000] [176.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 01:33:28,361 - Train: 17.61% [870200/4942000] [176.1/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 01:34:01,156 - Train: 17.61% [870300/4942000] [176.1/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 01:34:33,958 - Train: 17.61% [870400/4942000] [176.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:35:07,743 - Train: 17.61% [870500/4942000] [176.1/1000.0] [batch_t 0.324 (0.338)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 01:35:40,564 - Train: 17.62% [870600/4942000] [176.2/1000.0] [batch_t 0.331 (0.328)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 01:36:13,486 - Train: 17.62% [870700/4942000] [176.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:36:46,366 - Train: 17.62% [870800/4942000] [176.2/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 01:37:20,080 - Train: 17.62% [870900/4942000] [176.2/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 01:37:52,818 - Train: 17.62% [871000/4942000] [176.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:38:25,594 - Train: 17.63% [871100/4942000] [176.3/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 01:38:58,422 - Train: 17.63% [871200/4942000] [176.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 01:39:31,335 - Train: 17.63% [871300/4942000] [176.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 01:40:04,187 - Train: 17.63% [871400/4942000] [176.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 01:40:37,055 - Train: 17.63% [871500/4942000] [176.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 01:41:09,888 - Train: 17.64% [871600/4942000] [176.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 01:41:42,794 - Train: 17.64% [871700/4942000] [176.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:42:15,653 - Train: 17.64% [871800/4942000] [176.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:42:48,477 - Train: 17.64% [871900/4942000] [176.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:43:21,391 - Train: 17.64% [872000/4942000] [176.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 01:43:54,227 - Train: 17.65% [872100/4942000] [176.5/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 01:44:27,002 - Train: 17.65% [872200/4942000] [176.5/1000.0] [batch_t 0.321 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 01:44:59,702 - Train: 17.65% [872300/4942000] [176.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 01:45:32,457 - Train: 17.65% [872400/4942000] [176.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:46:05,239 - Train: 17.65% [872500/4942000] [176.5/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 01:46:37,981 - Train: 17.66% [872600/4942000] [176.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:47:10,789 - Train: 17.66% [872700/4942000] [176.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:47:43,603 - Train: 17.66% [872800/4942000] [176.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 01:48:16,362 - Train: 17.66% [872900/4942000] [176.6/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 01:48:49,079 - Train: 17.66% [873000/4942000] [176.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 01:49:21,896 - Train: 17.67% [873100/4942000] [176.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 01:49:54,636 - Train: 17.67% [873200/4942000] [176.7/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 01:50:27,447 - Train: 17.67% [873300/4942000] [176.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:51:00,330 - Train: 17.67% [873400/4942000] [176.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:51:33,162 - Train: 17.68% [873500/4942000] [176.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:52:07,015 - Train: 17.68% [873600/4942000] [176.8/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:52:39,860 - Train: 17.68% [873700/4942000] [176.8/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 01:53:12,659 - Train: 17.68% [873800/4942000] [176.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 01:53:45,500 - Train: 17.68% [873900/4942000] [176.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 01:54:18,395 - Train: 17.69% [874000/4942000] [176.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:54:51,129 - Train: 17.69% [874100/4942000] [176.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 01:55:23,991 - Train: 17.69% [874200/4942000] [176.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:55:56,785 - Train: 17.69% [874300/4942000] [176.9/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 01:56:29,567 - Train: 17.69% [874400/4942000] [176.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 01:57:02,493 - Train: 17.70% [874500/4942000] [177.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:57:35,314 - Train: 17.70% [874600/4942000] [177.0/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 01:58:08,170 - Train: 17.70% [874700/4942000] [177.0/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 01:58:19,359 - ==> Total time: 5 days, 8:00:58 Eta: 24 days, 19:14:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 01:58:43,543 - Train: 17.70% [874800/4942000] [177.0/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 01:59:18,402 - Train: 17.70% [874900/4942000] [177.0/1000.0] [batch_t 0.338 (0.348)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-08 01:59:51,183 - Train: 17.71% [875000/4942000] [177.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 02:00:24,051 - Train: 17.71% [875100/4942000] [177.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 02:00:56,834 - Train: 17.71% [875200/4942000] [177.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 02:01:29,636 - Train: 17.71% [875300/4942000] [177.1/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 02:02:02,393 - Train: 17.71% [875400/4942000] [177.1/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 02:02:35,271 - Train: 17.72% [875500/4942000] [177.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 02:03:08,186 - Train: 17.72% [875600/4942000] [177.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 02:03:41,008 - Train: 17.72% [875700/4942000] [177.2/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 02:04:13,832 - Train: 17.72% [875800/4942000] [177.2/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 02:04:46,701 - Train: 17.72% [875900/4942000] [177.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 02:05:19,617 - Train: 17.73% [876000/4942000] [177.3/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 02:05:52,457 - Train: 17.73% [876100/4942000] [177.3/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 02:06:25,370 - Train: 17.73% [876200/4942000] [177.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 02:06:58,251 - Train: 17.73% [876300/4942000] [177.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 02:07:31,056 - Train: 17.73% [876400/4942000] [177.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 02:08:03,875 - Train: 17.74% [876500/4942000] [177.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 02:08:36,762 - Train: 17.74% [876600/4942000] [177.4/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 02:09:10,297 - Train: 17.74% [876700/4942000] [177.4/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 02:09:43,095 - Train: 17.74% [876800/4942000] [177.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 02:10:15,882 - Train: 17.74% [876900/4942000] [177.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 02:10:48,756 - Train: 17.75% [877000/4942000] [177.5/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 02:11:21,477 - Train: 17.75% [877100/4942000] [177.5/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 02:11:54,251 - Train: 17.75% [877200/4942000] [177.5/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 02:12:27,847 - Train: 17.75% [877300/4942000] [177.5/1000.0] [batch_t 0.332 (0.336)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 02:13:00,686 - Train: 17.75% [877400/4942000] [177.5/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 02:13:34,265 - Train: 17.76% [877500/4942000] [177.6/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 02:14:07,538 - Train: 17.76% [877600/4942000] [177.6/1000.0] [batch_t 0.326 (0.333)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 02:14:40,326 - Train: 17.76% [877700/4942000] [177.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 02:15:13,150 - Train: 17.76% [877800/4942000] [177.6/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 02:15:45,980 - Train: 17.76% [877900/4942000] [177.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 02:16:21,002 - Train: 17.77% [878000/4942000] [177.7/1000.0] [batch_t 0.323 (0.350)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 02:16:53,914 - Train: 17.77% [878100/4942000] [177.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 02:17:26,870 - Train: 17.77% [878200/4942000] [177.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 02:17:59,672 - Train: 17.77% [878300/4942000] [177.7/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 02:18:33,450 - Train: 17.77% [878400/4942000] [177.7/1000.0] [batch_t 0.333 (0.338)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 02:19:06,681 - Train: 17.78% [878500/4942000] [177.8/1000.0] [batch_t 0.319 (0.332)] [data_t 0.002] [optim_t 0.317] [lr 0.005000] 2024-04-08 02:19:39,536 - Train: 17.78% [878600/4942000] [177.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 02:20:12,324 - Train: 17.78% [878700/4942000] [177.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 02:20:45,187 - Train: 17.78% [878800/4942000] [177.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 02:21:18,006 - Train: 17.78% [878900/4942000] [177.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 02:21:50,807 - Train: 17.79% [879000/4942000] [177.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 02:22:23,926 - Train: 17.79% [879100/4942000] [177.9/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 02:22:56,857 - Train: 17.79% [879200/4942000] [177.9/1000.0] [batch_t 0.340 (0.329)] [data_t 0.002] [optim_t 0.338] [lr 0.005000] 2024-04-08 02:23:29,777 - Train: 17.79% [879300/4942000] [177.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 02:24:02,632 - Train: 17.79% [879400/4942000] [177.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 02:24:35,564 - Train: 17.80% [879500/4942000] [178.0/1000.0] [batch_t 0.322 (0.329)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 02:25:08,340 - Train: 17.80% [879600/4942000] [178.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 02:25:33,282 - ==> Total time: 5 days, 8:28:12 Eta: 24 days, 17:16:19 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 02:25:43,312 - Train: 17.80% [879700/4942000] [178.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 02:26:16,892 - Train: 17.80% [879800/4942000] [178.0/1000.0] [batch_t 0.325 (0.336)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 02:26:49,704 - Train: 17.80% [879900/4942000] [178.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 02:27:22,428 - Train: 17.81% [880000/4942000] [178.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 02:27:55,096 - Train: 17.81% [880100/4942000] [178.1/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 02:28:28,215 - Train: 17.81% [880200/4942000] [178.1/1000.0] [batch_t 0.322 (0.331)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 02:29:01,001 - Train: 17.81% [880300/4942000] [178.1/1000.0] [batch_t 0.336 (0.328)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-08 02:29:34,956 - Train: 17.81% [880400/4942000] [178.1/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 02:30:09,376 - Train: 17.82% [880500/4942000] [178.2/1000.0] [batch_t 0.327 (0.344)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 02:30:42,257 - Train: 17.82% [880600/4942000] [178.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 02:31:16,246 - Train: 17.82% [880700/4942000] [178.2/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 02:31:49,106 - Train: 17.82% [880800/4942000] [178.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 02:32:22,692 - Train: 17.82% [880900/4942000] [178.2/1000.0] [batch_t 0.332 (0.336)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 02:32:55,619 - Train: 17.83% [881000/4942000] [178.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 02:33:29,584 - Train: 17.83% [881100/4942000] [178.3/1000.0] [batch_t 0.324 (0.340)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 02:34:02,510 - Train: 17.83% [881200/4942000] [178.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 02:34:35,445 - Train: 17.83% [881300/4942000] [178.3/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 02:35:08,892 - Train: 17.83% [881400/4942000] [178.3/1000.0] [batch_t 0.334 (0.334)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 02:35:41,634 - Train: 17.84% [881500/4942000] [178.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 02:36:15,204 - Train: 17.84% [881600/4942000] [178.4/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 02:36:48,022 - Train: 17.84% [881700/4942000] [178.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 02:37:22,277 - Train: 17.84% [881800/4942000] [178.4/1000.0] [batch_t 0.330 (0.342)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 02:37:55,122 - Train: 17.85% [881900/4942000] [178.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 02:38:27,925 - Train: 17.85% [882000/4942000] [178.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 02:39:00,827 - Train: 17.85% [882100/4942000] [178.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 02:39:33,890 - Train: 17.85% [882200/4942000] [178.5/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 02:40:07,240 - Train: 17.85% [882300/4942000] [178.5/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 02:40:40,053 - Train: 17.86% [882400/4942000] [178.6/1000.0] [batch_t 0.338 (0.328)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-08 02:41:13,040 - Train: 17.86% [882500/4942000] [178.6/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 02:41:45,918 - Train: 17.86% [882600/4942000] [178.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 02:42:18,790 - Train: 17.86% [882700/4942000] [178.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 02:42:51,648 - Train: 17.86% [882800/4942000] [178.6/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 02:43:25,067 - Train: 17.87% [882900/4942000] [178.7/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 02:43:57,948 - Train: 17.87% [883000/4942000] [178.7/1000.0] [batch_t 0.320 (0.329)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-08 02:44:30,752 - Train: 17.87% [883100/4942000] [178.7/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 02:45:03,564 - Train: 17.87% [883200/4942000] [178.7/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 02:45:36,464 - Train: 17.87% [883300/4942000] [178.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 02:46:09,228 - Train: 17.88% [883400/4942000] [178.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 02:46:41,997 - Train: 17.88% [883500/4942000] [178.8/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 02:47:14,791 - Train: 17.88% [883600/4942000] [178.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 02:47:47,572 - Train: 17.88% [883700/4942000] [178.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 02:48:21,503 - Train: 17.88% [883800/4942000] [178.8/1000.0] [batch_t 0.332 (0.339)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 02:48:54,277 - Train: 17.89% [883900/4942000] [178.9/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 02:49:26,964 - Train: 17.89% [884000/4942000] [178.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 02:49:59,731 - Train: 17.89% [884100/4942000] [178.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 02:50:32,515 - Train: 17.89% [884200/4942000] [178.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 02:51:05,265 - Train: 17.89% [884300/4942000] [178.9/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 02:51:38,194 - Train: 17.90% [884400/4942000] [179.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 02:52:11,135 - Train: 17.90% [884500/4942000] [179.0/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 02:52:44,015 - Train: 17.90% [884600/4942000] [179.0/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 02:52:49,928 - ==> Total time: 5 days, 8:55:29 Eta: 24 days, 15:19:30 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 02:53:19,505 - Train: 17.90% [884700/4942000] [179.0/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 02:53:52,324 - Train: 17.90% [884800/4942000] [179.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 02:54:25,181 - Train: 17.91% [884900/4942000] [179.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 02:54:57,994 - Train: 17.91% [885000/4942000] [179.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 02:55:30,719 - Train: 17.91% [885100/4942000] [179.1/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 02:56:03,521 - Train: 17.91% [885200/4942000] [179.1/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 02:56:36,360 - Train: 17.91% [885300/4942000] [179.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 02:57:09,093 - Train: 17.92% [885400/4942000] [179.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 02:57:41,841 - Train: 17.92% [885500/4942000] [179.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 02:58:14,668 - Train: 17.92% [885600/4942000] [179.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 02:58:47,477 - Train: 17.92% [885700/4942000] [179.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 02:59:20,270 - Train: 17.92% [885800/4942000] [179.2/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 02:59:53,108 - Train: 17.93% [885900/4942000] [179.3/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 03:00:25,909 - Train: 17.93% [886000/4942000] [179.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 03:00:58,917 - Train: 17.93% [886100/4942000] [179.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 03:01:31,718 - Train: 17.93% [886200/4942000] [179.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 03:02:04,520 - Train: 17.93% [886300/4942000] [179.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 03:02:38,054 - Train: 17.94% [886400/4942000] [179.4/1000.0] [batch_t 0.326 (0.335)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 03:03:10,878 - Train: 17.94% [886500/4942000] [179.4/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 03:03:43,654 - Train: 17.94% [886600/4942000] [179.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 03:04:16,512 - Train: 17.94% [886700/4942000] [179.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 03:04:49,338 - Train: 17.94% [886800/4942000] [179.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 03:05:22,203 - Train: 17.95% [886900/4942000] [179.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 03:05:54,991 - Train: 17.95% [887000/4942000] [179.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 03:06:27,783 - Train: 17.95% [887100/4942000] [179.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 03:07:00,556 - Train: 17.95% [887200/4942000] [179.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 03:07:33,370 - Train: 17.95% [887300/4942000] [179.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 03:08:06,144 - Train: 17.96% [887400/4942000] [179.6/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 03:08:39,132 - Train: 17.96% [887500/4942000] [179.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 03:09:11,881 - Train: 17.96% [887600/4942000] [179.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 03:09:44,666 - Train: 17.96% [887700/4942000] [179.6/1000.0] [batch_t 0.337 (0.328)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-08 03:10:17,472 - Train: 17.96% [887800/4942000] [179.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 03:10:50,248 - Train: 17.97% [887900/4942000] [179.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 03:11:23,542 - Train: 17.97% [888000/4942000] [179.7/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 03:11:56,338 - Train: 17.97% [888100/4942000] [179.7/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 03:12:30,358 - Train: 17.97% [888200/4942000] [179.7/1000.0] [batch_t 0.334 (0.340)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 03:13:03,234 - Train: 17.97% [888300/4942000] [179.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 03:13:37,010 - Train: 17.98% [888400/4942000] [179.8/1000.0] [batch_t 0.324 (0.338)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 03:14:09,859 - Train: 17.98% [888500/4942000] [179.8/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 03:14:42,603 - Train: 17.98% [888600/4942000] [179.8/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 03:15:15,456 - Train: 17.98% [888700/4942000] [179.8/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 03:15:48,247 - Train: 17.98% [888800/4942000] [179.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 03:16:21,075 - Train: 17.99% [888900/4942000] [179.9/1000.0] [batch_t 0.321 (0.328)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-08 03:16:54,061 - Train: 17.99% [889000/4942000] [179.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 03:17:28,045 - Train: 17.99% [889100/4942000] [179.9/1000.0] [batch_t 0.324 (0.340)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 03:18:00,861 - Train: 17.99% [889200/4942000] [179.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 03:18:33,616 - Train: 17.99% [889300/4942000] [179.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 03:19:06,399 - Train: 18.00% [889400/4942000] [180.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 03:19:39,154 - Train: 18.00% [889500/4942000] [180.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 03:19:58,854 - ==> Total time: 5 days, 9:22:38 Eta: 24 days, 13:23:06 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 03:20:15,370 - Train: 18.00% [889600/4942000] [180.0/1000.0] [batch_t 0.330 (0.351)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 03:20:48,222 - Train: 18.00% [889700/4942000] [180.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 03:21:20,963 - Train: 18.00% [889800/4942000] [180.0/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 03:21:53,700 - Train: 18.01% [889900/4942000] [180.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 03:22:27,665 - Train: 18.01% [890000/4942000] [180.1/1000.0] [batch_t 0.325 (0.340)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 03:23:00,421 - Train: 18.01% [890100/4942000] [180.1/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 03:23:33,248 - Train: 18.01% [890200/4942000] [180.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 03:24:06,150 - Train: 18.01% [890300/4942000] [180.1/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 03:24:38,999 - Train: 18.02% [890400/4942000] [180.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 03:25:11,841 - Train: 18.02% [890500/4942000] [180.2/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 03:25:44,679 - Train: 18.02% [890600/4942000] [180.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 03:26:17,472 - Train: 18.02% [890700/4942000] [180.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 03:26:50,283 - Train: 18.03% [890800/4942000] [180.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 03:27:23,123 - Train: 18.03% [890900/4942000] [180.3/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 03:27:56,007 - Train: 18.03% [891000/4942000] [180.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 03:28:29,878 - Train: 18.03% [891100/4942000] [180.3/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 03:29:02,721 - Train: 18.03% [891200/4942000] [180.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 03:29:35,487 - Train: 18.04% [891300/4942000] [180.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 03:30:08,251 - Train: 18.04% [891400/4942000] [180.4/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 03:30:40,969 - Train: 18.04% [891500/4942000] [180.4/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 03:31:13,723 - Train: 18.04% [891600/4942000] [180.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 03:31:46,661 - Train: 18.04% [891700/4942000] [180.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 03:32:19,504 - Train: 18.05% [891800/4942000] [180.5/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 03:32:52,358 - Train: 18.05% [891900/4942000] [180.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 03:33:25,130 - Train: 18.05% [892000/4942000] [180.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 03:33:57,884 - Train: 18.05% [892100/4942000] [180.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 03:34:30,655 - Train: 18.05% [892200/4942000] [180.5/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 03:35:03,420 - Train: 18.06% [892300/4942000] [180.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 03:35:36,184 - Train: 18.06% [892400/4942000] [180.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 03:36:08,951 - Train: 18.06% [892500/4942000] [180.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 03:36:41,724 - Train: 18.06% [892600/4942000] [180.6/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 03:37:14,506 - Train: 18.06% [892700/4942000] [180.6/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 03:37:47,180 - Train: 18.07% [892800/4942000] [180.7/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 03:38:19,962 - Train: 18.07% [892900/4942000] [180.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 03:38:52,674 - Train: 18.07% [893000/4942000] [180.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 03:39:25,503 - Train: 18.07% [893100/4942000] [180.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 03:39:58,255 - Train: 18.07% [893200/4942000] [180.7/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 03:40:30,974 - Train: 18.08% [893300/4942000] [180.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 03:41:03,728 - Train: 18.08% [893400/4942000] [180.8/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 03:41:36,611 - Train: 18.08% [893500/4942000] [180.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 03:42:09,437 - Train: 18.08% [893600/4942000] [180.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 03:42:42,210 - Train: 18.08% [893700/4942000] [180.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 03:43:14,984 - Train: 18.09% [893800/4942000] [180.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 03:43:47,807 - Train: 18.09% [893900/4942000] [180.9/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 03:44:20,590 - Train: 18.09% [894000/4942000] [180.9/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 03:44:53,338 - Train: 18.09% [894100/4942000] [180.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 03:45:26,081 - Train: 18.09% [894200/4942000] [180.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 03:45:58,804 - Train: 18.10% [894300/4942000] [181.0/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 03:46:31,610 - Train: 18.10% [894400/4942000] [181.0/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 03:47:04,459 - Train: 18.10% [894500/4942000] [181.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 03:47:05,116 - ==> Total time: 5 days, 9:49:44 Eta: 24 days, 11:27:29 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 03:47:39,933 - Train: 18.10% [894600/4942000] [181.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 03:48:13,653 - Train: 18.10% [894700/4942000] [181.0/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 03:48:46,567 - Train: 18.11% [894800/4942000] [181.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 03:49:19,437 - Train: 18.11% [894900/4942000] [181.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 03:49:52,332 - Train: 18.11% [895000/4942000] [181.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 03:50:25,249 - Train: 18.11% [895100/4942000] [181.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 03:50:58,129 - Train: 18.11% [895200/4942000] [181.1/1000.0] [batch_t 0.337 (0.329)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-08 03:51:31,581 - Train: 18.12% [895300/4942000] [181.2/1000.0] [batch_t 0.326 (0.334)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 03:52:05,138 - Train: 18.12% [895400/4942000] [181.2/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 03:52:38,079 - Train: 18.12% [895500/4942000] [181.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 03:53:10,970 - Train: 18.12% [895600/4942000] [181.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 03:53:43,838 - Train: 18.12% [895700/4942000] [181.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 03:54:16,851 - Train: 18.13% [895800/4942000] [181.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 03:54:49,766 - Train: 18.13% [895900/4942000] [181.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 03:55:22,616 - Train: 18.13% [896000/4942000] [181.3/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 03:55:55,392 - Train: 18.13% [896100/4942000] [181.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 03:56:28,248 - Train: 18.13% [896200/4942000] [181.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 03:57:01,159 - Train: 18.14% [896300/4942000] [181.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 03:57:34,599 - Train: 18.14% [896400/4942000] [181.4/1000.0] [batch_t 0.324 (0.334)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 03:58:08,145 - Train: 18.14% [896500/4942000] [181.4/1000.0] [batch_t 0.336 (0.335)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-08 03:58:40,939 - Train: 18.14% [896600/4942000] [181.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 03:59:14,709 - Train: 18.14% [896700/4942000] [181.4/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 03:59:47,583 - Train: 18.15% [896800/4942000] [181.5/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 04:00:20,520 - Train: 18.15% [896900/4942000] [181.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:00:53,449 - Train: 18.15% [897000/4942000] [181.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 04:01:27,253 - Train: 18.15% [897100/4942000] [181.5/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 04:02:00,061 - Train: 18.15% [897200/4942000] [181.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 04:02:32,901 - Train: 18.16% [897300/4942000] [181.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 04:03:05,938 - Train: 18.16% [897400/4942000] [181.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 04:03:38,838 - Train: 18.16% [897500/4942000] [181.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 04:04:12,081 - Train: 18.16% [897600/4942000] [181.6/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 04:04:44,932 - Train: 18.16% [897700/4942000] [181.6/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 04:05:17,723 - Train: 18.17% [897800/4942000] [181.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:05:50,450 - Train: 18.17% [897900/4942000] [181.7/1000.0] [batch_t 0.334 (0.327)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 04:06:23,625 - Train: 18.17% [898000/4942000] [181.7/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 04:06:56,439 - Train: 18.17% [898100/4942000] [181.7/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 04:07:29,300 - Train: 18.17% [898200/4942000] [181.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 04:08:02,141 - Train: 18.18% [898300/4942000] [181.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 04:08:35,738 - Train: 18.18% [898400/4942000] [181.8/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 04:09:08,661 - Train: 18.18% [898500/4942000] [181.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 04:09:41,531 - Train: 18.18% [898600/4942000] [181.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 04:10:14,902 - Train: 18.18% [898700/4942000] [181.8/1000.0] [batch_t 0.334 (0.334)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 04:10:47,719 - Train: 18.19% [898800/4942000] [181.9/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 04:11:21,451 - Train: 18.19% [898900/4942000] [181.9/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 04:11:54,271 - Train: 18.19% [899000/4942000] [181.9/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 04:12:27,725 - Train: 18.19% [899100/4942000] [181.9/1000.0] [batch_t 0.330 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 04:13:00,587 - Train: 18.20% [899200/4942000] [182.0/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 04:13:35,329 - Train: 18.20% [899300/4942000] [182.0/1000.0] [batch_t 0.325 (0.347)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 04:14:08,164 - Train: 18.20% [899400/4942000] [182.0/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 04:14:22,609 - ==> Total time: 5 days, 10:17:01 Eta: 24 days, 9:33:41 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 04:14:43,296 - Train: 18.20% [899500/4942000] [182.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:15:19,025 - Train: 18.20% [899600/4942000] [182.0/1000.0] [batch_t 0.331 (0.357)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 04:15:51,840 - Train: 18.21% [899700/4942000] [182.1/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 04:16:26,188 - Train: 18.21% [899800/4942000] [182.1/1000.0] [batch_t 0.331 (0.343)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 04:16:58,944 - Train: 18.21% [899900/4942000] [182.1/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 04:17:31,784 - Train: 18.21% [900000/4942000] [182.1/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 04:18:05,040 - Train: 18.21% [900100/4942000] [182.1/1000.0] [batch_t 0.756 (0.332)] [data_t 0.426] [optim_t 0.330] [lr 0.005000] 2024-04-08 04:18:37,898 - Train: 18.22% [900200/4942000] [182.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 04:19:10,670 - Train: 18.22% [900300/4942000] [182.2/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 04:19:43,511 - Train: 18.22% [900400/4942000] [182.2/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 04:20:16,405 - Train: 18.22% [900500/4942000] [182.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 04:20:49,251 - Train: 18.22% [900600/4942000] [182.2/1000.0] [batch_t 0.319 (0.328)] [data_t 0.002] [optim_t 0.317] [lr 0.005000] 2024-04-08 04:21:22,089 - Train: 18.23% [900700/4942000] [182.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 04:21:54,893 - Train: 18.23% [900800/4942000] [182.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:22:27,716 - Train: 18.23% [900900/4942000] [182.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 04:23:00,536 - Train: 18.23% [901000/4942000] [182.3/1000.0] [batch_t 0.336 (0.328)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-08 04:23:33,388 - Train: 18.23% [901100/4942000] [182.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:24:06,274 - Train: 18.24% [901200/4942000] [182.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 04:24:39,142 - Train: 18.24% [901300/4942000] [182.4/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 04:25:11,916 - Train: 18.24% [901400/4942000] [182.4/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 04:25:44,720 - Train: 18.24% [901500/4942000] [182.4/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 04:26:17,626 - Train: 18.24% [901600/4942000] [182.4/1000.0] [batch_t 0.322 (0.329)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 04:26:50,363 - Train: 18.25% [901700/4942000] [182.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:27:23,078 - Train: 18.25% [901800/4942000] [182.5/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 04:27:55,870 - Train: 18.25% [901900/4942000] [182.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:28:28,595 - Train: 18.25% [902000/4942000] [182.5/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 04:29:01,429 - Train: 18.25% [902100/4942000] [182.5/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 04:29:34,239 - Train: 18.26% [902200/4942000] [182.6/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 04:30:06,976 - Train: 18.26% [902300/4942000] [182.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 04:30:39,742 - Train: 18.26% [902400/4942000] [182.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 04:31:12,493 - Train: 18.26% [902500/4942000] [182.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 04:31:45,299 - Train: 18.26% [902600/4942000] [182.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 04:32:18,128 - Train: 18.27% [902700/4942000] [182.7/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 04:32:50,830 - Train: 18.27% [902800/4942000] [182.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 04:33:23,673 - Train: 18.27% [902900/4942000] [182.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 04:33:56,427 - Train: 18.27% [903000/4942000] [182.7/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 04:34:29,330 - Train: 18.27% [903100/4942000] [182.7/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 04:35:02,157 - Train: 18.28% [903200/4942000] [182.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 04:35:35,981 - Train: 18.28% [903300/4942000] [182.8/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 04:36:08,770 - Train: 18.28% [903400/4942000] [182.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:36:41,597 - Train: 18.28% [903500/4942000] [182.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 04:37:14,475 - Train: 18.28% [903600/4942000] [182.8/1000.0] [batch_t 0.338 (0.329)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-08 04:37:47,297 - Train: 18.29% [903700/4942000] [182.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:38:20,528 - Train: 18.29% [903800/4942000] [182.9/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 04:38:53,335 - Train: 18.29% [903900/4942000] [182.9/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 04:39:26,122 - Train: 18.29% [904000/4942000] [182.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 04:39:58,927 - Train: 18.29% [904100/4942000] [182.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 04:40:32,214 - Train: 18.30% [904200/4942000] [183.0/1000.0] [batch_t 0.334 (0.333)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 04:41:05,060 - Train: 18.30% [904300/4942000] [183.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 04:41:33,255 - ==> Total time: 5 days, 10:44:12 Eta: 24 days, 7:40:18 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 04:41:40,229 - Train: 18.30% [904400/4942000] [183.0/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 04:42:13,778 - Train: 18.30% [904500/4942000] [183.0/1000.0] [batch_t 0.323 (0.335)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 04:42:46,559 - Train: 18.30% [904600/4942000] [183.0/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 04:43:19,267 - Train: 18.31% [904700/4942000] [183.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 04:43:52,114 - Train: 18.31% [904800/4942000] [183.1/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 04:44:25,345 - Train: 18.31% [904900/4942000] [183.1/1000.0] [batch_t 0.321 (0.332)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-08 04:44:58,157 - Train: 18.31% [905000/4942000] [183.1/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 04:45:31,047 - Train: 18.31% [905100/4942000] [183.1/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 04:46:03,787 - Train: 18.32% [905200/4942000] [183.2/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 04:46:36,563 - Train: 18.32% [905300/4942000] [183.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 04:47:09,487 - Train: 18.32% [905400/4942000] [183.2/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 04:47:42,223 - Train: 18.32% [905500/4942000] [183.2/1000.0] [batch_t 0.321 (0.327)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-08 04:48:14,983 - Train: 18.32% [905600/4942000] [183.2/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 04:48:47,799 - Train: 18.33% [905700/4942000] [183.3/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 04:49:20,545 - Train: 18.33% [905800/4942000] [183.3/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 04:49:53,396 - Train: 18.33% [905900/4942000] [183.3/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 04:50:26,581 - Train: 18.33% [906000/4942000] [183.3/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:50:59,362 - Train: 18.33% [906100/4942000] [183.3/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 04:51:32,102 - Train: 18.34% [906200/4942000] [183.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:52:05,708 - Train: 18.34% [906300/4942000] [183.4/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:52:38,431 - Train: 18.34% [906400/4942000] [183.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 04:53:11,187 - Train: 18.34% [906500/4942000] [183.4/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 04:53:43,932 - Train: 18.34% [906600/4942000] [183.4/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 04:54:17,557 - Train: 18.35% [906700/4942000] [183.5/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 04:54:50,486 - Train: 18.35% [906800/4942000] [183.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:55:24,137 - Train: 18.35% [906900/4942000] [183.5/1000.0] [batch_t 0.325 (0.336)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 04:55:57,111 - Train: 18.35% [907000/4942000] [183.5/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 04:56:30,782 - Train: 18.35% [907100/4942000] [183.5/1000.0] [batch_t 0.333 (0.337)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 04:57:03,666 - Train: 18.36% [907200/4942000] [183.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 04:57:39,690 - Train: 18.36% [907300/4942000] [183.6/1000.0] [batch_t 0.328 (0.360)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 04:58:12,624 - Train: 18.36% [907400/4942000] [183.6/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 04:58:45,554 - Train: 18.36% [907500/4942000] [183.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:59:18,493 - Train: 18.37% [907600/4942000] [183.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 04:59:51,371 - Train: 18.37% [907700/4942000] [183.7/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 05:00:24,349 - Train: 18.37% [907800/4942000] [183.7/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 05:00:57,209 - Train: 18.37% [907900/4942000] [183.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 05:01:30,076 - Train: 18.37% [908000/4942000] [183.7/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 05:02:02,934 - Train: 18.38% [908100/4942000] [183.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 05:02:36,518 - Train: 18.38% [908200/4942000] [183.8/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 05:03:09,507 - Train: 18.38% [908300/4942000] [183.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 05:03:42,414 - Train: 18.38% [908400/4942000] [183.8/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 05:04:15,368 - Train: 18.38% [908500/4942000] [183.8/1000.0] [batch_t 0.338 (0.329)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-08 05:04:48,270 - Train: 18.39% [908600/4942000] [183.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 05:05:21,390 - Train: 18.39% [908700/4942000] [183.9/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 05:05:54,303 - Train: 18.39% [908800/4942000] [183.9/1000.0] [batch_t 0.337 (0.329)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-08 05:06:27,895 - Train: 18.39% [908900/4942000] [183.9/1000.0] [batch_t 0.326 (0.336)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 05:07:00,806 - Train: 18.39% [909000/4942000] [183.9/1000.0] [batch_t 0.338 (0.329)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-08 05:07:34,627 - Train: 18.40% [909100/4942000] [184.0/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 05:08:07,479 - Train: 18.40% [909200/4942000] [184.0/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 05:08:40,387 - Train: 18.40% [909300/4942000] [184.0/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 05:08:49,607 - ==> Total time: 5 days, 11:11:28 Eta: 24 days, 5:48:18 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 05:09:17,705 - Train: 18.40% [909400/4942000] [184.0/1000.0] [batch_t 0.330 (0.356)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 05:09:50,454 - Train: 18.40% [909500/4942000] [184.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 05:10:24,301 - Train: 18.41% [909600/4942000] [184.1/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 05:10:57,046 - Train: 18.41% [909700/4942000] [184.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 05:11:29,815 - Train: 18.41% [909800/4942000] [184.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 05:12:02,660 - Train: 18.41% [909900/4942000] [184.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 05:12:35,581 - Train: 18.41% [910000/4942000] [184.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 05:13:08,587 - Train: 18.42% [910100/4942000] [184.2/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 05:13:41,425 - Train: 18.42% [910200/4942000] [184.2/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 05:14:14,378 - Train: 18.42% [910300/4942000] [184.2/1000.0] [batch_t 0.339 (0.329)] [data_t 0.002] [optim_t 0.337] [lr 0.005000] 2024-04-08 05:14:47,260 - Train: 18.42% [910400/4942000] [184.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 05:15:20,213 - Train: 18.42% [910500/4942000] [184.2/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 05:15:53,056 - Train: 18.43% [910600/4942000] [184.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 05:16:26,527 - Train: 18.43% [910700/4942000] [184.3/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 05:16:59,431 - Train: 18.43% [910800/4942000] [184.3/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 05:17:32,312 - Train: 18.43% [910900/4942000] [184.3/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 05:18:05,989 - Train: 18.43% [911000/4942000] [184.3/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 05:18:38,823 - Train: 18.44% [911100/4942000] [184.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 05:19:12,595 - Train: 18.44% [911200/4942000] [184.4/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 05:19:45,479 - Train: 18.44% [911300/4942000] [184.4/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 05:20:18,341 - Train: 18.44% [911400/4942000] [184.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 05:20:51,282 - Train: 18.44% [911500/4942000] [184.4/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 05:21:24,767 - Train: 18.45% [911600/4942000] [184.5/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 05:21:57,672 - Train: 18.45% [911700/4942000] [184.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 05:22:30,505 - Train: 18.45% [911800/4942000] [184.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 05:23:03,368 - Train: 18.45% [911900/4942000] [184.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 05:23:36,715 - Train: 18.45% [912000/4942000] [184.5/1000.0] [batch_t 0.324 (0.333)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 05:24:09,601 - Train: 18.46% [912100/4942000] [184.6/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 05:24:42,545 - Train: 18.46% [912200/4942000] [184.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 05:25:15,433 - Train: 18.46% [912300/4942000] [184.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 05:25:48,284 - Train: 18.46% [912400/4942000] [184.6/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 05:26:21,809 - Train: 18.46% [912500/4942000] [184.6/1000.0] [batch_t 0.324 (0.335)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 05:26:54,685 - Train: 18.47% [912600/4942000] [184.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 05:27:28,146 - Train: 18.47% [912700/4942000] [184.7/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 05:28:00,942 - Train: 18.47% [912800/4942000] [184.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 05:28:34,889 - Train: 18.47% [912900/4942000] [184.7/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 05:29:07,728 - Train: 18.47% [913000/4942000] [184.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 05:29:40,553 - Train: 18.48% [913100/4942000] [184.8/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 05:30:14,253 - Train: 18.48% [913200/4942000] [184.8/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 05:30:47,094 - Train: 18.48% [913300/4942000] [184.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 05:31:19,958 - Train: 18.48% [913400/4942000] [184.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 05:31:52,854 - Train: 18.48% [913500/4942000] [184.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 05:32:25,719 - Train: 18.49% [913600/4942000] [184.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 05:32:58,587 - Train: 18.49% [913700/4942000] [184.9/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 05:33:31,533 - Train: 18.49% [913800/4942000] [184.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 05:34:04,435 - Train: 18.49% [913900/4942000] [184.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 05:34:37,403 - Train: 18.49% [914000/4942000] [184.9/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 05:35:10,321 - Train: 18.50% [914100/4942000] [185.0/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 05:35:43,327 - Train: 18.50% [914200/4942000] [185.0/1000.0] [batch_t 0.337 (0.330)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-08 05:36:07,051 - ==> Total time: 5 days, 11:38:46 Eta: 24 days, 3:57:17 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 05:36:19,615 - Train: 18.50% [914300/4942000] [185.0/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 05:36:52,515 - Train: 18.50% [914400/4942000] [185.0/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 05:37:26,076 - Train: 18.50% [914500/4942000] [185.0/1000.0] [batch_t 0.323 (0.336)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 05:37:58,917 - Train: 18.51% [914600/4942000] [185.1/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 05:38:32,560 - Train: 18.51% [914700/4942000] [185.1/1000.0] [batch_t 0.333 (0.336)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 05:39:06,152 - Train: 18.51% [914800/4942000] [185.1/1000.0] [batch_t 1.054 (0.336)] [data_t 0.732] [optim_t 0.323] [lr 0.005000] 2024-04-08 05:39:38,969 - Train: 18.51% [914900/4942000] [185.1/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 05:40:12,322 - Train: 18.51% [915000/4942000] [185.1/1000.0] [batch_t 0.324 (0.333)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 05:40:45,162 - Train: 18.52% [915100/4942000] [185.2/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 05:41:19,145 - Train: 18.52% [915200/4942000] [185.2/1000.0] [batch_t 0.325 (0.340)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 05:41:51,906 - Train: 18.52% [915300/4942000] [185.2/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 05:42:25,380 - Train: 18.52% [915400/4942000] [185.2/1000.0] [batch_t 0.331 (0.335)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 05:42:58,207 - Train: 18.52% [915500/4942000] [185.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 05:43:31,952 - Train: 18.53% [915600/4942000] [185.3/1000.0] [batch_t 0.326 (0.337)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 05:44:05,478 - Train: 18.53% [915700/4942000] [185.3/1000.0] [batch_t 1.088 (0.335)] [data_t 0.765] [optim_t 0.323] [lr 0.005000] 2024-04-08 05:44:38,400 - Train: 18.53% [915800/4942000] [185.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 05:45:12,468 - Train: 18.53% [915900/4942000] [185.3/1000.0] [batch_t 0.335 (0.341)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 05:45:45,256 - Train: 18.54% [916000/4942000] [185.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 05:46:18,056 - Train: 18.54% [916100/4942000] [185.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 05:46:50,823 - Train: 18.54% [916200/4942000] [185.4/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 05:47:24,644 - Train: 18.54% [916300/4942000] [185.4/1000.0] [batch_t 0.324 (0.338)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 05:47:57,520 - Train: 18.54% [916400/4942000] [185.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 05:48:30,926 - Train: 18.55% [916500/4942000] [185.5/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 05:49:03,698 - Train: 18.55% [916600/4942000] [185.5/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 05:49:37,826 - Train: 18.55% [916700/4942000] [185.5/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 05:50:11,380 - Train: 18.55% [916800/4942000] [185.5/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 05:50:44,209 - Train: 18.55% [916900/4942000] [185.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 05:51:18,100 - Train: 18.56% [917000/4942000] [185.6/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 05:51:50,843 - Train: 18.56% [917100/4942000] [185.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 05:52:23,817 - Train: 18.56% [917200/4942000] [185.6/1000.0] [batch_t 0.323 (0.330)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 05:52:56,573 - Train: 18.56% [917300/4942000] [185.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 05:53:29,402 - Train: 18.56% [917400/4942000] [185.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 05:54:02,307 - Train: 18.57% [917500/4942000] [185.7/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 05:54:36,581 - Train: 18.57% [917600/4942000] [185.7/1000.0] [batch_t 0.323 (0.343)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 05:55:10,326 - Train: 18.57% [917700/4942000] [185.7/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 05:55:43,155 - Train: 18.57% [917800/4942000] [185.7/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 05:56:16,034 - Train: 18.57% [917900/4942000] [185.7/1000.0] [batch_t 0.340 (0.329)] [data_t 0.002] [optim_t 0.337] [lr 0.005000] 2024-04-08 05:56:48,838 - Train: 18.58% [918000/4942000] [185.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 05:57:22,594 - Train: 18.58% [918100/4942000] [185.8/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 05:57:55,580 - Train: 18.58% [918200/4942000] [185.8/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 05:58:29,685 - Train: 18.58% [918300/4942000] [185.8/1000.0] [batch_t 0.325 (0.341)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 05:59:02,458 - Train: 18.58% [918400/4942000] [185.8/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 05:59:35,289 - Train: 18.59% [918500/4942000] [185.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 06:00:10,342 - Train: 18.59% [918600/4942000] [185.9/1000.0] [batch_t 0.327 (0.350)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 06:00:43,323 - Train: 18.59% [918700/4942000] [185.9/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 06:01:16,162 - Train: 18.59% [918800/4942000] [185.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 06:01:48,949 - Train: 18.59% [918900/4942000] [185.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 06:02:22,785 - Train: 18.60% [919000/4942000] [186.0/1000.0] [batch_t 0.324 (0.338)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 06:02:55,567 - Train: 18.60% [919100/4942000] [186.0/1000.0] [batch_t 0.340 (0.328)] [data_t 0.002] [optim_t 0.338] [lr 0.005000] 2024-04-08 06:03:29,431 - Train: 18.60% [919200/4942000] [186.0/1000.0] [batch_t 0.331 (0.339)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 06:03:33,378 - ==> Total time: 5 days, 12:06:12 Eta: 24 days, 2:07:49 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 06:04:04,996 - Train: 18.60% [919300/4942000] [186.0/1000.0] [batch_t 0.542 (0.333)] [data_t 0.216] [optim_t 0.325] [lr 0.005000] 2024-04-08 06:04:38,562 - Train: 18.60% [919400/4942000] [186.0/1000.0] [batch_t 0.325 (0.336)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 06:05:12,120 - Train: 18.61% [919500/4942000] [186.1/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 06:05:44,811 - Train: 18.61% [919600/4942000] [186.1/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 06:06:18,867 - Train: 18.61% [919700/4942000] [186.1/1000.0] [batch_t 0.325 (0.340)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 06:06:51,556 - Train: 18.61% [919800/4942000] [186.1/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 06:07:24,869 - Train: 18.61% [919900/4942000] [186.1/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 06:07:57,995 - Train: 18.62% [920000/4942000] [186.2/1000.0] [batch_t 0.323 (0.331)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 06:08:31,536 - Train: 18.62% [920100/4942000] [186.2/1000.0] [batch_t 0.324 (0.335)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 06:09:04,474 - Train: 18.62% [920200/4942000] [186.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:09:38,635 - Train: 18.62% [920300/4942000] [186.2/1000.0] [batch_t 0.330 (0.342)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 06:10:12,486 - Train: 18.62% [920400/4942000] [186.2/1000.0] [batch_t 0.338 (0.338)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-08 06:10:45,381 - Train: 18.63% [920500/4942000] [186.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 06:11:18,817 - Train: 18.63% [920600/4942000] [186.3/1000.0] [batch_t 0.322 (0.334)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 06:11:51,714 - Train: 18.63% [920700/4942000] [186.3/1000.0] [batch_t 0.337 (0.329)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-08 06:12:25,621 - Train: 18.63% [920800/4942000] [186.3/1000.0] [batch_t 0.333 (0.339)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 06:12:58,421 - Train: 18.63% [920900/4942000] [186.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 06:13:31,183 - Train: 18.64% [921000/4942000] [186.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:14:04,032 - Train: 18.64% [921100/4942000] [186.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:14:37,922 - Train: 18.64% [921200/4942000] [186.4/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 06:15:10,634 - Train: 18.64% [921300/4942000] [186.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 06:15:43,632 - Train: 18.64% [921400/4942000] [186.4/1000.0] [batch_t 0.319 (0.330)] [data_t 0.002] [optim_t 0.317] [lr 0.005000] 2024-04-08 06:16:17,118 - Train: 18.65% [921500/4942000] [186.5/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 06:16:49,906 - Train: 18.65% [921600/4942000] [186.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 06:17:23,601 - Train: 18.65% [921700/4942000] [186.5/1000.0] [batch_t 0.324 (0.337)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 06:17:56,321 - Train: 18.65% [921800/4942000] [186.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 06:18:30,073 - Train: 18.65% [921900/4942000] [186.5/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 06:19:02,744 - Train: 18.66% [922000/4942000] [186.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:19:37,065 - Train: 18.66% [922100/4942000] [186.6/1000.0] [batch_t 0.329 (0.343)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:20:09,986 - Train: 18.66% [922200/4942000] [186.6/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 06:20:42,968 - Train: 18.66% [922300/4942000] [186.6/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 06:21:15,935 - Train: 18.66% [922400/4942000] [186.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:21:48,832 - Train: 18.67% [922500/4942000] [186.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:22:22,775 - Train: 18.67% [922600/4942000] [186.7/1000.0] [batch_t 0.325 (0.339)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 06:22:55,517 - Train: 18.67% [922700/4942000] [186.7/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 06:23:29,597 - Train: 18.67% [922800/4942000] [186.7/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:24:02,496 - Train: 18.67% [922900/4942000] [186.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 06:24:35,660 - Train: 18.68% [923000/4942000] [186.8/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 06:25:09,512 - Train: 18.68% [923100/4942000] [186.8/1000.0] [batch_t 0.324 (0.338)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 06:25:42,395 - Train: 18.68% [923200/4942000] [186.8/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 06:26:16,047 - Train: 18.68% [923300/4942000] [186.8/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:26:48,960 - Train: 18.68% [923400/4942000] [186.8/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 06:27:22,432 - Train: 18.69% [923500/4942000] [186.9/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:27:55,248 - Train: 18.69% [923600/4942000] [186.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 06:28:28,974 - Train: 18.69% [923700/4942000] [186.9/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 06:29:01,871 - Train: 18.69% [923800/4942000] [186.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 06:29:35,724 - Train: 18.69% [923900/4942000] [186.9/1000.0] [batch_t 0.331 (0.338)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 06:30:09,274 - Train: 18.70% [924000/4942000] [187.0/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:30:42,174 - Train: 18.70% [924100/4942000] [187.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:30:59,902 - ==> Total time: 5 days, 12:33:39 Eta: 24 days, 0:19:14 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 06:31:20,746 - Train: 18.70% [924200/4942000] [187.0/1000.0] [batch_t 0.337 (0.401)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-08 06:31:53,610 - Train: 18.70% [924300/4942000] [187.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:32:26,858 - Train: 18.70% [924400/4942000] [187.0/1000.0] [batch_t 0.332 (0.332)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 06:32:59,683 - Train: 18.71% [924500/4942000] [187.1/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 06:33:33,738 - Train: 18.71% [924600/4942000] [187.1/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:34:07,472 - Train: 18.71% [924700/4942000] [187.1/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:34:40,435 - Train: 18.71% [924800/4942000] [187.1/1000.0] [batch_t 0.341 (0.330)] [data_t 0.002] [optim_t 0.339] [lr 0.005000] 2024-04-08 06:35:14,014 - Train: 18.72% [924900/4942000] [187.2/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 06:35:46,809 - Train: 18.72% [925000/4942000] [187.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 06:36:19,653 - Train: 18.72% [925100/4942000] [187.2/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 06:36:52,494 - Train: 18.72% [925200/4942000] [187.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 06:37:26,577 - Train: 18.72% [925300/4942000] [187.2/1000.0] [batch_t 0.341 (0.341)] [data_t 0.002] [optim_t 0.339] [lr 0.005000] 2024-04-08 06:37:59,454 - Train: 18.73% [925400/4942000] [187.3/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 06:38:33,629 - Train: 18.73% [925500/4942000] [187.3/1000.0] [batch_t 0.334 (0.342)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 06:39:07,584 - Train: 18.73% [925600/4942000] [187.3/1000.0] [batch_t 0.322 (0.339)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 06:39:40,307 - Train: 18.73% [925700/4942000] [187.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 06:40:14,225 - Train: 18.73% [925800/4942000] [187.3/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 06:40:47,029 - Train: 18.74% [925900/4942000] [187.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 06:41:21,081 - Train: 18.74% [926000/4942000] [187.4/1000.0] [batch_t 0.325 (0.340)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 06:41:53,899 - Train: 18.74% [926100/4942000] [187.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 06:42:27,633 - Train: 18.74% [926200/4942000] [187.4/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:43:00,505 - Train: 18.74% [926300/4942000] [187.4/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 06:43:33,399 - Train: 18.75% [926400/4942000] [187.5/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 06:44:07,464 - Train: 18.75% [926500/4942000] [187.5/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 06:44:40,381 - Train: 18.75% [926600/4942000] [187.5/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 06:45:13,241 - Train: 18.75% [926700/4942000] [187.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 06:45:46,034 - Train: 18.75% [926800/4942000] [187.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 06:46:18,852 - Train: 18.76% [926900/4942000] [187.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 06:46:51,648 - Train: 18.76% [927000/4942000] [187.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:47:24,654 - Train: 18.76% [927100/4942000] [187.6/1000.0] [batch_t 0.323 (0.330)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 06:47:57,522 - Train: 18.76% [927200/4942000] [187.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 06:48:30,759 - Train: 18.76% [927300/4942000] [187.6/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:49:03,558 - Train: 18.77% [927400/4942000] [187.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:49:37,045 - Train: 18.77% [927500/4942000] [187.7/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 06:50:09,769 - Train: 18.77% [927600/4942000] [187.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 06:50:42,532 - Train: 18.77% [927700/4942000] [187.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 06:51:15,950 - Train: 18.77% [927800/4942000] [187.7/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 06:51:48,713 - Train: 18.78% [927900/4942000] [187.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:52:22,695 - Train: 18.78% [928000/4942000] [187.8/1000.0] [batch_t 0.325 (0.340)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 06:52:55,555 - Train: 18.78% [928100/4942000] [187.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 06:53:28,710 - Train: 18.78% [928200/4942000] [187.8/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 06:54:01,461 - Train: 18.78% [928300/4942000] [187.8/1000.0] [batch_t 0.336 (0.327)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-08 06:54:34,239 - Train: 18.79% [928400/4942000] [187.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 06:55:07,156 - Train: 18.79% [928500/4942000] [187.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 06:55:39,851 - Train: 18.79% [928600/4942000] [187.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 06:56:12,626 - Train: 18.79% [928700/4942000] [187.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 06:56:45,352 - Train: 18.79% [928800/4942000] [187.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 06:57:18,170 - Train: 18.80% [928900/4942000] [188.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:57:50,880 - Train: 18.80% [929000/4942000] [188.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 06:58:22,386 - ==> Total time: 5 days, 13:01:01 Eta: 23 days, 22:31:14 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 06:58:25,965 - Train: 18.80% [929100/4942000] [188.0/1000.0] [batch_t 0.323 (0.332)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 06:58:58,732 - Train: 18.80% [929200/4942000] [188.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 06:59:32,156 - Train: 18.80% [929300/4942000] [188.0/1000.0] [batch_t 0.324 (0.334)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 07:00:04,870 - Train: 18.81% [929400/4942000] [188.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 07:00:37,597 - Train: 18.81% [929500/4942000] [188.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:01:10,815 - Train: 18.81% [929600/4942000] [188.1/1000.0] [batch_t 0.325 (0.332)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 07:01:43,593 - Train: 18.81% [929700/4942000] [188.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 07:02:16,643 - Train: 18.81% [929800/4942000] [188.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 07:02:49,530 - Train: 18.82% [929900/4942000] [188.2/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 07:03:23,384 - Train: 18.82% [930000/4942000] [188.2/1000.0] [batch_t 0.333 (0.338)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 07:03:56,312 - Train: 18.82% [930100/4942000] [188.2/1000.0] [batch_t 0.336 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-08 07:04:29,157 - Train: 18.82% [930200/4942000] [188.2/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 07:05:02,149 - Train: 18.82% [930300/4942000] [188.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 07:05:35,689 - Train: 18.83% [930400/4942000] [188.3/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 07:06:08,398 - Train: 18.83% [930500/4942000] [188.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:06:41,193 - Train: 18.83% [930600/4942000] [188.3/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 07:07:14,247 - Train: 18.83% [930700/4942000] [188.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 07:07:47,000 - Train: 18.83% [930800/4942000] [188.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 07:08:20,746 - Train: 18.84% [930900/4942000] [188.4/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:08:53,468 - Train: 18.84% [931000/4942000] [188.4/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 07:09:26,561 - Train: 18.84% [931100/4942000] [188.4/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 07:09:59,488 - Train: 18.84% [931200/4942000] [188.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 07:10:32,221 - Train: 18.84% [931300/4942000] [188.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:11:04,985 - Train: 18.85% [931400/4942000] [188.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 07:11:37,677 - Train: 18.85% [931500/4942000] [188.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:12:10,518 - Train: 18.85% [931600/4942000] [188.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 07:12:43,398 - Train: 18.85% [931700/4942000] [188.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:13:17,134 - Train: 18.85% [931800/4942000] [188.5/1000.0] [batch_t 0.324 (0.337)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 07:13:50,020 - Train: 18.86% [931900/4942000] [188.6/1000.0] [batch_t 0.325 (0.329)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-08 07:14:22,861 - Train: 18.86% [932000/4942000] [188.6/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 07:14:55,772 - Train: 18.86% [932100/4942000] [188.6/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 07:15:28,563 - Train: 18.86% [932200/4942000] [188.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 07:16:01,403 - Train: 18.86% [932300/4942000] [188.6/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 07:16:34,340 - Train: 18.87% [932400/4942000] [188.7/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 07:17:07,218 - Train: 18.87% [932500/4942000] [188.7/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 07:17:40,106 - Train: 18.87% [932600/4942000] [188.7/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 07:18:13,120 - Train: 18.87% [932700/4942000] [188.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 07:18:46,011 - Train: 18.87% [932800/4942000] [188.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:19:19,348 - Train: 18.88% [932900/4942000] [188.8/1000.0] [batch_t 0.326 (0.333)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 07:19:52,135 - Train: 18.88% [933000/4942000] [188.8/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 07:20:25,566 - Train: 18.88% [933100/4942000] [188.8/1000.0] [batch_t 0.330 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 07:20:58,344 - Train: 18.88% [933200/4942000] [188.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 07:21:31,464 - Train: 18.89% [933300/4942000] [188.9/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:22:04,390 - Train: 18.89% [933400/4942000] [188.9/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 07:22:37,269 - Train: 18.89% [933500/4942000] [188.9/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 07:23:10,388 - Train: 18.89% [933600/4942000] [188.9/1000.0] [batch_t 0.325 (0.331)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 07:23:43,193 - Train: 18.89% [933700/4942000] [188.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 07:24:16,555 - Train: 18.90% [933800/4942000] [189.0/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 07:24:49,377 - Train: 18.90% [933900/4942000] [189.0/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 07:25:23,392 - Train: 18.90% [934000/4942000] [189.0/1000.0] [batch_t 0.325 (0.340)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 07:25:35,900 - ==> Total time: 5 days, 13:28:15 Eta: 23 days, 20:43:26 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 07:25:58,675 - Train: 18.90% [934100/4942000] [189.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 07:26:31,457 - Train: 18.90% [934200/4942000] [189.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 07:27:04,230 - Train: 18.91% [934300/4942000] [189.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 07:27:37,005 - Train: 18.91% [934400/4942000] [189.1/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 07:28:09,779 - Train: 18.91% [934500/4942000] [189.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 07:28:42,572 - Train: 18.91% [934600/4942000] [189.1/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 07:29:15,424 - Train: 18.91% [934700/4942000] [189.1/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 07:29:48,265 - Train: 18.92% [934800/4942000] [189.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:30:20,995 - Train: 18.92% [934900/4942000] [189.2/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 07:30:53,807 - Train: 18.92% [935000/4942000] [189.2/1000.0] [batch_t 0.337 (0.328)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-08 07:31:26,586 - Train: 18.92% [935100/4942000] [189.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 07:31:59,376 - Train: 18.92% [935200/4942000] [189.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 07:32:33,347 - Train: 18.93% [935300/4942000] [189.3/1000.0] [batch_t 0.331 (0.340)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 07:33:06,123 - Train: 18.93% [935400/4942000] [189.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 07:33:39,104 - Train: 18.93% [935500/4942000] [189.3/1000.0] [batch_t 0.323 (0.330)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 07:34:11,927 - Train: 18.93% [935600/4942000] [189.3/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 07:34:44,667 - Train: 18.93% [935700/4942000] [189.3/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 07:35:19,283 - Train: 18.94% [935800/4942000] [189.4/1000.0] [batch_t 0.329 (0.346)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:35:52,078 - Train: 18.94% [935900/4942000] [189.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 07:36:26,618 - Train: 18.94% [936000/4942000] [189.4/1000.0] [batch_t 0.332 (0.345)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 07:36:59,422 - Train: 18.94% [936100/4942000] [189.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 07:37:32,189 - Train: 18.94% [936200/4942000] [189.4/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 07:38:05,016 - Train: 18.95% [936300/4942000] [189.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 07:38:37,804 - Train: 18.95% [936400/4942000] [189.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 07:39:11,794 - Train: 18.95% [936500/4942000] [189.5/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 07:39:44,573 - Train: 18.95% [936600/4942000] [189.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:40:17,429 - Train: 18.95% [936700/4942000] [189.5/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 07:40:50,254 - Train: 18.96% [936800/4942000] [189.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 07:41:23,858 - Train: 18.96% [936900/4942000] [189.6/1000.0] [batch_t 0.338 (0.336)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-08 07:41:56,664 - Train: 18.96% [937000/4942000] [189.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 07:42:29,496 - Train: 18.96% [937100/4942000] [189.6/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 07:43:02,255 - Train: 18.96% [937200/4942000] [189.6/1000.0] [batch_t 0.339 (0.327)] [data_t 0.002] [optim_t 0.337] [lr 0.005000] 2024-04-08 07:43:35,015 - Train: 18.97% [937300/4942000] [189.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 07:44:08,419 - Train: 18.97% [937400/4942000] [189.7/1000.0] [batch_t 0.325 (0.334)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 07:44:41,199 - Train: 18.97% [937500/4942000] [189.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 07:45:14,013 - Train: 18.97% [937600/4942000] [189.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 07:45:46,847 - Train: 18.97% [937700/4942000] [189.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 07:46:19,689 - Train: 18.98% [937800/4942000] [189.8/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 07:46:52,398 - Train: 18.98% [937900/4942000] [189.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 07:47:26,399 - Train: 18.98% [938000/4942000] [189.8/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 07:47:59,210 - Train: 18.98% [938100/4942000] [189.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 07:48:32,830 - Train: 18.98% [938200/4942000] [189.8/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 07:49:05,705 - Train: 18.99% [938300/4942000] [189.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 07:49:38,510 - Train: 18.99% [938400/4942000] [189.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:50:11,311 - Train: 18.99% [938500/4942000] [189.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 07:50:44,119 - Train: 18.99% [938600/4942000] [189.9/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 07:51:17,532 - Train: 18.99% [938700/4942000] [189.9/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 07:51:50,342 - Train: 19.00% [938800/4942000] [190.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:52:23,630 - Train: 19.00% [938900/4942000] [190.0/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 07:52:49,898 - ==> Total time: 5 days, 13:55:29 Eta: 23 days, 18:56:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 07:52:58,795 - Train: 19.00% [939000/4942000] [190.0/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 07:53:32,365 - Train: 19.00% [939100/4942000] [190.0/1000.0] [batch_t 0.333 (0.336)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 07:54:05,114 - Train: 19.00% [939200/4942000] [190.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 07:54:37,843 - Train: 19.01% [939300/4942000] [190.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:55:11,402 - Train: 19.01% [939400/4942000] [190.1/1000.0] [batch_t 0.324 (0.335)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 07:55:44,172 - Train: 19.01% [939500/4942000] [190.1/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 07:56:17,899 - Train: 19.01% [939600/4942000] [190.1/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:56:50,653 - Train: 19.01% [939700/4942000] [190.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:57:23,578 - Train: 19.02% [939800/4942000] [190.2/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 07:57:56,327 - Train: 19.02% [939900/4942000] [190.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 07:58:29,044 - Train: 19.02% [940000/4942000] [190.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 07:59:01,779 - Train: 19.02% [940100/4942000] [190.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 07:59:35,446 - Train: 19.02% [940200/4942000] [190.2/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:00:08,204 - Train: 19.03% [940300/4942000] [190.3/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 08:00:40,969 - Train: 19.03% [940400/4942000] [190.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 08:01:13,720 - Train: 19.03% [940500/4942000] [190.3/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 08:01:46,433 - Train: 19.03% [940600/4942000] [190.3/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 08:02:19,240 - Train: 19.03% [940700/4942000] [190.3/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 08:02:52,033 - Train: 19.04% [940800/4942000] [190.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 08:03:25,938 - Train: 19.04% [940900/4942000] [190.4/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 08:03:58,744 - Train: 19.04% [941000/4942000] [190.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:04:32,133 - Train: 19.04% [941100/4942000] [190.4/1000.0] [batch_t 0.333 (0.334)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 08:05:04,876 - Train: 19.04% [941200/4942000] [190.4/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 08:05:38,314 - Train: 19.05% [941300/4942000] [190.5/1000.0] [batch_t 0.333 (0.334)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 08:06:11,902 - Train: 19.05% [941400/4942000] [190.5/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:06:44,808 - Train: 19.05% [941500/4942000] [190.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 08:07:17,634 - Train: 19.05% [941600/4942000] [190.5/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 08:07:50,465 - Train: 19.06% [941700/4942000] [190.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:08:23,310 - Train: 19.06% [941800/4942000] [190.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:08:56,217 - Train: 19.06% [941900/4942000] [190.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:09:29,666 - Train: 19.06% [942000/4942000] [190.6/1000.0] [batch_t 0.326 (0.334)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 08:10:02,472 - Train: 19.06% [942100/4942000] [190.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 08:10:35,309 - Train: 19.07% [942200/4942000] [190.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 08:11:08,853 - Train: 19.07% [942300/4942000] [190.7/1000.0] [batch_t 0.321 (0.335)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-08 08:11:41,678 - Train: 19.07% [942400/4942000] [190.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 08:12:15,398 - Train: 19.07% [942500/4942000] [190.7/1000.0] [batch_t 0.336 (0.337)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-08 08:12:48,223 - Train: 19.07% [942600/4942000] [190.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 08:13:21,841 - Train: 19.08% [942700/4942000] [190.8/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:13:54,613 - Train: 19.08% [942800/4942000] [190.8/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 08:14:28,009 - Train: 19.08% [942900/4942000] [190.8/1000.0] [batch_t 0.330 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 08:15:00,802 - Train: 19.08% [943000/4942000] [190.8/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 08:15:33,649 - Train: 19.08% [943100/4942000] [190.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:16:07,594 - Train: 19.09% [943200/4942000] [190.9/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 08:16:40,481 - Train: 19.09% [943300/4942000] [190.9/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 08:17:13,316 - Train: 19.09% [943400/4942000] [190.9/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 08:17:46,069 - Train: 19.09% [943500/4942000] [190.9/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 08:18:18,895 - Train: 19.09% [943600/4942000] [190.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 08:18:51,627 - Train: 19.10% [943700/4942000] [191.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:19:26,045 - Train: 19.10% [943800/4942000] [191.0/1000.0] [batch_t 0.329 (0.344)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:19:58,926 - Train: 19.10% [943900/4942000] [191.0/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 08:20:06,136 - ==> Total time: 5 days, 14:22:45 Eta: 23 days, 17:10:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 08:20:34,281 - Train: 19.10% [944000/4942000] [191.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:21:07,118 - Train: 19.10% [944100/4942000] [191.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 08:21:39,895 - Train: 19.11% [944200/4942000] [191.1/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 08:22:12,646 - Train: 19.11% [944300/4942000] [191.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 08:22:45,362 - Train: 19.11% [944400/4942000] [191.1/1000.0] [batch_t 0.335 (0.327)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 08:23:18,176 - Train: 19.11% [944500/4942000] [191.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 08:23:50,976 - Train: 19.11% [944600/4942000] [191.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 08:24:23,714 - Train: 19.12% [944700/4942000] [191.2/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 08:24:56,558 - Train: 19.12% [944800/4942000] [191.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:25:30,016 - Train: 19.12% [944900/4942000] [191.2/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:26:02,842 - Train: 19.12% [945000/4942000] [191.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:26:36,965 - Train: 19.12% [945100/4942000] [191.2/1000.0] [batch_t 0.331 (0.341)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 08:27:09,970 - Train: 19.13% [945200/4942000] [191.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:27:42,870 - Train: 19.13% [945300/4942000] [191.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 08:28:15,736 - Train: 19.13% [945400/4942000] [191.3/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 08:28:48,576 - Train: 19.13% [945500/4942000] [191.3/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 08:29:21,379 - Train: 19.13% [945600/4942000] [191.3/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 08:29:54,265 - Train: 19.14% [945700/4942000] [191.4/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 08:30:27,217 - Train: 19.14% [945800/4942000] [191.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 08:31:00,041 - Train: 19.14% [945900/4942000] [191.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 08:31:33,474 - Train: 19.14% [946000/4942000] [191.4/1000.0] [batch_t 0.326 (0.334)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 08:32:06,936 - Train: 19.14% [946100/4942000] [191.4/1000.0] [batch_t 0.324 (0.335)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 08:32:39,817 - Train: 19.15% [946200/4942000] [191.5/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 08:33:12,813 - Train: 19.15% [946300/4942000] [191.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 08:33:45,657 - Train: 19.15% [946400/4942000] [191.5/1000.0] [batch_t 0.321 (0.328)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-08 08:34:19,917 - Train: 19.15% [946500/4942000] [191.5/1000.0] [batch_t 0.328 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:34:52,852 - Train: 19.15% [946600/4942000] [191.5/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 08:35:25,752 - Train: 19.16% [946700/4942000] [191.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 08:35:58,672 - Train: 19.16% [946800/4942000] [191.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 08:36:31,631 - Train: 19.16% [946900/4942000] [191.6/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 08:37:04,543 - Train: 19.16% [947000/4942000] [191.6/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 08:37:37,468 - Train: 19.16% [947100/4942000] [191.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 08:38:10,388 - Train: 19.17% [947200/4942000] [191.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:38:43,233 - Train: 19.17% [947300/4942000] [191.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 08:39:16,477 - Train: 19.17% [947400/4942000] [191.7/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 08:39:49,318 - Train: 19.17% [947500/4942000] [191.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:40:22,212 - Train: 19.17% [947600/4942000] [191.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:40:55,060 - Train: 19.18% [947700/4942000] [191.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 08:41:28,754 - Train: 19.18% [947800/4942000] [191.8/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:42:01,692 - Train: 19.18% [947900/4942000] [191.8/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 08:42:34,599 - Train: 19.18% [948000/4942000] [191.8/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 08:43:07,607 - Train: 19.18% [948100/4942000] [191.8/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 08:43:40,454 - Train: 19.19% [948200/4942000] [191.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 08:44:14,778 - Train: 19.19% [948300/4942000] [191.9/1000.0] [batch_t 0.324 (0.343)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 08:44:47,634 - Train: 19.19% [948400/4942000] [191.9/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 08:45:20,537 - Train: 19.19% [948500/4942000] [191.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:45:53,503 - Train: 19.19% [948600/4942000] [191.9/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 08:46:26,953 - Train: 19.20% [948700/4942000] [192.0/1000.0] [batch_t 0.323 (0.334)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 08:46:59,666 - Train: 19.20% [948800/4942000] [192.0/1000.0] [batch_t 0.319 (0.327)] [data_t 0.002] [optim_t 0.317] [lr 0.005000] 2024-04-08 08:47:20,605 - ==> Total time: 5 days, 14:49:59 Eta: 23 days, 15:25:24 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 08:47:34,600 - Train: 19.20% [948900/4942000] [192.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 08:48:07,431 - Train: 19.20% [949000/4942000] [192.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:48:40,279 - Train: 19.20% [949100/4942000] [192.0/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 08:49:14,102 - Train: 19.21% [949200/4942000] [192.1/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:49:46,920 - Train: 19.21% [949300/4942000] [192.1/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 08:50:20,784 - Train: 19.21% [949400/4942000] [192.1/1000.0] [batch_t 0.324 (0.339)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 08:50:53,666 - Train: 19.21% [949500/4942000] [192.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 08:51:27,070 - Train: 19.21% [949600/4942000] [192.1/1000.0] [batch_t 0.326 (0.334)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 08:51:59,796 - Train: 19.22% [949700/4942000] [192.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 08:52:33,360 - Train: 19.22% [949800/4942000] [192.2/1000.0] [batch_t 0.324 (0.336)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 08:53:06,132 - Train: 19.22% [949900/4942000] [192.2/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 08:53:38,932 - Train: 19.22% [950000/4942000] [192.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 08:54:12,699 - Train: 19.23% [950100/4942000] [192.3/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:54:45,494 - Train: 19.23% [950200/4942000] [192.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:55:18,245 - Train: 19.23% [950300/4942000] [192.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:55:51,006 - Train: 19.23% [950400/4942000] [192.3/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 08:56:24,728 - Train: 19.23% [950500/4942000] [192.3/1000.0] [batch_t 0.326 (0.337)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 08:56:57,472 - Train: 19.24% [950600/4942000] [192.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:57:30,889 - Train: 19.24% [950700/4942000] [192.4/1000.0] [batch_t 0.325 (0.334)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 08:58:03,744 - Train: 19.24% [950800/4942000] [192.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 08:58:36,486 - Train: 19.24% [950900/4942000] [192.4/1000.0] [batch_t 0.319 (0.327)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-08 08:59:10,377 - Train: 19.24% [951000/4942000] [192.4/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 08:59:43,096 - Train: 19.25% [951100/4942000] [192.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 09:00:16,106 - Train: 19.25% [951200/4942000] [192.5/1000.0] [batch_t 0.338 (0.330)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-08 09:00:48,941 - Train: 19.25% [951300/4942000] [192.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 09:01:21,653 - Train: 19.25% [951400/4942000] [192.5/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 09:01:54,465 - Train: 19.25% [951500/4942000] [192.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 09:02:27,956 - Train: 19.26% [951600/4942000] [192.6/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 09:03:00,727 - Train: 19.26% [951700/4942000] [192.6/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 09:03:33,610 - Train: 19.26% [951800/4942000] [192.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 09:04:07,897 - Train: 19.26% [951900/4942000] [192.6/1000.0] [batch_t 0.328 (0.343)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 09:04:40,648 - Train: 19.26% [952000/4942000] [192.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 09:05:13,510 - Train: 19.27% [952100/4942000] [192.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:05:46,296 - Train: 19.27% [952200/4942000] [192.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 09:06:20,920 - Train: 19.27% [952300/4942000] [192.7/1000.0] [batch_t 0.333 (0.346)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 09:06:53,780 - Train: 19.27% [952400/4942000] [192.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:07:26,499 - Train: 19.27% [952500/4942000] [192.7/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 09:07:59,366 - Train: 19.28% [952600/4942000] [192.8/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 09:08:33,399 - Train: 19.28% [952700/4942000] [192.8/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 09:09:06,187 - Train: 19.28% [952800/4942000] [192.8/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 09:09:38,942 - Train: 19.28% [952900/4942000] [192.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 09:10:12,900 - Train: 19.28% [953000/4942000] [192.8/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:10:45,667 - Train: 19.29% [953100/4942000] [192.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:11:18,419 - Train: 19.29% [953200/4942000] [192.9/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 09:11:51,148 - Train: 19.29% [953300/4942000] [192.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 09:12:23,934 - Train: 19.29% [953400/4942000] [192.9/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 09:12:56,730 - Train: 19.29% [953500/4942000] [192.9/1000.0] [batch_t 0.336 (0.328)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-08 09:13:30,047 - Train: 19.30% [953600/4942000] [193.0/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 09:14:02,962 - Train: 19.30% [953700/4942000] [193.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 09:14:36,788 - Train: 19.30% [953800/4942000] [193.0/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:14:38,768 - ==> Total time: 5 days, 15:17:17 Eta: 23 days, 13:41:14 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 09:15:11,870 - Train: 19.30% [953900/4942000] [193.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 09:15:44,614 - Train: 19.30% [954000/4942000] [193.0/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 09:16:17,354 - Train: 19.31% [954100/4942000] [193.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 09:16:50,154 - Train: 19.31% [954200/4942000] [193.1/1000.0] [batch_t 0.320 (0.328)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-08 09:17:22,863 - Train: 19.31% [954300/4942000] [193.1/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 09:17:55,658 - Train: 19.31% [954400/4942000] [193.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 09:18:28,429 - Train: 19.31% [954500/4942000] [193.1/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 09:19:01,197 - Train: 19.32% [954600/4942000] [193.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 09:19:34,004 - Train: 19.32% [954700/4942000] [193.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 09:20:06,763 - Train: 19.32% [954800/4942000] [193.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 09:20:39,474 - Train: 19.32% [954900/4942000] [193.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 09:21:12,399 - Train: 19.32% [955000/4942000] [193.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 09:21:45,114 - Train: 19.33% [955100/4942000] [193.3/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 09:22:17,872 - Train: 19.33% [955200/4942000] [193.3/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:22:50,611 - Train: 19.33% [955300/4942000] [193.3/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 09:23:23,363 - Train: 19.33% [955400/4942000] [193.3/1000.0] [batch_t 0.325 (0.327)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-08 09:23:56,086 - Train: 19.33% [955500/4942000] [193.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:24:28,935 - Train: 19.34% [955600/4942000] [193.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 09:25:01,696 - Train: 19.34% [955700/4942000] [193.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 09:25:34,501 - Train: 19.34% [955800/4942000] [193.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 09:26:07,282 - Train: 19.34% [955900/4942000] [193.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 09:26:40,019 - Train: 19.34% [956000/4942000] [193.4/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 09:27:12,841 - Train: 19.35% [956100/4942000] [193.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:27:45,536 - Train: 19.35% [956200/4942000] [193.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:28:18,260 - Train: 19.35% [956300/4942000] [193.5/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 09:28:51,067 - Train: 19.35% [956400/4942000] [193.5/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 09:29:23,844 - Train: 19.35% [956500/4942000] [193.5/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 09:29:56,594 - Train: 19.36% [956600/4942000] [193.6/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 09:30:29,530 - Train: 19.36% [956700/4942000] [193.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:31:02,334 - Train: 19.36% [956800/4942000] [193.6/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 09:31:35,114 - Train: 19.36% [956900/4942000] [193.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:32:08,385 - Train: 19.36% [957000/4942000] [193.6/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 09:32:41,135 - Train: 19.37% [957100/4942000] [193.7/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 09:33:14,974 - Train: 19.37% [957200/4942000] [193.7/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 09:33:47,671 - Train: 19.37% [957300/4942000] [193.7/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 09:34:21,837 - Train: 19.37% [957400/4942000] [193.7/1000.0] [batch_t 0.325 (0.342)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 09:34:54,534 - Train: 19.37% [957500/4942000] [193.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 09:35:33,193 - Train: 19.38% [957600/4942000] [193.8/1000.0] [batch_t 0.329 (0.386)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:36:08,203 - Train: 19.38% [957700/4942000] [193.8/1000.0] [batch_t 0.322 (0.350)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 09:36:41,074 - Train: 19.38% [957800/4942000] [193.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 09:37:14,733 - Train: 19.38% [957900/4942000] [193.8/1000.0] [batch_t 0.326 (0.336)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 09:37:47,514 - Train: 19.38% [958000/4942000] [193.8/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 09:38:20,747 - Train: 19.39% [958100/4942000] [193.9/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 09:38:53,463 - Train: 19.39% [958200/4942000] [193.9/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 09:39:26,253 - Train: 19.39% [958300/4942000] [193.9/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 09:39:59,068 - Train: 19.39% [958400/4942000] [193.9/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 09:40:31,795 - Train: 19.39% [958500/4942000] [193.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 09:41:04,596 - Train: 19.40% [958600/4942000] [194.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 09:41:38,616 - Train: 19.40% [958700/4942000] [194.0/1000.0] [batch_t 0.326 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 09:41:54,326 - ==> Total time: 5 days, 15:44:33 Eta: 23 days, 11:57:42 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 09:42:14,141 - Train: 19.40% [958800/4942000] [194.0/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 09:42:50,569 - Train: 19.40% [958900/4942000] [194.0/1000.0] [batch_t 0.332 (0.364)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 09:43:23,445 - Train: 19.41% [959000/4942000] [194.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 09:43:56,288 - Train: 19.41% [959100/4942000] [194.1/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 09:44:29,272 - Train: 19.41% [959200/4942000] [194.1/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 09:45:02,116 - Train: 19.41% [959300/4942000] [194.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 09:45:39,950 - Train: 19.41% [959400/4942000] [194.1/1000.0] [batch_t 0.327 (0.378)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 09:46:13,626 - Train: 19.42% [959500/4942000] [194.2/1000.0] [batch_t 0.322 (0.337)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 09:46:46,478 - Train: 19.42% [959600/4942000] [194.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 09:47:20,332 - Train: 19.42% [959700/4942000] [194.2/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 09:47:53,122 - Train: 19.42% [959800/4942000] [194.2/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 09:48:25,922 - Train: 19.42% [959900/4942000] [194.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:48:58,712 - Train: 19.43% [960000/4942000] [194.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 09:49:31,431 - Train: 19.43% [960100/4942000] [194.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 09:50:04,203 - Train: 19.43% [960200/4942000] [194.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 09:50:38,233 - Train: 19.43% [960300/4942000] [194.3/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:51:12,168 - Train: 19.43% [960400/4942000] [194.3/1000.0] [batch_t 0.325 (0.339)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 09:51:44,906 - Train: 19.44% [960500/4942000] [194.4/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 09:52:18,349 - Train: 19.44% [960600/4942000] [194.4/1000.0] [batch_t 0.322 (0.334)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 09:52:51,088 - Train: 19.44% [960700/4942000] [194.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:53:24,446 - Train: 19.44% [960800/4942000] [194.4/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:53:57,175 - Train: 19.44% [960900/4942000] [194.4/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 09:54:30,992 - Train: 19.45% [961000/4942000] [194.5/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 09:55:08,068 - Train: 19.45% [961100/4942000] [194.5/1000.0] [batch_t 0.331 (0.371)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 09:55:40,781 - Train: 19.45% [961200/4942000] [194.5/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 09:56:14,090 - Train: 19.45% [961300/4942000] [194.5/1000.0] [batch_t 0.326 (0.333)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 09:56:47,591 - Train: 19.45% [961400/4942000] [194.5/1000.0] [batch_t 0.325 (0.335)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 09:57:20,388 - Train: 19.46% [961500/4942000] [194.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 09:57:54,568 - Train: 19.46% [961600/4942000] [194.6/1000.0] [batch_t 0.327 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 09:58:28,160 - Train: 19.46% [961700/4942000] [194.6/1000.0] [batch_t 0.324 (0.336)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 09:59:00,930 - Train: 19.46% [961800/4942000] [194.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:59:34,905 - Train: 19.46% [961900/4942000] [194.6/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 10:00:07,690 - Train: 19.47% [962000/4942000] [194.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 10:00:40,609 - Train: 19.47% [962100/4942000] [194.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 10:01:13,420 - Train: 19.47% [962200/4942000] [194.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:01:46,174 - Train: 19.47% [962300/4942000] [194.7/1000.0] [batch_t 0.324 (0.327)] [data_t 0.003] [optim_t 0.321] [lr 0.005000] 2024-04-08 10:02:20,229 - Train: 19.47% [962400/4942000] [194.7/1000.0] [batch_t 0.326 (0.340)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 10:02:53,082 - Train: 19.48% [962500/4942000] [194.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 10:03:26,616 - Train: 19.48% [962600/4942000] [194.8/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 10:03:59,445 - Train: 19.48% [962700/4942000] [194.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 10:04:32,674 - Train: 19.48% [962800/4942000] [194.8/1000.0] [batch_t 0.324 (0.332)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 10:05:05,630 - Train: 19.48% [962900/4942000] [194.8/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 10:05:38,702 - Train: 19.49% [963000/4942000] [194.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 10:06:11,613 - Train: 19.49% [963100/4942000] [194.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 10:06:44,384 - Train: 19.49% [963200/4942000] [194.9/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 10:07:19,127 - Train: 19.49% [963300/4942000] [194.9/1000.0] [batch_t 0.327 (0.347)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:07:51,921 - Train: 19.49% [963400/4942000] [194.9/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 10:08:24,864 - Train: 19.50% [963500/4942000] [195.0/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 10:08:57,535 - Train: 19.50% [963600/4942000] [195.0/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:09:27,035 - ==> Total time: 5 days, 16:12:06 Eta: 23 days, 10:16:07 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 10:09:32,495 - Train: 19.50% [963700/4942000] [195.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:10:06,445 - Train: 19.50% [963800/4942000] [195.0/1000.0] [batch_t 0.324 (0.339)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 10:10:39,316 - Train: 19.50% [963900/4942000] [195.0/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 10:11:13,628 - Train: 19.51% [964000/4942000] [195.1/1000.0] [batch_t 0.327 (0.343)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:11:46,702 - Train: 19.51% [964100/4942000] [195.1/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 10:12:19,869 - Train: 19.51% [964200/4942000] [195.1/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 10:12:52,796 - Train: 19.51% [964300/4942000] [195.1/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 10:13:27,456 - Train: 19.51% [964400/4942000] [195.1/1000.0] [batch_t 0.329 (0.347)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 10:14:00,436 - Train: 19.52% [964500/4942000] [195.2/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 10:14:41,461 - Train: 19.52% [964600/4942000] [195.2/1000.0] [batch_t 0.328 (0.410)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 10:15:16,261 - Train: 19.52% [964700/4942000] [195.2/1000.0] [batch_t 0.327 (0.348)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:15:49,191 - Train: 19.52% [964800/4942000] [195.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 10:16:23,882 - Train: 19.52% [964900/4942000] [195.2/1000.0] [batch_t 0.329 (0.347)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 10:16:56,774 - Train: 19.53% [965000/4942000] [195.3/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 10:17:31,179 - Train: 19.53% [965100/4942000] [195.3/1000.0] [batch_t 0.326 (0.344)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 10:18:03,946 - Train: 19.53% [965200/4942000] [195.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 10:18:37,946 - Train: 19.53% [965300/4942000] [195.3/1000.0] [batch_t 0.332 (0.340)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 10:19:11,853 - Train: 19.53% [965400/4942000] [195.3/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 10:19:44,630 - Train: 19.54% [965500/4942000] [195.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:20:20,308 - Train: 19.54% [965600/4942000] [195.4/1000.0] [batch_t 0.325 (0.357)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 10:20:53,038 - Train: 19.54% [965700/4942000] [195.4/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 10:21:27,499 - Train: 19.54% [965800/4942000] [195.4/1000.0] [batch_t 0.324 (0.345)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 10:22:00,211 - Train: 19.54% [965900/4942000] [195.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 10:22:34,253 - Train: 19.55% [966000/4942000] [195.5/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:23:08,995 - Train: 19.55% [966100/4942000] [195.5/1000.0] [batch_t 0.318 (0.347)] [data_t 0.002] [optim_t 0.316] [lr 0.005000] 2024-04-08 10:23:41,721 - Train: 19.55% [966200/4942000] [195.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:24:16,202 - Train: 19.55% [966300/4942000] [195.5/1000.0] [batch_t 0.327 (0.345)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:24:48,875 - Train: 19.55% [966400/4942000] [195.5/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 10:25:22,520 - Train: 19.56% [966500/4942000] [195.6/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:25:55,264 - Train: 19.56% [966600/4942000] [195.6/1000.0] [batch_t 0.321 (0.327)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-08 10:26:29,749 - Train: 19.56% [966700/4942000] [195.6/1000.0] [batch_t 0.324 (0.345)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 10:27:02,455 - Train: 19.56% [966800/4942000] [195.6/1000.0] [batch_t 0.336 (0.327)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-08 10:27:37,167 - Train: 19.56% [966900/4942000] [195.6/1000.0] [batch_t 0.327 (0.347)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:28:12,308 - Train: 19.57% [967000/4942000] [195.7/1000.0] [batch_t 0.327 (0.351)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:28:45,187 - Train: 19.57% [967100/4942000] [195.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 10:29:19,040 - Train: 19.57% [967200/4942000] [195.7/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:29:52,006 - Train: 19.57% [967300/4942000] [195.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 10:30:28,356 - Train: 19.58% [967400/4942000] [195.8/1000.0] [batch_t 0.329 (0.363)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 10:31:01,204 - Train: 19.58% [967500/4942000] [195.8/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 10:31:37,307 - Train: 19.58% [967600/4942000] [195.8/1000.0] [batch_t 0.327 (0.361)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:32:12,454 - Train: 19.58% [967700/4942000] [195.8/1000.0] [batch_t 0.326 (0.351)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 10:32:51,970 - Train: 19.58% [967800/4942000] [195.8/1000.0] [batch_t 0.913 (0.395)] [data_t 0.592] [optim_t 0.322] [lr 0.005000] 2024-04-08 10:33:32,840 - Train: 19.59% [967900/4942000] [195.9/1000.0] [batch_t 0.323 (0.409)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 10:34:09,085 - Train: 19.59% [968000/4942000] [195.9/1000.0] [batch_t 0.329 (0.362)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 10:34:41,867 - Train: 19.59% [968100/4942000] [195.9/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 10:35:19,994 - Train: 19.59% [968200/4942000] [195.9/1000.0] [batch_t 0.326 (0.381)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:35:52,815 - Train: 19.59% [968300/4942000] [195.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:36:28,802 - Train: 19.60% [968400/4942000] [196.0/1000.0] [batch_t 0.322 (0.360)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 10:37:02,264 - Train: 19.60% [968500/4942000] [196.0/1000.0] [batch_t 0.326 (0.335)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 10:37:38,595 - Train: 19.60% [968600/4942000] [196.0/1000.0] [batch_t 0.322 (0.363)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 10:37:49,070 - ==> Total time: 5 days, 16:40:28 Eta: 23 days, 8:38:39 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 10:38:16,542 - Train: 19.60% [968700/4942000] [196.0/1000.0] [batch_t 0.324 (0.367)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 10:38:49,358 - Train: 19.60% [968800/4942000] [196.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 10:39:24,907 - Train: 19.61% [968900/4942000] [196.1/1000.0] [batch_t 0.328 (0.355)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 10:39:57,812 - Train: 19.61% [969000/4942000] [196.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 10:40:31,586 - Train: 19.61% [969100/4942000] [196.1/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 10:41:04,526 - Train: 19.61% [969200/4942000] [196.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 10:41:37,440 - Train: 19.61% [969300/4942000] [196.1/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 10:42:11,745 - Train: 19.62% [969400/4942000] [196.2/1000.0] [batch_t 0.325 (0.343)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 10:42:44,591 - Train: 19.62% [969500/4942000] [196.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 10:43:19,954 - Train: 19.62% [969600/4942000] [196.2/1000.0] [batch_t 0.322 (0.354)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 10:43:52,737 - Train: 19.62% [969700/4942000] [196.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:44:26,864 - Train: 19.62% [969800/4942000] [196.2/1000.0] [batch_t 0.326 (0.341)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 10:44:59,631 - Train: 19.63% [969900/4942000] [196.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:45:34,908 - Train: 19.63% [970000/4942000] [196.3/1000.0] [batch_t 0.328 (0.353)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 10:46:09,263 - Train: 19.63% [970100/4942000] [196.3/1000.0] [batch_t 0.326 (0.343)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 10:46:42,127 - Train: 19.63% [970200/4942000] [196.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 10:47:16,188 - Train: 19.63% [970300/4942000] [196.3/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 10:47:49,155 - Train: 19.64% [970400/4942000] [196.4/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 10:48:23,856 - Train: 19.64% [970500/4942000] [196.4/1000.0] [batch_t 0.329 (0.347)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 10:48:56,917 - Train: 19.64% [970600/4942000] [196.4/1000.0] [batch_t 0.324 (0.331)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 10:49:34,254 - Train: 19.64% [970700/4942000] [196.4/1000.0] [batch_t 0.325 (0.373)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 10:50:10,894 - Train: 19.64% [970800/4942000] [196.4/1000.0] [batch_t 0.319 (0.366)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-08 10:50:43,636 - Train: 19.65% [970900/4942000] [196.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 10:51:18,267 - Train: 19.65% [971000/4942000] [196.5/1000.0] [batch_t 0.326 (0.346)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 10:51:50,979 - Train: 19.65% [971100/4942000] [196.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 10:52:26,062 - Train: 19.65% [971200/4942000] [196.5/1000.0] [batch_t 0.322 (0.351)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 10:52:58,839 - Train: 19.65% [971300/4942000] [196.5/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 10:53:32,996 - Train: 19.66% [971400/4942000] [196.6/1000.0] [batch_t 0.326 (0.341)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 10:54:07,113 - Train: 19.66% [971500/4942000] [196.6/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 10:54:39,868 - Train: 19.66% [971600/4942000] [196.6/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 10:55:15,016 - Train: 19.66% [971700/4942000] [196.6/1000.0] [batch_t 0.327 (0.351)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:55:47,724 - Train: 19.66% [971800/4942000] [196.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 10:56:21,880 - Train: 19.67% [971900/4942000] [196.7/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:56:54,622 - Train: 19.67% [972000/4942000] [196.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 10:57:28,427 - Train: 19.67% [972100/4942000] [196.7/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:58:01,155 - Train: 19.67% [972200/4942000] [196.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:58:35,619 - Train: 19.67% [972300/4942000] [196.7/1000.0] [batch_t 0.333 (0.345)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 10:59:10,833 - Train: 19.68% [972400/4942000] [196.8/1000.0] [batch_t 0.330 (0.352)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 10:59:43,599 - Train: 19.68% [972500/4942000] [196.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 11:00:19,193 - Train: 19.68% [972600/4942000] [196.8/1000.0] [batch_t 0.323 (0.356)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 11:00:51,987 - Train: 19.68% [972700/4942000] [196.8/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 11:01:26,461 - Train: 19.68% [972800/4942000] [196.8/1000.0] [batch_t 0.324 (0.345)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 11:01:59,199 - Train: 19.69% [972900/4942000] [196.9/1000.0] [batch_t 0.337 (0.327)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-08 11:02:35,771 - Train: 19.69% [973000/4942000] [196.9/1000.0] [batch_t 0.329 (0.366)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 11:03:10,161 - Train: 19.69% [973100/4942000] [196.9/1000.0] [batch_t 0.328 (0.344)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 11:03:42,926 - Train: 19.69% [973200/4942000] [196.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 11:04:17,609 - Train: 19.69% [973300/4942000] [196.9/1000.0] [batch_t 0.329 (0.347)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 11:04:59,623 - Train: 19.70% [973400/4942000] [197.0/1000.0] [batch_t 0.327 (0.420)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 11:05:39,024 - Train: 19.70% [973500/4942000] [197.0/1000.0] [batch_t 0.327 (0.394)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 11:06:03,253 - ==> Total time: 5 days, 17:08:42 Eta: 23 days, 7:01:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 11:06:15,745 - Train: 19.70% [973600/4942000] [197.0/1000.0] [batch_t 0.323 (0.335)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 11:06:48,596 - Train: 19.70% [973700/4942000] [197.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 11:07:24,647 - Train: 19.70% [973800/4942000] [197.0/1000.0] [batch_t 0.324 (0.360)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 11:07:57,531 - Train: 19.71% [973900/4942000] [197.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 11:08:33,149 - Train: 19.71% [974000/4942000] [197.1/1000.0] [batch_t 0.328 (0.356)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 11:09:08,772 - Train: 19.71% [974100/4942000] [197.1/1000.0] [batch_t 0.333 (0.356)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 11:09:41,552 - Train: 19.71% [974200/4942000] [197.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 11:10:15,850 - Train: 19.71% [974300/4942000] [197.1/1000.0] [batch_t 0.328 (0.343)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 11:10:48,679 - Train: 19.72% [974400/4942000] [197.2/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 11:11:23,190 - Train: 19.72% [974500/4942000] [197.2/1000.0] [batch_t 0.335 (0.345)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 11:11:55,914 - Train: 19.72% [974600/4942000] [197.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 11:12:31,529 - Train: 19.72% [974700/4942000] [197.2/1000.0] [batch_t 0.329 (0.356)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 11:13:04,243 - Train: 19.72% [974800/4942000] [197.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 11:13:39,746 - Train: 19.73% [974900/4942000] [197.3/1000.0] [batch_t 0.326 (0.355)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 11:14:14,848 - Train: 19.73% [975000/4942000] [197.3/1000.0] [batch_t 0.328 (0.351)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 11:14:47,618 - Train: 19.73% [975100/4942000] [197.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 11:15:22,844 - Train: 19.73% [975200/4942000] [197.3/1000.0] [batch_t 0.324 (0.352)] [data_t 0.004] [optim_t 0.321] [lr 0.005000] 2024-04-08 11:15:55,553 - Train: 19.73% [975300/4942000] [197.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 11:16:29,125 - Train: 19.74% [975400/4942000] [197.4/1000.0] [batch_t 0.323 (0.336)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 11:17:01,829 - Train: 19.74% [975500/4942000] [197.4/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 11:17:35,142 - Train: 19.74% [975600/4942000] [197.4/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 11:18:10,271 - Train: 19.74% [975700/4942000] [197.4/1000.0] [batch_t 0.331 (0.351)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 11:18:42,959 - Train: 19.75% [975800/4942000] [197.5/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 11:19:17,125 - Train: 19.75% [975900/4942000] [197.5/1000.0] [batch_t 0.325 (0.342)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 11:19:49,964 - Train: 19.75% [976000/4942000] [197.5/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 11:20:24,093 - Train: 19.75% [976100/4942000] [197.5/1000.0] [batch_t 0.326 (0.341)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 11:20:56,885 - Train: 19.75% [976200/4942000] [197.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 11:21:31,624 - Train: 19.76% [976300/4942000] [197.6/1000.0] [batch_t 0.324 (0.347)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 11:22:04,457 - Train: 19.76% [976400/4942000] [197.6/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 11:22:38,240 - Train: 19.76% [976500/4942000] [197.6/1000.0] [batch_t 0.322 (0.338)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 11:23:12,062 - Train: 19.76% [976600/4942000] [197.6/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 11:23:44,992 - Train: 19.76% [976700/4942000] [197.6/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 11:24:24,065 - Train: 19.77% [976800/4942000] [197.7/1000.0] [batch_t 0.334 (0.391)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 11:24:56,926 - Train: 19.77% [976900/4942000] [197.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 11:25:31,185 - Train: 19.77% [977000/4942000] [197.7/1000.0] [batch_t 0.323 (0.342)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 11:26:04,078 - Train: 19.77% [977100/4942000] [197.7/1000.0] [batch_t 0.338 (0.329)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-08 11:26:39,527 - Train: 19.77% [977200/4942000] [197.7/1000.0] [batch_t 0.329 (0.354)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 11:27:14,566 - Train: 19.78% [977300/4942000] [197.8/1000.0] [batch_t 0.327 (0.350)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 11:27:47,349 - Train: 19.78% [977400/4942000] [197.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 11:28:23,079 - Train: 19.78% [977500/4942000] [197.8/1000.0] [batch_t 0.329 (0.357)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 11:29:08,120 - Train: 19.78% [977600/4942000] [197.8/1000.0] [batch_t 0.469 (0.450)] [data_t 0.145] [optim_t 0.324] [lr 0.005000] 2024-04-08 11:29:41,888 - Train: 19.78% [977700/4942000] [197.8/1000.0] [batch_t 0.336 (0.338)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-08 11:30:15,326 - Train: 19.79% [977800/4942000] [197.9/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 11:30:48,235 - Train: 19.79% [977900/4942000] [197.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 11:31:22,663 - Train: 19.79% [978000/4942000] [197.9/1000.0] [batch_t 0.338 (0.344)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-08 11:31:58,492 - Train: 19.79% [978100/4942000] [197.9/1000.0] [batch_t 0.331 (0.358)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 11:32:33,196 - Train: 19.79% [978200/4942000] [197.9/1000.0] [batch_t 0.333 (0.347)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 11:33:07,173 - Train: 19.80% [978300/4942000] [198.0/1000.0] [batch_t 0.325 (0.340)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 11:33:40,015 - Train: 19.80% [978400/4942000] [198.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 11:34:14,545 - Train: 19.80% [978500/4942000] [198.0/1000.0] [batch_t 0.333 (0.345)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 11:34:19,820 - ==> Total time: 5 days, 17:36:59 Eta: 23 days, 5:24:57 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 11:34:49,750 - Train: 19.80% [978600/4942000] [198.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 11:35:23,802 - Train: 19.80% [978700/4942000] [198.0/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 11:35:56,670 - Train: 19.81% [978800/4942000] [198.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 11:36:31,210 - Train: 19.81% [978900/4942000] [198.1/1000.0] [batch_t 0.330 (0.345)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 11:37:03,942 - Train: 19.81% [979000/4942000] [198.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 11:37:38,412 - Train: 19.81% [979100/4942000] [198.1/1000.0] [batch_t 0.325 (0.345)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 11:38:13,418 - Train: 19.81% [979200/4942000] [198.1/1000.0] [batch_t 0.327 (0.350)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 11:38:46,144 - Train: 19.82% [979300/4942000] [198.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 11:39:19,519 - Train: 19.82% [979400/4942000] [198.2/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 11:39:52,240 - Train: 19.82% [979500/4942000] [198.2/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 11:40:26,513 - Train: 19.82% [979600/4942000] [198.2/1000.0] [batch_t 0.325 (0.343)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 11:40:59,316 - Train: 19.82% [979700/4942000] [198.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 11:41:33,099 - Train: 19.83% [979800/4942000] [198.3/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 11:42:07,724 - Train: 19.83% [979900/4942000] [198.3/1000.0] [batch_t 0.335 (0.346)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 11:42:40,522 - Train: 19.83% [980000/4942000] [198.3/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 11:43:15,302 - Train: 19.83% [980100/4942000] [198.3/1000.0] [batch_t 0.324 (0.348)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 11:43:48,086 - Train: 19.83% [980200/4942000] [198.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 11:44:23,047 - Train: 19.84% [980300/4942000] [198.4/1000.0] [batch_t 0.330 (0.350)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 11:44:55,886 - Train: 19.84% [980400/4942000] [198.4/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 11:45:30,965 - Train: 19.84% [980500/4942000] [198.4/1000.0] [batch_t 0.327 (0.351)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 11:46:03,827 - Train: 19.84% [980600/4942000] [198.4/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 11:46:38,170 - Train: 19.84% [980700/4942000] [198.4/1000.0] [batch_t 0.325 (0.343)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 11:47:12,287 - Train: 19.85% [980800/4942000] [198.5/1000.0] [batch_t 0.321 (0.341)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-08 11:47:45,159 - Train: 19.85% [980900/4942000] [198.5/1000.0] [batch_t 0.337 (0.329)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-08 11:48:19,952 - Train: 19.85% [981000/4942000] [198.5/1000.0] [batch_t 0.323 (0.348)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 11:48:52,801 - Train: 19.85% [981100/4942000] [198.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 11:49:27,355 - Train: 19.85% [981200/4942000] [198.5/1000.0] [batch_t 0.331 (0.345)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 11:50:00,288 - Train: 19.86% [981300/4942000] [198.6/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 11:50:34,256 - Train: 19.86% [981400/4942000] [198.6/1000.0] [batch_t 0.326 (0.340)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 11:51:07,279 - Train: 19.86% [981500/4942000] [198.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 11:51:40,110 - Train: 19.86% [981600/4942000] [198.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 11:52:14,192 - Train: 19.86% [981700/4942000] [198.6/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 11:52:46,939 - Train: 19.87% [981800/4942000] [198.7/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 11:53:22,230 - Train: 19.87% [981900/4942000] [198.7/1000.0] [batch_t 0.326 (0.353)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 11:53:55,073 - Train: 19.87% [982000/4942000] [198.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 11:54:28,032 - Train: 19.87% [982100/4942000] [198.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 11:55:00,816 - Train: 19.87% [982200/4942000] [198.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 11:55:35,090 - Train: 19.88% [982300/4942000] [198.8/1000.0] [batch_t 0.329 (0.343)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 11:56:08,973 - Train: 19.88% [982400/4942000] [198.8/1000.0] [batch_t 0.323 (0.339)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 11:56:41,732 - Train: 19.88% [982500/4942000] [198.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 11:57:15,725 - Train: 19.88% [982600/4942000] [198.8/1000.0] [batch_t 0.331 (0.340)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 11:57:48,513 - Train: 19.88% [982700/4942000] [198.8/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 11:58:21,365 - Train: 19.89% [982800/4942000] [198.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 11:58:54,815 - Train: 19.89% [982900/4942000] [198.9/1000.0] [batch_t 0.333 (0.334)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 11:59:30,186 - Train: 19.89% [983000/4942000] [198.9/1000.0] [batch_t 0.330 (0.354)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 12:00:08,697 - Train: 19.89% [983100/4942000] [198.9/1000.0] [batch_t 4.371 (0.385)] [data_t 4.047] [optim_t 0.324] [lr 0.005000] 2024-04-08 12:00:43,328 - Train: 19.89% [983200/4942000] [198.9/1000.0] [batch_t 0.323 (0.346)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 12:01:17,172 - Train: 19.90% [983300/4942000] [199.0/1000.0] [batch_t 0.324 (0.338)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 12:01:50,028 - Train: 19.90% [983400/4942000] [199.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 12:02:10,134 - ==> Total time: 5 days, 18:04:49 Eta: 23 days, 3:47:26 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 12:02:26,227 - Train: 19.90% [983500/4942000] [199.0/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 12:02:58,939 - Train: 19.90% [983600/4942000] [199.0/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 12:03:35,732 - Train: 19.90% [983700/4942000] [199.0/1000.0] [batch_t 0.993 (0.368)] [data_t 0.662] [optim_t 0.331] [lr 0.005000] 2024-04-08 12:04:17,640 - Train: 19.91% [983800/4942000] [199.1/1000.0] [batch_t 0.327 (0.419)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 12:04:50,409 - Train: 19.91% [983900/4942000] [199.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 12:05:24,489 - Train: 19.91% [984000/4942000] [199.1/1000.0] [batch_t 0.323 (0.341)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 12:05:57,470 - Train: 19.91% [984100/4942000] [199.1/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 12:06:36,067 - Train: 19.92% [984200/4942000] [199.2/1000.0] [batch_t 0.919 (0.386)] [data_t 0.602] [optim_t 0.317] [lr 0.005000] 2024-04-08 12:07:37,777 - Train: 19.92% [984300/4942000] [199.2/1000.0] [batch_t 0.322 (0.617)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 12:08:42,241 - Train: 19.92% [984400/4942000] [199.2/1000.0] [batch_t 0.326 (0.645)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 12:09:28,686 - Train: 19.92% [984500/4942000] [199.2/1000.0] [batch_t 0.331 (0.464)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 12:10:02,778 - Train: 19.92% [984600/4942000] [199.2/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 12:10:42,173 - Train: 19.93% [984700/4942000] [199.3/1000.0] [batch_t 0.322 (0.394)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 12:11:15,072 - Train: 19.93% [984800/4942000] [199.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 12:11:48,036 - Train: 19.93% [984900/4942000] [199.3/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 12:12:21,748 - Train: 19.93% [985000/4942000] [199.3/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 12:12:54,644 - Train: 19.93% [985100/4942000] [199.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 12:13:29,043 - Train: 19.94% [985200/4942000] [199.4/1000.0] [batch_t 0.330 (0.344)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 12:14:08,495 - Train: 19.94% [985300/4942000] [199.4/1000.0] [batch_t 0.329 (0.394)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 12:14:41,325 - Train: 19.94% [985400/4942000] [199.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 12:15:38,708 - Train: 19.94% [985500/4942000] [199.4/1000.0] [batch_t 0.361 (0.574)] [data_t 0.037] [optim_t 0.324] [lr 0.005000] 2024-04-08 12:16:24,729 - Train: 19.94% [985600/4942000] [199.4/1000.0] [batch_t 0.328 (0.460)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 12:16:57,563 - Train: 19.95% [985700/4942000] [199.5/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 12:17:32,302 - Train: 19.95% [985800/4942000] [199.5/1000.0] [batch_t 0.324 (0.347)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 12:18:05,230 - Train: 19.95% [985900/4942000] [199.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.008] [optim_t 0.322] [lr 0.005000] 2024-04-08 12:18:38,119 - Train: 19.95% [986000/4942000] [199.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 12:19:13,130 - Train: 19.95% [986100/4942000] [199.5/1000.0] [batch_t 0.327 (0.350)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 12:19:50,280 - Train: 19.96% [986200/4942000] [199.6/1000.0] [batch_t 0.335 (0.371)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 12:20:23,826 - Train: 19.96% [986300/4942000] [199.6/1000.0] [batch_t 0.334 (0.335)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 12:20:56,708 - Train: 19.96% [986400/4942000] [199.6/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 12:21:30,958 - Train: 19.96% [986500/4942000] [199.6/1000.0] [batch_t 0.494 (0.342)] [data_t 0.166] [optim_t 0.328] [lr 0.005000] 2024-04-08 12:22:08,272 - Train: 19.96% [986600/4942000] [199.6/1000.0] [batch_t 0.327 (0.373)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 12:22:41,031 - Train: 19.97% [986700/4942000] [199.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 12:23:13,964 - Train: 19.97% [986800/4942000] [199.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 12:23:46,808 - Train: 19.97% [986900/4942000] [199.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 12:24:20,369 - Train: 19.97% [987000/4942000] [199.7/1000.0] [batch_t 0.325 (0.335)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 12:24:53,231 - Train: 19.97% [987100/4942000] [199.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 12:25:26,042 - Train: 19.98% [987200/4942000] [199.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 12:26:13,140 - Train: 19.98% [987300/4942000] [199.8/1000.0] [batch_t 0.330 (0.471)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 12:26:58,966 - Train: 19.98% [987400/4942000] [199.8/1000.0] [batch_t 0.326 (0.458)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 12:27:48,913 - Train: 19.98% [987500/4942000] [199.8/1000.0] [batch_t 0.321 (0.499)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-08 12:28:28,069 - Train: 19.98% [987600/4942000] [199.8/1000.0] [batch_t 0.329 (0.391)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 12:29:01,072 - Train: 19.99% [987700/4942000] [199.9/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 12:29:33,958 - Train: 19.99% [987800/4942000] [199.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 12:30:06,853 - Train: 19.99% [987900/4942000] [199.9/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 12:30:39,747 - Train: 19.99% [988000/4942000] [199.9/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 12:31:12,868 - Train: 19.99% [988100/4942000] [199.9/1000.0] [batch_t 0.335 (0.331)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 12:31:45,771 - Train: 20.00% [988200/4942000] [200.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 12:32:19,383 - Train: 20.00% [988300/4942000] [200.0/1000.0] [batch_t 0.332 (0.336)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 12:32:52,131 - Train: 20.00% [988400/4942000] [200.0/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 12:33:01,522 - Test: 16.13% [50/310] [batch_t 0.165 (0.168)] 2024-04-08 12:33:10,051 - Test: 32.26% [100/310] [batch_t 0.158 (0.169)] 2024-04-08 12:33:17,983 - Test: 48.39% [150/310] [batch_t 0.156 (0.166)] 2024-04-08 12:33:25,866 - Test: 64.52% [200/310] [batch_t 0.157 (0.164)] 2024-04-08 12:33:34,384 - Test: 80.65% [250/310] [batch_t 0.154 (0.165)] 2024-04-08 12:33:42,968 - Test: 96.77% [300/310] [batch_t 0.153 (0.166)] 2024-04-08 12:33:44,447 - Test: 100.00% [310/310] [batch_t 0.086 (0.165)] 2024-04-08 13:00:28,954 - ==> Metric Time for coco : 0.004 (mAUROC_sp_max) 0.002 (mAP_sp_max) 0.001 (mF1_max_sp_max) 348.700 (mAUROC_px) 295.453 (mAP_px) 35.126 (mF1_max_px) 849.591 (mAUPRO_px) 11.288 (mF1_px_0.2_0.8_0.1) 11.109 (mAcc_px_0.2_0.8_0.1) 11.101 (mIoU_px_0.2_0.8_0.1) 32.479 (mIoU_max_px) 2024-04-08 13:00:29,706 - | Name | mAUROC_sp_max | mAUROC_sp_max (Max) | mAP_sp_max | mAP_sp_max (Max) | mF1_max_sp_max | mF1_max_sp_max (Max) | mAUROC_px | mAUROC_px (Max) | mAP_px | mAP_px (Max) | mF1_max_px | mF1_max_px (Max) | mAUPRO_px | mAUPRO_px (Max) | mF1_px_0.2_0.8_0.1 | mF1_px_0.2_0.8_0.1 (Max) | mAcc_px_0.2_0.8_0.1 | mAcc_px_0.2_0.8_0.1 (Max) | mIoU_px_0.2_0.8_0.1 | mIoU_px_0.2_0.8_0.1 (Max) | mIoU_max_px | mIoU_max_px (Max) | |:------:|:---------------:|:---------------------:|:------------:|:------------------:|:----------------:|:----------------------:|:-----------:|:------------------:|:--------:|:------------------:|:------------:|:------------------:|:-----------:|:------------------:|:--------------------:|:--------------------------:|:---------------------:|:---------------------------:|:---------------------:|:---------------------------:|:-------------:|:-------------------:| | coco | 64.780 | 66.882 (50 epoch) | 44.742 | 46.681 (100 epoch) | 53.242 | 54.576 (50 epoch) | 71.889 | 71.889 (200 epoch) | 14.810 | 14.810 (200 epoch) | 22.265 | 22.265 (200 epoch) | 42.855 | 44.441 (50 epoch) | 11.720 | 11.792 (50 epoch) | 43.761 | 44.665 (50 epoch) | 6.397 | 6.432 (100 epoch) | 12.527 | 12.527 (200 epoch) | 2024-04-08 13:00:30,348 - ==> Total time: 5 days, 19:03:09 Eta: 23 days, 4:12:38 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 13:01:50,053 - Train: 20.00% [988500/4942000] [200.0/1000.0] [batch_t 0.768 (0.758)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-08 13:03:05,560 - Train: 20.00% [988600/4942000] [200.0/1000.0] [batch_t 0.766 (0.755)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-08 13:04:21,172 - Train: 20.01% [988700/4942000] [200.1/1000.0] [batch_t 0.754 (0.756)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-08 13:05:36,765 - Train: 20.01% [988800/4942000] [200.1/1000.0] [batch_t 0.764 (0.756)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-08 13:06:52,356 - Train: 20.01% [988900/4942000] [200.1/1000.0] [batch_t 0.765 (0.756)] [data_t 0.003] [optim_t 0.762] [lr 0.005000] 2024-04-08 13:08:08,144 - Train: 20.01% [989000/4942000] [200.1/1000.0] [batch_t 0.750 (0.758)] [data_t 0.003] [optim_t 0.747] [lr 0.005000] 2024-04-08 13:09:23,684 - Train: 20.01% [989100/4942000] [200.1/1000.0] [batch_t 0.759 (0.755)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-08 13:10:39,233 - Train: 20.02% [989200/4942000] [200.2/1000.0] [batch_t 0.756 (0.755)] [data_t 0.002] [optim_t 0.754] [lr 0.005000] 2024-04-08 13:11:54,851 - Train: 20.02% [989300/4942000] [200.2/1000.0] [batch_t 0.755 (0.756)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-08 13:13:10,464 - Train: 20.02% [989400/4942000] [200.2/1000.0] [batch_t 0.755 (0.756)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-08 13:14:26,061 - Train: 20.02% [989500/4942000] [200.2/1000.0] [batch_t 0.750 (0.756)] [data_t 0.003] [optim_t 0.747] [lr 0.005000] 2024-04-08 13:15:41,761 - Train: 20.02% [989600/4942000] [200.2/1000.0] [batch_t 0.756 (0.757)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-08 13:16:57,412 - Train: 20.03% [989700/4942000] [200.3/1000.0] [batch_t 0.755 (0.756)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-08 13:18:12,865 - Train: 20.03% [989800/4942000] [200.3/1000.0] [batch_t 0.746 (0.754)] [data_t 0.002] [optim_t 0.744] [lr 0.005000] 2024-04-08 13:19:28,409 - Train: 20.03% [989900/4942000] [200.3/1000.0] [batch_t 0.763 (0.755)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-08 13:20:43,836 - Train: 20.03% [990000/4942000] [200.3/1000.0] [batch_t 0.768 (0.754)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-08 13:21:59,773 - Train: 20.03% [990100/4942000] [200.3/1000.0] [batch_t 0.759 (0.754)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-08 13:23:20,318 - Train: 20.04% [990200/4942000] [200.4/1000.0] [batch_t 0.758 (0.805)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-08 13:24:35,885 - Train: 20.04% [990300/4942000] [200.4/1000.0] [batch_t 0.751 (0.756)] [data_t 0.002] [optim_t 0.749] [lr 0.005000] 2024-04-08 13:25:51,367 - Train: 20.04% [990400/4942000] [200.4/1000.0] [batch_t 0.758 (0.755)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-08 13:27:06,972 - Train: 20.04% [990500/4942000] [200.4/1000.0] [batch_t 0.755 (0.756)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-08 13:28:22,486 - Train: 20.04% [990600/4942000] [200.4/1000.0] [batch_t 0.762 (0.755)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-08 13:29:38,045 - Train: 20.05% [990700/4942000] [200.5/1000.0] [batch_t 0.750 (0.755)] [data_t 0.002] [optim_t 0.748] [lr 0.005000] 2024-04-08 13:30:53,505 - Train: 20.05% [990800/4942000] [200.5/1000.0] [batch_t 0.759 (0.754)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-08 13:32:08,998 - Train: 20.05% [990900/4942000] [200.5/1000.0] [batch_t 0.754 (0.755)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-08 13:33:24,520 - Train: 20.05% [991000/4942000] [200.5/1000.0] [batch_t 0.755 (0.755)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-08 13:34:39,967 - Train: 20.05% [991100/4942000] [200.5/1000.0] [batch_t 0.754 (0.754)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-08 13:35:55,474 - Train: 20.06% [991200/4942000] [200.6/1000.0] [batch_t 0.753 (0.755)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-08 13:37:11,008 - Train: 20.06% [991300/4942000] [200.6/1000.0] [batch_t 0.754 (0.755)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-08 13:38:26,458 - Train: 20.06% [991400/4942000] [200.6/1000.0] [batch_t 0.746 (0.754)] [data_t 0.002] [optim_t 0.743] [lr 0.005000] 2024-04-08 13:39:41,974 - Train: 20.06% [991500/4942000] [200.6/1000.0] [batch_t 0.765 (0.755)] [data_t 0.002] [optim_t 0.763] [lr 0.005000] 2024-04-08 13:40:57,501 - Train: 20.06% [991600/4942000] [200.6/1000.0] [batch_t 0.758 (0.755)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-08 13:42:12,981 - Train: 20.07% [991700/4942000] [200.7/1000.0] [batch_t 0.762 (0.755)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-08 13:43:28,626 - Train: 20.07% [991800/4942000] [200.7/1000.0] [batch_t 0.757 (0.756)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-08 13:44:44,090 - Train: 20.07% [991900/4942000] [200.7/1000.0] [batch_t 0.754 (0.755)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-08 13:45:59,603 - Train: 20.07% [992000/4942000] [200.7/1000.0] [batch_t 0.764 (0.755)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-08 13:47:15,109 - Train: 20.07% [992100/4942000] [200.7/1000.0] [batch_t 0.763 (0.755)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-08 13:48:30,662 - Train: 20.08% [992200/4942000] [200.8/1000.0] [batch_t 0.742 (0.755)] [data_t 0.002] [optim_t 0.740] [lr 0.005000] 2024-04-08 13:49:46,175 - Train: 20.08% [992300/4942000] [200.8/1000.0] [batch_t 0.753 (0.755)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-08 13:51:01,735 - Train: 20.08% [992400/4942000] [200.8/1000.0] [batch_t 0.754 (0.755)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-08 13:52:17,289 - Train: 20.08% [992500/4942000] [200.8/1000.0] [batch_t 0.762 (0.755)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-08 13:53:32,732 - Train: 20.08% [992600/4942000] [200.8/1000.0] [batch_t 0.750 (0.754)] [data_t 0.003] [optim_t 0.747] [lr 0.005000] 2024-04-08 13:54:48,220 - Train: 20.09% [992700/4942000] [200.9/1000.0] [batch_t 0.759 (0.755)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-08 13:56:03,635 - Train: 20.09% [992800/4942000] [200.9/1000.0] [batch_t 0.754 (0.754)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-08 13:57:18,970 - Train: 20.09% [992900/4942000] [200.9/1000.0] [batch_t 0.760 (0.753)] [data_t 0.002] [optim_t 0.758] [lr 0.005000] 2024-04-08 13:58:34,484 - Train: 20.09% [993000/4942000] [200.9/1000.0] [batch_t 0.768 (0.755)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-08 13:59:50,089 - Train: 20.10% [993100/4942000] [201.0/1000.0] [batch_t 0.755 (0.756)] [data_t 0.003] [optim_t 0.753] [lr 0.005000] 2024-04-08 14:01:05,668 - Train: 20.10% [993200/4942000] [201.0/1000.0] [batch_t 0.754 (0.756)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-08 14:02:21,186 - Train: 20.10% [993300/4942000] [201.0/1000.0] [batch_t 0.760 (0.755)] [data_t 0.002] [optim_t 0.758] [lr 0.005000] 2024-04-08 14:02:52,925 - ==> Total time: 5 days, 20:05:32 Eta: 23 days, 4:53:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 14:03:39,427 - Train: 20.10% [993400/4942000] [201.0/1000.0] [batch_t 0.765 (0.763)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-08 14:04:54,865 - Train: 20.10% [993500/4942000] [201.0/1000.0] [batch_t 0.746 (0.754)] [data_t 0.003] [optim_t 0.743] [lr 0.005000] 2024-04-08 14:06:10,337 - Train: 20.11% [993600/4942000] [201.1/1000.0] [batch_t 0.759 (0.754)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-08 14:07:25,848 - Train: 20.11% [993700/4942000] [201.1/1000.0] [batch_t 0.768 (0.755)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-08 14:08:41,383 - Train: 20.11% [993800/4942000] [201.1/1000.0] [batch_t 0.759 (0.755)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-08 14:09:56,816 - Train: 20.11% [993900/4942000] [201.1/1000.0] [batch_t 0.762 (0.754)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-08 14:11:12,289 - Train: 20.11% [994000/4942000] [201.1/1000.0] [batch_t 0.759 (0.755)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-08 14:12:27,727 - Train: 20.12% [994100/4942000] [201.2/1000.0] [batch_t 0.750 (0.754)] [data_t 0.002] [optim_t 0.748] [lr 0.005000] 2024-04-08 14:13:43,345 - Train: 20.12% [994200/4942000] [201.2/1000.0] [batch_t 0.761 (0.756)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-08 14:14:58,864 - Train: 20.12% [994300/4942000] [201.2/1000.0] [batch_t 0.754 (0.755)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-08 14:16:14,391 - Train: 20.12% [994400/4942000] [201.2/1000.0] [batch_t 0.739 (0.755)] [data_t 0.002] [optim_t 0.737] [lr 0.005000] 2024-04-08 14:17:30,425 - Train: 20.12% [994500/4942000] [201.2/1000.0] [batch_t 0.755 (0.760)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-08 14:18:46,010 - Train: 20.13% [994600/4942000] [201.3/1000.0] [batch_t 0.749 (0.755)] [data_t 0.002] [optim_t 0.747] [lr 0.005000] 2024-04-08 14:20:01,581 - Train: 20.13% [994700/4942000] [201.3/1000.0] [batch_t 0.761 (0.756)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-08 14:21:17,069 - Train: 20.13% [994800/4942000] [201.3/1000.0] [batch_t 0.755 (0.755)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-08 14:22:32,626 - Train: 20.13% [994900/4942000] [201.3/1000.0] [batch_t 0.755 (0.755)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-08 14:23:48,919 - Train: 20.13% [995000/4942000] [201.3/1000.0] [batch_t 0.754 (0.763)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-08 14:25:17,377 - Train: 20.14% [995100/4942000] [201.4/1000.0] [batch_t 0.758 (0.884)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-08 14:26:38,065 - Train: 20.14% [995200/4942000] [201.4/1000.0] [batch_t 0.754 (0.807)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-08 14:27:53,625 - Train: 20.14% [995300/4942000] [201.4/1000.0] [batch_t 0.755 (0.756)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-08 14:29:08,953 - Train: 20.14% [995400/4942000] [201.4/1000.0] [batch_t 0.751 (0.753)] [data_t 0.002] [optim_t 0.748] [lr 0.005000] 2024-04-08 14:30:24,509 - Train: 20.14% [995500/4942000] [201.4/1000.0] [batch_t 0.760 (0.755)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-08 14:31:39,989 - Train: 20.15% [995600/4942000] [201.5/1000.0] [batch_t 0.764 (0.755)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-08 14:32:55,413 - Train: 20.15% [995700/4942000] [201.5/1000.0] [batch_t 0.750 (0.754)] [data_t 0.002] [optim_t 0.748] [lr 0.005000] 2024-04-08 14:34:10,900 - Train: 20.15% [995800/4942000] [201.5/1000.0] [batch_t 0.764 (0.755)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-08 14:35:26,410 - Train: 20.15% [995900/4942000] [201.5/1000.0] [batch_t 0.754 (0.755)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-08 14:36:41,978 - Train: 20.15% [996000/4942000] [201.5/1000.0] [batch_t 0.767 (0.756)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-08 14:37:57,435 - Train: 20.16% [996100/4942000] [201.6/1000.0] [batch_t 0.756 (0.754)] [data_t 0.002] [optim_t 0.754] [lr 0.005000] 2024-04-08 14:39:12,888 - Train: 20.16% [996200/4942000] [201.6/1000.0] [batch_t 0.765 (0.754)] [data_t 0.002] [optim_t 0.763] [lr 0.005000] 2024-04-08 14:40:28,354 - Train: 20.16% [996300/4942000] [201.6/1000.0] [batch_t 0.750 (0.755)] [data_t 0.002] [optim_t 0.748] [lr 0.005000] 2024-04-08 14:41:43,898 - Train: 20.16% [996400/4942000] [201.6/1000.0] [batch_t 0.759 (0.755)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-08 14:43:01,069 - Train: 20.16% [996500/4942000] [201.6/1000.0] [batch_t 0.759 (0.772)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-08 14:44:16,540 - Train: 20.17% [996600/4942000] [201.7/1000.0] [batch_t 0.749 (0.755)] [data_t 0.002] [optim_t 0.747] [lr 0.005000] 2024-04-08 14:45:31,987 - Train: 20.17% [996700/4942000] [201.7/1000.0] [batch_t 0.755 (0.754)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-08 14:46:47,489 - Train: 20.17% [996800/4942000] [201.7/1000.0] [batch_t 0.750 (0.755)] [data_t 0.002] [optim_t 0.748] [lr 0.005000] 2024-04-08 14:48:02,933 - Train: 20.17% [996900/4942000] [201.7/1000.0] [batch_t 0.762 (0.754)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-08 14:49:18,427 - Train: 20.17% [997000/4942000] [201.7/1000.0] [batch_t 0.758 (0.755)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-08 14:50:33,924 - Train: 20.18% [997100/4942000] [201.8/1000.0] [batch_t 0.755 (0.755)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-08 14:51:49,398 - Train: 20.18% [997200/4942000] [201.8/1000.0] [batch_t 0.757 (0.755)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-08 14:53:04,950 - Train: 20.18% [997300/4942000] [201.8/1000.0] [batch_t 0.759 (0.755)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-08 14:54:20,694 - Train: 20.18% [997400/4942000] [201.8/1000.0] [batch_t 0.760 (0.755)] [data_t 0.002] [optim_t 0.758] [lr 0.005000] 2024-04-08 14:55:36,338 - Train: 20.18% [997500/4942000] [201.8/1000.0] [batch_t 0.765 (0.756)] [data_t 0.002] [optim_t 0.763] [lr 0.005000] 2024-04-08 14:56:51,852 - Train: 20.19% [997600/4942000] [201.9/1000.0] [batch_t 0.762 (0.755)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-08 14:58:07,354 - Train: 20.19% [997700/4942000] [201.9/1000.0] [batch_t 0.750 (0.755)] [data_t 0.002] [optim_t 0.748] [lr 0.005000] 2024-04-08 14:59:22,825 - Train: 20.19% [997800/4942000] [201.9/1000.0] [batch_t 0.761 (0.755)] [data_t 0.002] [optim_t 0.758] [lr 0.005000] 2024-04-08 15:00:38,338 - Train: 20.19% [997900/4942000] [201.9/1000.0] [batch_t 0.764 (0.755)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-08 15:01:56,257 - Train: 20.19% [998000/4942000] [201.9/1000.0] [batch_t 0.763 (0.779)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-08 15:03:11,860 - Train: 20.20% [998100/4942000] [202.0/1000.0] [batch_t 0.750 (0.756)] [data_t 0.002] [optim_t 0.748] [lr 0.005000] 2024-04-08 15:04:27,467 - Train: 20.20% [998200/4942000] [202.0/1000.0] [batch_t 0.754 (0.756)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-08 15:05:30,813 - ==> Total time: 5 days, 21:08:09 Eta: 23 days, 5:33:27 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 15:05:47,613 - Train: 20.20% [998300/4942000] [202.0/1000.0] [batch_t 0.763 (0.893)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-08 15:07:03,175 - Train: 20.20% [998400/4942000] [202.0/1000.0] [batch_t 0.757 (0.756)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-08 15:08:18,784 - Train: 20.20% [998500/4942000] [202.0/1000.0] [batch_t 0.751 (0.756)] [data_t 0.002] [optim_t 0.748] [lr 0.005000] 2024-04-08 15:09:34,552 - Train: 20.21% [998600/4942000] [202.1/1000.0] [batch_t 0.748 (0.755)] [data_t 0.002] [optim_t 0.746] [lr 0.005000] 2024-04-08 15:10:49,949 - Train: 20.21% [998700/4942000] [202.1/1000.0] [batch_t 0.759 (0.754)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-08 15:12:05,373 - Train: 20.21% [998800/4942000] [202.1/1000.0] [batch_t 0.754 (0.754)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-08 15:13:20,900 - Train: 20.21% [998900/4942000] [202.1/1000.0] [batch_t 0.755 (0.755)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-08 15:14:36,342 - Train: 20.21% [999000/4942000] [202.1/1000.0] [batch_t 0.759 (0.754)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-08 15:15:51,878 - Train: 20.22% [999100/4942000] [202.2/1000.0] [batch_t 0.755 (0.755)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-08 15:17:07,331 - Train: 20.22% [999200/4942000] [202.2/1000.0] [batch_t 0.735 (0.754)] [data_t 0.002] [optim_t 0.733] [lr 0.005000] 2024-04-08 15:18:22,904 - Train: 20.22% [999300/4942000] [202.2/1000.0] [batch_t 0.759 (0.756)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-08 15:19:39,614 - Train: 20.22% [999400/4942000] [202.2/1000.0] [batch_t 0.763 (0.767)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-08 15:21:01,388 - Train: 20.22% [999500/4942000] [202.2/1000.0] [batch_t 0.753 (0.818)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-08 15:23:18,722 - Train: 20.23% [999600/4942000] [202.3/1000.0] [batch_t 0.759 (1.373)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-08 15:25:57,555 - Train: 20.23% [999700/4942000] [202.3/1000.0] [batch_t 8.614 (1.588)] [data_t 7.854] [optim_t 0.761] [lr 0.005000] 2024-04-08 15:28:29,474 - Train: 20.23% [999800/4942000] [202.3/1000.0] [batch_t 0.763 (1.519)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-08 15:29:45,618 - Train: 20.23% [999900/4942000] [202.3/1000.0] [batch_t 0.768 (0.761)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-08 15:35:43,094 - Train: 20.23% [1000000/4942000] [202.3/1000.0] [batch_t 0.327 (3.575)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 15:41:13,038 - Train: 20.24% [1000100/4942000] [202.4/1000.0] [batch_t 0.767 (3.299)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-08 15:45:09,166 - Train: 20.24% [1000200/4942000] [202.4/1000.0] [batch_t 0.767 (2.361)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-08 15:46:37,375 - Train: 20.24% [1000300/4942000] [202.4/1000.0] [batch_t 0.749 (0.882)] [data_t 0.003] [optim_t 0.747] [lr 0.005000] 2024-04-08 15:47:54,281 - Train: 20.24% [1000400/4942000] [202.4/1000.0] [batch_t 0.769 (0.769)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-08 15:49:11,725 - Train: 20.24% [1000500/4942000] [202.4/1000.0] [batch_t 0.771 (0.774)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-08 15:50:29,683 - Train: 20.25% [1000600/4942000] [202.5/1000.0] [batch_t 0.759 (0.779)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-08 15:51:46,984 - Train: 20.25% [1000700/4942000] [202.5/1000.0] [batch_t 0.752 (0.772)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-08 15:53:08,424 - Train: 20.25% [1000800/4942000] [202.5/1000.0] [batch_t 0.762 (0.814)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-08 15:54:33,730 - Train: 20.25% [1000900/4942000] [202.5/1000.0] [batch_t 0.771 (0.853)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-08 15:56:13,539 - Train: 20.25% [1001000/4942000] [202.5/1000.0] [batch_t 0.745 (0.998)] [data_t 0.003] [optim_t 0.742] [lr 0.005000] 2024-04-08 15:57:42,923 - Train: 20.26% [1001100/4942000] [202.6/1000.0] [batch_t 0.769 (0.894)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-08 15:58:59,428 - Train: 20.26% [1001200/4942000] [202.6/1000.0] [batch_t 0.768 (0.765)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-08 16:00:15,758 - Train: 20.26% [1001300/4942000] [202.6/1000.0] [batch_t 0.758 (0.763)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-08 16:01:32,027 - Train: 20.26% [1001400/4942000] [202.6/1000.0] [batch_t 0.772 (0.763)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-08 16:02:48,987 - Train: 20.27% [1001500/4942000] [202.7/1000.0] [batch_t 0.778 (0.770)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-08 16:04:05,265 - Train: 20.27% [1001600/4942000] [202.7/1000.0] [batch_t 0.755 (0.763)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-08 16:05:21,593 - Train: 20.27% [1001700/4942000] [202.7/1000.0] [batch_t 0.771 (0.763)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-08 16:06:37,838 - Train: 20.27% [1001800/4942000] [202.7/1000.0] [batch_t 0.763 (0.762)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-08 16:07:54,051 - Train: 20.27% [1001900/4942000] [202.7/1000.0] [batch_t 0.763 (0.762)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-08 16:09:10,478 - Train: 20.28% [1002000/4942000] [202.8/1000.0] [batch_t 0.772 (0.764)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-08 16:10:26,805 - Train: 20.28% [1002100/4942000] [202.8/1000.0] [batch_t 0.773 (0.763)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-08 16:11:43,140 - Train: 20.28% [1002200/4942000] [202.8/1000.0] [batch_t 0.760 (0.763)] [data_t 0.002] [optim_t 0.758] [lr 0.005000] 2024-04-08 16:12:59,508 - Train: 20.28% [1002300/4942000] [202.8/1000.0] [batch_t 0.770 (0.764)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-08 16:14:15,895 - Train: 20.28% [1002400/4942000] [202.8/1000.0] [batch_t 0.769 (0.764)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-08 16:15:32,165 - Train: 20.29% [1002500/4942000] [202.9/1000.0] [batch_t 0.756 (0.763)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-08 16:16:48,484 - Train: 20.29% [1002600/4942000] [202.9/1000.0] [batch_t 0.772 (0.763)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-08 16:18:04,677 - Train: 20.29% [1002700/4942000] [202.9/1000.0] [batch_t 0.768 (0.762)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-08 16:19:21,100 - Train: 20.29% [1002800/4942000] [202.9/1000.0] [batch_t 0.777 (0.764)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-08 16:20:37,385 - Train: 20.29% [1002900/4942000] [202.9/1000.0] [batch_t 0.758 (0.763)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-08 16:21:53,794 - Train: 20.30% [1003000/4942000] [203.0/1000.0] [batch_t 0.774 (0.764)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-08 16:23:10,055 - Train: 20.30% [1003100/4942000] [203.0/1000.0] [batch_t 0.757 (0.763)] [data_t 0.003] [optim_t 0.754] [lr 0.005000] 2024-04-08 16:24:26,438 - Train: 20.30% [1003200/4942000] [203.0/1000.0] [batch_t 0.769 (0.764)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-08 16:24:46,320 - ==> Total time: 5 days, 22:27:25 Eta: 23 days, 7:18:07 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 16:25:46,258 - Train: 20.30% [1003300/4942000] [203.0/1000.0] [batch_t 0.772 (0.778)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-08 16:27:02,694 - Train: 20.30% [1003400/4942000] [203.0/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-08 16:28:18,999 - Train: 20.31% [1003500/4942000] [203.1/1000.0] [batch_t 0.768 (0.763)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-08 16:29:35,251 - Train: 20.31% [1003600/4942000] [203.1/1000.0] [batch_t 0.768 (0.762)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-08 16:30:51,546 - Train: 20.31% [1003700/4942000] [203.1/1000.0] [batch_t 0.745 (0.763)] [data_t 0.003] [optim_t 0.742] [lr 0.005000] 2024-04-08 16:32:07,976 - Train: 20.31% [1003800/4942000] [203.1/1000.0] [batch_t 0.778 (0.764)] [data_t 0.002] [optim_t 0.776] [lr 0.005000] 2024-04-08 16:33:24,340 - Train: 20.31% [1003900/4942000] [203.1/1000.0] [batch_t 0.772 (0.764)] [data_t 0.002] [optim_t 0.769] [lr 0.005000] 2024-04-08 16:34:40,707 - Train: 20.32% [1004000/4942000] [203.2/1000.0] [batch_t 0.769 (0.764)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-08 16:35:57,066 - Train: 20.32% [1004100/4942000] [203.2/1000.0] [batch_t 0.774 (0.763)] [data_t 0.002] [optim_t 0.772] [lr 0.005000] 2024-04-08 16:37:13,344 - Train: 20.32% [1004200/4942000] [203.2/1000.0] [batch_t 0.763 (0.763)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-08 16:38:29,743 - Train: 20.32% [1004300/4942000] [203.2/1000.0] [batch_t 0.774 (0.764)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-08 16:40:02,057 - Train: 20.32% [1004400/4942000] [203.2/1000.0] [batch_t 0.760 (0.923)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-08 16:41:18,588 - Train: 20.33% [1004500/4942000] [203.3/1000.0] [batch_t 0.759 (0.765)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-08 16:42:35,007 - Train: 20.33% [1004600/4942000] [203.3/1000.0] [batch_t 0.768 (0.764)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-08 16:43:51,325 - Train: 20.33% [1004700/4942000] [203.3/1000.0] [batch_t 0.758 (0.763)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-08 16:45:07,589 - Train: 20.33% [1004800/4942000] [203.3/1000.0] [batch_t 0.773 (0.763)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-08 16:46:23,844 - Train: 20.33% [1004900/4942000] [203.3/1000.0] [batch_t 0.758 (0.762)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-08 16:47:40,191 - Train: 20.34% [1005000/4942000] [203.4/1000.0] [batch_t 0.768 (0.763)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-08 16:48:56,426 - Train: 20.34% [1005100/4942000] [203.4/1000.0] [batch_t 0.779 (0.762)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-08 16:50:12,788 - Train: 20.34% [1005200/4942000] [203.4/1000.0] [batch_t 0.772 (0.764)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-08 16:51:29,117 - Train: 20.34% [1005300/4942000] [203.4/1000.0] [batch_t 0.768 (0.763)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-08 16:52:45,425 - Train: 20.34% [1005400/4942000] [203.4/1000.0] [batch_t 0.750 (0.763)] [data_t 0.003] [optim_t 0.747] [lr 0.005000] 2024-04-08 16:54:01,730 - Train: 20.35% [1005500/4942000] [203.5/1000.0] [batch_t 0.759 (0.763)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-08 16:55:18,643 - Train: 20.35% [1005600/4942000] [203.5/1000.0] [batch_t 0.772 (0.769)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-08 16:56:34,962 - Train: 20.35% [1005700/4942000] [203.5/1000.0] [batch_t 0.759 (0.763)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-08 16:57:51,207 - Train: 20.35% [1005800/4942000] [203.5/1000.0] [batch_t 0.752 (0.762)] [data_t 0.003] [optim_t 0.749] [lr 0.005000] 2024-04-08 16:59:07,551 - Train: 20.35% [1005900/4942000] [203.5/1000.0] [batch_t 0.769 (0.763)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-08 17:00:23,992 - Train: 20.36% [1006000/4942000] [203.6/1000.0] [batch_t 0.751 (0.764)] [data_t 0.002] [optim_t 0.748] [lr 0.005000] 2024-04-08 17:01:40,362 - Train: 20.36% [1006100/4942000] [203.6/1000.0] [batch_t 0.759 (0.764)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-08 17:02:56,789 - Train: 20.36% [1006200/4942000] [203.6/1000.0] [batch_t 0.771 (0.764)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-08 17:04:13,228 - Train: 20.36% [1006300/4942000] [203.6/1000.0] [batch_t 0.762 (0.764)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-08 17:05:29,628 - Train: 20.36% [1006400/4942000] [203.6/1000.0] [batch_t 0.768 (0.764)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-08 17:06:45,920 - Train: 20.37% [1006500/4942000] [203.7/1000.0] [batch_t 0.772 (0.763)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-08 17:08:02,328 - Train: 20.37% [1006600/4942000] [203.7/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-08 17:09:18,541 - Train: 20.37% [1006700/4942000] [203.7/1000.0] [batch_t 0.764 (0.762)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-08 17:10:34,870 - Train: 20.37% [1006800/4942000] [203.7/1000.0] [batch_t 0.768 (0.763)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-08 17:11:51,207 - Train: 20.37% [1006900/4942000] [203.7/1000.0] [batch_t 0.758 (0.763)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-08 17:13:07,576 - Train: 20.38% [1007000/4942000] [203.8/1000.0] [batch_t 0.754 (0.764)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-08 17:14:23,963 - Train: 20.38% [1007100/4942000] [203.8/1000.0] [batch_t 0.773 (0.764)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-08 17:15:40,253 - Train: 20.38% [1007200/4942000] [203.8/1000.0] [batch_t 0.749 (0.763)] [data_t 0.003] [optim_t 0.746] [lr 0.005000] 2024-04-08 17:16:56,671 - Train: 20.38% [1007300/4942000] [203.8/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-08 17:18:12,983 - Train: 20.38% [1007400/4942000] [203.8/1000.0] [batch_t 0.767 (0.763)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-08 17:19:29,424 - Train: 20.39% [1007500/4942000] [203.9/1000.0] [batch_t 0.760 (0.764)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-08 17:20:45,718 - Train: 20.39% [1007600/4942000] [203.9/1000.0] [batch_t 0.763 (0.763)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-08 17:22:02,136 - Train: 20.39% [1007700/4942000] [203.9/1000.0] [batch_t 0.768 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-08 17:23:18,318 - Train: 20.39% [1007800/4942000] [203.9/1000.0] [batch_t 0.758 (0.762)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-08 17:24:34,566 - Train: 20.39% [1007900/4942000] [203.9/1000.0] [batch_t 0.758 (0.762)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-08 17:25:50,777 - Train: 20.40% [1008000/4942000] [204.0/1000.0] [batch_t 0.769 (0.762)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-08 17:27:07,138 - Train: 20.40% [1008100/4942000] [204.0/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-08 17:27:59,060 - ==> Total time: 5 days, 23:30:38 Eta: 23 days, 7:58:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 17:28:25,662 - Train: 20.40% [1008200/4942000] [204.0/1000.0] [batch_t 0.767 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-08 17:29:41,921 - Train: 20.40% [1008300/4942000] [204.0/1000.0] [batch_t 0.758 (0.762)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-08 17:30:58,032 - Train: 20.40% [1008400/4942000] [204.0/1000.0] [batch_t 0.768 (0.761)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-08 17:32:14,365 - Train: 20.41% [1008500/4942000] [204.1/1000.0] [batch_t 0.767 (0.763)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-08 17:33:30,572 - Train: 20.41% [1008600/4942000] [204.1/1000.0] [batch_t 0.759 (0.762)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-08 17:34:46,868 - Train: 20.41% [1008700/4942000] [204.1/1000.0] [batch_t 0.758 (0.763)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-08 17:36:03,236 - Train: 20.41% [1008800/4942000] [204.1/1000.0] [batch_t 0.755 (0.764)] [data_t 0.003] [optim_t 0.752] [lr 0.005000] 2024-04-08 17:37:19,469 - Train: 20.41% [1008900/4942000] [204.1/1000.0] [batch_t 0.766 (0.762)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-08 17:38:35,724 - Train: 20.42% [1009000/4942000] [204.2/1000.0] [batch_t 0.752 (0.762)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-08 17:39:51,988 - Train: 20.42% [1009100/4942000] [204.2/1000.0] [batch_t 0.769 (0.763)] [data_t 0.003] [optim_t 0.767] [lr 0.005000] 2024-04-08 17:41:08,295 - Train: 20.42% [1009200/4942000] [204.2/1000.0] [batch_t 0.764 (0.763)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-08 17:42:33,970 - Train: 20.42% [1009300/4942000] [204.2/1000.0] [batch_t 0.776 (0.857)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-08 17:43:50,199 - Train: 20.42% [1009400/4942000] [204.2/1000.0] [batch_t 0.772 (0.762)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-08 17:45:06,415 - Train: 20.43% [1009500/4942000] [204.3/1000.0] [batch_t 0.768 (0.762)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-08 17:46:31,912 - Train: 20.43% [1009600/4942000] [204.3/1000.0] [batch_t 0.759 (0.855)] [data_t 0.003] [optim_t 0.756] [lr 0.005000] 2024-04-08 17:47:48,163 - Train: 20.43% [1009700/4942000] [204.3/1000.0] [batch_t 0.763 (0.762)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-08 17:49:16,176 - Train: 20.43% [1009800/4942000] [204.3/1000.0] [batch_t 0.770 (0.880)] [data_t 0.002] [optim_t 0.768] [lr 0.005000] 2024-04-08 17:50:39,585 - Train: 20.44% [1009900/4942000] [204.4/1000.0] [batch_t 0.758 (0.834)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-08 17:51:55,919 - Train: 20.44% [1010000/4942000] [204.4/1000.0] [batch_t 0.773 (0.763)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-08 17:53:22,209 - Train: 20.44% [1010100/4942000] [204.4/1000.0] [batch_t 10.738 (0.863)] [data_t 9.973] [optim_t 0.766] [lr 0.005000] 2024-04-08 17:54:38,674 - Train: 20.44% [1010200/4942000] [204.4/1000.0] [batch_t 0.767 (0.765)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-08 17:55:57,435 - Train: 20.44% [1010300/4942000] [204.4/1000.0] [batch_t 0.757 (0.787)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-08 17:57:13,849 - Train: 20.45% [1010400/4942000] [204.5/1000.0] [batch_t 0.772 (0.764)] [data_t 0.002] [optim_t 0.770] [lr 0.005000] 2024-04-08 17:58:39,548 - Train: 20.45% [1010500/4942000] [204.5/1000.0] [batch_t 0.759 (0.857)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-08 18:00:00,771 - Train: 20.45% [1010600/4942000] [204.5/1000.0] [batch_t 0.762 (0.812)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-08 18:01:28,861 - Train: 20.45% [1010700/4942000] [204.5/1000.0] [batch_t 0.772 (0.881)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-08 18:02:57,695 - Train: 20.45% [1010800/4942000] [204.5/1000.0] [batch_t 0.760 (0.888)] [data_t 0.003] [optim_t 0.758] [lr 0.005000] 2024-04-08 18:04:18,900 - Train: 20.46% [1010900/4942000] [204.6/1000.0] [batch_t 0.766 (0.812)] [data_t 0.002] [optim_t 0.763] [lr 0.005000] 2024-04-08 18:05:38,037 - Train: 20.46% [1011000/4942000] [204.6/1000.0] [batch_t 0.773 (0.791)] [data_t 0.002] [optim_t 0.771] [lr 0.005000] 2024-04-08 18:06:57,318 - Train: 20.46% [1011100/4942000] [204.6/1000.0] [batch_t 0.766 (0.793)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-08 18:08:27,738 - Train: 20.46% [1011200/4942000] [204.6/1000.0] [batch_t 0.756 (0.904)] [data_t 0.003] [optim_t 0.753] [lr 0.005000] 2024-04-08 18:09:44,004 - Train: 20.46% [1011300/4942000] [204.6/1000.0] [batch_t 0.753 (0.763)] [data_t 0.003] [optim_t 0.750] [lr 0.005000] 2024-04-08 18:11:09,120 - Train: 20.47% [1011400/4942000] [204.7/1000.0] [batch_t 0.777 (0.851)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-08 18:12:26,727 - Train: 20.47% [1011500/4942000] [204.7/1000.0] [batch_t 0.755 (0.776)] [data_t 0.003] [optim_t 0.753] [lr 0.005000] 2024-04-08 18:13:48,316 - Train: 20.47% [1011600/4942000] [204.7/1000.0] [batch_t 0.769 (0.816)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-08 18:15:04,489 - Train: 20.47% [1011700/4942000] [204.7/1000.0] [batch_t 0.776 (0.762)] [data_t 0.002] [optim_t 0.774] [lr 0.005000] 2024-04-08 18:16:20,823 - Train: 20.47% [1011800/4942000] [204.7/1000.0] [batch_t 0.770 (0.763)] [data_t 0.002] [optim_t 0.767] [lr 0.005000] 2024-04-08 18:17:37,070 - Train: 20.48% [1011900/4942000] [204.8/1000.0] [batch_t 0.764 (0.762)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-08 18:18:53,883 - Train: 20.48% [1012000/4942000] [204.8/1000.0] [batch_t 0.772 (0.768)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-08 18:20:10,043 - Train: 20.48% [1012100/4942000] [204.8/1000.0] [batch_t 0.762 (0.761)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-08 18:21:26,408 - Train: 20.48% [1012200/4942000] [204.8/1000.0] [batch_t 0.769 (0.764)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-08 18:22:42,799 - Train: 20.48% [1012300/4942000] [204.8/1000.0] [batch_t 0.758 (0.764)] [data_t 0.003] [optim_t 0.755] [lr 0.005000] 2024-04-08 18:23:58,933 - Train: 20.49% [1012400/4942000] [204.9/1000.0] [batch_t 0.750 (0.761)] [data_t 0.003] [optim_t 0.747] [lr 0.005000] 2024-04-08 18:25:15,322 - Train: 20.49% [1012500/4942000] [204.9/1000.0] [batch_t 0.769 (0.764)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-08 18:26:31,572 - Train: 20.49% [1012600/4942000] [204.9/1000.0] [batch_t 0.759 (0.762)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-08 18:27:47,764 - Train: 20.49% [1012700/4942000] [204.9/1000.0] [batch_t 0.774 (0.762)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-08 18:29:04,108 - Train: 20.49% [1012800/4942000] [204.9/1000.0] [batch_t 0.753 (0.763)] [data_t 0.003] [optim_t 0.751] [lr 0.005000] 2024-04-08 18:30:20,358 - Train: 20.50% [1012900/4942000] [205.0/1000.0] [batch_t 0.766 (0.762)] [data_t 0.003] [optim_t 0.763] [lr 0.005000] 2024-04-08 18:31:36,671 - Train: 20.50% [1013000/4942000] [205.0/1000.0] [batch_t 0.741 (0.763)] [data_t 0.003] [optim_t 0.738] [lr 0.005000] 2024-04-08 18:32:53,038 - Train: 20.50% [1013100/4942000] [205.0/1000.0] [batch_t 0.764 (0.764)] [data_t 0.003] [optim_t 0.760] [lr 0.005000] 2024-04-08 18:33:00,638 - ==> Total time: 6 days, 0:35:39 Eta: 23 days, 8:44:38 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 18:33:48,019 - Train: 20.50% [1013200/4942000] [205.0/1000.0] [batch_t 0.324 (0.498)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 18:34:21,653 - Train: 20.50% [1013300/4942000] [205.0/1000.0] [batch_t 0.320 (0.336)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-08 18:34:54,402 - Train: 20.51% [1013400/4942000] [205.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 18:35:28,612 - Train: 20.51% [1013500/4942000] [205.1/1000.0] [batch_t 0.325 (0.342)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 18:36:01,413 - Train: 20.51% [1013600/4942000] [205.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 18:36:35,788 - Train: 20.51% [1013700/4942000] [205.1/1000.0] [batch_t 0.330 (0.344)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 18:37:09,796 - Train: 20.51% [1013800/4942000] [205.1/1000.0] [batch_t 0.333 (0.340)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 18:37:42,564 - Train: 20.52% [1013900/4942000] [205.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 18:38:15,483 - Train: 20.52% [1014000/4942000] [205.2/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 18:38:48,202 - Train: 20.52% [1014100/4942000] [205.2/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 18:39:21,391 - Train: 20.52% [1014200/4942000] [205.2/1000.0] [batch_t 0.326 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 18:39:54,076 - Train: 20.52% [1014300/4942000] [205.2/1000.0] [batch_t 0.334 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 18:40:26,943 - Train: 20.53% [1014400/4942000] [205.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 18:40:59,635 - Train: 20.53% [1014500/4942000] [205.3/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 18:41:32,557 - Train: 20.53% [1014600/4942000] [205.3/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 18:42:05,370 - Train: 20.53% [1014700/4942000] [205.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 18:42:38,110 - Train: 20.53% [1014800/4942000] [205.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 18:43:10,953 - Train: 20.54% [1014900/4942000] [205.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 18:43:43,748 - Train: 20.54% [1015000/4942000] [205.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 18:44:16,932 - Train: 20.54% [1015100/4942000] [205.4/1000.0] [batch_t 0.326 (0.332)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 18:44:49,628 - Train: 20.54% [1015200/4942000] [205.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 18:45:23,534 - Train: 20.54% [1015300/4942000] [205.4/1000.0] [batch_t 0.324 (0.339)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 18:45:56,309 - Train: 20.55% [1015400/4942000] [205.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 18:46:29,070 - Train: 20.55% [1015500/4942000] [205.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 18:47:01,838 - Train: 20.55% [1015600/4942000] [205.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 18:47:34,531 - Train: 20.55% [1015700/4942000] [205.5/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 18:48:07,444 - Train: 20.55% [1015800/4942000] [205.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 18:48:40,118 - Train: 20.56% [1015900/4942000] [205.6/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 18:49:12,872 - Train: 20.56% [1016000/4942000] [205.6/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 18:49:45,546 - Train: 20.56% [1016100/4942000] [205.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 18:50:18,345 - Train: 20.56% [1016200/4942000] [205.6/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 18:50:51,129 - Train: 20.56% [1016300/4942000] [205.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 18:51:23,840 - Train: 20.57% [1016400/4942000] [205.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 18:51:56,529 - Train: 20.57% [1016500/4942000] [205.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 18:52:29,199 - Train: 20.57% [1016600/4942000] [205.7/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 18:53:01,964 - Train: 20.57% [1016700/4942000] [205.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 18:53:34,674 - Train: 20.57% [1016800/4942000] [205.7/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 18:54:07,400 - Train: 20.58% [1016900/4942000] [205.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 18:54:40,079 - Train: 20.58% [1017000/4942000] [205.8/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 18:55:12,853 - Train: 20.58% [1017100/4942000] [205.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 18:56:01,596 - Train: 20.58% [1017200/4942000] [205.8/1000.0] [batch_t 0.328 (0.487)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 18:56:36,880 - Train: 20.58% [1017300/4942000] [205.8/1000.0] [batch_t 0.329 (0.352)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 18:57:10,621 - Train: 20.59% [1017400/4942000] [205.9/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 18:57:43,327 - Train: 20.59% [1017500/4942000] [205.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 18:58:17,306 - Train: 20.59% [1017600/4942000] [205.9/1000.0] [batch_t 0.330 (0.340)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 18:58:49,963 - Train: 20.59% [1017700/4942000] [205.9/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 18:59:22,673 - Train: 20.59% [1017800/4942000] [205.9/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 18:59:55,390 - Train: 20.60% [1017900/4942000] [206.0/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:00:29,755 - Train: 20.60% [1018000/4942000] [206.0/1000.0] [batch_t 0.330 (0.344)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:00:46,730 - ==> Total time: 6 days, 1:03:25 Eta: 23 days, 7:06:14 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 19:01:06,387 - Train: 20.60% [1018100/4942000] [206.0/1000.0] [batch_t 1.586 (0.359)] [data_t 1.263] [optim_t 0.324] [lr 0.005000] 2024-04-08 19:01:43,326 - Train: 20.60% [1018200/4942000] [206.0/1000.0] [batch_t 0.331 (0.369)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 19:02:21,655 - Train: 20.61% [1018300/4942000] [206.1/1000.0] [batch_t 0.324 (0.383)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 19:02:54,362 - Train: 20.61% [1018400/4942000] [206.1/1000.0] [batch_t 0.336 (0.327)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-08 19:03:36,490 - Train: 20.61% [1018500/4942000] [206.1/1000.0] [batch_t 0.329 (0.421)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:04:10,341 - Train: 20.61% [1018600/4942000] [206.1/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:04:43,087 - Train: 20.61% [1018700/4942000] [206.1/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 19:05:16,565 - Train: 20.62% [1018800/4942000] [206.2/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:05:49,275 - Train: 20.62% [1018900/4942000] [206.2/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:06:21,950 - Train: 20.62% [1019000/4942000] [206.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:06:54,688 - Train: 20.62% [1019100/4942000] [206.2/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 19:07:27,332 - Train: 20.62% [1019200/4942000] [206.2/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 19:08:00,128 - Train: 20.63% [1019300/4942000] [206.3/1000.0] [batch_t 0.315 (0.328)] [data_t 0.002] [optim_t 0.313] [lr 0.005000] 2024-04-08 19:08:32,786 - Train: 20.63% [1019400/4942000] [206.3/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 19:09:07,992 - Train: 20.63% [1019500/4942000] [206.3/1000.0] [batch_t 0.325 (0.352)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 19:09:40,734 - Train: 20.63% [1019600/4942000] [206.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 19:10:13,424 - Train: 20.63% [1019700/4942000] [206.3/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 19:10:46,127 - Train: 20.64% [1019800/4942000] [206.4/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 19:11:18,845 - Train: 20.64% [1019900/4942000] [206.4/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 19:11:51,726 - Train: 20.64% [1020000/4942000] [206.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 19:12:24,592 - Train: 20.64% [1020100/4942000] [206.4/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 19:12:57,376 - Train: 20.64% [1020200/4942000] [206.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 19:13:30,147 - Train: 20.65% [1020300/4942000] [206.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:14:02,827 - Train: 20.65% [1020400/4942000] [206.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:14:35,510 - Train: 20.65% [1020500/4942000] [206.5/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 19:15:08,176 - Train: 20.65% [1020600/4942000] [206.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 19:15:40,963 - Train: 20.65% [1020700/4942000] [206.5/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 19:16:13,695 - Train: 20.66% [1020800/4942000] [206.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:16:46,391 - Train: 20.66% [1020900/4942000] [206.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 19:17:20,374 - Train: 20.66% [1021000/4942000] [206.6/1000.0] [batch_t 0.326 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 19:17:53,080 - Train: 20.66% [1021100/4942000] [206.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:18:26,750 - Train: 20.66% [1021200/4942000] [206.6/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 19:18:59,558 - Train: 20.67% [1021300/4942000] [206.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 19:19:32,553 - Train: 20.67% [1021400/4942000] [206.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 19:20:05,409 - Train: 20.67% [1021500/4942000] [206.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 19:20:38,192 - Train: 20.67% [1021600/4942000] [206.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:21:10,941 - Train: 20.67% [1021700/4942000] [206.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 19:21:43,658 - Train: 20.68% [1021800/4942000] [206.8/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:22:17,538 - Train: 20.68% [1021900/4942000] [206.8/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:22:50,255 - Train: 20.68% [1022000/4942000] [206.8/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 19:23:23,346 - Train: 20.68% [1022100/4942000] [206.8/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 19:23:56,113 - Train: 20.68% [1022200/4942000] [206.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 19:24:29,332 - Train: 20.69% [1022300/4942000] [206.9/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 19:25:02,066 - Train: 20.69% [1022400/4942000] [206.9/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 19:25:35,963 - Train: 20.69% [1022500/4942000] [206.9/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 19:26:08,691 - Train: 20.69% [1022600/4942000] [206.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 19:26:41,446 - Train: 20.69% [1022700/4942000] [206.9/1000.0] [batch_t 0.338 (0.327)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-08 19:27:16,376 - Train: 20.70% [1022800/4942000] [207.0/1000.0] [batch_t 0.328 (0.349)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 19:27:49,764 - Train: 20.70% [1022900/4942000] [207.0/1000.0] [batch_t 0.323 (0.334)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 19:28:20,538 - ==> Total time: 6 days, 1:30:59 Eta: 23 days, 5:27:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 19:28:25,394 - Train: 20.70% [1023000/4942000] [207.0/1000.0] [batch_t 0.336 (0.437)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-08 19:28:58,127 - Train: 20.70% [1023100/4942000] [207.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:29:30,780 - Train: 20.70% [1023200/4942000] [207.0/1000.0] [batch_t 0.327 (0.326)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 19:30:03,464 - Train: 20.71% [1023300/4942000] [207.1/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 19:30:36,155 - Train: 20.71% [1023400/4942000] [207.1/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 19:31:08,833 - Train: 20.71% [1023500/4942000] [207.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 19:31:41,482 - Train: 20.71% [1023600/4942000] [207.1/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 19:32:14,283 - Train: 20.71% [1023700/4942000] [207.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 19:32:47,164 - Train: 20.72% [1023800/4942000] [207.2/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 19:33:19,839 - Train: 20.72% [1023900/4942000] [207.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 19:33:52,453 - Train: 20.72% [1024000/4942000] [207.2/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:34:26,104 - Train: 20.72% [1024100/4942000] [207.2/1000.0] [batch_t 0.326 (0.336)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 19:34:58,935 - Train: 20.72% [1024200/4942000] [207.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:35:31,627 - Train: 20.73% [1024300/4942000] [207.3/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 19:36:04,240 - Train: 20.73% [1024400/4942000] [207.3/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 19:36:36,934 - Train: 20.73% [1024500/4942000] [207.3/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:37:10,105 - Train: 20.73% [1024600/4942000] [207.3/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 19:37:42,842 - Train: 20.73% [1024700/4942000] [207.3/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 19:38:15,945 - Train: 20.74% [1024800/4942000] [207.4/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:38:48,683 - Train: 20.74% [1024900/4942000] [207.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:39:22,283 - Train: 20.74% [1025000/4942000] [207.4/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 19:39:54,952 - Train: 20.74% [1025100/4942000] [207.4/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 19:40:28,278 - Train: 20.74% [1025200/4942000] [207.4/1000.0] [batch_t 0.325 (0.333)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 19:41:01,010 - Train: 20.75% [1025300/4942000] [207.5/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 19:41:33,850 - Train: 20.75% [1025400/4942000] [207.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 19:42:07,861 - Train: 20.75% [1025500/4942000] [207.5/1000.0] [batch_t 0.325 (0.340)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 19:42:40,643 - Train: 20.75% [1025600/4942000] [207.5/1000.0] [batch_t 0.338 (0.328)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-08 19:43:14,494 - Train: 20.75% [1025700/4942000] [207.5/1000.0] [batch_t 0.325 (0.338)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 19:43:47,359 - Train: 20.76% [1025800/4942000] [207.6/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 19:44:21,355 - Train: 20.76% [1025900/4942000] [207.6/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 19:44:54,093 - Train: 20.76% [1026000/4942000] [207.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 19:45:27,595 - Train: 20.76% [1026100/4942000] [207.6/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 19:46:00,389 - Train: 20.76% [1026200/4942000] [207.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:46:33,233 - Train: 20.77% [1026300/4942000] [207.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:47:06,014 - Train: 20.77% [1026400/4942000] [207.7/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 19:47:38,724 - Train: 20.77% [1026500/4942000] [207.7/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:48:11,520 - Train: 20.77% [1026600/4942000] [207.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 19:48:44,334 - Train: 20.77% [1026700/4942000] [207.7/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 19:49:17,593 - Train: 20.78% [1026800/4942000] [207.8/1000.0] [batch_t 0.332 (0.332)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 19:49:50,469 - Train: 20.78% [1026900/4942000] [207.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 19:50:23,675 - Train: 20.78% [1027000/4942000] [207.8/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 19:50:56,745 - Train: 20.78% [1027100/4942000] [207.8/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 19:51:29,537 - Train: 20.79% [1027200/4942000] [207.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 19:52:02,289 - Train: 20.79% [1027300/4942000] [207.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 19:52:36,796 - Train: 20.79% [1027400/4942000] [207.9/1000.0] [batch_t 0.326 (0.345)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 19:53:09,638 - Train: 20.79% [1027500/4942000] [207.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 19:53:42,366 - Train: 20.79% [1027600/4942000] [207.9/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:54:16,894 - Train: 20.80% [1027700/4942000] [208.0/1000.0] [batch_t 0.328 (0.345)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 19:54:49,589 - Train: 20.80% [1027800/4942000] [208.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 19:55:22,490 - Train: 20.80% [1027900/4942000] [208.0/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 19:55:34,297 - ==> Total time: 6 days, 1:58:13 Eta: 23 days, 3:48:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 19:55:57,696 - Train: 20.80% [1028000/4942000] [208.0/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 19:56:31,702 - Train: 20.80% [1028100/4942000] [208.0/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:57:04,526 - Train: 20.81% [1028200/4942000] [208.1/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 19:57:37,392 - Train: 20.81% [1028300/4942000] [208.1/1000.0] [batch_t 0.336 (0.329)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-08 19:58:12,882 - Train: 20.81% [1028400/4942000] [208.1/1000.0] [batch_t 0.332 (0.355)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 19:58:45,618 - Train: 20.81% [1028500/4942000] [208.1/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 19:59:21,103 - Train: 20.81% [1028600/4942000] [208.1/1000.0] [batch_t 0.325 (0.355)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 19:59:53,818 - Train: 20.82% [1028700/4942000] [208.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 20:00:27,675 - Train: 20.82% [1028800/4942000] [208.2/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 20:01:00,493 - Train: 20.82% [1028900/4942000] [208.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 20:01:34,265 - Train: 20.82% [1029000/4942000] [208.2/1000.0] [batch_t 0.325 (0.338)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 20:02:06,962 - Train: 20.82% [1029100/4942000] [208.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 20:02:39,591 - Train: 20.83% [1029200/4942000] [208.3/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 20:03:12,280 - Train: 20.83% [1029300/4942000] [208.3/1000.0] [batch_t 0.335 (0.327)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 20:03:45,103 - Train: 20.83% [1029400/4942000] [208.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 20:04:18,715 - Train: 20.83% [1029500/4942000] [208.3/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 20:04:51,395 - Train: 20.83% [1029600/4942000] [208.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 20:05:24,691 - Train: 20.84% [1029700/4942000] [208.4/1000.0] [batch_t 0.324 (0.333)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 20:05:57,372 - Train: 20.84% [1029800/4942000] [208.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 20:06:30,178 - Train: 20.84% [1029900/4942000] [208.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 20:07:02,867 - Train: 20.84% [1030000/4942000] [208.4/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 20:07:36,523 - Train: 20.84% [1030100/4942000] [208.4/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 20:08:10,734 - Train: 20.85% [1030200/4942000] [208.5/1000.0] [batch_t 0.331 (0.342)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 20:08:43,427 - Train: 20.85% [1030300/4942000] [208.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 20:09:17,143 - Train: 20.85% [1030400/4942000] [208.5/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 20:09:49,950 - Train: 20.85% [1030500/4942000] [208.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:10:23,170 - Train: 20.85% [1030600/4942000] [208.5/1000.0] [batch_t 0.325 (0.332)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 20:10:55,881 - Train: 20.86% [1030700/4942000] [208.6/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 20:11:31,459 - Train: 20.86% [1030800/4942000] [208.6/1000.0] [batch_t 0.327 (0.356)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 20:12:04,109 - Train: 20.86% [1030900/4942000] [208.6/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 20:12:38,194 - Train: 20.86% [1031000/4942000] [208.6/1000.0] [batch_t 0.325 (0.341)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 20:13:11,435 - Train: 20.86% [1031100/4942000] [208.6/1000.0] [batch_t 0.325 (0.332)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 20:13:44,167 - Train: 20.87% [1031200/4942000] [208.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 20:14:19,671 - Train: 20.87% [1031300/4942000] [208.7/1000.0] [batch_t 0.332 (0.355)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 20:14:52,382 - Train: 20.87% [1031400/4942000] [208.7/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:15:25,861 - Train: 20.87% [1031500/4942000] [208.7/1000.0] [batch_t 0.331 (0.335)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 20:15:58,607 - Train: 20.87% [1031600/4942000] [208.7/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 20:16:32,153 - Train: 20.88% [1031700/4942000] [208.8/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 20:17:06,472 - Train: 20.88% [1031800/4942000] [208.8/1000.0] [batch_t 2.008 (0.343)] [data_t 1.677] [optim_t 0.331] [lr 0.005000] 2024-04-08 20:17:39,121 - Train: 20.88% [1031900/4942000] [208.8/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 20:18:13,034 - Train: 20.88% [1032000/4942000] [208.8/1000.0] [batch_t 0.328 (0.339)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 20:18:45,645 - Train: 20.88% [1032100/4942000] [208.8/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 20:19:18,370 - Train: 20.89% [1032200/4942000] [208.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 20:19:51,005 - Train: 20.89% [1032300/4942000] [208.9/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 20:20:24,711 - Train: 20.89% [1032400/4942000] [208.9/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:20:57,415 - Train: 20.89% [1032500/4942000] [208.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 20:21:30,193 - Train: 20.89% [1032600/4942000] [208.9/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 20:22:02,833 - Train: 20.90% [1032700/4942000] [209.0/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 20:22:38,561 - Train: 20.90% [1032800/4942000] [209.0/1000.0] [batch_t 0.329 (0.357)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 20:23:06,554 - ==> Total time: 6 days, 2:25:45 Eta: 23 days, 2:11:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 20:23:17,579 - Train: 20.90% [1032900/4942000] [209.0/1000.0] [batch_t 0.327 (0.390)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 20:23:50,266 - Train: 20.90% [1033000/4942000] [209.0/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 20:24:27,902 - Train: 20.90% [1033100/4942000] [209.0/1000.0] [batch_t 0.327 (0.376)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 20:25:00,579 - Train: 20.91% [1033200/4942000] [209.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 20:25:35,774 - Train: 20.91% [1033300/4942000] [209.1/1000.0] [batch_t 0.327 (0.352)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 20:26:12,193 - Train: 20.91% [1033400/4942000] [209.1/1000.0] [batch_t 0.328 (0.364)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 20:26:44,973 - Train: 20.91% [1033500/4942000] [209.1/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 20:27:19,206 - Train: 20.91% [1033600/4942000] [209.1/1000.0] [batch_t 0.328 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 20:27:51,967 - Train: 20.92% [1033700/4942000] [209.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 20:28:26,476 - Train: 20.92% [1033800/4942000] [209.2/1000.0] [batch_t 0.330 (0.345)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:28:59,284 - Train: 20.92% [1033900/4942000] [209.2/1000.0] [batch_t 0.336 (0.328)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-08 20:29:33,870 - Train: 20.92% [1034000/4942000] [209.2/1000.0] [batch_t 0.329 (0.346)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:30:09,819 - Train: 20.92% [1034100/4942000] [209.2/1000.0] [batch_t 0.329 (0.359)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 20:30:42,489 - Train: 20.93% [1034200/4942000] [209.3/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 20:31:16,676 - Train: 20.93% [1034300/4942000] [209.3/1000.0] [batch_t 0.325 (0.342)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 20:31:49,458 - Train: 20.93% [1034400/4942000] [209.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 20:32:23,421 - Train: 20.93% [1034500/4942000] [209.3/1000.0] [batch_t 0.323 (0.340)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 20:32:56,190 - Train: 20.93% [1034600/4942000] [209.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 20:33:30,153 - Train: 20.94% [1034700/4942000] [209.4/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:34:02,878 - Train: 20.94% [1034800/4942000] [209.4/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 20:34:37,330 - Train: 20.94% [1034900/4942000] [209.4/1000.0] [batch_t 0.324 (0.344)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 20:35:10,894 - Train: 20.94% [1035000/4942000] [209.4/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 20:35:43,727 - Train: 20.94% [1035100/4942000] [209.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 20:36:17,165 - Train: 20.95% [1035200/4942000] [209.5/1000.0] [batch_t 0.324 (0.334)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 20:36:50,203 - Train: 20.95% [1035300/4942000] [209.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 20:37:24,012 - Train: 20.95% [1035400/4942000] [209.5/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 20:37:56,770 - Train: 20.95% [1035500/4942000] [209.5/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 20:38:30,354 - Train: 20.96% [1035600/4942000] [209.6/1000.0] [batch_t 0.336 (0.336)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-08 20:39:03,064 - Train: 20.96% [1035700/4942000] [209.6/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 20:39:36,413 - Train: 20.96% [1035800/4942000] [209.6/1000.0] [batch_t 0.326 (0.333)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 20:40:10,075 - Train: 20.96% [1035900/4942000] [209.6/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 20:40:42,892 - Train: 20.96% [1036000/4942000] [209.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 20:41:15,677 - Train: 20.97% [1036100/4942000] [209.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 20:41:48,533 - Train: 20.97% [1036200/4942000] [209.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:42:21,431 - Train: 20.97% [1036300/4942000] [209.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 20:42:54,211 - Train: 20.97% [1036400/4942000] [209.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 20:43:26,954 - Train: 20.97% [1036500/4942000] [209.7/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:43:59,728 - Train: 20.98% [1036600/4942000] [209.8/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 20:44:32,546 - Train: 20.98% [1036700/4942000] [209.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 20:45:05,247 - Train: 20.98% [1036800/4942000] [209.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 20:45:37,857 - Train: 20.98% [1036900/4942000] [209.8/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 20:46:10,682 - Train: 20.98% [1037000/4942000] [209.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 20:46:43,420 - Train: 20.99% [1037100/4942000] [209.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 20:47:16,247 - Train: 20.99% [1037200/4942000] [209.9/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 20:47:49,083 - Train: 20.99% [1037300/4942000] [209.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 20:48:21,886 - Train: 20.99% [1037400/4942000] [209.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 20:48:54,684 - Train: 20.99% [1037500/4942000] [209.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 20:49:27,499 - Train: 21.00% [1037600/4942000] [210.0/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 20:50:00,312 - Train: 21.00% [1037700/4942000] [210.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 20:50:33,109 - Train: 21.00% [1037800/4942000] [210.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 20:50:39,674 - ==> Total time: 6 days, 2:53:18 Eta: 23 days, 0:34:50 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 20:51:08,002 - Train: 21.00% [1037900/4942000] [210.0/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 20:51:40,778 - Train: 21.00% [1038000/4942000] [210.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:52:13,439 - Train: 21.01% [1038100/4942000] [210.1/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 20:52:46,070 - Train: 21.01% [1038200/4942000] [210.1/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 20:53:18,939 - Train: 21.01% [1038300/4942000] [210.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 20:53:51,622 - Train: 21.01% [1038400/4942000] [210.1/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:54:24,329 - Train: 21.01% [1038500/4942000] [210.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 20:54:57,037 - Train: 21.02% [1038600/4942000] [210.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 20:55:29,771 - Train: 21.02% [1038700/4942000] [210.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 20:56:02,448 - Train: 21.02% [1038800/4942000] [210.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 20:56:35,212 - Train: 21.02% [1038900/4942000] [210.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 20:57:07,951 - Train: 21.02% [1039000/4942000] [210.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 20:57:40,650 - Train: 21.03% [1039100/4942000] [210.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 20:58:13,305 - Train: 21.03% [1039200/4942000] [210.3/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 20:58:46,008 - Train: 21.03% [1039300/4942000] [210.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 20:59:18,718 - Train: 21.03% [1039400/4942000] [210.3/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 20:59:51,382 - Train: 21.03% [1039500/4942000] [210.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 21:00:24,154 - Train: 21.04% [1039600/4942000] [210.4/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 21:00:56,983 - Train: 21.04% [1039700/4942000] [210.4/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 21:01:29,719 - Train: 21.04% [1039800/4942000] [210.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 21:02:02,499 - Train: 21.04% [1039900/4942000] [210.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 21:02:35,185 - Train: 21.04% [1040000/4942000] [210.4/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 21:03:07,918 - Train: 21.05% [1040100/4942000] [210.5/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 21:03:40,589 - Train: 21.05% [1040200/4942000] [210.5/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 21:04:13,263 - Train: 21.05% [1040300/4942000] [210.5/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 21:04:45,951 - Train: 21.05% [1040400/4942000] [210.5/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 21:05:18,740 - Train: 21.05% [1040500/4942000] [210.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 21:05:51,410 - Train: 21.06% [1040600/4942000] [210.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 21:06:24,095 - Train: 21.06% [1040700/4942000] [210.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 21:06:56,782 - Train: 21.06% [1040800/4942000] [210.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 21:07:29,476 - Train: 21.06% [1040900/4942000] [210.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 21:08:02,222 - Train: 21.06% [1041000/4942000] [210.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 21:08:34,909 - Train: 21.07% [1041100/4942000] [210.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 21:09:08,106 - Train: 21.07% [1041200/4942000] [210.7/1000.0] [batch_t 0.333 (0.332)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 21:09:40,827 - Train: 21.07% [1041300/4942000] [210.7/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 21:10:13,546 - Train: 21.07% [1041400/4942000] [210.7/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 21:10:46,236 - Train: 21.07% [1041500/4942000] [210.7/1000.0] [batch_t 0.321 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 21:11:18,938 - Train: 21.08% [1041600/4942000] [210.8/1000.0] [batch_t 0.335 (0.327)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 21:11:51,620 - Train: 21.08% [1041700/4942000] [210.8/1000.0] [batch_t 0.344 (0.327)] [data_t 0.002] [optim_t 0.342] [lr 0.005000] 2024-04-08 21:12:24,363 - Train: 21.08% [1041800/4942000] [210.8/1000.0] [batch_t 0.336 (0.327)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-08 21:12:57,024 - Train: 21.08% [1041900/4942000] [210.8/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 21:13:29,734 - Train: 21.08% [1042000/4942000] [210.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 21:14:02,443 - Train: 21.09% [1042100/4942000] [210.9/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 21:14:35,091 - Train: 21.09% [1042200/4942000] [210.9/1000.0] [batch_t 0.337 (0.326)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-08 21:15:07,917 - Train: 21.09% [1042300/4942000] [210.9/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 21:15:40,592 - Train: 21.09% [1042400/4942000] [210.9/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 21:16:13,355 - Train: 21.09% [1042500/4942000] [210.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 21:16:46,111 - Train: 21.10% [1042600/4942000] [211.0/1000.0] [batch_t 0.321 (0.327)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-08 21:17:18,840 - Train: 21.10% [1042700/4942000] [211.0/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 21:17:39,113 - ==> Total time: 6 days, 3:20:18 Eta: 22 days, 22:56:52 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 21:17:53,928 - Train: 21.10% [1042800/4942000] [211.0/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 21:18:26,615 - Train: 21.10% [1042900/4942000] [211.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 21:18:59,205 - Train: 21.10% [1043000/4942000] [211.0/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 21:19:31,907 - Train: 21.11% [1043100/4942000] [211.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 21:20:04,591 - Train: 21.11% [1043200/4942000] [211.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 21:20:37,264 - Train: 21.11% [1043300/4942000] [211.1/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 21:21:09,965 - Train: 21.11% [1043400/4942000] [211.1/1000.0] [batch_t 0.334 (0.327)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 21:21:42,618 - Train: 21.11% [1043500/4942000] [211.1/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 21:22:15,518 - Train: 21.12% [1043600/4942000] [211.2/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 21:22:48,362 - Train: 21.12% [1043700/4942000] [211.2/1000.0] [batch_t 0.339 (0.328)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-08 21:23:21,192 - Train: 21.12% [1043800/4942000] [211.2/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 21:23:54,134 - Train: 21.12% [1043900/4942000] [211.2/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 21:24:27,020 - Train: 21.13% [1044000/4942000] [211.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 21:24:59,843 - Train: 21.13% [1044100/4942000] [211.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 21:25:32,749 - Train: 21.13% [1044200/4942000] [211.3/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 21:26:05,489 - Train: 21.13% [1044300/4942000] [211.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 21:26:38,285 - Train: 21.13% [1044400/4942000] [211.3/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 21:27:11,052 - Train: 21.14% [1044500/4942000] [211.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 21:27:43,852 - Train: 21.14% [1044600/4942000] [211.4/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 21:28:16,692 - Train: 21.14% [1044700/4942000] [211.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 21:28:49,544 - Train: 21.14% [1044800/4942000] [211.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 21:29:22,339 - Train: 21.14% [1044900/4942000] [211.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 21:29:55,147 - Train: 21.15% [1045000/4942000] [211.5/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 21:30:27,982 - Train: 21.15% [1045100/4942000] [211.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 21:31:00,777 - Train: 21.15% [1045200/4942000] [211.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 21:31:33,749 - Train: 21.15% [1045300/4942000] [211.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 21:32:06,574 - Train: 21.15% [1045400/4942000] [211.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 21:32:39,225 - Train: 21.16% [1045500/4942000] [211.6/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 21:33:13,162 - Train: 21.16% [1045600/4942000] [211.6/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 21:33:46,911 - Train: 21.16% [1045700/4942000] [211.6/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 21:34:19,580 - Train: 21.16% [1045800/4942000] [211.6/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 21:34:52,284 - Train: 21.16% [1045900/4942000] [211.6/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 21:35:24,981 - Train: 21.17% [1046000/4942000] [211.7/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 21:35:57,708 - Train: 21.17% [1046100/4942000] [211.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 21:36:30,397 - Train: 21.17% [1046200/4942000] [211.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 21:37:03,113 - Train: 21.17% [1046300/4942000] [211.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 21:37:35,809 - Train: 21.17% [1046400/4942000] [211.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 21:38:08,459 - Train: 21.18% [1046500/4942000] [211.8/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 21:38:41,175 - Train: 21.18% [1046600/4942000] [211.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 21:39:13,920 - Train: 21.18% [1046700/4942000] [211.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 21:39:46,737 - Train: 21.18% [1046800/4942000] [211.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 21:40:19,470 - Train: 21.18% [1046900/4942000] [211.8/1000.0] [batch_t 0.336 (0.327)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-08 21:40:52,274 - Train: 21.19% [1047000/4942000] [211.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 21:41:24,996 - Train: 21.19% [1047100/4942000] [211.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 21:41:57,782 - Train: 21.19% [1047200/4942000] [211.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 21:42:30,604 - Train: 21.19% [1047300/4942000] [211.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 21:43:03,414 - Train: 21.19% [1047400/4942000] [211.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 21:43:36,235 - Train: 21.20% [1047500/4942000] [212.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 21:44:09,041 - Train: 21.20% [1047600/4942000] [212.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 21:44:41,864 - Train: 21.20% [1047700/4942000] [212.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 21:44:43,171 - ==> Total time: 6 days, 3:47:22 Eta: 22 days, 21:19:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 21:45:16,783 - Train: 21.20% [1047800/4942000] [212.0/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 21:45:49,453 - Train: 21.20% [1047900/4942000] [212.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 21:46:22,121 - Train: 21.21% [1048000/4942000] [212.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 21:46:54,879 - Train: 21.21% [1048100/4942000] [212.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 21:47:27,505 - Train: 21.21% [1048200/4942000] [212.1/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 21:48:00,151 - Train: 21.21% [1048300/4942000] [212.1/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 21:48:32,950 - Train: 21.21% [1048400/4942000] [212.1/1000.0] [batch_t 0.325 (0.328)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-08 21:49:05,832 - Train: 21.22% [1048500/4942000] [212.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 21:49:38,635 - Train: 21.22% [1048600/4942000] [212.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 21:50:11,445 - Train: 21.22% [1048700/4942000] [212.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 21:50:44,187 - Train: 21.22% [1048800/4942000] [212.2/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 21:51:17,094 - Train: 21.22% [1048900/4942000] [212.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 21:51:50,122 - Train: 21.23% [1049000/4942000] [212.3/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 21:52:23,075 - Train: 21.23% [1049100/4942000] [212.3/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 21:52:56,129 - Train: 21.23% [1049200/4942000] [212.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 21:53:29,227 - Train: 21.23% [1049300/4942000] [212.3/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 21:54:02,265 - Train: 21.23% [1049400/4942000] [212.3/1000.0] [batch_t 0.323 (0.330)] [data_t 0.003] [optim_t 0.321] [lr 0.005000] 2024-04-08 21:54:35,520 - Train: 21.24% [1049500/4942000] [212.4/1000.0] [batch_t 0.329 (0.332)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 21:55:08,568 - Train: 21.24% [1049600/4942000] [212.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 21:55:41,560 - Train: 21.24% [1049700/4942000] [212.4/1000.0] [batch_t 0.340 (0.330)] [data_t 0.003] [optim_t 0.337] [lr 0.005000] 2024-04-08 21:56:14,465 - Train: 21.24% [1049800/4942000] [212.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 21:56:47,418 - Train: 21.24% [1049900/4942000] [212.4/1000.0] [batch_t 0.325 (0.329)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 21:57:20,414 - Train: 21.25% [1050000/4942000] [212.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 21:57:53,332 - Train: 21.25% [1050100/4942000] [212.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 21:58:26,187 - Train: 21.25% [1050200/4942000] [212.5/1000.0] [batch_t 0.331 (0.328)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 21:58:59,040 - Train: 21.25% [1050300/4942000] [212.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 21:59:31,897 - Train: 21.25% [1050400/4942000] [212.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 22:00:04,690 - Train: 21.26% [1050500/4942000] [212.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 22:00:37,573 - Train: 21.26% [1050600/4942000] [212.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 22:01:10,483 - Train: 21.26% [1050700/4942000] [212.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 22:01:43,315 - Train: 21.26% [1050800/4942000] [212.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 22:02:16,271 - Train: 21.26% [1050900/4942000] [212.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 22:02:49,009 - Train: 21.27% [1051000/4942000] [212.7/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-08 22:03:21,844 - Train: 21.27% [1051100/4942000] [212.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 22:03:54,671 - Train: 21.27% [1051200/4942000] [212.7/1000.0] [batch_t 0.331 (0.328)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 22:04:27,435 - Train: 21.27% [1051300/4942000] [212.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 22:05:00,305 - Train: 21.27% [1051400/4942000] [212.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 22:05:33,066 - Train: 21.28% [1051500/4942000] [212.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-08 22:06:05,971 - Train: 21.28% [1051600/4942000] [212.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 22:06:38,742 - Train: 21.28% [1051700/4942000] [212.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 22:07:11,554 - Train: 21.28% [1051800/4942000] [212.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 22:07:44,400 - Train: 21.28% [1051900/4942000] [212.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 22:08:17,267 - Train: 21.29% [1052000/4942000] [212.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 22:08:50,044 - Train: 21.29% [1052100/4942000] [212.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 22:09:22,846 - Train: 21.29% [1052200/4942000] [212.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 22:09:55,591 - Train: 21.29% [1052300/4942000] [212.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 22:10:28,483 - Train: 21.30% [1052400/4942000] [213.0/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 22:11:01,213 - Train: 21.30% [1052500/4942000] [213.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 22:11:34,007 - Train: 21.30% [1052600/4942000] [213.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 22:11:49,087 - ==> Total time: 6 days, 4:14:28 Eta: 22 days, 19:43:36 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 22:12:08,781 - Train: 21.30% [1052700/4942000] [213.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 22:12:41,497 - Train: 21.30% [1052800/4942000] [213.0/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 22:13:14,249 - Train: 21.31% [1052900/4942000] [213.1/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 22:13:46,969 - Train: 21.31% [1053000/4942000] [213.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 22:14:19,663 - Train: 21.31% [1053100/4942000] [213.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 22:14:52,326 - Train: 21.31% [1053200/4942000] [213.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 22:15:24,996 - Train: 21.31% [1053300/4942000] [213.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 22:15:57,692 - Train: 21.32% [1053400/4942000] [213.2/1000.0] [batch_t 0.334 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 22:16:30,475 - Train: 21.32% [1053500/4942000] [213.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 22:17:03,129 - Train: 21.32% [1053600/4942000] [213.2/1000.0] [batch_t 0.328 (0.326)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 22:17:36,035 - Train: 21.32% [1053700/4942000] [213.2/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 22:18:08,711 - Train: 21.32% [1053800/4942000] [213.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 22:18:41,421 - Train: 21.33% [1053900/4942000] [213.3/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 22:19:14,070 - Train: 21.33% [1054000/4942000] [213.3/1000.0] [batch_t 0.332 (0.326)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 22:19:46,745 - Train: 21.33% [1054100/4942000] [213.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 22:20:19,447 - Train: 21.33% [1054200/4942000] [213.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 22:20:52,241 - Train: 21.33% [1054300/4942000] [213.3/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 22:21:24,965 - Train: 21.34% [1054400/4942000] [213.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 22:21:57,643 - Train: 21.34% [1054500/4942000] [213.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 22:22:30,368 - Train: 21.34% [1054600/4942000] [213.4/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 22:23:03,010 - Train: 21.34% [1054700/4942000] [213.4/1000.0] [batch_t 0.331 (0.326)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 22:23:35,705 - Train: 21.34% [1054800/4942000] [213.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 22:24:08,378 - Train: 21.35% [1054900/4942000] [213.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 22:24:41,074 - Train: 21.35% [1055000/4942000] [213.5/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 22:25:13,840 - Train: 21.35% [1055100/4942000] [213.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 22:25:46,568 - Train: 21.35% [1055200/4942000] [213.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 22:26:19,278 - Train: 21.35% [1055300/4942000] [213.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 22:26:51,995 - Train: 21.36% [1055400/4942000] [213.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 22:27:24,682 - Train: 21.36% [1055500/4942000] [213.6/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 22:27:57,412 - Train: 21.36% [1055600/4942000] [213.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 22:28:30,158 - Train: 21.36% [1055700/4942000] [213.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 22:29:02,871 - Train: 21.36% [1055800/4942000] [213.6/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 22:29:35,736 - Train: 21.37% [1055900/4942000] [213.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 22:30:08,490 - Train: 21.37% [1056000/4942000] [213.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 22:30:41,175 - Train: 21.37% [1056100/4942000] [213.7/1000.0] [batch_t 0.330 (0.327)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 22:31:13,935 - Train: 21.37% [1056200/4942000] [213.7/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 22:31:46,758 - Train: 21.37% [1056300/4942000] [213.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 22:32:19,508 - Train: 21.38% [1056400/4942000] [213.8/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 22:32:52,278 - Train: 21.38% [1056500/4942000] [213.8/1000.0] [batch_t 0.337 (0.328)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-08 22:33:25,064 - Train: 21.38% [1056600/4942000] [213.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 22:33:57,939 - Train: 21.38% [1056700/4942000] [213.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 22:34:30,752 - Train: 21.38% [1056800/4942000] [213.8/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 22:35:03,522 - Train: 21.39% [1056900/4942000] [213.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 22:35:36,241 - Train: 21.39% [1057000/4942000] [213.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 22:36:09,038 - Train: 21.39% [1057100/4942000] [213.9/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 22:36:41,892 - Train: 21.39% [1057200/4942000] [213.9/1000.0] [batch_t 0.325 (0.328)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-08 22:37:14,710 - Train: 21.39% [1057300/4942000] [213.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 22:37:47,580 - Train: 21.40% [1057400/4942000] [214.0/1000.0] [batch_t 0.337 (0.329)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-08 22:38:20,447 - Train: 21.40% [1057500/4942000] [214.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 22:38:49,257 - ==> Total time: 6 days, 4:41:28 Eta: 22 days, 18:07:39 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 22:38:55,326 - Train: 21.40% [1057600/4942000] [214.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 22:39:28,089 - Train: 21.40% [1057700/4942000] [214.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 22:40:00,807 - Train: 21.40% [1057800/4942000] [214.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 22:40:33,641 - Train: 21.41% [1057900/4942000] [214.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 22:41:06,391 - Train: 21.41% [1058000/4942000] [214.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 22:41:39,439 - Train: 21.41% [1058100/4942000] [214.1/1000.0] [batch_t 0.324 (0.330)] [data_t 0.004] [optim_t 0.320] [lr 0.005000] 2024-04-08 22:42:12,186 - Train: 21.41% [1058200/4942000] [214.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 22:42:44,881 - Train: 21.41% [1058300/4942000] [214.1/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 22:43:17,567 - Train: 21.42% [1058400/4942000] [214.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 22:43:50,199 - Train: 21.42% [1058500/4942000] [214.2/1000.0] [batch_t 0.331 (0.326)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 22:44:22,883 - Train: 21.42% [1058600/4942000] [214.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 22:44:55,589 - Train: 21.42% [1058700/4942000] [214.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 22:45:28,331 - Train: 21.42% [1058800/4942000] [214.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 22:46:00,987 - Train: 21.43% [1058900/4942000] [214.3/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 22:46:33,764 - Train: 21.43% [1059000/4942000] [214.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 22:47:06,386 - Train: 21.43% [1059100/4942000] [214.3/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 22:47:39,057 - Train: 21.43% [1059200/4942000] [214.3/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 22:48:11,924 - Train: 21.43% [1059300/4942000] [214.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 22:48:44,602 - Train: 21.44% [1059400/4942000] [214.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 22:49:17,277 - Train: 21.44% [1059500/4942000] [214.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 22:49:49,978 - Train: 21.44% [1059600/4942000] [214.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 22:50:22,663 - Train: 21.44% [1059700/4942000] [214.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 22:50:55,315 - Train: 21.44% [1059800/4942000] [214.4/1000.0] [batch_t 0.321 (0.326)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-08 22:51:27,986 - Train: 21.45% [1059900/4942000] [214.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 22:52:00,649 - Train: 21.45% [1060000/4942000] [214.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 22:52:33,331 - Train: 21.45% [1060100/4942000] [214.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 22:53:06,000 - Train: 21.45% [1060200/4942000] [214.5/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 22:53:38,687 - Train: 21.45% [1060300/4942000] [214.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 22:54:11,356 - Train: 21.46% [1060400/4942000] [214.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 22:54:44,012 - Train: 21.46% [1060500/4942000] [214.6/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 22:55:16,639 - Train: 21.46% [1060600/4942000] [214.6/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 22:55:49,350 - Train: 21.46% [1060700/4942000] [214.6/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 22:56:22,151 - Train: 21.46% [1060800/4942000] [214.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 22:56:54,816 - Train: 21.47% [1060900/4942000] [214.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 22:57:27,527 - Train: 21.47% [1061000/4942000] [214.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 22:58:00,173 - Train: 21.47% [1061100/4942000] [214.7/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 22:58:32,853 - Train: 21.47% [1061200/4942000] [214.7/1000.0] [batch_t 0.334 (0.327)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 22:59:05,521 - Train: 21.48% [1061300/4942000] [214.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 22:59:38,171 - Train: 21.48% [1061400/4942000] [214.8/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 23:00:10,851 - Train: 21.48% [1061500/4942000] [214.8/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 23:00:43,517 - Train: 21.48% [1061600/4942000] [214.8/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 23:01:16,195 - Train: 21.48% [1061700/4942000] [214.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 23:01:52,037 - Train: 21.49% [1061800/4942000] [214.9/1000.0] [batch_t 0.326 (0.358)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 23:02:29,698 - Train: 21.49% [1061900/4942000] [214.9/1000.0] [batch_t 0.328 (0.377)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 23:03:03,793 - Train: 21.49% [1062000/4942000] [214.9/1000.0] [batch_t 0.330 (0.341)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 23:03:36,506 - Train: 21.49% [1062100/4942000] [214.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 23:04:09,355 - Train: 21.49% [1062200/4942000] [214.9/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 23:04:42,089 - Train: 21.50% [1062300/4942000] [215.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 23:05:14,758 - Train: 21.50% [1062400/4942000] [215.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 23:05:47,408 - Train: 21.50% [1062500/4942000] [215.0/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 23:05:57,214 - ==> Total time: 6 days, 5:08:36 Eta: 22 days, 16:32:49 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 23:06:22,158 - Train: 21.50% [1062600/4942000] [215.0/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 23:06:54,838 - Train: 21.50% [1062700/4942000] [215.0/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 23:07:27,509 - Train: 21.51% [1062800/4942000] [215.1/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 23:08:00,202 - Train: 21.51% [1062900/4942000] [215.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 23:08:32,910 - Train: 21.51% [1063000/4942000] [215.1/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 23:09:05,601 - Train: 21.51% [1063100/4942000] [215.1/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 23:09:38,289 - Train: 21.51% [1063200/4942000] [215.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 23:10:11,183 - Train: 21.52% [1063300/4942000] [215.2/1000.0] [batch_t 0.321 (0.329)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-08 23:10:43,829 - Train: 21.52% [1063400/4942000] [215.2/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 23:11:18,174 - Train: 21.52% [1063500/4942000] [215.2/1000.0] [batch_t 0.329 (0.343)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 23:11:50,816 - Train: 21.52% [1063600/4942000] [215.2/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 23:12:23,549 - Train: 21.52% [1063700/4942000] [215.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 23:12:56,267 - Train: 21.53% [1063800/4942000] [215.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 23:13:29,832 - Train: 21.53% [1063900/4942000] [215.3/1000.0] [batch_t 0.326 (0.336)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 23:14:02,485 - Train: 21.53% [1064000/4942000] [215.3/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 23:14:35,895 - Train: 21.53% [1064100/4942000] [215.3/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 23:15:08,555 - Train: 21.53% [1064200/4942000] [215.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 23:15:41,220 - Train: 21.54% [1064300/4942000] [215.4/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 23:16:13,924 - Train: 21.54% [1064400/4942000] [215.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 23:16:46,626 - Train: 21.54% [1064500/4942000] [215.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 23:17:19,315 - Train: 21.54% [1064600/4942000] [215.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 23:17:51,995 - Train: 21.54% [1064700/4942000] [215.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 23:18:24,666 - Train: 21.55% [1064800/4942000] [215.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 23:18:57,327 - Train: 21.55% [1064900/4942000] [215.5/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 23:19:30,944 - Train: 21.55% [1065000/4942000] [215.5/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 23:20:03,591 - Train: 21.55% [1065100/4942000] [215.5/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 23:20:37,497 - Train: 21.55% [1065200/4942000] [215.5/1000.0] [batch_t 0.331 (0.339)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 23:21:10,196 - Train: 21.56% [1065300/4942000] [215.6/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 23:21:42,827 - Train: 21.56% [1065400/4942000] [215.6/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 23:22:15,528 - Train: 21.56% [1065500/4942000] [215.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 23:22:48,138 - Train: 21.56% [1065600/4942000] [215.6/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 23:23:21,841 - Train: 21.56% [1065700/4942000] [215.6/1000.0] [batch_t 0.325 (0.337)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 23:23:54,528 - Train: 21.57% [1065800/4942000] [215.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 23:24:27,210 - Train: 21.57% [1065900/4942000] [215.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 23:24:59,849 - Train: 21.57% [1066000/4942000] [215.7/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 23:25:32,474 - Train: 21.57% [1066100/4942000] [215.7/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 23:26:05,182 - Train: 21.57% [1066200/4942000] [215.7/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 23:26:37,861 - Train: 21.58% [1066300/4942000] [215.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 23:27:10,668 - Train: 21.58% [1066400/4942000] [215.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 23:27:43,359 - Train: 21.58% [1066500/4942000] [215.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 23:28:16,060 - Train: 21.58% [1066600/4942000] [215.8/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 23:28:48,731 - Train: 21.58% [1066700/4942000] [215.8/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 23:29:21,435 - Train: 21.59% [1066800/4942000] [215.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 23:29:54,108 - Train: 21.59% [1066900/4942000] [215.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 23:30:26,818 - Train: 21.59% [1067000/4942000] [215.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 23:30:59,473 - Train: 21.59% [1067100/4942000] [215.9/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 23:31:32,134 - Train: 21.59% [1067200/4942000] [215.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 23:32:04,818 - Train: 21.60% [1067300/4942000] [216.0/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 23:32:37,726 - Train: 21.60% [1067400/4942000] [216.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 23:33:01,299 - ==> Total time: 6 days, 5:35:40 Eta: 22 days, 14:58:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-08 23:33:14,093 - Train: 21.60% [1067500/4942000] [216.0/1000.0] [batch_t 0.324 (0.331)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 23:33:46,787 - Train: 21.60% [1067600/4942000] [216.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 23:34:20,262 - Train: 21.60% [1067700/4942000] [216.0/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 23:34:53,045 - Train: 21.61% [1067800/4942000] [216.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 23:35:25,812 - Train: 21.61% [1067900/4942000] [216.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 23:35:58,544 - Train: 21.61% [1068000/4942000] [216.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 23:36:31,292 - Train: 21.61% [1068100/4942000] [216.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 23:37:03,980 - Train: 21.61% [1068200/4942000] [216.1/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 23:37:36,686 - Train: 21.62% [1068300/4942000] [216.2/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 23:38:09,366 - Train: 21.62% [1068400/4942000] [216.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 23:38:42,117 - Train: 21.62% [1068500/4942000] [216.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 23:39:15,131 - Train: 21.62% [1068600/4942000] [216.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 23:39:47,831 - Train: 21.62% [1068700/4942000] [216.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 23:40:20,542 - Train: 21.63% [1068800/4942000] [216.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 23:40:53,246 - Train: 21.63% [1068900/4942000] [216.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 23:41:25,929 - Train: 21.63% [1069000/4942000] [216.3/1000.0] [batch_t 0.335 (0.327)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 23:41:58,638 - Train: 21.63% [1069100/4942000] [216.3/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 23:42:31,600 - Train: 21.63% [1069200/4942000] [216.3/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 23:43:04,343 - Train: 21.64% [1069300/4942000] [216.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 23:43:37,196 - Train: 21.64% [1069400/4942000] [216.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 23:44:09,935 - Train: 21.64% [1069500/4942000] [216.4/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 23:44:42,726 - Train: 21.64% [1069600/4942000] [216.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 23:45:15,485 - Train: 21.65% [1069700/4942000] [216.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 23:45:48,244 - Train: 21.65% [1069800/4942000] [216.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 23:46:20,932 - Train: 21.65% [1069900/4942000] [216.5/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 23:46:53,628 - Train: 21.65% [1070000/4942000] [216.5/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 23:47:26,339 - Train: 21.65% [1070100/4942000] [216.5/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 23:47:59,086 - Train: 21.66% [1070200/4942000] [216.6/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 23:48:31,780 - Train: 21.66% [1070300/4942000] [216.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 23:49:04,537 - Train: 21.66% [1070400/4942000] [216.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 23:49:37,242 - Train: 21.66% [1070500/4942000] [216.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 23:50:10,003 - Train: 21.66% [1070600/4942000] [216.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 23:50:42,842 - Train: 21.67% [1070700/4942000] [216.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 23:51:15,561 - Train: 21.67% [1070800/4942000] [216.7/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 23:51:48,276 - Train: 21.67% [1070900/4942000] [216.7/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 23:52:20,913 - Train: 21.67% [1071000/4942000] [216.7/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 23:52:53,563 - Train: 21.67% [1071100/4942000] [216.7/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 23:53:26,285 - Train: 21.68% [1071200/4942000] [216.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 23:53:59,001 - Train: 21.68% [1071300/4942000] [216.8/1000.0] [batch_t 0.335 (0.327)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 23:54:31,659 - Train: 21.68% [1071400/4942000] [216.8/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 23:55:04,429 - Train: 21.68% [1071500/4942000] [216.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 23:55:37,117 - Train: 21.68% [1071600/4942000] [216.8/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 23:56:09,859 - Train: 21.69% [1071700/4942000] [216.9/1000.0] [batch_t 0.334 (0.327)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 23:56:42,568 - Train: 21.69% [1071800/4942000] [216.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 23:57:15,279 - Train: 21.69% [1071900/4942000] [216.9/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 23:57:48,001 - Train: 21.69% [1072000/4942000] [216.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 23:58:21,131 - Train: 21.69% [1072100/4942000] [216.9/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 23:58:53,903 - Train: 21.70% [1072200/4942000] [217.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 23:59:26,576 - Train: 21.70% [1072300/4942000] [217.0/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 23:59:59,288 - Train: 21.70% [1072400/4942000] [217.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 00:00:03,872 - ==> Total time: 6 days, 6:02:43 Eta: 22 days, 13:24:27 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 00:00:34,895 - Train: 21.70% [1072500/4942000] [217.0/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 00:01:07,754 - Train: 21.70% [1072600/4942000] [217.0/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 00:01:40,418 - Train: 21.71% [1072700/4942000] [217.1/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 00:02:13,043 - Train: 21.71% [1072800/4942000] [217.1/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 00:02:45,715 - Train: 21.71% [1072900/4942000] [217.1/1000.0] [batch_t 0.334 (0.327)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 00:03:18,407 - Train: 21.71% [1073000/4942000] [217.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 00:03:51,116 - Train: 21.71% [1073100/4942000] [217.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 00:04:23,771 - Train: 21.72% [1073200/4942000] [217.2/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 00:04:56,475 - Train: 21.72% [1073300/4942000] [217.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 00:05:29,291 - Train: 21.72% [1073400/4942000] [217.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 00:06:01,914 - Train: 21.72% [1073500/4942000] [217.2/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 00:06:34,656 - Train: 21.72% [1073600/4942000] [217.2/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 00:07:07,294 - Train: 21.73% [1073700/4942000] [217.3/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 00:07:39,970 - Train: 21.73% [1073800/4942000] [217.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 00:08:12,650 - Train: 21.73% [1073900/4942000] [217.3/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 00:08:45,340 - Train: 21.73% [1074000/4942000] [217.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 00:09:17,976 - Train: 21.73% [1074100/4942000] [217.3/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 00:09:50,584 - Train: 21.74% [1074200/4942000] [217.4/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 00:10:23,328 - Train: 21.74% [1074300/4942000] [217.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 00:10:56,002 - Train: 21.74% [1074400/4942000] [217.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 00:11:28,720 - Train: 21.74% [1074500/4942000] [217.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 00:12:01,483 - Train: 21.74% [1074600/4942000] [217.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-09 00:12:34,287 - Train: 21.75% [1074700/4942000] [217.5/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 00:13:06,993 - Train: 21.75% [1074800/4942000] [217.5/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 00:13:39,825 - Train: 21.75% [1074900/4942000] [217.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 00:14:12,528 - Train: 21.75% [1075000/4942000] [217.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 00:14:45,356 - Train: 21.75% [1075100/4942000] [217.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 00:15:18,092 - Train: 21.76% [1075200/4942000] [217.6/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 00:15:50,833 - Train: 21.76% [1075300/4942000] [217.6/1000.0] [batch_t 0.335 (0.327)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-09 00:16:24,512 - Train: 21.76% [1075400/4942000] [217.6/1000.0] [batch_t 0.332 (0.337)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-09 00:16:57,221 - Train: 21.76% [1075500/4942000] [217.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 00:17:29,934 - Train: 21.76% [1075600/4942000] [217.6/1000.0] [batch_t 0.332 (0.327)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-09 00:18:02,548 - Train: 21.77% [1075700/4942000] [217.7/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 00:18:35,685 - Train: 21.77% [1075800/4942000] [217.7/1000.0] [batch_t 0.325 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 00:19:08,332 - Train: 21.77% [1075900/4942000] [217.7/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 00:19:40,976 - Train: 21.77% [1076000/4942000] [217.7/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 00:20:13,824 - Train: 21.77% [1076100/4942000] [217.7/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 00:20:46,657 - Train: 21.78% [1076200/4942000] [217.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 00:21:19,509 - Train: 21.78% [1076300/4942000] [217.8/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 00:21:52,360 - Train: 21.78% [1076400/4942000] [217.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 00:22:25,019 - Train: 21.78% [1076500/4942000] [217.8/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 00:22:57,704 - Train: 21.78% [1076600/4942000] [217.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 00:23:30,346 - Train: 21.79% [1076700/4942000] [217.9/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 00:24:03,056 - Train: 21.79% [1076800/4942000] [217.9/1000.0] [batch_t 0.321 (0.327)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-09 00:24:35,755 - Train: 21.79% [1076900/4942000] [217.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 00:25:08,436 - Train: 21.79% [1077000/4942000] [217.9/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 00:25:41,150 - Train: 21.79% [1077100/4942000] [217.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 00:26:13,837 - Train: 21.80% [1077200/4942000] [218.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 00:26:46,503 - Train: 21.80% [1077300/4942000] [218.0/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 00:27:04,805 - ==> Total time: 6 days, 6:29:43 Eta: 22 days, 11:51:03 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 00:27:21,601 - Train: 21.80% [1077400/4942000] [218.0/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 00:27:54,348 - Train: 21.80% [1077500/4942000] [218.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 00:28:27,230 - Train: 21.80% [1077600/4942000] [218.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 00:28:59,943 - Train: 21.81% [1077700/4942000] [218.1/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 00:29:32,697 - Train: 21.81% [1077800/4942000] [218.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 00:30:05,524 - Train: 21.81% [1077900/4942000] [218.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 00:30:38,253 - Train: 21.81% [1078000/4942000] [218.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 00:31:10,958 - Train: 21.82% [1078100/4942000] [218.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 00:31:43,612 - Train: 21.82% [1078200/4942000] [218.2/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 00:32:16,266 - Train: 21.82% [1078300/4942000] [218.2/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 00:32:49,010 - Train: 21.82% [1078400/4942000] [218.2/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 00:33:22,706 - Train: 21.82% [1078500/4942000] [218.2/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 00:33:55,330 - Train: 21.83% [1078600/4942000] [218.3/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 00:34:27,943 - Train: 21.83% [1078700/4942000] [218.3/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 00:35:00,589 - Train: 21.83% [1078800/4942000] [218.3/1000.0] [batch_t 0.331 (0.326)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 00:35:34,180 - Train: 21.83% [1078900/4942000] [218.3/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 00:36:06,903 - Train: 21.83% [1079000/4942000] [218.3/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 00:36:40,103 - Train: 21.84% [1079100/4942000] [218.4/1000.0] [batch_t 0.319 (0.332)] [data_t 0.002] [optim_t 0.317] [lr 0.005000] 2024-04-09 00:37:12,790 - Train: 21.84% [1079200/4942000] [218.4/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 00:37:45,411 - Train: 21.84% [1079300/4942000] [218.4/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 00:38:18,120 - Train: 21.84% [1079400/4942000] [218.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 00:38:50,857 - Train: 21.84% [1079500/4942000] [218.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 00:39:23,494 - Train: 21.85% [1079600/4942000] [218.5/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 00:39:56,255 - Train: 21.85% [1079700/4942000] [218.5/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 00:40:28,975 - Train: 21.85% [1079800/4942000] [218.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 00:41:01,696 - Train: 21.85% [1079900/4942000] [218.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 00:41:34,458 - Train: 21.85% [1080000/4942000] [218.5/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 00:42:07,275 - Train: 21.86% [1080100/4942000] [218.6/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 00:42:39,977 - Train: 21.86% [1080200/4942000] [218.6/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 00:43:12,671 - Train: 21.86% [1080300/4942000] [218.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 00:43:45,365 - Train: 21.86% [1080400/4942000] [218.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 00:44:18,183 - Train: 21.86% [1080500/4942000] [218.6/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 00:44:50,969 - Train: 21.87% [1080600/4942000] [218.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 00:45:23,674 - Train: 21.87% [1080700/4942000] [218.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 00:45:56,358 - Train: 21.87% [1080800/4942000] [218.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 00:46:29,050 - Train: 21.87% [1080900/4942000] [218.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 00:47:01,684 - Train: 21.87% [1081000/4942000] [218.7/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 00:47:34,327 - Train: 21.88% [1081100/4942000] [218.8/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 00:48:07,955 - Train: 21.88% [1081200/4942000] [218.8/1000.0] [batch_t 0.324 (0.336)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 00:48:40,629 - Train: 21.88% [1081300/4942000] [218.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 00:49:13,768 - Train: 21.88% [1081400/4942000] [218.8/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 00:49:46,412 - Train: 21.88% [1081500/4942000] [218.8/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 00:50:19,110 - Train: 21.89% [1081600/4942000] [218.9/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 00:50:51,808 - Train: 21.89% [1081700/4942000] [218.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 00:51:24,405 - Train: 21.89% [1081800/4942000] [218.9/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 00:51:57,173 - Train: 21.89% [1081900/4942000] [218.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 00:52:29,883 - Train: 21.89% [1082000/4942000] [218.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 00:53:02,460 - Train: 21.90% [1082100/4942000] [219.0/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 00:53:35,727 - Train: 21.90% [1082200/4942000] [219.0/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 00:54:08,599 - ==> Total time: 6 days, 6:56:47 Eta: 22 days, 10:18:26 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 00:54:11,321 - Train: 21.90% [1082300/4942000] [219.0/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 00:54:43,964 - Train: 21.90% [1082400/4942000] [219.0/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 00:55:16,611 - Train: 21.90% [1082500/4942000] [219.0/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 00:55:49,257 - Train: 21.91% [1082600/4942000] [219.1/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 00:56:21,851 - Train: 21.91% [1082700/4942000] [219.1/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 00:56:54,436 - Train: 21.91% [1082800/4942000] [219.1/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 00:57:27,056 - Train: 21.91% [1082900/4942000] [219.1/1000.0] [batch_t 0.322 (0.326)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 00:57:59,713 - Train: 21.91% [1083000/4942000] [219.1/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 00:58:32,310 - Train: 21.92% [1083100/4942000] [219.2/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 00:59:04,935 - Train: 21.92% [1083200/4942000] [219.2/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 00:59:37,790 - Train: 21.92% [1083300/4942000] [219.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 01:00:10,457 - Train: 21.92% [1083400/4942000] [219.2/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 01:00:43,260 - Train: 21.92% [1083500/4942000] [219.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 01:01:16,018 - Train: 21.93% [1083600/4942000] [219.3/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 01:01:48,843 - Train: 21.93% [1083700/4942000] [219.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 01:02:21,630 - Train: 21.93% [1083800/4942000] [219.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 01:02:54,273 - Train: 21.93% [1083900/4942000] [219.3/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 01:03:26,974 - Train: 21.93% [1084000/4942000] [219.3/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 01:03:59,674 - Train: 21.94% [1084100/4942000] [219.4/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 01:04:32,317 - Train: 21.94% [1084200/4942000] [219.4/1000.0] [batch_t 0.332 (0.326)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 01:05:04,921 - Train: 21.94% [1084300/4942000] [219.4/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 01:05:37,515 - Train: 21.94% [1084400/4942000] [219.4/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 01:06:10,095 - Train: 21.94% [1084500/4942000] [219.4/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 01:06:42,697 - Train: 21.95% [1084600/4942000] [219.5/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 01:07:15,460 - Train: 21.95% [1084700/4942000] [219.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 01:07:48,079 - Train: 21.95% [1084800/4942000] [219.5/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 01:08:20,665 - Train: 21.95% [1084900/4942000] [219.5/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 01:08:53,314 - Train: 21.95% [1085000/4942000] [219.5/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 01:09:25,915 - Train: 21.96% [1085100/4942000] [219.6/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 01:09:58,514 - Train: 21.96% [1085200/4942000] [219.6/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 01:10:31,178 - Train: 21.96% [1085300/4942000] [219.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 01:11:03,825 - Train: 21.96% [1085400/4942000] [219.6/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 01:11:36,431 - Train: 21.96% [1085500/4942000] [219.6/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 01:12:09,064 - Train: 21.97% [1085600/4942000] [219.7/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 01:12:41,701 - Train: 21.97% [1085700/4942000] [219.7/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 01:13:14,313 - Train: 21.97% [1085800/4942000] [219.7/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 01:13:46,933 - Train: 21.97% [1085900/4942000] [219.7/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 01:14:19,474 - Train: 21.97% [1086000/4942000] [219.7/1000.0] [batch_t 0.325 (0.325)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 01:14:52,253 - Train: 21.98% [1086100/4942000] [219.8/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 01:15:24,893 - Train: 21.98% [1086200/4942000] [219.8/1000.0] [batch_t 0.321 (0.326)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 01:15:57,462 - Train: 21.98% [1086300/4942000] [219.8/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 01:16:30,083 - Train: 21.98% [1086400/4942000] [219.8/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 01:17:02,795 - Train: 21.99% [1086500/4942000] [219.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 01:17:35,381 - Train: 21.99% [1086600/4942000] [219.9/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 01:18:16,860 - Train: 21.99% [1086700/4942000] [219.9/1000.0] [batch_t 0.336 (0.415)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-09 01:19:02,283 - Train: 21.99% [1086800/4942000] [219.9/1000.0] [batch_t 0.326 (0.454)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 01:19:34,925 - Train: 21.99% [1086900/4942000] [219.9/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 01:20:07,530 - Train: 22.00% [1087000/4942000] [220.0/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 01:20:40,129 - Train: 22.00% [1087100/4942000] [220.0/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 01:21:12,759 - Train: 22.00% [1087200/4942000] [220.0/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 01:21:25,784 - ==> Total time: 6 days, 7:24:04 Eta: 22 days, 8:47:12 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 01:21:48,232 - Train: 22.00% [1087300/4942000] [220.0/1000.0] [batch_t 0.323 (0.333)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 01:22:20,896 - Train: 22.00% [1087400/4942000] [220.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 01:22:53,646 - Train: 22.01% [1087500/4942000] [220.1/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 01:23:26,239 - Train: 22.01% [1087600/4942000] [220.1/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 01:23:58,889 - Train: 22.01% [1087700/4942000] [220.1/1000.0] [batch_t 0.322 (0.326)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 01:24:31,472 - Train: 22.01% [1087800/4942000] [220.1/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 01:25:04,053 - Train: 22.01% [1087900/4942000] [220.1/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 01:25:36,654 - Train: 22.02% [1088000/4942000] [220.2/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 01:26:09,973 - Train: 22.02% [1088100/4942000] [220.2/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 01:26:42,653 - Train: 22.02% [1088200/4942000] [220.2/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 01:27:15,284 - Train: 22.02% [1088300/4942000] [220.2/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 01:27:47,991 - Train: 22.02% [1088400/4942000] [220.2/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 01:28:20,639 - Train: 22.03% [1088500/4942000] [220.3/1000.0] [batch_t 0.322 (0.326)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 01:28:53,337 - Train: 22.03% [1088600/4942000] [220.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 01:29:25,976 - Train: 22.03% [1088700/4942000] [220.3/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 01:29:58,603 - Train: 22.03% [1088800/4942000] [220.3/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 01:30:31,324 - Train: 22.03% [1088900/4942000] [220.3/1000.0] [batch_t 0.320 (0.327)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-09 01:31:03,995 - Train: 22.04% [1089000/4942000] [220.4/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 01:31:36,618 - Train: 22.04% [1089100/4942000] [220.4/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 01:32:09,309 - Train: 22.04% [1089200/4942000] [220.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 01:32:41,905 - Train: 22.04% [1089300/4942000] [220.4/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 01:33:14,558 - Train: 22.04% [1089400/4942000] [220.4/1000.0] [batch_t 0.322 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 01:33:47,150 - Train: 22.05% [1089500/4942000] [220.5/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 01:34:19,760 - Train: 22.05% [1089600/4942000] [220.5/1000.0] [batch_t 0.330 (0.326)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-09 01:34:52,380 - Train: 22.05% [1089700/4942000] [220.5/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 01:35:24,952 - Train: 22.05% [1089800/4942000] [220.5/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 01:35:57,568 - Train: 22.05% [1089900/4942000] [220.5/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 01:36:30,189 - Train: 22.06% [1090000/4942000] [220.6/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 01:37:02,781 - Train: 22.06% [1090100/4942000] [220.6/1000.0] [batch_t 0.331 (0.326)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 01:37:35,510 - Train: 22.06% [1090200/4942000] [220.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 01:38:08,285 - Train: 22.06% [1090300/4942000] [220.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 01:38:40,960 - Train: 22.06% [1090400/4942000] [220.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 01:39:13,685 - Train: 22.07% [1090500/4942000] [220.7/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 01:39:46,315 - Train: 22.07% [1090600/4942000] [220.7/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 01:40:18,994 - Train: 22.07% [1090700/4942000] [220.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 01:40:51,591 - Train: 22.07% [1090800/4942000] [220.7/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 01:41:24,180 - Train: 22.07% [1090900/4942000] [220.7/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 01:41:56,860 - Train: 22.08% [1091000/4942000] [220.8/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 01:42:29,607 - Train: 22.08% [1091100/4942000] [220.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 01:43:02,336 - Train: 22.08% [1091200/4942000] [220.8/1000.0] [batch_t 0.318 (0.327)] [data_t 0.002] [optim_t 0.317] [lr 0.005000] 2024-04-09 01:43:34,998 - Train: 22.08% [1091300/4942000] [220.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-09 01:44:07,742 - Train: 22.08% [1091400/4942000] [220.8/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 01:44:40,351 - Train: 22.09% [1091500/4942000] [220.9/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 01:45:13,013 - Train: 22.09% [1091600/4942000] [220.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 01:45:45,712 - Train: 22.09% [1091700/4942000] [220.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 01:46:18,430 - Train: 22.09% [1091800/4942000] [220.9/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 01:46:51,098 - Train: 22.09% [1091900/4942000] [220.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 01:47:23,725 - Train: 22.10% [1092000/4942000] [221.0/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 01:47:56,335 - Train: 22.10% [1092100/4942000] [221.0/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 01:48:23,096 - ==> Total time: 6 days, 7:51:02 Eta: 22 days, 7:15:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 01:48:31,117 - Train: 22.10% [1092200/4942000] [221.0/1000.0] [batch_t 0.326 (0.336)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 01:49:03,773 - Train: 22.10% [1092300/4942000] [221.0/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 01:49:36,387 - Train: 22.10% [1092400/4942000] [221.0/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 01:50:09,064 - Train: 22.11% [1092500/4942000] [221.1/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 01:50:41,783 - Train: 22.11% [1092600/4942000] [221.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 01:51:14,394 - Train: 22.11% [1092700/4942000] [221.1/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 01:51:46,990 - Train: 22.11% [1092800/4942000] [221.1/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 01:52:19,649 - Train: 22.11% [1092900/4942000] [221.1/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 01:52:52,205 - Train: 22.12% [1093000/4942000] [221.2/1000.0] [batch_t 0.321 (0.325)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-09 01:53:24,961 - Train: 22.12% [1093100/4942000] [221.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 01:53:57,591 - Train: 22.12% [1093200/4942000] [221.2/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 01:54:30,241 - Train: 22.12% [1093300/4942000] [221.2/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 01:55:02,904 - Train: 22.12% [1093400/4942000] [221.2/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 01:55:35,571 - Train: 22.13% [1093500/4942000] [221.3/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 01:56:08,235 - Train: 22.13% [1093600/4942000] [221.3/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 01:56:40,933 - Train: 22.13% [1093700/4942000] [221.3/1000.0] [batch_t 0.335 (0.327)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-09 01:57:13,584 - Train: 22.13% [1093800/4942000] [221.3/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 01:57:46,251 - Train: 22.13% [1093900/4942000] [221.3/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 01:58:18,846 - Train: 22.14% [1094000/4942000] [221.4/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 01:58:51,446 - Train: 22.14% [1094100/4942000] [221.4/1000.0] [batch_t 0.334 (0.326)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 01:59:24,093 - Train: 22.14% [1094200/4942000] [221.4/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 01:59:56,717 - Train: 22.14% [1094300/4942000] [221.4/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 02:00:29,304 - Train: 22.14% [1094400/4942000] [221.4/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 02:01:02,031 - Train: 22.15% [1094500/4942000] [221.5/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 02:01:34,693 - Train: 22.15% [1094600/4942000] [221.5/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 02:02:07,397 - Train: 22.15% [1094700/4942000] [221.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 02:02:40,105 - Train: 22.15% [1094800/4942000] [221.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 02:03:12,740 - Train: 22.15% [1094900/4942000] [221.5/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 02:03:45,327 - Train: 22.16% [1095000/4942000] [221.6/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 02:04:17,959 - Train: 22.16% [1095100/4942000] [221.6/1000.0] [batch_t 0.331 (0.326)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 02:04:50,630 - Train: 22.16% [1095200/4942000] [221.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 02:05:23,275 - Train: 22.16% [1095300/4942000] [221.6/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 02:05:55,943 - Train: 22.17% [1095400/4942000] [221.7/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 02:06:28,579 - Train: 22.17% [1095500/4942000] [221.7/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 02:07:01,220 - Train: 22.17% [1095600/4942000] [221.7/1000.0] [batch_t 0.331 (0.326)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 02:07:33,849 - Train: 22.17% [1095700/4942000] [221.7/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 02:08:06,472 - Train: 22.17% [1095800/4942000] [221.7/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 02:08:39,159 - Train: 22.18% [1095900/4942000] [221.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 02:09:11,766 - Train: 22.18% [1096000/4942000] [221.8/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 02:09:44,404 - Train: 22.18% [1096100/4942000] [221.8/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 02:10:17,099 - Train: 22.18% [1096200/4942000] [221.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 02:10:49,684 - Train: 22.18% [1096300/4942000] [221.8/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 02:11:22,307 - Train: 22.19% [1096400/4942000] [221.9/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 02:11:54,920 - Train: 22.19% [1096500/4942000] [221.9/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 02:12:27,572 - Train: 22.19% [1096600/4942000] [221.9/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 02:13:00,204 - Train: 22.19% [1096700/4942000] [221.9/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 02:13:32,969 - Train: 22.19% [1096800/4942000] [221.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 02:14:05,656 - Train: 22.20% [1096900/4942000] [222.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 02:14:38,311 - Train: 22.20% [1097000/4942000] [222.0/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 02:15:10,933 - Train: 22.20% [1097100/4942000] [222.0/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 02:15:18,754 - ==> Total time: 6 days, 8:17:57 Eta: 22 days, 5:44:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 02:15:45,659 - Train: 22.20% [1097200/4942000] [222.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 02:16:18,315 - Train: 22.20% [1097300/4942000] [222.0/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 02:16:50,921 - Train: 22.21% [1097400/4942000] [222.1/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 02:17:23,559 - Train: 22.21% [1097500/4942000] [222.1/1000.0] [batch_t 0.322 (0.326)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 02:17:58,001 - Train: 22.21% [1097600/4942000] [222.1/1000.0] [batch_t 0.323 (0.344)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 02:18:31,907 - Train: 22.21% [1097700/4942000] [222.1/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 02:19:04,556 - Train: 22.21% [1097800/4942000] [222.1/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 02:19:37,248 - Train: 22.22% [1097900/4942000] [222.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 02:20:09,939 - Train: 22.22% [1098000/4942000] [222.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 02:20:42,685 - Train: 22.22% [1098100/4942000] [222.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 02:21:15,302 - Train: 22.22% [1098200/4942000] [222.2/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 02:21:47,906 - Train: 22.22% [1098300/4942000] [222.2/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 02:22:20,553 - Train: 22.23% [1098400/4942000] [222.3/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 02:22:53,167 - Train: 22.23% [1098500/4942000] [222.3/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 02:23:26,611 - Train: 22.23% [1098600/4942000] [222.3/1000.0] [batch_t 0.326 (0.334)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 02:23:59,387 - Train: 22.23% [1098700/4942000] [222.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 02:24:31,921 - Train: 22.23% [1098800/4942000] [222.3/1000.0] [batch_t 0.328 (0.325)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 02:25:05,411 - Train: 22.24% [1098900/4942000] [222.4/1000.0] [batch_t 1.208 (0.335)] [data_t 0.883] [optim_t 0.324] [lr 0.005000] 2024-04-09 02:25:38,058 - Train: 22.24% [1099000/4942000] [222.4/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 02:26:10,709 - Train: 22.24% [1099100/4942000] [222.4/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 02:26:43,361 - Train: 22.24% [1099200/4942000] [222.4/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 02:27:15,937 - Train: 22.24% [1099300/4942000] [222.4/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 02:27:48,572 - Train: 22.25% [1099400/4942000] [222.5/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 02:28:21,202 - Train: 22.25% [1099500/4942000] [222.5/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 02:28:53,951 - Train: 22.25% [1099600/4942000] [222.5/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 02:29:26,790 - Train: 22.25% [1099700/4942000] [222.5/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 02:29:59,459 - Train: 22.25% [1099800/4942000] [222.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 02:30:32,068 - Train: 22.26% [1099900/4942000] [222.6/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 02:31:04,715 - Train: 22.26% [1100000/4942000] [222.6/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 02:31:37,631 - Train: 22.26% [1100100/4942000] [222.6/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 02:32:10,299 - Train: 22.26% [1100200/4942000] [222.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 02:32:42,962 - Train: 22.26% [1100300/4942000] [222.6/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 02:33:15,728 - Train: 22.27% [1100400/4942000] [222.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 02:33:48,470 - Train: 22.27% [1100500/4942000] [222.7/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 02:34:21,128 - Train: 22.27% [1100600/4942000] [222.7/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 02:34:53,795 - Train: 22.27% [1100700/4942000] [222.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 02:35:26,474 - Train: 22.27% [1100800/4942000] [222.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 02:35:59,114 - Train: 22.28% [1100900/4942000] [222.8/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 02:36:31,931 - Train: 22.28% [1101000/4942000] [222.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 02:37:04,635 - Train: 22.28% [1101100/4942000] [222.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 02:37:37,317 - Train: 22.28% [1101200/4942000] [222.8/1000.0] [batch_t 0.321 (0.327)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-09 02:38:09,969 - Train: 22.28% [1101300/4942000] [222.8/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 02:38:42,691 - Train: 22.29% [1101400/4942000] [222.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 02:39:15,636 - Train: 22.29% [1101500/4942000] [222.9/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 02:39:48,323 - Train: 22.29% [1101600/4942000] [222.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 02:40:21,017 - Train: 22.29% [1101700/4942000] [222.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 02:40:53,671 - Train: 22.29% [1101800/4942000] [222.9/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 02:41:26,417 - Train: 22.30% [1101900/4942000] [223.0/1000.0] [batch_t 0.335 (0.327)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-09 02:41:59,126 - Train: 22.30% [1102000/4942000] [223.0/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 02:42:20,690 - ==> Total time: 6 days, 8:44:59 Eta: 22 days, 4:13:38 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 02:42:33,929 - Train: 22.30% [1102100/4942000] [223.0/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 02:43:06,591 - Train: 22.30% [1102200/4942000] [223.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 02:43:39,215 - Train: 22.30% [1102300/4942000] [223.0/1000.0] [batch_t 0.331 (0.326)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 02:44:11,854 - Train: 22.31% [1102400/4942000] [223.1/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 02:44:44,518 - Train: 22.31% [1102500/4942000] [223.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 02:45:17,204 - Train: 22.31% [1102600/4942000] [223.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 02:45:49,913 - Train: 22.31% [1102700/4942000] [223.1/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 02:46:22,758 - Train: 22.31% [1102800/4942000] [223.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 02:46:55,456 - Train: 22.32% [1102900/4942000] [223.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 02:47:28,136 - Train: 22.32% [1103000/4942000] [223.2/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 02:48:00,833 - Train: 22.32% [1103100/4942000] [223.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 02:48:33,483 - Train: 22.32% [1103200/4942000] [223.2/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 02:49:06,136 - Train: 22.32% [1103300/4942000] [223.2/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 02:49:38,809 - Train: 22.33% [1103400/4942000] [223.3/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 02:50:11,472 - Train: 22.33% [1103500/4942000] [223.3/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 02:50:44,097 - Train: 22.33% [1103600/4942000] [223.3/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 02:51:16,818 - Train: 22.33% [1103700/4942000] [223.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 02:51:49,452 - Train: 22.34% [1103800/4942000] [223.4/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 02:52:22,142 - Train: 22.34% [1103900/4942000] [223.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 02:52:54,801 - Train: 22.34% [1104000/4942000] [223.4/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 02:53:27,512 - Train: 22.34% [1104100/4942000] [223.4/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 02:54:00,347 - Train: 22.34% [1104200/4942000] [223.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 02:54:33,334 - Train: 22.35% [1104300/4942000] [223.5/1000.0] [batch_t 0.337 (0.330)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-09 02:55:06,156 - Train: 22.35% [1104400/4942000] [223.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 02:55:38,842 - Train: 22.35% [1104500/4942000] [223.5/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 02:56:11,504 - Train: 22.35% [1104600/4942000] [223.5/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 02:56:44,163 - Train: 22.35% [1104700/4942000] [223.5/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 02:57:16,865 - Train: 22.36% [1104800/4942000] [223.6/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 02:57:49,579 - Train: 22.36% [1104900/4942000] [223.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 02:58:22,287 - Train: 22.36% [1105000/4942000] [223.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 02:58:55,144 - Train: 22.36% [1105100/4942000] [223.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 02:59:27,785 - Train: 22.36% [1105200/4942000] [223.6/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 03:00:00,364 - Train: 22.37% [1105300/4942000] [223.7/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 03:00:33,015 - Train: 22.37% [1105400/4942000] [223.7/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 03:01:05,637 - Train: 22.37% [1105500/4942000] [223.7/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 03:01:38,246 - Train: 22.37% [1105600/4942000] [223.7/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 03:02:11,128 - Train: 22.37% [1105700/4942000] [223.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 03:02:43,745 - Train: 22.38% [1105800/4942000] [223.8/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 03:03:16,442 - Train: 22.38% [1105900/4942000] [223.8/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 03:03:49,098 - Train: 22.38% [1106000/4942000] [223.8/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 03:04:21,685 - Train: 22.38% [1106100/4942000] [223.8/1000.0] [batch_t 0.331 (0.326)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 03:04:54,330 - Train: 22.38% [1106200/4942000] [223.8/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 03:05:27,001 - Train: 22.39% [1106300/4942000] [223.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 03:05:59,598 - Train: 22.39% [1106400/4942000] [223.9/1000.0] [batch_t 0.321 (0.326)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-09 03:06:32,403 - Train: 22.39% [1106500/4942000] [223.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 03:07:05,032 - Train: 22.39% [1106600/4942000] [223.9/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 03:07:37,584 - Train: 22.39% [1106700/4942000] [223.9/1000.0] [batch_t 0.324 (0.325)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 03:08:10,203 - Train: 22.40% [1106800/4942000] [224.0/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 03:08:42,776 - Train: 22.40% [1106900/4942000] [224.0/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 03:09:15,398 - Train: 22.40% [1107000/4942000] [224.0/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 03:09:18,009 - ==> Total time: 6 days, 9:11:57 Eta: 22 days, 2:43:33 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 03:09:50,333 - Train: 22.40% [1107100/4942000] [224.0/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 03:10:23,028 - Train: 22.40% [1107200/4942000] [224.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 03:10:55,683 - Train: 22.41% [1107300/4942000] [224.1/1000.0] [batch_t 0.333 (0.326)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 03:11:28,400 - Train: 22.41% [1107400/4942000] [224.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 03:12:01,045 - Train: 22.41% [1107500/4942000] [224.1/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 03:12:33,779 - Train: 22.41% [1107600/4942000] [224.1/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 03:13:06,412 - Train: 22.41% [1107700/4942000] [224.1/1000.0] [batch_t 0.332 (0.326)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 03:13:39,112 - Train: 22.42% [1107800/4942000] [224.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 03:14:11,724 - Train: 22.42% [1107900/4942000] [224.2/1000.0] [batch_t 0.322 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 03:14:44,384 - Train: 22.42% [1108000/4942000] [224.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 03:15:17,655 - Train: 22.42% [1108100/4942000] [224.2/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 03:15:50,432 - Train: 22.42% [1108200/4942000] [224.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 03:16:23,174 - Train: 22.43% [1108300/4942000] [224.3/1000.0] [batch_t 0.335 (0.327)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-09 03:16:56,063 - Train: 22.43% [1108400/4942000] [224.3/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 03:17:28,810 - Train: 22.43% [1108500/4942000] [224.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 03:18:01,560 - Train: 22.43% [1108600/4942000] [224.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 03:18:34,305 - Train: 22.43% [1108700/4942000] [224.3/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 03:19:06,972 - Train: 22.44% [1108800/4942000] [224.4/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 03:19:39,640 - Train: 22.44% [1108900/4942000] [224.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 03:20:12,302 - Train: 22.44% [1109000/4942000] [224.4/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 03:20:44,944 - Train: 22.44% [1109100/4942000] [224.4/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 03:21:17,643 - Train: 22.44% [1109200/4942000] [224.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 03:21:50,316 - Train: 22.45% [1109300/4942000] [224.5/1000.0] [batch_t 0.336 (0.327)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-09 03:22:23,043 - Train: 22.45% [1109400/4942000] [224.5/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 03:22:55,808 - Train: 22.45% [1109500/4942000] [224.5/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 03:23:28,473 - Train: 22.45% [1109600/4942000] [224.5/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 03:24:01,136 - Train: 22.45% [1109700/4942000] [224.5/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 03:24:34,064 - Train: 22.46% [1109800/4942000] [224.6/1000.0] [batch_t 0.464 (0.329)] [data_t 0.002] [optim_t 0.462] [lr 0.005000] 2024-04-09 03:25:06,710 - Train: 22.46% [1109900/4942000] [224.6/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 03:25:39,413 - Train: 22.46% [1110000/4942000] [224.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 03:26:12,087 - Train: 22.46% [1110100/4942000] [224.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 03:26:44,848 - Train: 22.46% [1110200/4942000] [224.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 03:27:17,555 - Train: 22.47% [1110300/4942000] [224.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 03:27:50,294 - Train: 22.47% [1110400/4942000] [224.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 03:28:22,919 - Train: 22.47% [1110500/4942000] [224.7/1000.0] [batch_t 0.320 (0.326)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-09 03:28:55,653 - Train: 22.47% [1110600/4942000] [224.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 03:29:28,366 - Train: 22.47% [1110700/4942000] [224.7/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 03:30:01,072 - Train: 22.48% [1110800/4942000] [224.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 03:30:33,750 - Train: 22.48% [1110900/4942000] [224.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 03:31:07,254 - Train: 22.48% [1111000/4942000] [224.8/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 03:31:39,912 - Train: 22.48% [1111100/4942000] [224.8/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 03:32:12,531 - Train: 22.48% [1111200/4942000] [224.8/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 03:32:45,261 - Train: 22.49% [1111300/4942000] [224.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 03:33:17,922 - Train: 22.49% [1111400/4942000] [224.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 03:33:50,613 - Train: 22.49% [1111500/4942000] [224.9/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 03:34:23,325 - Train: 22.49% [1111600/4942000] [224.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 03:34:56,049 - Train: 22.49% [1111700/4942000] [224.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 03:35:28,765 - Train: 22.50% [1111800/4942000] [225.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 03:36:01,492 - Train: 22.50% [1111900/4942000] [225.0/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 03:36:17,820 - ==> Total time: 6 days, 9:38:57 Eta: 22 days, 1:14:09 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 03:36:36,191 - Train: 22.50% [1112000/4942000] [225.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 03:37:08,843 - Train: 22.50% [1112100/4942000] [225.0/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 03:37:41,501 - Train: 22.51% [1112200/4942000] [225.1/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 03:38:14,133 - Train: 22.51% [1112300/4942000] [225.1/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 03:38:46,767 - Train: 22.51% [1112400/4942000] [225.1/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 03:39:19,497 - Train: 22.51% [1112500/4942000] [225.1/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 03:39:52,268 - Train: 22.51% [1112600/4942000] [225.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 03:40:24,917 - Train: 22.52% [1112700/4942000] [225.2/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 03:40:57,568 - Train: 22.52% [1112800/4942000] [225.2/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 03:41:30,266 - Train: 22.52% [1112900/4942000] [225.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 03:42:02,922 - Train: 22.52% [1113000/4942000] [225.2/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 03:42:35,592 - Train: 22.52% [1113100/4942000] [225.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 03:43:08,510 - Train: 22.53% [1113200/4942000] [225.3/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 03:43:41,186 - Train: 22.53% [1113300/4942000] [225.3/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 03:44:13,860 - Train: 22.53% [1113400/4942000] [225.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 03:44:46,670 - Train: 22.53% [1113500/4942000] [225.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 03:45:19,324 - Train: 22.53% [1113600/4942000] [225.3/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 03:45:51,952 - Train: 22.54% [1113700/4942000] [225.4/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 03:46:25,613 - Train: 22.54% [1113800/4942000] [225.4/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 03:46:58,311 - Train: 22.54% [1113900/4942000] [225.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 03:47:31,279 - Train: 22.54% [1114000/4942000] [225.4/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 03:48:03,947 - Train: 22.54% [1114100/4942000] [225.4/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 03:48:36,601 - Train: 22.55% [1114200/4942000] [225.5/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 03:49:09,208 - Train: 22.55% [1114300/4942000] [225.5/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 03:49:41,894 - Train: 22.55% [1114400/4942000] [225.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 03:50:14,607 - Train: 22.55% [1114500/4942000] [225.5/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 03:50:47,259 - Train: 22.55% [1114600/4942000] [225.5/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 03:51:19,958 - Train: 22.56% [1114700/4942000] [225.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 03:51:52,534 - Train: 22.56% [1114800/4942000] [225.6/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 03:52:25,208 - Train: 22.56% [1114900/4942000] [225.6/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 03:52:57,844 - Train: 22.56% [1115000/4942000] [225.6/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 03:53:30,438 - Train: 22.56% [1115100/4942000] [225.6/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 03:54:03,057 - Train: 22.57% [1115200/4942000] [225.7/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 03:54:35,735 - Train: 22.57% [1115300/4942000] [225.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 03:55:08,390 - Train: 22.57% [1115400/4942000] [225.7/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 03:55:41,119 - Train: 22.57% [1115500/4942000] [225.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 03:56:13,723 - Train: 22.57% [1115600/4942000] [225.7/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 03:56:46,368 - Train: 22.58% [1115700/4942000] [225.8/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 03:57:19,388 - Train: 22.58% [1115800/4942000] [225.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 03:57:52,043 - Train: 22.58% [1115900/4942000] [225.8/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 03:58:24,697 - Train: 22.58% [1116000/4942000] [225.8/1000.0] [batch_t 0.331 (0.326)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 03:58:57,290 - Train: 22.58% [1116100/4942000] [225.8/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 03:59:30,019 - Train: 22.59% [1116200/4942000] [225.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 04:00:02,734 - Train: 22.59% [1116300/4942000] [225.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 04:00:36,035 - Train: 22.59% [1116400/4942000] [225.9/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 04:01:08,878 - Train: 22.59% [1116500/4942000] [225.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 04:01:41,601 - Train: 22.59% [1116600/4942000] [225.9/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 04:02:14,249 - Train: 22.60% [1116700/4942000] [226.0/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 04:02:46,930 - Train: 22.60% [1116800/4942000] [226.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 04:03:17,146 - ==> Total time: 6 days, 10:05:56 Eta: 21 days, 23:45:17 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 04:03:21,948 - Train: 22.60% [1116900/4942000] [226.0/1000.0] [batch_t 0.327 (0.345)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 04:03:54,593 - Train: 22.60% [1117000/4942000] [226.0/1000.0] [batch_t 0.321 (0.326)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-09 04:04:27,412 - Train: 22.60% [1117100/4942000] [226.0/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 04:05:00,053 - Train: 22.61% [1117200/4942000] [226.1/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 04:05:32,698 - Train: 22.61% [1117300/4942000] [226.1/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 04:06:05,382 - Train: 22.61% [1117400/4942000] [226.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 04:06:38,120 - Train: 22.61% [1117500/4942000] [226.1/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 04:07:10,950 - Train: 22.61% [1117600/4942000] [226.1/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 04:07:43,670 - Train: 22.62% [1117700/4942000] [226.2/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 04:08:16,447 - Train: 22.62% [1117800/4942000] [226.2/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 04:08:49,231 - Train: 22.62% [1117900/4942000] [226.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 04:09:21,941 - Train: 22.62% [1118000/4942000] [226.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 04:09:54,823 - Train: 22.62% [1118100/4942000] [226.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 04:10:27,564 - Train: 22.63% [1118200/4942000] [226.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 04:11:00,426 - Train: 22.63% [1118300/4942000] [226.3/1000.0] [batch_t 0.322 (0.329)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 04:11:33,425 - Train: 22.63% [1118400/4942000] [226.3/1000.0] [batch_t 0.323 (0.330)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 04:12:07,663 - Train: 22.63% [1118500/4942000] [226.3/1000.0] [batch_t 0.327 (0.342)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 04:12:40,344 - Train: 22.63% [1118600/4942000] [226.3/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 04:13:13,319 - Train: 22.64% [1118700/4942000] [226.4/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 04:13:45,976 - Train: 22.64% [1118800/4942000] [226.4/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 04:14:19,222 - Train: 22.64% [1118900/4942000] [226.4/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 04:14:51,945 - Train: 22.64% [1119000/4942000] [226.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 04:15:25,320 - Train: 22.64% [1119100/4942000] [226.4/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 04:15:58,008 - Train: 22.65% [1119200/4942000] [226.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 04:16:30,907 - Train: 22.65% [1119300/4942000] [226.5/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 04:17:03,686 - Train: 22.65% [1119400/4942000] [226.5/1000.0] [batch_t 0.336 (0.328)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-09 04:17:36,465 - Train: 22.65% [1119500/4942000] [226.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 04:18:09,191 - Train: 22.65% [1119600/4942000] [226.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 04:18:41,998 - Train: 22.66% [1119700/4942000] [226.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 04:19:14,703 - Train: 22.66% [1119800/4942000] [226.6/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 04:19:47,423 - Train: 22.66% [1119900/4942000] [226.6/1000.0] [batch_t 0.336 (0.327)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-09 04:20:20,119 - Train: 22.66% [1120000/4942000] [226.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 04:20:52,778 - Train: 22.66% [1120100/4942000] [226.6/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 04:21:25,386 - Train: 22.67% [1120200/4942000] [226.7/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 04:21:58,022 - Train: 22.67% [1120300/4942000] [226.7/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 04:22:30,676 - Train: 22.67% [1120400/4942000] [226.7/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 04:23:03,365 - Train: 22.67% [1120500/4942000] [226.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 04:23:36,125 - Train: 22.68% [1120600/4942000] [226.8/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 04:24:08,759 - Train: 22.68% [1120700/4942000] [226.8/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 04:24:41,374 - Train: 22.68% [1120800/4942000] [226.8/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 04:25:14,001 - Train: 22.68% [1120900/4942000] [226.8/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 04:25:46,749 - Train: 22.68% [1121000/4942000] [226.8/1000.0] [batch_t 0.334 (0.327)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 04:26:19,587 - Train: 22.69% [1121100/4942000] [226.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 04:26:52,353 - Train: 22.69% [1121200/4942000] [226.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 04:27:24,964 - Train: 22.69% [1121300/4942000] [226.9/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 04:27:57,608 - Train: 22.69% [1121400/4942000] [226.9/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 04:28:30,240 - Train: 22.69% [1121500/4942000] [226.9/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 04:29:02,870 - Train: 22.70% [1121600/4942000] [227.0/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 04:29:35,493 - Train: 22.70% [1121700/4942000] [227.0/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 04:30:08,088 - Train: 22.70% [1121800/4942000] [227.0/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 04:30:19,196 - ==> Total time: 6 days, 10:32:58 Eta: 21 days, 22:17:07 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 04:30:42,766 - Train: 22.70% [1121900/4942000] [227.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 04:31:15,509 - Train: 22.70% [1122000/4942000] [227.0/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 04:31:48,122 - Train: 22.71% [1122100/4942000] [227.1/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 04:32:20,746 - Train: 22.71% [1122200/4942000] [227.1/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 04:32:53,383 - Train: 22.71% [1122300/4942000] [227.1/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 04:33:26,034 - Train: 22.71% [1122400/4942000] [227.1/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 04:33:58,833 - Train: 22.71% [1122500/4942000] [227.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 04:34:31,458 - Train: 22.72% [1122600/4942000] [227.2/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 04:35:04,131 - Train: 22.72% [1122700/4942000] [227.2/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 04:35:36,775 - Train: 22.72% [1122800/4942000] [227.2/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 04:36:09,445 - Train: 22.72% [1122900/4942000] [227.2/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 04:36:42,320 - Train: 22.72% [1123000/4942000] [227.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 04:37:14,932 - Train: 22.73% [1123100/4942000] [227.3/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 04:37:47,569 - Train: 22.73% [1123200/4942000] [227.3/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 04:38:20,229 - Train: 22.73% [1123300/4942000] [227.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 04:38:52,843 - Train: 22.73% [1123400/4942000] [227.3/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 04:39:25,523 - Train: 22.73% [1123500/4942000] [227.3/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 04:39:58,186 - Train: 22.74% [1123600/4942000] [227.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 04:40:30,820 - Train: 22.74% [1123700/4942000] [227.4/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 04:41:03,448 - Train: 22.74% [1123800/4942000] [227.4/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 04:41:36,156 - Train: 22.74% [1123900/4942000] [227.4/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 04:42:08,851 - Train: 22.74% [1124000/4942000] [227.4/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 04:42:41,507 - Train: 22.75% [1124100/4942000] [227.5/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 04:43:14,169 - Train: 22.75% [1124200/4942000] [227.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 04:43:46,852 - Train: 22.75% [1124300/4942000] [227.5/1000.0] [batch_t 0.321 (0.327)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-09 04:44:19,524 - Train: 22.75% [1124400/4942000] [227.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 04:44:52,241 - Train: 22.75% [1124500/4942000] [227.5/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 04:45:24,887 - Train: 22.76% [1124600/4942000] [227.6/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 04:45:57,509 - Train: 22.76% [1124700/4942000] [227.6/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 04:46:30,129 - Train: 22.76% [1124800/4942000] [227.6/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 04:47:02,683 - Train: 22.76% [1124900/4942000] [227.6/1000.0] [batch_t 0.326 (0.325)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 04:47:35,297 - Train: 22.76% [1125000/4942000] [227.6/1000.0] [batch_t 0.336 (0.326)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-09 04:48:07,915 - Train: 22.77% [1125100/4942000] [227.7/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 04:48:40,557 - Train: 22.77% [1125200/4942000] [227.7/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 04:49:13,258 - Train: 22.77% [1125300/4942000] [227.7/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 04:49:45,879 - Train: 22.77% [1125400/4942000] [227.7/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 04:50:18,507 - Train: 22.77% [1125500/4942000] [227.7/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 04:50:51,111 - Train: 22.78% [1125600/4942000] [227.8/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 04:51:23,775 - Train: 22.78% [1125700/4942000] [227.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 04:51:56,444 - Train: 22.78% [1125800/4942000] [227.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 04:52:29,092 - Train: 22.78% [1125900/4942000] [227.8/1000.0] [batch_t 0.333 (0.326)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 04:53:01,736 - Train: 22.78% [1126000/4942000] [227.8/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 04:53:34,424 - Train: 22.79% [1126100/4942000] [227.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 04:54:07,016 - Train: 22.79% [1126200/4942000] [227.9/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 04:54:39,703 - Train: 22.79% [1126300/4942000] [227.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 04:55:12,348 - Train: 22.79% [1126400/4942000] [227.9/1000.0] [batch_t 0.317 (0.326)] [data_t 0.002] [optim_t 0.315] [lr 0.005000] 2024-04-09 04:55:44,941 - Train: 22.79% [1126500/4942000] [227.9/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 04:56:17,836 - Train: 22.80% [1126600/4942000] [228.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 04:56:50,557 - Train: 22.80% [1126700/4942000] [228.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 04:57:15,565 - ==> Total time: 6 days, 10:59:54 Eta: 21 days, 20:49:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 04:57:25,363 - Train: 22.80% [1126800/4942000] [228.0/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 04:57:57,910 - Train: 22.80% [1126900/4942000] [228.0/1000.0] [batch_t 0.329 (0.325)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 04:58:30,610 - Train: 22.80% [1127000/4942000] [228.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 04:59:03,259 - Train: 22.81% [1127100/4942000] [228.1/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 04:59:35,985 - Train: 22.81% [1127200/4942000] [228.1/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 05:00:08,604 - Train: 22.81% [1127300/4942000] [228.1/1000.0] [batch_t 0.321 (0.326)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-09 05:00:41,280 - Train: 22.81% [1127400/4942000] [228.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 05:01:13,874 - Train: 22.81% [1127500/4942000] [228.1/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 05:01:46,552 - Train: 22.82% [1127600/4942000] [228.2/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 05:02:19,169 - Train: 22.82% [1127700/4942000] [228.2/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 05:02:51,870 - Train: 22.82% [1127800/4942000] [228.2/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 05:03:24,679 - Train: 22.82% [1127900/4942000] [228.2/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 05:03:57,352 - Train: 22.82% [1128000/4942000] [228.2/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 05:04:30,146 - Train: 22.83% [1128100/4942000] [228.3/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 05:05:02,863 - Train: 22.83% [1128200/4942000] [228.3/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 05:05:35,450 - Train: 22.83% [1128300/4942000] [228.3/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 05:06:08,067 - Train: 22.83% [1128400/4942000] [228.3/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 05:06:40,650 - Train: 22.83% [1128500/4942000] [228.3/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 05:07:13,289 - Train: 22.84% [1128600/4942000] [228.4/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 05:07:45,896 - Train: 22.84% [1128700/4942000] [228.4/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 05:08:18,506 - Train: 22.84% [1128800/4942000] [228.4/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 05:08:51,113 - Train: 22.84% [1128900/4942000] [228.4/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 05:09:23,705 - Train: 22.85% [1129000/4942000] [228.5/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 05:09:56,315 - Train: 22.85% [1129100/4942000] [228.5/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 05:10:28,986 - Train: 22.85% [1129200/4942000] [228.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 05:11:01,660 - Train: 22.85% [1129300/4942000] [228.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 05:11:34,325 - Train: 22.85% [1129400/4942000] [228.5/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 05:12:07,293 - Train: 22.86% [1129500/4942000] [228.6/1000.0] [batch_t 0.323 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 05:12:40,013 - Train: 22.86% [1129600/4942000] [228.6/1000.0] [batch_t 0.335 (0.327)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-09 05:13:12,743 - Train: 22.86% [1129700/4942000] [228.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 05:13:45,385 - Train: 22.86% [1129800/4942000] [228.6/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 05:14:18,001 - Train: 22.86% [1129900/4942000] [228.6/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 05:14:50,657 - Train: 22.87% [1130000/4942000] [228.7/1000.0] [batch_t 0.331 (0.326)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 05:15:23,421 - Train: 22.87% [1130100/4942000] [228.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 05:15:55,992 - Train: 22.87% [1130200/4942000] [228.7/1000.0] [batch_t 0.321 (0.326)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 05:16:28,667 - Train: 22.87% [1130300/4942000] [228.7/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 05:17:01,345 - Train: 22.87% [1130400/4942000] [228.7/1000.0] [batch_t 0.335 (0.327)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-09 05:17:34,178 - Train: 22.88% [1130500/4942000] [228.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 05:18:06,935 - Train: 22.88% [1130600/4942000] [228.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 05:18:39,697 - Train: 22.88% [1130700/4942000] [228.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 05:19:12,364 - Train: 22.88% [1130800/4942000] [228.8/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 05:19:45,169 - Train: 22.88% [1130900/4942000] [228.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 05:20:18,036 - Train: 22.89% [1131000/4942000] [228.9/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 05:20:50,690 - Train: 22.89% [1131100/4942000] [228.9/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 05:21:23,346 - Train: 22.89% [1131200/4942000] [228.9/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 05:21:56,038 - Train: 22.89% [1131300/4942000] [228.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 05:22:28,883 - Train: 22.89% [1131400/4942000] [228.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 05:23:01,578 - Train: 22.90% [1131500/4942000] [229.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 05:23:34,289 - Train: 22.90% [1131600/4942000] [229.0/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 05:24:06,980 - Train: 22.90% [1131700/4942000] [229.0/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 05:24:12,900 - ==> Total time: 6 days, 11:26:52 Eta: 21 days, 19:21:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 05:24:41,631 - Train: 22.90% [1131800/4942000] [229.0/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 05:25:14,243 - Train: 22.90% [1131900/4942000] [229.0/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 05:25:46,935 - Train: 22.91% [1132000/4942000] [229.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 05:26:19,614 - Train: 22.91% [1132100/4942000] [229.1/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 05:26:52,339 - Train: 22.91% [1132200/4942000] [229.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 05:27:25,151 - Train: 22.91% [1132300/4942000] [229.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 05:27:57,840 - Train: 22.91% [1132400/4942000] [229.1/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 05:28:30,495 - Train: 22.92% [1132500/4942000] [229.2/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 05:29:03,180 - Train: 22.92% [1132600/4942000] [229.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 05:29:35,858 - Train: 22.92% [1132700/4942000] [229.2/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 05:30:08,514 - Train: 22.92% [1132800/4942000] [229.2/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 05:30:41,134 - Train: 22.92% [1132900/4942000] [229.2/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 05:31:13,855 - Train: 22.93% [1133000/4942000] [229.3/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 05:31:46,624 - Train: 22.93% [1133100/4942000] [229.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 05:32:19,249 - Train: 22.93% [1133200/4942000] [229.3/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 05:32:51,880 - Train: 22.93% [1133300/4942000] [229.3/1000.0] [batch_t 0.321 (0.326)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-09 05:33:24,570 - Train: 22.93% [1133400/4942000] [229.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 05:33:57,210 - Train: 22.94% [1133500/4942000] [229.4/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 05:34:29,920 - Train: 22.94% [1133600/4942000] [229.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 05:35:02,723 - Train: 22.94% [1133700/4942000] [229.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 05:35:35,350 - Train: 22.94% [1133800/4942000] [229.4/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 05:36:08,005 - Train: 22.94% [1133900/4942000] [229.4/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 05:36:40,594 - Train: 22.95% [1134000/4942000] [229.5/1000.0] [batch_t 0.322 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 05:37:13,222 - Train: 22.95% [1134100/4942000] [229.5/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 05:37:45,928 - Train: 22.95% [1134200/4942000] [229.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 05:38:18,568 - Train: 22.95% [1134300/4942000] [229.5/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 05:38:51,234 - Train: 22.95% [1134400/4942000] [229.5/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 05:39:24,055 - Train: 22.96% [1134500/4942000] [229.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 05:39:56,755 - Train: 22.96% [1134600/4942000] [229.6/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 05:40:29,714 - Train: 22.96% [1134700/4942000] [229.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 05:41:02,416 - Train: 22.96% [1134800/4942000] [229.6/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 05:41:35,045 - Train: 22.96% [1134900/4942000] [229.6/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 05:42:07,724 - Train: 22.97% [1135000/4942000] [229.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 05:42:40,351 - Train: 22.97% [1135100/4942000] [229.7/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 05:43:13,332 - Train: 22.97% [1135200/4942000] [229.7/1000.0] [batch_t 0.336 (0.330)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-09 05:43:46,108 - Train: 22.97% [1135300/4942000] [229.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 05:44:18,886 - Train: 22.97% [1135400/4942000] [229.7/1000.0] [batch_t 0.321 (0.328)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-09 05:44:51,562 - Train: 22.98% [1135500/4942000] [229.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 05:45:24,921 - Train: 22.98% [1135600/4942000] [229.8/1000.0] [batch_t 0.331 (0.333)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 05:45:57,588 - Train: 22.98% [1135700/4942000] [229.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 05:46:30,276 - Train: 22.98% [1135800/4942000] [229.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 05:47:02,918 - Train: 22.98% [1135900/4942000] [229.8/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 05:47:35,617 - Train: 22.99% [1136000/4942000] [229.9/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 05:48:08,265 - Train: 22.99% [1136100/4942000] [229.9/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 05:48:40,933 - Train: 22.99% [1136200/4942000] [229.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 05:49:13,605 - Train: 22.99% [1136300/4942000] [229.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 05:49:46,351 - Train: 22.99% [1136400/4942000] [229.9/1000.0] [batch_t 0.334 (0.327)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 05:50:19,046 - Train: 23.00% [1136500/4942000] [230.0/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 05:50:51,918 - Train: 23.00% [1136600/4942000] [230.0/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 05:51:11,507 - ==> Total time: 6 days, 11:53:50 Eta: 21 days, 17:55:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 05:51:26,627 - Train: 23.00% [1136700/4942000] [230.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 05:51:59,261 - Train: 23.00% [1136800/4942000] [230.0/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 05:52:31,898 - Train: 23.00% [1136900/4942000] [230.0/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 05:53:04,602 - Train: 23.01% [1137000/4942000] [230.1/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 05:53:37,399 - Train: 23.01% [1137100/4942000] [230.1/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 05:54:10,163 - Train: 23.01% [1137200/4942000] [230.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 05:54:42,966 - Train: 23.01% [1137300/4942000] [230.1/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 05:55:15,666 - Train: 23.01% [1137400/4942000] [230.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-09 05:55:48,392 - Train: 23.02% [1137500/4942000] [230.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 05:56:21,090 - Train: 23.02% [1137600/4942000] [230.2/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 05:56:53,818 - Train: 23.02% [1137700/4942000] [230.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 05:57:26,561 - Train: 23.02% [1137800/4942000] [230.2/1000.0] [batch_t 0.335 (0.327)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-09 05:57:59,507 - Train: 23.03% [1137900/4942000] [230.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 05:58:32,177 - Train: 23.03% [1138000/4942000] [230.3/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 05:59:04,838 - Train: 23.03% [1138100/4942000] [230.3/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 05:59:37,447 - Train: 23.03% [1138200/4942000] [230.3/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 06:00:10,034 - Train: 23.03% [1138300/4942000] [230.3/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 06:00:42,749 - Train: 23.04% [1138400/4942000] [230.4/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 06:01:15,402 - Train: 23.04% [1138500/4942000] [230.4/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 06:01:48,016 - Train: 23.04% [1138600/4942000] [230.4/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 06:02:20,681 - Train: 23.04% [1138700/4942000] [230.4/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 06:02:53,366 - Train: 23.04% [1138800/4942000] [230.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 06:03:26,051 - Train: 23.05% [1138900/4942000] [230.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 06:03:58,814 - Train: 23.05% [1139000/4942000] [230.5/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 06:04:31,505 - Train: 23.05% [1139100/4942000] [230.5/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 06:05:04,164 - Train: 23.05% [1139200/4942000] [230.5/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 06:05:36,951 - Train: 23.05% [1139300/4942000] [230.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 06:06:09,590 - Train: 23.06% [1139400/4942000] [230.6/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 06:06:42,285 - Train: 23.06% [1139500/4942000] [230.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 06:07:14,961 - Train: 23.06% [1139600/4942000] [230.6/1000.0] [batch_t 0.336 (0.327)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-09 06:07:47,594 - Train: 23.06% [1139700/4942000] [230.6/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 06:08:20,234 - Train: 23.06% [1139800/4942000] [230.6/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 06:08:52,959 - Train: 23.07% [1139900/4942000] [230.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 06:09:25,628 - Train: 23.07% [1140000/4942000] [230.7/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 06:09:58,378 - Train: 23.07% [1140100/4942000] [230.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 06:10:31,047 - Train: 23.07% [1140200/4942000] [230.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 06:11:03,730 - Train: 23.07% [1140300/4942000] [230.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 06:11:36,390 - Train: 23.08% [1140400/4942000] [230.8/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 06:12:09,096 - Train: 23.08% [1140500/4942000] [230.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 06:12:41,719 - Train: 23.08% [1140600/4942000] [230.8/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 06:13:14,390 - Train: 23.08% [1140700/4942000] [230.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 06:13:47,176 - Train: 23.08% [1140800/4942000] [230.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 06:14:19,824 - Train: 23.09% [1140900/4942000] [230.9/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 06:14:52,475 - Train: 23.09% [1141000/4942000] [230.9/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 06:15:25,104 - Train: 23.09% [1141100/4942000] [230.9/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 06:15:57,700 - Train: 23.09% [1141200/4942000] [230.9/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 06:16:30,299 - Train: 23.09% [1141300/4942000] [230.9/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 06:17:02,958 - Train: 23.10% [1141400/4942000] [231.0/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 06:17:35,578 - Train: 23.10% [1141500/4942000] [231.0/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 06:18:08,396 - Train: 23.10% [1141600/4942000] [231.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 06:18:09,056 - ==> Total time: 6 days, 12:20:48 Eta: 21 days, 16:28:44 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 06:18:43,189 - Train: 23.10% [1141700/4942000] [231.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 06:19:15,854 - Train: 23.10% [1141800/4942000] [231.0/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 06:19:48,435 - Train: 23.11% [1141900/4942000] [231.1/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 06:20:21,015 - Train: 23.11% [1142000/4942000] [231.1/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 06:20:53,808 - Train: 23.11% [1142100/4942000] [231.1/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 06:21:26,564 - Train: 23.11% [1142200/4942000] [231.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 06:21:59,202 - Train: 23.11% [1142300/4942000] [231.1/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 06:22:31,829 - Train: 23.12% [1142400/4942000] [231.2/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 06:23:04,425 - Train: 23.12% [1142500/4942000] [231.2/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 06:23:37,065 - Train: 23.12% [1142600/4942000] [231.2/1000.0] [batch_t 0.321 (0.326)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 06:24:09,701 - Train: 23.12% [1142700/4942000] [231.2/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 06:24:42,260 - Train: 23.12% [1142800/4942000] [231.2/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 06:25:14,902 - Train: 23.13% [1142900/4942000] [231.3/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 06:25:47,504 - Train: 23.13% [1143000/4942000] [231.3/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 06:26:20,167 - Train: 23.13% [1143100/4942000] [231.3/1000.0] [batch_t 0.320 (0.327)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-09 06:26:52,792 - Train: 23.13% [1143200/4942000] [231.3/1000.0] [batch_t 0.322 (0.326)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 06:27:25,417 - Train: 23.13% [1143300/4942000] [231.3/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 06:27:58,034 - Train: 23.14% [1143400/4942000] [231.4/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 06:28:30,677 - Train: 23.14% [1143500/4942000] [231.4/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 06:29:03,330 - Train: 23.14% [1143600/4942000] [231.4/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 06:29:35,953 - Train: 23.14% [1143700/4942000] [231.4/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 06:30:08,741 - Train: 23.14% [1143800/4942000] [231.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 06:30:41,409 - Train: 23.15% [1143900/4942000] [231.5/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 06:31:14,022 - Train: 23.15% [1144000/4942000] [231.5/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 06:31:46,650 - Train: 23.15% [1144100/4942000] [231.5/1000.0] [batch_t 0.322 (0.326)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 06:32:19,211 - Train: 23.15% [1144200/4942000] [231.5/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 06:32:51,881 - Train: 23.15% [1144300/4942000] [231.5/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 06:33:24,550 - Train: 23.16% [1144400/4942000] [231.6/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 06:33:57,126 - Train: 23.16% [1144500/4942000] [231.6/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 06:34:29,845 - Train: 23.16% [1144600/4942000] [231.6/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 06:35:02,527 - Train: 23.16% [1144700/4942000] [231.6/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 06:35:35,255 - Train: 23.16% [1144800/4942000] [231.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 06:36:07,979 - Train: 23.17% [1144900/4942000] [231.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 06:36:40,887 - Train: 23.17% [1145000/4942000] [231.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 06:37:13,554 - Train: 23.17% [1145100/4942000] [231.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 06:37:46,262 - Train: 23.17% [1145200/4942000] [231.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 06:38:18,969 - Train: 23.17% [1145300/4942000] [231.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 06:38:51,712 - Train: 23.18% [1145400/4942000] [231.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 06:39:24,424 - Train: 23.18% [1145500/4942000] [231.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 06:39:57,104 - Train: 23.18% [1145600/4942000] [231.8/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 06:40:29,782 - Train: 23.18% [1145700/4942000] [231.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 06:41:02,516 - Train: 23.18% [1145800/4942000] [231.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 06:41:35,315 - Train: 23.19% [1145900/4942000] [231.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 06:42:07,961 - Train: 23.19% [1146000/4942000] [231.9/1000.0] [batch_t 0.332 (0.326)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 06:42:40,655 - Train: 23.19% [1146100/4942000] [231.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 06:43:13,330 - Train: 23.19% [1146200/4942000] [231.9/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 06:43:46,029 - Train: 23.20% [1146300/4942000] [232.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 06:44:18,895 - Train: 23.20% [1146400/4942000] [232.0/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 06:44:51,588 - Train: 23.20% [1146500/4942000] [232.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 06:45:05,964 - ==> Total time: 6 days, 12:47:45 Eta: 21 days, 15:02:54 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 06:45:26,241 - Train: 23.20% [1146600/4942000] [232.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 06:45:58,877 - Train: 23.20% [1146700/4942000] [232.0/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 06:46:31,856 - Train: 23.21% [1146800/4942000] [232.1/1000.0] [batch_t 0.323 (0.330)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 06:47:04,517 - Train: 23.21% [1146900/4942000] [232.1/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 06:47:37,235 - Train: 23.21% [1147000/4942000] [232.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 06:48:09,979 - Train: 23.21% [1147100/4942000] [232.1/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 06:48:42,589 - Train: 23.21% [1147200/4942000] [232.1/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 06:49:15,246 - Train: 23.22% [1147300/4942000] [232.2/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 06:49:47,934 - Train: 23.22% [1147400/4942000] [232.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 06:50:20,810 - Train: 23.22% [1147500/4942000] [232.2/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 06:50:53,506 - Train: 23.22% [1147600/4942000] [232.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 06:51:26,149 - Train: 23.22% [1147700/4942000] [232.2/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 06:51:58,946 - Train: 23.23% [1147800/4942000] [232.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 06:52:31,637 - Train: 23.23% [1147900/4942000] [232.3/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 06:53:04,346 - Train: 23.23% [1148000/4942000] [232.3/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 06:53:37,013 - Train: 23.23% [1148100/4942000] [232.3/1000.0] [batch_t 0.335 (0.327)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-09 06:54:10,019 - Train: 23.23% [1148200/4942000] [232.3/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 06:54:42,715 - Train: 23.24% [1148300/4942000] [232.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 06:55:15,351 - Train: 23.24% [1148400/4942000] [232.4/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 06:55:48,054 - Train: 23.24% [1148500/4942000] [232.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 06:56:20,741 - Train: 23.24% [1148600/4942000] [232.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 06:56:53,441 - Train: 23.24% [1148700/4942000] [232.4/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 06:57:26,066 - Train: 23.25% [1148800/4942000] [232.5/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 06:57:58,675 - Train: 23.25% [1148900/4942000] [232.5/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 06:58:31,321 - Train: 23.25% [1149000/4942000] [232.5/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 06:59:04,096 - Train: 23.25% [1149100/4942000] [232.5/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 06:59:36,856 - Train: 23.25% [1149200/4942000] [232.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 07:00:09,503 - Train: 23.26% [1149300/4942000] [232.6/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 07:00:42,225 - Train: 23.26% [1149400/4942000] [232.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 07:01:14,888 - Train: 23.26% [1149500/4942000] [232.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 07:01:47,538 - Train: 23.26% [1149600/4942000] [232.6/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 07:02:20,139 - Train: 23.26% [1149700/4942000] [232.6/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 07:02:52,793 - Train: 23.27% [1149800/4942000] [232.7/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 07:03:25,405 - Train: 23.27% [1149900/4942000] [232.7/1000.0] [batch_t 0.334 (0.326)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 07:03:58,105 - Train: 23.27% [1150000/4942000] [232.7/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 07:04:30,800 - Train: 23.27% [1150100/4942000] [232.7/1000.0] [batch_t 0.321 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 07:05:03,486 - Train: 23.27% [1150200/4942000] [232.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 07:05:36,203 - Train: 23.28% [1150300/4942000] [232.8/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 07:06:08,861 - Train: 23.28% [1150400/4942000] [232.8/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 07:06:41,579 - Train: 23.28% [1150500/4942000] [232.8/1000.0] [batch_t 0.337 (0.327)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-09 07:07:14,395 - Train: 23.28% [1150600/4942000] [232.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-09 07:07:47,038 - Train: 23.28% [1150700/4942000] [232.8/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 07:08:19,591 - Train: 23.29% [1150800/4942000] [232.9/1000.0] [batch_t 0.329 (0.325)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 07:08:52,187 - Train: 23.29% [1150900/4942000] [232.9/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 07:09:24,799 - Train: 23.29% [1151000/4942000] [232.9/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 07:09:57,393 - Train: 23.29% [1151100/4942000] [232.9/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 07:10:30,019 - Train: 23.29% [1151200/4942000] [232.9/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 07:11:02,639 - Train: 23.30% [1151300/4942000] [233.0/1000.0] [batch_t 0.334 (0.326)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 07:11:35,308 - Train: 23.30% [1151400/4942000] [233.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 07:12:03,470 - ==> Total time: 6 days, 13:14:42 Eta: 21 days, 13:37:36 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 07:12:10,914 - Train: 23.30% [1151500/4942000] [233.0/1000.0] [batch_t 0.334 (0.356)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 07:12:43,679 - Train: 23.30% [1151600/4942000] [233.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 07:13:16,468 - Train: 23.30% [1151700/4942000] [233.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 07:13:49,171 - Train: 23.31% [1151800/4942000] [233.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 07:14:21,985 - Train: 23.31% [1151900/4942000] [233.1/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 07:14:54,624 - Train: 23.31% [1152000/4942000] [233.1/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 07:15:27,293 - Train: 23.31% [1152100/4942000] [233.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 07:16:00,036 - Train: 23.31% [1152200/4942000] [233.1/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 07:16:32,756 - Train: 23.32% [1152300/4942000] [233.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 07:17:05,393 - Train: 23.32% [1152400/4942000] [233.2/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 07:17:38,060 - Train: 23.32% [1152500/4942000] [233.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 07:18:10,801 - Train: 23.32% [1152600/4942000] [233.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 07:18:43,480 - Train: 23.32% [1152700/4942000] [233.2/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 07:19:16,201 - Train: 23.33% [1152800/4942000] [233.3/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 07:19:48,951 - Train: 23.33% [1152900/4942000] [233.3/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 07:20:21,614 - Train: 23.33% [1153000/4942000] [233.3/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 07:20:54,228 - Train: 23.33% [1153100/4942000] [233.3/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 07:21:26,905 - Train: 23.33% [1153200/4942000] [233.3/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 07:21:59,703 - Train: 23.34% [1153300/4942000] [233.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 07:22:32,486 - Train: 23.34% [1153400/4942000] [233.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 07:23:05,144 - Train: 23.34% [1153500/4942000] [233.4/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 07:23:37,826 - Train: 23.34% [1153600/4942000] [233.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 07:24:10,532 - Train: 23.34% [1153700/4942000] [233.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 07:24:43,322 - Train: 23.35% [1153800/4942000] [233.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 07:25:15,993 - Train: 23.35% [1153900/4942000] [233.5/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 07:25:48,617 - Train: 23.35% [1154000/4942000] [233.5/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 07:26:21,265 - Train: 23.35% [1154100/4942000] [233.5/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 07:26:53,922 - Train: 23.35% [1154200/4942000] [233.5/1000.0] [batch_t 0.322 (0.326)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 07:27:26,555 - Train: 23.36% [1154300/4942000] [233.6/1000.0] [batch_t 0.321 (0.326)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 07:27:59,243 - Train: 23.36% [1154400/4942000] [233.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 07:28:31,805 - Train: 23.36% [1154500/4942000] [233.6/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 07:29:04,590 - Train: 23.36% [1154600/4942000] [233.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 07:29:37,276 - Train: 23.37% [1154700/4942000] [233.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 07:30:10,034 - Train: 23.37% [1154800/4942000] [233.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 07:30:42,648 - Train: 23.37% [1154900/4942000] [233.7/1000.0] [batch_t 0.322 (0.326)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 07:31:15,284 - Train: 23.37% [1155000/4942000] [233.7/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 07:31:47,946 - Train: 23.37% [1155100/4942000] [233.7/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 07:32:20,571 - Train: 23.38% [1155200/4942000] [233.8/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 07:32:53,207 - Train: 23.38% [1155300/4942000] [233.8/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 07:33:25,855 - Train: 23.38% [1155400/4942000] [233.8/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 07:33:58,476 - Train: 23.38% [1155500/4942000] [233.8/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 07:34:31,253 - Train: 23.38% [1155600/4942000] [233.8/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 07:35:04,029 - Train: 23.39% [1155700/4942000] [233.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 07:35:36,712 - Train: 23.39% [1155800/4942000] [233.9/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 07:36:09,336 - Train: 23.39% [1155900/4942000] [233.9/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 07:36:42,028 - Train: 23.39% [1156000/4942000] [233.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 07:37:14,759 - Train: 23.39% [1156100/4942000] [233.9/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 07:37:47,580 - Train: 23.40% [1156200/4942000] [234.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 07:38:20,271 - Train: 23.40% [1156300/4942000] [234.0/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 07:38:52,873 - Train: 23.40% [1156400/4942000] [234.0/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 07:39:02,012 - ==> Total time: 6 days, 13:41:41 Eta: 21 days, 12:12:52 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 07:39:29,515 - Train: 23.40% [1156500/4942000] [234.0/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 07:40:02,161 - Train: 23.40% [1156600/4942000] [234.0/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 07:40:34,790 - Train: 23.41% [1156700/4942000] [234.1/1000.0] [batch_t 0.334 (0.326)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 07:41:07,454 - Train: 23.41% [1156800/4942000] [234.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 07:41:40,183 - Train: 23.41% [1156900/4942000] [234.1/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 07:42:12,907 - Train: 23.41% [1157000/4942000] [234.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 07:42:45,644 - Train: 23.41% [1157100/4942000] [234.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 07:43:18,299 - Train: 23.42% [1157200/4942000] [234.2/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 07:43:51,040 - Train: 23.42% [1157300/4942000] [234.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 07:44:23,692 - Train: 23.42% [1157400/4942000] [234.2/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 07:44:56,534 - Train: 23.42% [1157500/4942000] [234.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 07:45:29,210 - Train: 23.42% [1157600/4942000] [234.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 07:46:01,940 - Train: 23.43% [1157700/4942000] [234.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 07:46:34,656 - Train: 23.43% [1157800/4942000] [234.3/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 07:47:07,348 - Train: 23.43% [1157900/4942000] [234.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 07:47:40,020 - Train: 23.43% [1158000/4942000] [234.3/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 07:48:12,650 - Train: 23.43% [1158100/4942000] [234.3/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 07:48:45,244 - Train: 23.44% [1158200/4942000] [234.4/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 07:49:17,858 - Train: 23.44% [1158300/4942000] [234.4/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 07:49:50,536 - Train: 23.44% [1158400/4942000] [234.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 07:50:23,222 - Train: 23.44% [1158500/4942000] [234.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 07:50:55,878 - Train: 23.44% [1158600/4942000] [234.4/1000.0] [batch_t 0.322 (0.326)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 07:51:28,750 - Train: 23.45% [1158700/4942000] [234.5/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-09 07:52:01,549 - Train: 23.45% [1158800/4942000] [234.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 07:52:34,519 - Train: 23.45% [1158900/4942000] [234.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 07:53:07,178 - Train: 23.45% [1159000/4942000] [234.5/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 07:53:39,928 - Train: 23.45% [1159100/4942000] [234.5/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 07:54:12,603 - Train: 23.46% [1159200/4942000] [234.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 07:54:45,186 - Train: 23.46% [1159300/4942000] [234.6/1000.0] [batch_t 0.322 (0.326)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 07:55:17,808 - Train: 23.46% [1159400/4942000] [234.6/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 07:55:50,500 - Train: 23.46% [1159500/4942000] [234.6/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 07:56:23,236 - Train: 23.46% [1159600/4942000] [234.6/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 07:56:55,919 - Train: 23.47% [1159700/4942000] [234.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 07:57:28,620 - Train: 23.47% [1159800/4942000] [234.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 07:58:01,322 - Train: 23.47% [1159900/4942000] [234.7/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 07:58:34,051 - Train: 23.47% [1160000/4942000] [234.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 07:59:06,813 - Train: 23.47% [1160100/4942000] [234.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 07:59:39,457 - Train: 23.48% [1160200/4942000] [234.8/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 08:00:12,111 - Train: 23.48% [1160300/4942000] [234.8/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 08:00:44,943 - Train: 23.48% [1160400/4942000] [234.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 08:01:17,617 - Train: 23.48% [1160500/4942000] [234.8/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 08:01:50,424 - Train: 23.48% [1160600/4942000] [234.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 08:02:23,106 - Train: 23.49% [1160700/4942000] [234.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 08:02:55,797 - Train: 23.49% [1160800/4942000] [234.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 08:03:28,525 - Train: 23.49% [1160900/4942000] [234.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 08:04:01,197 - Train: 23.49% [1161000/4942000] [234.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 08:04:33,835 - Train: 23.49% [1161100/4942000] [234.9/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 08:05:06,502 - Train: 23.50% [1161200/4942000] [235.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 08:05:39,288 - Train: 23.50% [1161300/4942000] [235.0/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 08:06:02,230 - ==> Total time: 6 days, 14:08:41 Eta: 21 days, 10:48:42 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 08:06:15,496 - Train: 23.50% [1161400/4942000] [235.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 08:06:48,128 - Train: 23.50% [1161500/4942000] [235.0/1000.0] [batch_t 0.331 (0.326)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 08:07:20,926 - Train: 23.50% [1161600/4942000] [235.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 08:07:53,764 - Train: 23.51% [1161700/4942000] [235.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 08:08:26,443 - Train: 23.51% [1161800/4942000] [235.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 08:08:59,137 - Train: 23.51% [1161900/4942000] [235.1/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 08:09:31,771 - Train: 23.51% [1162000/4942000] [235.1/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 08:10:04,518 - Train: 23.51% [1162100/4942000] [235.1/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 08:10:37,114 - Train: 23.52% [1162200/4942000] [235.2/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 08:11:09,812 - Train: 23.52% [1162300/4942000] [235.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 08:11:42,474 - Train: 23.52% [1162400/4942000] [235.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 08:12:15,150 - Train: 23.52% [1162500/4942000] [235.2/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 08:12:47,826 - Train: 23.52% [1162600/4942000] [235.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 08:13:20,469 - Train: 23.53% [1162700/4942000] [235.3/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 08:13:53,163 - Train: 23.53% [1162800/4942000] [235.3/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 08:14:25,812 - Train: 23.53% [1162900/4942000] [235.3/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 08:14:58,456 - Train: 23.53% [1163000/4942000] [235.3/1000.0] [batch_t 0.321 (0.326)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 08:15:31,218 - Train: 23.54% [1163100/4942000] [235.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 08:16:03,812 - Train: 23.54% [1163200/4942000] [235.4/1000.0] [batch_t 0.322 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 08:16:36,419 - Train: 23.54% [1163300/4942000] [235.4/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 08:17:10,239 - Train: 23.54% [1163400/4942000] [235.4/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 08:17:42,921 - Train: 23.54% [1163500/4942000] [235.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 08:18:15,552 - Train: 23.55% [1163600/4942000] [235.5/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 08:18:48,238 - Train: 23.55% [1163700/4942000] [235.5/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 08:19:20,967 - Train: 23.55% [1163800/4942000] [235.5/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 08:19:53,605 - Train: 23.55% [1163900/4942000] [235.5/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 08:20:26,330 - Train: 23.55% [1164000/4942000] [235.5/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 08:20:59,001 - Train: 23.56% [1164100/4942000] [235.6/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 08:21:31,666 - Train: 23.56% [1164200/4942000] [235.6/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 08:22:04,328 - Train: 23.56% [1164300/4942000] [235.6/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 08:22:37,030 - Train: 23.56% [1164400/4942000] [235.6/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 08:23:09,748 - Train: 23.56% [1164500/4942000] [235.6/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 08:23:42,497 - Train: 23.57% [1164600/4942000] [235.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 08:24:15,236 - Train: 23.57% [1164700/4942000] [235.7/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 08:24:47,899 - Train: 23.57% [1164800/4942000] [235.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 08:25:20,654 - Train: 23.57% [1164900/4942000] [235.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 08:25:53,309 - Train: 23.57% [1165000/4942000] [235.7/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 08:26:25,979 - Train: 23.58% [1165100/4942000] [235.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 08:26:58,625 - Train: 23.58% [1165200/4942000] [235.8/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 08:27:31,249 - Train: 23.58% [1165300/4942000] [235.8/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 08:28:03,894 - Train: 23.58% [1165400/4942000] [235.8/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 08:28:36,563 - Train: 23.58% [1165500/4942000] [235.8/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 08:29:09,218 - Train: 23.59% [1165600/4942000] [235.9/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 08:29:41,876 - Train: 23.59% [1165700/4942000] [235.9/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 08:30:14,525 - Train: 23.59% [1165800/4942000] [235.9/1000.0] [batch_t 0.322 (0.326)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 08:30:47,275 - Train: 23.59% [1165900/4942000] [235.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 08:31:20,072 - Train: 23.59% [1166000/4942000] [235.9/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 08:31:52,757 - Train: 23.60% [1166100/4942000] [236.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 08:32:25,451 - Train: 23.60% [1166200/4942000] [236.0/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 08:32:58,106 - Train: 23.60% [1166300/4942000] [236.0/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 08:33:02,035 - ==> Total time: 6 days, 14:35:41 Eta: 21 days, 9:25:01 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 08:33:34,421 - Train: 23.60% [1166400/4942000] [236.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 08:34:07,074 - Train: 23.60% [1166500/4942000] [236.0/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 08:34:39,770 - Train: 23.61% [1166600/4942000] [236.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 08:35:12,499 - Train: 23.61% [1166700/4942000] [236.1/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 08:35:45,167 - Train: 23.61% [1166800/4942000] [236.1/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 08:36:17,859 - Train: 23.61% [1166900/4942000] [236.1/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 08:36:50,484 - Train: 23.61% [1167000/4942000] [236.1/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 08:37:23,090 - Train: 23.62% [1167100/4942000] [236.2/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 08:37:55,797 - Train: 23.62% [1167200/4942000] [236.2/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 08:38:28,557 - Train: 23.62% [1167300/4942000] [236.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 08:39:01,179 - Train: 23.62% [1167400/4942000] [236.2/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 08:39:33,871 - Train: 23.62% [1167500/4942000] [236.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 08:40:06,583 - Train: 23.63% [1167600/4942000] [236.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 08:40:39,333 - Train: 23.63% [1167700/4942000] [236.3/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 08:41:12,051 - Train: 23.63% [1167800/4942000] [236.3/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 08:41:44,691 - Train: 23.63% [1167900/4942000] [236.3/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 08:42:17,324 - Train: 23.63% [1168000/4942000] [236.3/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 08:42:50,065 - Train: 23.64% [1168100/4942000] [236.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 08:43:22,727 - Train: 23.64% [1168200/4942000] [236.4/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 08:43:55,338 - Train: 23.64% [1168300/4942000] [236.4/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 08:44:27,898 - Train: 23.64% [1168400/4942000] [236.4/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 08:45:00,567 - Train: 23.64% [1168500/4942000] [236.4/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 08:45:33,206 - Train: 23.65% [1168600/4942000] [236.5/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 08:46:05,891 - Train: 23.65% [1168700/4942000] [236.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 08:46:38,654 - Train: 23.65% [1168800/4942000] [236.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 08:47:11,407 - Train: 23.65% [1168900/4942000] [236.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 08:47:44,084 - Train: 23.65% [1169000/4942000] [236.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 08:48:16,692 - Train: 23.66% [1169100/4942000] [236.6/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 08:48:49,297 - Train: 23.66% [1169200/4942000] [236.6/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 08:49:22,125 - Train: 23.66% [1169300/4942000] [236.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 08:49:54,735 - Train: 23.66% [1169400/4942000] [236.6/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 08:50:27,405 - Train: 23.66% [1169500/4942000] [236.6/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 08:51:00,104 - Train: 23.67% [1169600/4942000] [236.7/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 08:51:32,761 - Train: 23.67% [1169700/4942000] [236.7/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 08:52:05,485 - Train: 23.67% [1169800/4942000] [236.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 08:52:38,110 - Train: 23.67% [1169900/4942000] [236.7/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 08:53:10,743 - Train: 23.67% [1170000/4942000] [236.7/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 08:53:43,576 - Train: 23.68% [1170100/4942000] [236.8/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 08:54:16,398 - Train: 23.68% [1170200/4942000] [236.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 08:54:49,056 - Train: 23.68% [1170300/4942000] [236.8/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 08:55:21,662 - Train: 23.68% [1170400/4942000] [236.8/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 08:55:54,249 - Train: 23.68% [1170500/4942000] [236.8/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 08:56:26,987 - Train: 23.69% [1170600/4942000] [236.9/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 08:56:59,614 - Train: 23.69% [1170700/4942000] [236.9/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 08:57:32,273 - Train: 23.69% [1170800/4942000] [236.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 08:58:04,898 - Train: 23.69% [1170900/4942000] [236.9/1000.0] [batch_t 0.321 (0.326)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-09 08:58:37,676 - Train: 23.69% [1171000/4942000] [236.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-09 08:59:10,281 - Train: 23.70% [1171100/4942000] [237.0/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 08:59:42,966 - Train: 23.70% [1171200/4942000] [237.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 09:00:00,757 - ==> Total time: 6 days, 15:02:39 Eta: 21 days, 8:01:44 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 09:00:19,303 - Train: 23.70% [1171300/4942000] [237.0/1000.0] [batch_t 0.321 (0.354)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-09 09:00:51,977 - Train: 23.70% [1171400/4942000] [237.0/1000.0] [batch_t 0.321 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 09:01:24,978 - Train: 23.70% [1171500/4942000] [237.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 09:01:57,702 - Train: 23.71% [1171600/4942000] [237.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 09:02:30,433 - Train: 23.71% [1171700/4942000] [237.1/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 09:03:03,101 - Train: 23.71% [1171800/4942000] [237.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 09:03:35,814 - Train: 23.71% [1171900/4942000] [237.1/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 09:04:08,497 - Train: 23.72% [1172000/4942000] [237.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 09:04:41,224 - Train: 23.72% [1172100/4942000] [237.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 09:05:14,043 - Train: 23.72% [1172200/4942000] [237.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 09:05:46,713 - Train: 23.72% [1172300/4942000] [237.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 09:06:19,341 - Train: 23.72% [1172400/4942000] [237.2/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 09:06:51,943 - Train: 23.73% [1172500/4942000] [237.3/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 09:07:25,098 - Train: 23.73% [1172600/4942000] [237.3/1000.0] [batch_t 0.321 (0.331)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-09 09:07:57,796 - Train: 23.73% [1172700/4942000] [237.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 09:08:30,462 - Train: 23.73% [1172800/4942000] [237.3/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 09:09:03,189 - Train: 23.73% [1172900/4942000] [237.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 09:09:35,813 - Train: 23.74% [1173000/4942000] [237.4/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 09:10:08,522 - Train: 23.74% [1173100/4942000] [237.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 09:10:41,121 - Train: 23.74% [1173200/4942000] [237.4/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 09:11:13,673 - Train: 23.74% [1173300/4942000] [237.4/1000.0] [batch_t 0.325 (0.325)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 09:11:46,302 - Train: 23.74% [1173400/4942000] [237.4/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 09:12:18,898 - Train: 23.75% [1173500/4942000] [237.5/1000.0] [batch_t 0.321 (0.326)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-09 09:12:51,586 - Train: 23.75% [1173600/4942000] [237.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 09:13:24,241 - Train: 23.75% [1173700/4942000] [237.5/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 09:13:56,821 - Train: 23.75% [1173800/4942000] [237.5/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 09:14:29,780 - Train: 23.75% [1173900/4942000] [237.5/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 09:15:02,396 - Train: 23.76% [1174000/4942000] [237.6/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 09:15:35,036 - Train: 23.76% [1174100/4942000] [237.6/1000.0] [batch_t 0.334 (0.326)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 09:16:07,685 - Train: 23.76% [1174200/4942000] [237.6/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 09:16:40,324 - Train: 23.76% [1174300/4942000] [237.6/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 09:17:13,032 - Train: 23.76% [1174400/4942000] [237.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 09:17:45,623 - Train: 23.77% [1174500/4942000] [237.7/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 09:18:18,292 - Train: 23.77% [1174600/4942000] [237.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 09:18:50,962 - Train: 23.77% [1174700/4942000] [237.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 09:19:23,578 - Train: 23.77% [1174800/4942000] [237.7/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 09:19:56,161 - Train: 23.77% [1174900/4942000] [237.7/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 09:20:28,753 - Train: 23.78% [1175000/4942000] [237.8/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 09:21:01,482 - Train: 23.78% [1175100/4942000] [237.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 09:21:34,073 - Train: 23.78% [1175200/4942000] [237.8/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 09:22:06,689 - Train: 23.78% [1175300/4942000] [237.8/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 09:22:39,305 - Train: 23.78% [1175400/4942000] [237.8/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 09:23:11,917 - Train: 23.79% [1175500/4942000] [237.9/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 09:23:44,581 - Train: 23.79% [1175600/4942000] [237.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 09:24:17,205 - Train: 23.79% [1175700/4942000] [237.9/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 09:24:50,057 - Train: 23.79% [1175800/4942000] [237.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 09:25:22,684 - Train: 23.79% [1175900/4942000] [237.9/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 09:25:55,282 - Train: 23.80% [1176000/4942000] [238.0/1000.0] [batch_t 0.332 (0.326)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 09:26:27,876 - Train: 23.80% [1176100/4942000] [238.0/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 09:26:59,223 - ==> Total time: 6 days, 15:29:38 Eta: 21 days, 6:38:55 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 09:27:02,548 - Train: 23.80% [1176200/4942000] [238.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 09:27:35,200 - Train: 23.80% [1176300/4942000] [238.0/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 09:28:07,886 - Train: 23.80% [1176400/4942000] [238.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 09:28:40,562 - Train: 23.81% [1176500/4942000] [238.1/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 09:29:13,353 - Train: 23.81% [1176600/4942000] [238.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 09:29:46,048 - Train: 23.81% [1176700/4942000] [238.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 09:30:18,775 - Train: 23.81% [1176800/4942000] [238.1/1000.0] [batch_t 0.335 (0.327)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-09 09:30:51,530 - Train: 23.81% [1176900/4942000] [238.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 09:31:24,274 - Train: 23.82% [1177000/4942000] [238.2/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 09:31:57,114 - Train: 23.82% [1177100/4942000] [238.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 09:32:29,802 - Train: 23.82% [1177200/4942000] [238.2/1000.0] [batch_t 0.321 (0.327)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-09 09:33:02,494 - Train: 23.82% [1177300/4942000] [238.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 09:33:35,194 - Train: 23.82% [1177400/4942000] [238.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 09:34:07,791 - Train: 23.83% [1177500/4942000] [238.3/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 09:34:40,487 - Train: 23.83% [1177600/4942000] [238.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 09:35:13,171 - Train: 23.83% [1177700/4942000] [238.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 09:35:45,829 - Train: 23.83% [1177800/4942000] [238.3/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 09:36:18,473 - Train: 23.83% [1177900/4942000] [238.3/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 09:36:51,189 - Train: 23.84% [1178000/4942000] [238.4/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 09:37:23,818 - Train: 23.84% [1178100/4942000] [238.4/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 09:37:56,453 - Train: 23.84% [1178200/4942000] [238.4/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 09:38:29,047 - Train: 23.84% [1178300/4942000] [238.4/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 09:39:01,624 - Train: 23.84% [1178400/4942000] [238.4/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 09:39:34,394 - Train: 23.85% [1178500/4942000] [238.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 09:40:07,088 - Train: 23.85% [1178600/4942000] [238.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 09:40:39,680 - Train: 23.85% [1178700/4942000] [238.5/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 09:41:12,292 - Train: 23.85% [1178800/4942000] [238.5/1000.0] [batch_t 0.333 (0.326)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 09:41:45,068 - Train: 23.85% [1178900/4942000] [238.5/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 09:42:17,861 - Train: 23.86% [1179000/4942000] [238.6/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 09:42:50,625 - Train: 23.86% [1179100/4942000] [238.6/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 09:43:23,362 - Train: 23.86% [1179200/4942000] [238.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 09:43:56,188 - Train: 23.86% [1179300/4942000] [238.6/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 09:44:28,902 - Train: 23.86% [1179400/4942000] [238.6/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 09:45:01,645 - Train: 23.87% [1179500/4942000] [238.7/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 09:45:34,349 - Train: 23.87% [1179600/4942000] [238.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 09:46:07,122 - Train: 23.87% [1179700/4942000] [238.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 09:46:39,838 - Train: 23.87% [1179800/4942000] [238.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 09:47:12,784 - Train: 23.87% [1179900/4942000] [238.7/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 09:47:45,626 - Train: 23.88% [1180000/4942000] [238.8/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-09 09:48:18,378 - Train: 23.88% [1180100/4942000] [238.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 09:48:51,168 - Train: 23.88% [1180200/4942000] [238.8/1000.0] [batch_t 0.338 (0.328)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-09 09:49:24,005 - Train: 23.88% [1180300/4942000] [238.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 09:49:56,667 - Train: 23.89% [1180400/4942000] [238.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 09:50:29,369 - Train: 23.89% [1180500/4942000] [238.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 09:51:02,109 - Train: 23.89% [1180600/4942000] [238.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 09:51:34,776 - Train: 23.89% [1180700/4942000] [238.9/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 09:52:07,468 - Train: 23.89% [1180800/4942000] [238.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 09:52:40,191 - Train: 23.90% [1180900/4942000] [239.0/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 09:53:12,882 - Train: 23.90% [1181000/4942000] [239.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 09:53:45,605 - Train: 23.90% [1181100/4942000] [239.0/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 09:53:58,027 - ==> Total time: 6 days, 15:56:37 Eta: 21 days, 5:16:36 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 09:54:20,349 - Train: 23.90% [1181200/4942000] [239.0/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 09:54:53,223 - Train: 23.90% [1181300/4942000] [239.0/1000.0] [batch_t 0.322 (0.329)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 09:55:25,808 - Train: 23.91% [1181400/4942000] [239.1/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 09:55:58,445 - Train: 23.91% [1181500/4942000] [239.1/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 09:56:31,103 - Train: 23.91% [1181600/4942000] [239.1/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 09:57:03,683 - Train: 23.91% [1181700/4942000] [239.1/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 09:57:36,303 - Train: 23.91% [1181800/4942000] [239.1/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 09:58:08,879 - Train: 23.92% [1181900/4942000] [239.2/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 09:58:41,576 - Train: 23.92% [1182000/4942000] [239.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 09:59:14,153 - Train: 23.92% [1182100/4942000] [239.2/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 09:59:46,800 - Train: 23.92% [1182200/4942000] [239.2/1000.0] [batch_t 0.322 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 10:00:19,483 - Train: 23.92% [1182300/4942000] [239.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 10:00:52,270 - Train: 23.93% [1182400/4942000] [239.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 10:01:24,950 - Train: 23.93% [1182500/4942000] [239.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 10:01:57,745 - Train: 23.93% [1182600/4942000] [239.3/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 10:02:30,595 - Train: 23.93% [1182700/4942000] [239.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 10:03:03,296 - Train: 23.93% [1182800/4942000] [239.3/1000.0] [batch_t 0.320 (0.327)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-09 10:03:35,904 - Train: 23.94% [1182900/4942000] [239.4/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 10:04:08,501 - Train: 23.94% [1183000/4942000] [239.4/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 10:04:41,166 - Train: 23.94% [1183100/4942000] [239.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 10:05:13,801 - Train: 23.94% [1183200/4942000] [239.4/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 10:05:46,429 - Train: 23.94% [1183300/4942000] [239.4/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 10:06:19,082 - Train: 23.95% [1183400/4942000] [239.5/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 10:06:51,766 - Train: 23.95% [1183500/4942000] [239.5/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 10:07:24,436 - Train: 23.95% [1183600/4942000] [239.5/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 10:07:57,429 - Train: 23.95% [1183700/4942000] [239.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 10:08:30,199 - Train: 23.95% [1183800/4942000] [239.5/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 10:09:02,833 - Train: 23.96% [1183900/4942000] [239.6/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 10:09:35,512 - Train: 23.96% [1184000/4942000] [239.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 10:10:08,246 - Train: 23.96% [1184100/4942000] [239.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 10:10:40,869 - Train: 23.96% [1184200/4942000] [239.6/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 10:11:13,593 - Train: 23.96% [1184300/4942000] [239.6/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 10:11:46,361 - Train: 23.97% [1184400/4942000] [239.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 10:12:18,979 - Train: 23.97% [1184500/4942000] [239.7/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 10:12:51,651 - Train: 23.97% [1184600/4942000] [239.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 10:13:24,388 - Train: 23.97% [1184700/4942000] [239.7/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 10:13:57,055 - Train: 23.97% [1184800/4942000] [239.7/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 10:14:29,658 - Train: 23.98% [1184900/4942000] [239.8/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 10:15:02,367 - Train: 23.98% [1185000/4942000] [239.8/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 10:15:35,030 - Train: 23.98% [1185100/4942000] [239.8/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 10:16:07,697 - Train: 23.98% [1185200/4942000] [239.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 10:16:40,440 - Train: 23.98% [1185300/4942000] [239.8/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 10:17:13,797 - Train: 23.99% [1185400/4942000] [239.9/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 10:17:46,501 - Train: 23.99% [1185500/4942000] [239.9/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 10:18:19,303 - Train: 23.99% [1185600/4942000] [239.9/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 10:18:52,068 - Train: 23.99% [1185700/4942000] [239.9/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 10:19:24,669 - Train: 23.99% [1185800/4942000] [239.9/1000.0] [batch_t 0.322 (0.326)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 10:19:57,329 - Train: 24.00% [1185900/4942000] [240.0/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 10:20:29,971 - Train: 24.00% [1186000/4942000] [240.0/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 10:20:56,071 - ==> Total time: 6 days, 16:23:35 Eta: 21 days, 3:54:41 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 10:21:05,857 - Train: 24.00% [1186100/4942000] [240.0/1000.0] [batch_t 0.906 (0.375)] [data_t 0.579] [optim_t 0.327] [lr 0.005000] 2024-04-09 10:21:38,611 - Train: 24.00% [1186200/4942000] [240.0/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 10:22:11,283 - Train: 24.00% [1186300/4942000] [240.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 10:22:43,897 - Train: 24.01% [1186400/4942000] [240.1/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 10:23:16,511 - Train: 24.01% [1186500/4942000] [240.1/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 10:23:49,177 - Train: 24.01% [1186600/4942000] [240.1/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 10:24:21,847 - Train: 24.01% [1186700/4942000] [240.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 10:24:54,471 - Train: 24.01% [1186800/4942000] [240.1/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 10:25:27,292 - Train: 24.02% [1186900/4942000] [240.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 10:25:59,960 - Train: 24.02% [1187000/4942000] [240.2/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 10:26:32,638 - Train: 24.02% [1187100/4942000] [240.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 10:27:05,337 - Train: 24.02% [1187200/4942000] [240.2/1000.0] [batch_t 0.334 (0.327)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 10:27:38,005 - Train: 24.02% [1187300/4942000] [240.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 10:28:11,313 - Train: 24.03% [1187400/4942000] [240.3/1000.0] [batch_t 0.325 (0.333)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 10:28:44,033 - Train: 24.03% [1187500/4942000] [240.3/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 10:29:16,653 - Train: 24.03% [1187600/4942000] [240.3/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 10:29:49,299 - Train: 24.03% [1187700/4942000] [240.3/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 10:30:21,945 - Train: 24.03% [1187800/4942000] [240.3/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 10:30:54,718 - Train: 24.04% [1187900/4942000] [240.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 10:31:27,388 - Train: 24.04% [1188000/4942000] [240.4/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 10:32:00,076 - Train: 24.04% [1188100/4942000] [240.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 10:32:32,761 - Train: 24.04% [1188200/4942000] [240.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 10:33:05,344 - Train: 24.04% [1188300/4942000] [240.4/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 10:33:38,146 - Train: 24.05% [1188400/4942000] [240.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 10:34:10,912 - Train: 24.05% [1188500/4942000] [240.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 10:34:43,572 - Train: 24.05% [1188600/4942000] [240.5/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 10:35:16,277 - Train: 24.05% [1188700/4942000] [240.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 10:35:48,905 - Train: 24.06% [1188800/4942000] [240.6/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 10:36:21,651 - Train: 24.06% [1188900/4942000] [240.6/1000.0] [batch_t 0.334 (0.327)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 10:36:54,370 - Train: 24.06% [1189000/4942000] [240.6/1000.0] [batch_t 0.334 (0.327)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 10:37:27,115 - Train: 24.06% [1189100/4942000] [240.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 10:37:59,741 - Train: 24.06% [1189200/4942000] [240.6/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 10:38:32,385 - Train: 24.07% [1189300/4942000] [240.7/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 10:39:05,071 - Train: 24.07% [1189400/4942000] [240.7/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 10:39:37,807 - Train: 24.07% [1189500/4942000] [240.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 10:40:10,498 - Train: 24.07% [1189600/4942000] [240.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 10:40:43,202 - Train: 24.07% [1189700/4942000] [240.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 10:41:16,062 - Train: 24.08% [1189800/4942000] [240.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 10:41:48,796 - Train: 24.08% [1189900/4942000] [240.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 10:42:21,457 - Train: 24.08% [1190000/4942000] [240.8/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 10:42:54,083 - Train: 24.08% [1190100/4942000] [240.8/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 10:43:26,689 - Train: 24.08% [1190200/4942000] [240.8/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 10:43:59,343 - Train: 24.09% [1190300/4942000] [240.9/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 10:44:32,017 - Train: 24.09% [1190400/4942000] [240.9/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 10:45:04,669 - Train: 24.09% [1190500/4942000] [240.9/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 10:45:37,381 - Train: 24.09% [1190600/4942000] [240.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 10:46:09,969 - Train: 24.09% [1190700/4942000] [240.9/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 10:46:42,627 - Train: 24.10% [1190800/4942000] [241.0/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 10:47:15,226 - Train: 24.10% [1190900/4942000] [241.0/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 10:47:47,903 - Train: 24.10% [1191000/4942000] [241.0/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 10:47:55,074 - ==> Total time: 6 days, 16:50:34 Eta: 21 days, 2:33:17 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 10:48:22,769 - Train: 24.10% [1191100/4942000] [241.0/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 10:48:55,418 - Train: 24.10% [1191200/4942000] [241.0/1000.0] [batch_t 0.321 (0.326)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-09 10:49:28,080 - Train: 24.11% [1191300/4942000] [241.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 10:50:00,819 - Train: 24.11% [1191400/4942000] [241.1/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 10:50:33,506 - Train: 24.11% [1191500/4942000] [241.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 10:51:06,125 - Train: 24.11% [1191600/4942000] [241.1/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 10:51:38,846 - Train: 24.11% [1191700/4942000] [241.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 10:52:11,488 - Train: 24.12% [1191800/4942000] [241.2/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 10:52:44,107 - Train: 24.12% [1191900/4942000] [241.2/1000.0] [batch_t 0.322 (0.326)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 10:53:16,805 - Train: 24.12% [1192000/4942000] [241.2/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 10:53:49,532 - Train: 24.12% [1192100/4942000] [241.2/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 10:54:22,469 - Train: 24.12% [1192200/4942000] [241.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 10:54:55,207 - Train: 24.13% [1192300/4942000] [241.3/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 10:55:28,003 - Train: 24.13% [1192400/4942000] [241.3/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 10:56:00,764 - Train: 24.13% [1192500/4942000] [241.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 10:56:33,877 - Train: 24.13% [1192600/4942000] [241.3/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 10:57:06,568 - Train: 24.13% [1192700/4942000] [241.3/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 10:57:39,351 - Train: 24.14% [1192800/4942000] [241.4/1000.0] [batch_t 0.336 (0.328)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-09 10:58:12,105 - Train: 24.14% [1192900/4942000] [241.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 10:58:44,870 - Train: 24.14% [1193000/4942000] [241.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 10:59:17,617 - Train: 24.14% [1193100/4942000] [241.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 10:59:50,234 - Train: 24.14% [1193200/4942000] [241.4/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 11:00:23,014 - Train: 24.15% [1193300/4942000] [241.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 11:00:55,777 - Train: 24.15% [1193400/4942000] [241.5/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 11:01:28,479 - Train: 24.15% [1193500/4942000] [241.5/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 11:02:01,268 - Train: 24.15% [1193600/4942000] [241.5/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 11:02:33,865 - Train: 24.15% [1193700/4942000] [241.5/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 11:03:06,484 - Train: 24.16% [1193800/4942000] [241.6/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 11:03:39,101 - Train: 24.16% [1193900/4942000] [241.6/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 11:04:11,864 - Train: 24.16% [1194000/4942000] [241.6/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 11:04:44,447 - Train: 24.16% [1194100/4942000] [241.6/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 11:05:17,067 - Train: 24.16% [1194200/4942000] [241.6/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 11:05:49,912 - Train: 24.17% [1194300/4942000] [241.7/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 11:06:22,598 - Train: 24.17% [1194400/4942000] [241.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 11:06:55,309 - Train: 24.17% [1194500/4942000] [241.7/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 11:07:28,001 - Train: 24.17% [1194600/4942000] [241.7/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 11:08:00,800 - Train: 24.17% [1194700/4942000] [241.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 11:08:33,464 - Train: 24.18% [1194800/4942000] [241.8/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 11:09:06,107 - Train: 24.18% [1194900/4942000] [241.8/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 11:09:38,735 - Train: 24.18% [1195000/4942000] [241.8/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 11:10:11,383 - Train: 24.18% [1195100/4942000] [241.8/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 11:10:44,037 - Train: 24.18% [1195200/4942000] [241.8/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 11:11:16,723 - Train: 24.19% [1195300/4942000] [241.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 11:11:49,505 - Train: 24.19% [1195400/4942000] [241.9/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 11:12:22,295 - Train: 24.19% [1195500/4942000] [241.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 11:12:54,965 - Train: 24.19% [1195600/4942000] [241.9/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 11:13:27,649 - Train: 24.19% [1195700/4942000] [241.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 11:14:00,264 - Train: 24.20% [1195800/4942000] [242.0/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 11:14:32,955 - Train: 24.20% [1195900/4942000] [242.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 11:14:53,848 - ==> Total time: 6 days, 17:17:33 Eta: 21 days, 1:12:19 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 11:15:07,585 - Train: 24.20% [1196000/4942000] [242.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 11:15:40,168 - Train: 24.20% [1196100/4942000] [242.0/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 11:16:12,786 - Train: 24.20% [1196200/4942000] [242.0/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 11:16:45,455 - Train: 24.21% [1196300/4942000] [242.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 11:17:18,140 - Train: 24.21% [1196400/4942000] [242.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 11:17:50,798 - Train: 24.21% [1196500/4942000] [242.1/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 11:18:23,557 - Train: 24.21% [1196600/4942000] [242.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 11:18:56,373 - Train: 24.21% [1196700/4942000] [242.1/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-09 11:19:29,064 - Train: 24.22% [1196800/4942000] [242.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 11:20:01,739 - Train: 24.22% [1196900/4942000] [242.2/1000.0] [batch_t 0.334 (0.327)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 11:20:34,375 - Train: 24.22% [1197000/4942000] [242.2/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 11:21:07,055 - Train: 24.22% [1197100/4942000] [242.2/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 11:21:39,637 - Train: 24.23% [1197200/4942000] [242.3/1000.0] [batch_t 0.321 (0.326)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-09 11:22:12,376 - Train: 24.23% [1197300/4942000] [242.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 11:22:45,128 - Train: 24.23% [1197400/4942000] [242.3/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 11:23:17,866 - Train: 24.23% [1197500/4942000] [242.3/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 11:23:50,590 - Train: 24.23% [1197600/4942000] [242.3/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 11:24:23,204 - Train: 24.24% [1197700/4942000] [242.4/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 11:24:55,857 - Train: 24.24% [1197800/4942000] [242.4/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 11:25:28,556 - Train: 24.24% [1197900/4942000] [242.4/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 11:26:01,242 - Train: 24.24% [1198000/4942000] [242.4/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 11:26:33,985 - Train: 24.24% [1198100/4942000] [242.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 11:27:06,731 - Train: 24.25% [1198200/4942000] [242.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 11:27:39,353 - Train: 24.25% [1198300/4942000] [242.5/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 11:28:12,035 - Train: 24.25% [1198400/4942000] [242.5/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 11:28:44,698 - Train: 24.25% [1198500/4942000] [242.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 11:29:17,373 - Train: 24.25% [1198600/4942000] [242.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 11:29:50,071 - Train: 24.26% [1198700/4942000] [242.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 11:30:22,803 - Train: 24.26% [1198800/4942000] [242.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 11:30:55,512 - Train: 24.26% [1198900/4942000] [242.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 11:31:28,288 - Train: 24.26% [1199000/4942000] [242.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 11:32:00,929 - Train: 24.26% [1199100/4942000] [242.6/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 11:32:33,583 - Train: 24.27% [1199200/4942000] [242.7/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 11:33:06,256 - Train: 24.27% [1199300/4942000] [242.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 11:33:38,882 - Train: 24.27% [1199400/4942000] [242.7/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 11:34:11,475 - Train: 24.27% [1199500/4942000] [242.7/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 11:34:44,322 - Train: 24.27% [1199600/4942000] [242.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 11:35:16,944 - Train: 24.28% [1199700/4942000] [242.8/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 11:35:49,674 - Train: 24.28% [1199800/4942000] [242.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 11:36:22,388 - Train: 24.28% [1199900/4942000] [242.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 11:36:54,991 - Train: 24.28% [1200000/4942000] [242.8/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 11:37:27,660 - Train: 24.28% [1200100/4942000] [242.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 11:38:00,373 - Train: 24.29% [1200200/4942000] [242.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 11:38:33,015 - Train: 24.29% [1200300/4942000] [242.9/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 11:39:05,616 - Train: 24.29% [1200400/4942000] [242.9/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 11:39:38,295 - Train: 24.29% [1200500/4942000] [242.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-09 11:40:11,016 - Train: 24.29% [1200600/4942000] [242.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 11:40:43,677 - Train: 24.30% [1200700/4942000] [243.0/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 11:41:16,383 - Train: 24.30% [1200800/4942000] [243.0/1000.0] [batch_t 0.321 (0.327)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-09 11:41:49,084 - Train: 24.30% [1200900/4942000] [243.0/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 11:41:51,043 - ==> Total time: 6 days, 17:44:30 Eta: 20 days, 23:51:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 11:42:23,973 - Train: 24.30% [1201000/4942000] [243.0/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 11:42:56,550 - Train: 24.30% [1201100/4942000] [243.0/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 11:43:29,424 - Train: 24.31% [1201200/4942000] [243.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 11:44:02,195 - Train: 24.31% [1201300/4942000] [243.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 11:44:35,194 - Train: 24.31% [1201400/4942000] [243.1/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 11:45:08,270 - Train: 24.31% [1201500/4942000] [243.1/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 11:45:40,836 - Train: 24.31% [1201600/4942000] [243.1/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 11:46:15,539 - Train: 24.32% [1201700/4942000] [243.2/1000.0] [batch_t 0.327 (0.347)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 11:46:48,185 - Train: 24.32% [1201800/4942000] [243.2/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 11:47:21,553 - Train: 24.32% [1201900/4942000] [243.2/1000.0] [batch_t 0.326 (0.334)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 11:47:54,237 - Train: 24.32% [1202000/4942000] [243.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 11:48:30,026 - Train: 24.32% [1202100/4942000] [243.2/1000.0] [batch_t 0.329 (0.358)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 11:49:02,680 - Train: 24.33% [1202200/4942000] [243.3/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 11:49:36,312 - Train: 24.33% [1202300/4942000] [243.3/1000.0] [batch_t 0.325 (0.336)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 11:50:10,040 - Train: 24.33% [1202400/4942000] [243.3/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 11:50:42,646 - Train: 24.33% [1202500/4942000] [243.3/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 11:51:17,638 - Train: 24.33% [1202600/4942000] [243.3/1000.0] [batch_t 0.328 (0.350)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 11:51:50,236 - Train: 24.34% [1202700/4942000] [243.4/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 11:52:23,092 - Train: 24.34% [1202800/4942000] [243.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 11:52:55,885 - Train: 24.34% [1202900/4942000] [243.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 11:53:31,131 - Train: 24.34% [1203000/4942000] [243.4/1000.0] [batch_t 0.878 (0.352)] [data_t 0.551] [optim_t 0.327] [lr 0.005000] 2024-04-09 11:54:05,068 - Train: 24.34% [1203100/4942000] [243.4/1000.0] [batch_t 0.322 (0.339)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 11:54:37,687 - Train: 24.35% [1203200/4942000] [243.5/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 11:55:15,272 - Train: 24.35% [1203300/4942000] [243.5/1000.0] [batch_t 0.328 (0.376)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 11:55:47,979 - Train: 24.35% [1203400/4942000] [243.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 11:56:21,085 - Train: 24.35% [1203500/4942000] [243.5/1000.0] [batch_t 0.324 (0.331)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 11:56:53,736 - Train: 24.35% [1203600/4942000] [243.5/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 11:57:27,366 - Train: 24.36% [1203700/4942000] [243.6/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 11:58:02,915 - Train: 24.36% [1203800/4942000] [243.6/1000.0] [batch_t 0.327 (0.355)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 11:58:36,047 - Train: 24.36% [1203900/4942000] [243.6/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 11:59:09,428 - Train: 24.36% [1204000/4942000] [243.6/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 11:59:42,011 - Train: 24.36% [1204100/4942000] [243.6/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 12:00:14,639 - Train: 24.37% [1204200/4942000] [243.7/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 12:00:53,702 - Train: 24.37% [1204300/4942000] [243.7/1000.0] [batch_t 0.326 (0.391)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 12:01:26,348 - Train: 24.37% [1204400/4942000] [243.7/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 12:01:58,959 - Train: 24.37% [1204500/4942000] [243.7/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 12:02:31,598 - Train: 24.37% [1204600/4942000] [243.7/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 12:03:04,228 - Train: 24.38% [1204700/4942000] [243.8/1000.0] [batch_t 0.318 (0.326)] [data_t 0.002] [optim_t 0.316] [lr 0.005000] 2024-04-09 12:03:36,997 - Train: 24.38% [1204800/4942000] [243.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 12:04:09,608 - Train: 24.38% [1204900/4942000] [243.8/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 12:04:42,222 - Train: 24.38% [1205000/4942000] [243.8/1000.0] [batch_t 0.322 (0.326)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 12:05:14,833 - Train: 24.38% [1205100/4942000] [243.8/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 12:05:47,423 - Train: 24.39% [1205200/4942000] [243.9/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 12:06:20,173 - Train: 24.39% [1205300/4942000] [243.9/1000.0] [batch_t 0.335 (0.327)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-09 12:06:52,823 - Train: 24.39% [1205400/4942000] [243.9/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 12:07:25,578 - Train: 24.39% [1205500/4942000] [243.9/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 12:07:58,262 - Train: 24.39% [1205600/4942000] [243.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 12:08:31,186 - Train: 24.40% [1205700/4942000] [244.0/1000.0] [batch_t 0.336 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-09 12:09:03,893 - Train: 24.40% [1205800/4942000] [244.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 12:09:19,634 - ==> Total time: 6 days, 18:11:58 Eta: 20 days, 22:33:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 12:09:38,766 - Train: 24.40% [1205900/4942000] [244.0/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 12:10:11,526 - Train: 24.40% [1206000/4942000] [244.0/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 12:10:44,204 - Train: 24.41% [1206100/4942000] [244.1/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 12:11:16,892 - Train: 24.41% [1206200/4942000] [244.1/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 12:11:49,624 - Train: 24.41% [1206300/4942000] [244.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 12:12:22,298 - Train: 24.41% [1206400/4942000] [244.1/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 12:12:54,979 - Train: 24.41% [1206500/4942000] [244.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 12:13:27,839 - Train: 24.42% [1206600/4942000] [244.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 12:14:00,604 - Train: 24.42% [1206700/4942000] [244.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 12:14:33,350 - Train: 24.42% [1206800/4942000] [244.2/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 12:15:05,993 - Train: 24.42% [1206900/4942000] [244.2/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 12:15:38,577 - Train: 24.42% [1207000/4942000] [244.2/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 12:16:11,214 - Train: 24.43% [1207100/4942000] [244.3/1000.0] [batch_t 0.320 (0.326)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-09 12:16:43,906 - Train: 24.43% [1207200/4942000] [244.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 12:17:16,586 - Train: 24.43% [1207300/4942000] [244.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 12:17:49,266 - Train: 24.43% [1207400/4942000] [244.3/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 12:18:22,024 - Train: 24.43% [1207500/4942000] [244.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 12:18:54,678 - Train: 24.44% [1207600/4942000] [244.4/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 12:19:27,448 - Train: 24.44% [1207700/4942000] [244.4/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 12:20:00,097 - Train: 24.44% [1207800/4942000] [244.4/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 12:20:32,713 - Train: 24.44% [1207900/4942000] [244.4/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 12:21:05,464 - Train: 24.44% [1208000/4942000] [244.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 12:21:38,085 - Train: 24.45% [1208100/4942000] [244.5/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 12:22:10,793 - Train: 24.45% [1208200/4942000] [244.5/1000.0] [batch_t 0.334 (0.327)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 12:22:43,490 - Train: 24.45% [1208300/4942000] [244.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 12:23:16,124 - Train: 24.45% [1208400/4942000] [244.5/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 12:23:48,778 - Train: 24.45% [1208500/4942000] [244.5/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 12:24:21,415 - Train: 24.46% [1208600/4942000] [244.6/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 12:24:54,036 - Train: 24.46% [1208700/4942000] [244.6/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 12:25:26,641 - Train: 24.46% [1208800/4942000] [244.6/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 12:25:59,238 - Train: 24.46% [1208900/4942000] [244.6/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 12:26:31,885 - Train: 24.46% [1209000/4942000] [244.6/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 12:27:04,604 - Train: 24.47% [1209100/4942000] [244.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 12:27:37,219 - Train: 24.47% [1209200/4942000] [244.7/1000.0] [batch_t 0.319 (0.326)] [data_t 0.002] [optim_t 0.317] [lr 0.005000] 2024-04-09 12:28:09,843 - Train: 24.47% [1209300/4942000] [244.7/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 12:28:42,502 - Train: 24.47% [1209400/4942000] [244.7/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 12:29:15,248 - Train: 24.47% [1209500/4942000] [244.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 12:29:47,907 - Train: 24.48% [1209600/4942000] [244.8/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 12:30:20,551 - Train: 24.48% [1209700/4942000] [244.8/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 12:30:53,166 - Train: 24.48% [1209800/4942000] [244.8/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 12:31:25,762 - Train: 24.48% [1209900/4942000] [244.8/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 12:31:58,409 - Train: 24.48% [1210000/4942000] [244.8/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 12:32:31,025 - Train: 24.49% [1210100/4942000] [244.9/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 12:33:03,704 - Train: 24.49% [1210200/4942000] [244.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 12:33:36,617 - Train: 24.49% [1210300/4942000] [244.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 12:34:09,430 - Train: 24.49% [1210400/4942000] [244.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 12:34:42,068 - Train: 24.49% [1210500/4942000] [244.9/1000.0] [batch_t 0.332 (0.326)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 12:35:14,784 - Train: 24.50% [1210600/4942000] [245.0/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 12:35:47,411 - Train: 24.50% [1210700/4942000] [245.0/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 12:36:16,928 - ==> Total time: 6 days, 18:38:56 Eta: 20 days, 21:13:27 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 12:36:22,206 - Train: 24.50% [1210800/4942000] [245.0/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 12:36:54,812 - Train: 24.50% [1210900/4942000] [245.0/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 12:37:27,449 - Train: 24.50% [1211000/4942000] [245.0/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 12:38:00,135 - Train: 24.51% [1211100/4942000] [245.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 12:38:32,756 - Train: 24.51% [1211200/4942000] [245.1/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 12:39:05,325 - Train: 24.51% [1211300/4942000] [245.1/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 12:39:37,993 - Train: 24.51% [1211400/4942000] [245.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 12:40:10,701 - Train: 24.51% [1211500/4942000] [245.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 12:40:43,395 - Train: 24.52% [1211600/4942000] [245.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 12:41:16,011 - Train: 24.52% [1211700/4942000] [245.2/1000.0] [batch_t 0.333 (0.326)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 12:41:48,658 - Train: 24.52% [1211800/4942000] [245.2/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 12:42:21,284 - Train: 24.52% [1211900/4942000] [245.2/1000.0] [batch_t 0.320 (0.326)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-09 12:42:53,911 - Train: 24.52% [1212000/4942000] [245.2/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 12:43:26,565 - Train: 24.53% [1212100/4942000] [245.3/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 12:43:59,198 - Train: 24.53% [1212200/4942000] [245.3/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 12:44:31,950 - Train: 24.53% [1212300/4942000] [245.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 12:45:04,696 - Train: 24.53% [1212400/4942000] [245.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 12:45:37,321 - Train: 24.53% [1212500/4942000] [245.3/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 12:46:10,041 - Train: 24.54% [1212600/4942000] [245.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 12:46:42,657 - Train: 24.54% [1212700/4942000] [245.4/1000.0] [batch_t 0.335 (0.326)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-09 12:47:15,261 - Train: 24.54% [1212800/4942000] [245.4/1000.0] [batch_t 0.322 (0.326)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 12:47:47,882 - Train: 24.54% [1212900/4942000] [245.4/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 12:48:20,544 - Train: 24.54% [1213000/4942000] [245.4/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 12:48:53,131 - Train: 24.55% [1213100/4942000] [245.5/1000.0] [batch_t 0.321 (0.326)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-09 12:49:25,782 - Train: 24.55% [1213200/4942000] [245.5/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 12:49:58,368 - Train: 24.55% [1213300/4942000] [245.5/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 12:50:30,963 - Train: 24.55% [1213400/4942000] [245.5/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 12:51:03,739 - Train: 24.55% [1213500/4942000] [245.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 12:51:36,325 - Train: 24.56% [1213600/4942000] [245.6/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 12:52:09,025 - Train: 24.56% [1213700/4942000] [245.6/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 12:52:41,640 - Train: 24.56% [1213800/4942000] [245.6/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 12:53:14,314 - Train: 24.56% [1213900/4942000] [245.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 12:53:46,972 - Train: 24.56% [1214000/4942000] [245.6/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 12:54:19,666 - Train: 24.57% [1214100/4942000] [245.7/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 12:54:52,375 - Train: 24.57% [1214200/4942000] [245.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 12:55:25,118 - Train: 24.57% [1214300/4942000] [245.7/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 12:55:57,850 - Train: 24.57% [1214400/4942000] [245.7/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 12:56:30,703 - Train: 24.58% [1214500/4942000] [245.8/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 12:57:03,490 - Train: 24.58% [1214600/4942000] [245.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 12:57:36,121 - Train: 24.58% [1214700/4942000] [245.8/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 12:58:08,756 - Train: 24.58% [1214800/4942000] [245.8/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 12:58:41,455 - Train: 24.58% [1214900/4942000] [245.8/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 12:59:14,121 - Train: 24.59% [1215000/4942000] [245.9/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 12:59:46,852 - Train: 24.59% [1215100/4942000] [245.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 13:00:19,495 - Train: 24.59% [1215200/4942000] [245.9/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 13:00:52,090 - Train: 24.59% [1215300/4942000] [245.9/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 13:01:24,677 - Train: 24.59% [1215400/4942000] [245.9/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 13:01:57,322 - Train: 24.60% [1215500/4942000] [246.0/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 13:02:29,977 - Train: 24.60% [1215600/4942000] [246.0/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 13:03:02,697 - Train: 24.60% [1215700/4942000] [246.0/1000.0] [batch_t 0.338 (0.327)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-09 13:03:13,148 - ==> Total time: 6 days, 19:05:52 Eta: 20 days, 19:54:05 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 13:03:37,455 - Train: 24.60% [1215800/4942000] [246.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 13:04:10,157 - Train: 24.60% [1215900/4942000] [246.0/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 13:04:43,005 - Train: 24.61% [1216000/4942000] [246.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 13:05:15,763 - Train: 24.61% [1216100/4942000] [246.1/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 13:05:48,468 - Train: 24.61% [1216200/4942000] [246.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 13:06:21,208 - Train: 24.61% [1216300/4942000] [246.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 13:06:53,911 - Train: 24.61% [1216400/4942000] [246.1/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 13:07:26,754 - Train: 24.62% [1216500/4942000] [246.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 13:07:59,434 - Train: 24.62% [1216600/4942000] [246.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 13:08:32,086 - Train: 24.62% [1216700/4942000] [246.2/1000.0] [batch_t 0.322 (0.326)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 13:09:04,849 - Train: 24.62% [1216800/4942000] [246.2/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 13:09:37,485 - Train: 24.62% [1216900/4942000] [246.2/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 13:10:10,242 - Train: 24.63% [1217000/4942000] [246.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 13:10:42,897 - Train: 24.63% [1217100/4942000] [246.3/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 13:11:15,583 - Train: 24.63% [1217200/4942000] [246.3/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 13:11:48,178 - Train: 24.63% [1217300/4942000] [246.3/1000.0] [batch_t 0.322 (0.326)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 13:12:20,820 - Train: 24.63% [1217400/4942000] [246.3/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 13:12:53,484 - Train: 24.64% [1217500/4942000] [246.4/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 13:13:26,126 - Train: 24.64% [1217600/4942000] [246.4/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 13:13:58,800 - Train: 24.64% [1217700/4942000] [246.4/1000.0] [batch_t 0.335 (0.327)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-09 13:14:31,514 - Train: 24.64% [1217800/4942000] [246.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 13:15:04,348 - Train: 24.64% [1217900/4942000] [246.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 13:15:37,139 - Train: 24.65% [1218000/4942000] [246.5/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 13:16:09,800 - Train: 24.65% [1218100/4942000] [246.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 13:16:42,464 - Train: 24.65% [1218200/4942000] [246.5/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 13:17:15,154 - Train: 24.65% [1218300/4942000] [246.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 13:17:47,804 - Train: 24.65% [1218400/4942000] [246.5/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 13:18:20,445 - Train: 24.66% [1218500/4942000] [246.6/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 13:18:53,065 - Train: 24.66% [1218600/4942000] [246.6/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 13:19:25,699 - Train: 24.66% [1218700/4942000] [246.6/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 13:19:58,337 - Train: 24.66% [1218800/4942000] [246.6/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 13:20:30,988 - Train: 24.66% [1218900/4942000] [246.6/1000.0] [batch_t 0.333 (0.326)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 13:21:03,727 - Train: 24.67% [1219000/4942000] [246.7/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 13:21:36,366 - Train: 24.67% [1219100/4942000] [246.7/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 13:22:09,049 - Train: 24.67% [1219200/4942000] [246.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 13:22:41,802 - Train: 24.67% [1219300/4942000] [246.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 13:23:14,464 - Train: 24.67% [1219400/4942000] [246.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 13:23:47,168 - Train: 24.68% [1219500/4942000] [246.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 13:24:19,791 - Train: 24.68% [1219600/4942000] [246.8/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 13:24:52,467 - Train: 24.68% [1219700/4942000] [246.8/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 13:25:25,081 - Train: 24.68% [1219800/4942000] [246.8/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 13:25:57,756 - Train: 24.68% [1219900/4942000] [246.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 13:26:30,446 - Train: 24.69% [1220000/4942000] [246.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 13:27:03,139 - Train: 24.69% [1220100/4942000] [246.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 13:27:35,820 - Train: 24.69% [1220200/4942000] [246.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 13:28:08,535 - Train: 24.69% [1220300/4942000] [246.9/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 13:28:41,150 - Train: 24.69% [1220400/4942000] [246.9/1000.0] [batch_t 0.322 (0.326)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 13:29:13,793 - Train: 24.70% [1220500/4942000] [247.0/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 13:29:46,488 - Train: 24.70% [1220600/4942000] [247.0/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 13:30:10,751 - ==> Total time: 6 days, 19:32:49 Eta: 20 days, 18:35:14 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 13:30:21,798 - Train: 24.70% [1220700/4942000] [247.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 13:30:54,473 - Train: 24.70% [1220800/4942000] [247.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 13:31:27,217 - Train: 24.70% [1220900/4942000] [247.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 13:31:59,886 - Train: 24.71% [1221000/4942000] [247.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 13:32:32,547 - Train: 24.71% [1221100/4942000] [247.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 13:33:05,333 - Train: 24.71% [1221200/4942000] [247.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 13:33:37,970 - Train: 24.71% [1221300/4942000] [247.1/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 13:34:10,621 - Train: 24.71% [1221400/4942000] [247.1/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 13:34:43,263 - Train: 24.72% [1221500/4942000] [247.2/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 13:35:15,902 - Train: 24.72% [1221600/4942000] [247.2/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 13:35:48,542 - Train: 24.72% [1221700/4942000] [247.2/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 13:36:21,252 - Train: 24.72% [1221800/4942000] [247.2/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 13:36:53,874 - Train: 24.72% [1221900/4942000] [247.2/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 13:37:26,481 - Train: 24.73% [1222000/4942000] [247.3/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 13:37:59,132 - Train: 24.73% [1222100/4942000] [247.3/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 13:38:31,927 - Train: 24.73% [1222200/4942000] [247.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 13:39:04,551 - Train: 24.73% [1222300/4942000] [247.3/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 13:39:37,173 - Train: 24.73% [1222400/4942000] [247.3/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 13:40:09,803 - Train: 24.74% [1222500/4942000] [247.4/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 13:40:42,467 - Train: 24.74% [1222600/4942000] [247.4/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 13:41:15,080 - Train: 24.74% [1222700/4942000] [247.4/1000.0] [batch_t 0.321 (0.326)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 13:41:47,693 - Train: 24.74% [1222800/4942000] [247.4/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 13:42:20,343 - Train: 24.75% [1222900/4942000] [247.5/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 13:42:53,023 - Train: 24.75% [1223000/4942000] [247.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 13:43:25,673 - Train: 24.75% [1223100/4942000] [247.5/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 13:43:58,306 - Train: 24.75% [1223200/4942000] [247.5/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 13:44:30,951 - Train: 24.75% [1223300/4942000] [247.5/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 13:45:03,584 - Train: 24.76% [1223400/4942000] [247.6/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 13:45:36,218 - Train: 24.76% [1223500/4942000] [247.6/1000.0] [batch_t 0.319 (0.326)] [data_t 0.002] [optim_t 0.317] [lr 0.005000] 2024-04-09 13:46:09,164 - Train: 24.76% [1223600/4942000] [247.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 13:46:41,835 - Train: 24.76% [1223700/4942000] [247.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 13:47:14,578 - Train: 24.76% [1223800/4942000] [247.6/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 13:47:47,184 - Train: 24.77% [1223900/4942000] [247.7/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 13:48:19,794 - Train: 24.77% [1224000/4942000] [247.7/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 13:48:52,437 - Train: 24.77% [1224100/4942000] [247.7/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 13:49:25,084 - Train: 24.77% [1224200/4942000] [247.7/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 13:49:57,748 - Train: 24.77% [1224300/4942000] [247.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 13:50:30,434 - Train: 24.78% [1224400/4942000] [247.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 13:51:03,091 - Train: 24.78% [1224500/4942000] [247.8/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 13:51:35,729 - Train: 24.78% [1224600/4942000] [247.8/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 13:52:08,348 - Train: 24.78% [1224700/4942000] [247.8/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 13:52:40,962 - Train: 24.78% [1224800/4942000] [247.8/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 13:53:13,638 - Train: 24.79% [1224900/4942000] [247.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 13:53:46,289 - Train: 24.79% [1225000/4942000] [247.9/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 13:54:19,130 - Train: 24.79% [1225100/4942000] [247.9/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 13:54:51,794 - Train: 24.79% [1225200/4942000] [247.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 13:55:24,489 - Train: 24.79% [1225300/4942000] [247.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 13:55:57,159 - Train: 24.80% [1225400/4942000] [248.0/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 13:56:29,883 - Train: 24.80% [1225500/4942000] [248.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 13:57:02,569 - Train: 24.80% [1225600/4942000] [248.0/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 13:57:07,825 - ==> Total time: 6 days, 19:59:47 Eta: 20 days, 17:16:45 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 13:57:37,197 - Train: 24.80% [1225700/4942000] [248.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 13:58:10,010 - Train: 24.80% [1225800/4942000] [248.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 13:58:42,617 - Train: 24.81% [1225900/4942000] [248.1/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 13:59:15,229 - Train: 24.81% [1226000/4942000] [248.1/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 13:59:47,949 - Train: 24.81% [1226100/4942000] [248.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 14:00:20,573 - Train: 24.81% [1226200/4942000] [248.1/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 14:00:53,173 - Train: 24.81% [1226300/4942000] [248.1/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 14:01:25,976 - Train: 24.82% [1226400/4942000] [248.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 14:01:58,638 - Train: 24.82% [1226500/4942000] [248.2/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 14:02:31,494 - Train: 24.82% [1226600/4942000] [248.2/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 14:03:04,111 - Train: 24.82% [1226700/4942000] [248.2/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 14:03:36,701 - Train: 24.82% [1226800/4942000] [248.2/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 14:04:09,297 - Train: 24.83% [1226900/4942000] [248.3/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 14:04:41,913 - Train: 24.83% [1227000/4942000] [248.3/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 14:05:14,562 - Train: 24.83% [1227100/4942000] [248.3/1000.0] [batch_t 0.334 (0.326)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 14:05:47,350 - Train: 24.83% [1227200/4942000] [248.3/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 14:06:19,956 - Train: 24.83% [1227300/4942000] [248.3/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 14:06:52,522 - Train: 24.84% [1227400/4942000] [248.4/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 14:07:25,127 - Train: 24.84% [1227500/4942000] [248.4/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 14:07:57,725 - Train: 24.84% [1227600/4942000] [248.4/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 14:08:30,343 - Train: 24.84% [1227700/4942000] [248.4/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 14:09:03,031 - Train: 24.84% [1227800/4942000] [248.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 14:09:35,659 - Train: 24.85% [1227900/4942000] [248.5/1000.0] [batch_t 0.332 (0.326)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 14:10:08,299 - Train: 24.85% [1228000/4942000] [248.5/1000.0] [batch_t 0.331 (0.326)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 14:10:40,972 - Train: 24.85% [1228100/4942000] [248.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 14:11:13,601 - Train: 24.85% [1228200/4942000] [248.5/1000.0] [batch_t 0.331 (0.326)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 14:11:46,351 - Train: 24.85% [1228300/4942000] [248.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 14:12:18,974 - Train: 24.86% [1228400/4942000] [248.6/1000.0] [batch_t 0.331 (0.326)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 14:12:51,657 - Train: 24.86% [1228500/4942000] [248.6/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 14:13:24,293 - Train: 24.86% [1228600/4942000] [248.6/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 14:13:56,957 - Train: 24.86% [1228700/4942000] [248.6/1000.0] [batch_t 0.320 (0.327)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-09 14:14:29,678 - Train: 24.86% [1228800/4942000] [248.6/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 14:15:02,376 - Train: 24.87% [1228900/4942000] [248.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 14:15:35,097 - Train: 24.87% [1229000/4942000] [248.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 14:16:07,721 - Train: 24.87% [1229100/4942000] [248.7/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 14:16:40,406 - Train: 24.87% [1229200/4942000] [248.7/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 14:17:13,002 - Train: 24.87% [1229300/4942000] [248.7/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 14:17:45,590 - Train: 24.88% [1229400/4942000] [248.8/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 14:18:18,150 - Train: 24.88% [1229500/4942000] [248.8/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 14:18:50,743 - Train: 24.88% [1229600/4942000] [248.8/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 14:19:23,360 - Train: 24.88% [1229700/4942000] [248.8/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 14:19:55,961 - Train: 24.88% [1229800/4942000] [248.8/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 14:20:28,567 - Train: 24.89% [1229900/4942000] [248.9/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 14:21:01,147 - Train: 24.89% [1230000/4942000] [248.9/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 14:21:33,712 - Train: 24.89% [1230100/4942000] [248.9/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 14:22:06,316 - Train: 24.89% [1230200/4942000] [248.9/1000.0] [batch_t 0.320 (0.326)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-09 14:22:38,886 - Train: 24.89% [1230300/4942000] [248.9/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 14:23:11,491 - Train: 24.90% [1230400/4942000] [249.0/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 14:23:44,100 - Train: 24.90% [1230500/4942000] [249.0/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 14:24:03,030 - ==> Total time: 6 days, 20:26:42 Eta: 20 days, 15:58:36 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 14:24:19,715 - Train: 24.90% [1230600/4942000] [249.0/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 14:24:52,287 - Train: 24.90% [1230700/4942000] [249.0/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 14:25:24,894 - Train: 24.90% [1230800/4942000] [249.0/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 14:25:57,495 - Train: 24.91% [1230900/4942000] [249.1/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 14:26:30,113 - Train: 24.91% [1231000/4942000] [249.1/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 14:27:02,702 - Train: 24.91% [1231100/4942000] [249.1/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 14:27:35,269 - Train: 24.91% [1231200/4942000] [249.1/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 14:28:08,217 - Train: 24.92% [1231300/4942000] [249.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 14:28:40,814 - Train: 24.92% [1231400/4942000] [249.2/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 14:29:13,396 - Train: 24.92% [1231500/4942000] [249.2/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 14:29:45,983 - Train: 24.92% [1231600/4942000] [249.2/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 14:30:18,578 - Train: 24.92% [1231700/4942000] [249.2/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 14:30:51,149 - Train: 24.93% [1231800/4942000] [249.3/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 14:31:23,756 - Train: 24.93% [1231900/4942000] [249.3/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 14:31:56,341 - Train: 24.93% [1232000/4942000] [249.3/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 14:32:29,052 - Train: 24.93% [1232100/4942000] [249.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 14:33:01,650 - Train: 24.93% [1232200/4942000] [249.3/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 14:33:34,249 - Train: 24.94% [1232300/4942000] [249.4/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 14:34:06,840 - Train: 24.94% [1232400/4942000] [249.4/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 14:34:39,419 - Train: 24.94% [1232500/4942000] [249.4/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 14:35:12,014 - Train: 24.94% [1232600/4942000] [249.4/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 14:35:44,567 - Train: 24.94% [1232700/4942000] [249.4/1000.0] [batch_t 0.327 (0.325)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 14:36:17,168 - Train: 24.95% [1232800/4942000] [249.5/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 14:36:49,773 - Train: 24.95% [1232900/4942000] [249.5/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 14:37:22,407 - Train: 24.95% [1233000/4942000] [249.5/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 14:37:55,056 - Train: 24.95% [1233100/4942000] [249.5/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 14:38:27,664 - Train: 24.95% [1233200/4942000] [249.5/1000.0] [batch_t 0.322 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 14:39:00,228 - Train: 24.96% [1233300/4942000] [249.6/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 14:39:32,798 - Train: 24.96% [1233400/4942000] [249.6/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 14:40:05,511 - Train: 24.96% [1233500/4942000] [249.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 14:40:38,119 - Train: 24.96% [1233600/4942000] [249.6/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 14:41:10,670 - Train: 24.96% [1233700/4942000] [249.6/1000.0] [batch_t 0.323 (0.325)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 14:41:43,294 - Train: 24.97% [1233800/4942000] [249.7/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 14:42:15,855 - Train: 24.97% [1233900/4942000] [249.7/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 14:42:48,469 - Train: 24.97% [1234000/4942000] [249.7/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 14:43:21,084 - Train: 24.97% [1234100/4942000] [249.7/1000.0] [batch_t 0.324 (0.326)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 14:43:53,737 - Train: 24.97% [1234200/4942000] [249.7/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 14:44:26,329 - Train: 24.98% [1234300/4942000] [249.8/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 14:44:58,953 - Train: 24.98% [1234400/4942000] [249.8/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 14:45:31,544 - Train: 24.98% [1234500/4942000] [249.8/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 14:46:04,198 - Train: 24.98% [1234600/4942000] [249.8/1000.0] [batch_t 0.322 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 14:46:36,798 - Train: 24.98% [1234700/4942000] [249.8/1000.0] [batch_t 0.323 (0.326)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 14:47:09,364 - Train: 24.99% [1234800/4942000] [249.9/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 14:47:42,092 - Train: 24.99% [1234900/4942000] [249.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 14:48:15,304 - Train: 24.99% [1235000/4942000] [249.9/1000.0] [batch_t 0.324 (0.332)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 14:48:47,884 - Train: 24.99% [1235100/4942000] [249.9/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 14:49:20,512 - Train: 24.99% [1235200/4942000] [249.9/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 14:49:53,087 - Train: 25.00% [1235300/4942000] [250.0/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 14:50:25,757 - Train: 25.00% [1235400/4942000] [250.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 14:50:58,351 - Train: 25.00% [1235500/4942000] [250.0/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 14:51:08,486 - Test: 16.13% [50/310] [batch_t 0.152 (0.186)] 2024-04-09 14:51:16,093 - Test: 32.26% [100/310] [batch_t 0.152 (0.169)] 2024-04-09 14:51:23,692 - Test: 48.39% [150/310] [batch_t 0.153 (0.163)] 2024-04-09 14:51:31,264 - Test: 64.52% [200/310] [batch_t 0.153 (0.160)] 2024-04-09 14:51:38,884 - Test: 80.65% [250/310] [batch_t 0.172 (0.159)] 2024-04-09 14:51:46,825 - Test: 96.77% [300/310] [batch_t 0.150 (0.159)] 2024-04-09 14:51:48,253 - Test: 100.00% [310/310] [batch_t 0.082 (0.158)] 2024-04-09 15:18:03,010 - ==> Metric Time for coco : 0.004 (mAUROC_sp_max) 0.001 (mAP_sp_max) 0.001 (mF1_max_sp_max) 385.904 (mAUROC_px) 270.899 (mAP_px) 33.519 (mF1_max_px) 804.687 (mAUPRO_px) 11.629 (mF1_px_0.2_0.8_0.1) 11.562 (mAcc_px_0.2_0.8_0.1) 11.573 (mIoU_px_0.2_0.8_0.1) 33.719 (mIoU_max_px) 2024-04-09 15:18:03,607 - | Name | mAUROC_sp_max | mAUROC_sp_max (Max) | mAP_sp_max | mAP_sp_max (Max) | mF1_max_sp_max | mF1_max_sp_max (Max) | mAUROC_px | mAUROC_px (Max) | mAP_px | mAP_px (Max) | mF1_max_px | mF1_max_px (Max) | mAUPRO_px | mAUPRO_px (Max) | mF1_px_0.2_0.8_0.1 | mF1_px_0.2_0.8_0.1 (Max) | mAcc_px_0.2_0.8_0.1 | mAcc_px_0.2_0.8_0.1 (Max) | mIoU_px_0.2_0.8_0.1 | mIoU_px_0.2_0.8_0.1 (Max) | mIoU_max_px | mIoU_max_px (Max) | |:------:|:---------------:|:---------------------:|:------------:|:------------------:|:----------------:|:----------------------:|:-----------:|:------------------:|:--------:|:------------------:|:------------:|:------------------:|:-----------:|:------------------:|:--------------------:|:--------------------------:|:---------------------:|:---------------------------:|:---------------------:|:---------------------------:|:-------------:|:-------------------:| | coco | 64.720 | 66.882 (50 epoch) | 44.805 | 46.681 (100 epoch) | 53.234 | 54.576 (50 epoch) | 72.127 | 72.127 (250 epoch) | 14.954 | 14.954 (250 epoch) | 22.053 | 22.265 (200 epoch) | 42.777 | 44.441 (50 epoch) | 11.135 | 11.792 (50 epoch) | 41.112 | 44.665 (50 epoch) | 6.084 | 6.432 (100 epoch) | 12.393 | 12.527 (200 epoch) | 2024-04-09 15:18:04,243 - ==> Total time: 6 days, 21:20:43 Eta: 20 days, 16:02:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 15:19:22,629 - Train: 25.00% [1235600/4942000] [250.0/1000.0] [batch_t 0.763 (0.758)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-09 15:20:38,596 - Train: 25.00% [1235700/4942000] [250.0/1000.0] [batch_t 0.762 (0.760)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-09 15:21:54,543 - Train: 25.01% [1235800/4942000] [250.1/1000.0] [batch_t 0.768 (0.759)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-09 15:23:10,550 - Train: 25.01% [1235900/4942000] [250.1/1000.0] [batch_t 0.759 (0.760)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-09 15:24:26,524 - Train: 25.01% [1236000/4942000] [250.1/1000.0] [batch_t 0.762 (0.760)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-09 15:25:42,407 - Train: 25.01% [1236100/4942000] [250.1/1000.0] [batch_t 0.764 (0.759)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-09 15:26:58,340 - Train: 25.01% [1236200/4942000] [250.1/1000.0] [batch_t 0.755 (0.759)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-09 15:28:14,247 - Train: 25.02% [1236300/4942000] [250.2/1000.0] [batch_t 0.759 (0.759)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-09 15:29:30,130 - Train: 25.02% [1236400/4942000] [250.2/1000.0] [batch_t 0.760 (0.759)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-09 15:30:46,172 - Train: 25.02% [1236500/4942000] [250.2/1000.0] [batch_t 0.763 (0.760)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-09 15:32:02,131 - Train: 25.02% [1236600/4942000] [250.2/1000.0] [batch_t 0.764 (0.759)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-09 15:33:18,085 - Train: 25.02% [1236700/4942000] [250.2/1000.0] [batch_t 0.759 (0.759)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-09 15:34:20,747 - Train: 25.03% [1236800/4942000] [250.3/1000.0] [batch_t 0.327 (0.627)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 15:34:53,587 - Train: 25.03% [1236900/4942000] [250.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 15:35:26,673 - Train: 25.03% [1237000/4942000] [250.3/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 15:35:59,562 - Train: 25.03% [1237100/4942000] [250.3/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 15:36:34,208 - Train: 25.03% [1237200/4942000] [250.3/1000.0] [batch_t 0.329 (0.346)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 15:37:20,785 - Train: 25.04% [1237300/4942000] [250.4/1000.0] [batch_t 0.325 (0.466)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 15:37:53,682 - Train: 25.04% [1237400/4942000] [250.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 15:38:26,549 - Train: 25.04% [1237500/4942000] [250.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 15:38:59,524 - Train: 25.04% [1237600/4942000] [250.4/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 15:39:32,489 - Train: 25.04% [1237700/4942000] [250.4/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 15:40:05,310 - Train: 25.05% [1237800/4942000] [250.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 15:40:38,168 - Train: 25.05% [1237900/4942000] [250.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 15:41:10,993 - Train: 25.05% [1238000/4942000] [250.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 15:41:43,799 - Train: 25.05% [1238100/4942000] [250.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 15:42:16,613 - Train: 25.05% [1238200/4942000] [250.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 15:42:49,431 - Train: 25.06% [1238300/4942000] [250.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 15:43:22,278 - Train: 25.06% [1238400/4942000] [250.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 15:43:55,108 - Train: 25.06% [1238500/4942000] [250.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 15:44:28,157 - Train: 25.06% [1238600/4942000] [250.6/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-09 15:45:00,994 - Train: 25.06% [1238700/4942000] [250.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 15:45:33,819 - Train: 25.07% [1238800/4942000] [250.7/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 15:46:06,693 - Train: 25.07% [1238900/4942000] [250.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 15:46:39,581 - Train: 25.07% [1239000/4942000] [250.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 15:47:12,528 - Train: 25.07% [1239100/4942000] [250.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 15:47:45,413 - Train: 25.07% [1239200/4942000] [250.7/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 15:48:18,261 - Train: 25.08% [1239300/4942000] [250.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 15:48:51,132 - Train: 25.08% [1239400/4942000] [250.8/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 15:49:23,953 - Train: 25.08% [1239500/4942000] [250.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 15:49:56,802 - Train: 25.08% [1239600/4942000] [250.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 15:50:29,687 - Train: 25.08% [1239700/4942000] [250.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 15:51:02,526 - Train: 25.09% [1239800/4942000] [250.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 15:51:35,373 - Train: 25.09% [1239900/4942000] [250.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 15:52:08,181 - Train: 25.09% [1240000/4942000] [250.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 15:52:41,039 - Train: 25.09% [1240100/4942000] [250.9/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 15:53:13,865 - Train: 25.10% [1240200/4942000] [251.0/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 15:53:46,726 - Train: 25.10% [1240300/4942000] [251.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 15:54:19,553 - Train: 25.10% [1240400/4942000] [251.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 15:54:33,361 - ==> Total time: 6 days, 21:57:12 Eta: 20 days, 15:12:56 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 15:54:54,628 - Train: 25.10% [1240500/4942000] [251.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 15:55:27,513 - Train: 25.10% [1240600/4942000] [251.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 15:56:00,363 - Train: 25.11% [1240700/4942000] [251.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 15:56:33,164 - Train: 25.11% [1240800/4942000] [251.1/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 15:57:05,990 - Train: 25.11% [1240900/4942000] [251.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 15:57:38,810 - Train: 25.11% [1241000/4942000] [251.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 15:58:11,670 - Train: 25.11% [1241100/4942000] [251.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 15:58:44,528 - Train: 25.12% [1241200/4942000] [251.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 15:59:17,386 - Train: 25.12% [1241300/4942000] [251.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 15:59:50,246 - Train: 25.12% [1241400/4942000] [251.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-09 16:00:23,073 - Train: 25.12% [1241500/4942000] [251.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 16:00:55,913 - Train: 25.12% [1241600/4942000] [251.2/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 16:01:28,754 - Train: 25.13% [1241700/4942000] [251.3/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 16:02:01,637 - Train: 25.13% [1241800/4942000] [251.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 16:02:34,640 - Train: 25.13% [1241900/4942000] [251.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 16:03:07,461 - Train: 25.13% [1242000/4942000] [251.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 16:03:40,304 - Train: 25.13% [1242100/4942000] [251.3/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 16:04:13,169 - Train: 25.14% [1242200/4942000] [251.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 16:04:46,012 - Train: 25.14% [1242300/4942000] [251.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 16:05:18,825 - Train: 25.14% [1242400/4942000] [251.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 16:05:51,663 - Train: 25.14% [1242500/4942000] [251.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 16:06:24,495 - Train: 25.14% [1242600/4942000] [251.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 16:06:57,302 - Train: 25.15% [1242700/4942000] [251.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 16:07:38,074 - Train: 25.15% [1242800/4942000] [251.5/1000.0] [batch_t 0.327 (0.408)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 16:08:11,020 - Train: 25.15% [1242900/4942000] [251.5/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 16:08:43,893 - Train: 25.15% [1243000/4942000] [251.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 16:09:16,732 - Train: 25.15% [1243100/4942000] [251.5/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 16:09:49,552 - Train: 25.16% [1243200/4942000] [251.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 16:10:22,528 - Train: 25.16% [1243300/4942000] [251.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 16:10:55,431 - Train: 25.16% [1243400/4942000] [251.6/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 16:11:28,352 - Train: 25.16% [1243500/4942000] [251.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 16:12:01,219 - Train: 25.16% [1243600/4942000] [251.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 16:12:34,065 - Train: 25.17% [1243700/4942000] [251.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 16:13:06,920 - Train: 25.17% [1243800/4942000] [251.7/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 16:13:39,738 - Train: 25.17% [1243900/4942000] [251.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 16:14:12,614 - Train: 25.17% [1244000/4942000] [251.7/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 16:14:45,460 - Train: 25.17% [1244100/4942000] [251.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 16:15:18,285 - Train: 25.18% [1244200/4942000] [251.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 16:15:51,151 - Train: 25.18% [1244300/4942000] [251.8/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 16:16:23,954 - Train: 25.18% [1244400/4942000] [251.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 16:16:56,781 - Train: 25.18% [1244500/4942000] [251.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 16:17:29,642 - Train: 25.18% [1244600/4942000] [251.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 16:18:02,609 - Train: 25.19% [1244700/4942000] [251.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 16:18:35,465 - Train: 25.19% [1244800/4942000] [251.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 16:19:08,332 - Train: 25.19% [1244900/4942000] [251.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 16:19:41,185 - Train: 25.19% [1245000/4942000] [251.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 16:20:14,045 - Train: 25.19% [1245100/4942000] [251.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 16:20:46,872 - Train: 25.20% [1245200/4942000] [252.0/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 16:21:20,064 - Train: 25.20% [1245300/4942000] [252.0/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 16:21:47,659 - ==> Total time: 6 days, 22:24:26 Eta: 20 days, 13:56:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 16:21:55,603 - Train: 25.20% [1245400/4942000] [252.0/1000.0] [batch_t 0.323 (0.333)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 16:22:28,979 - Train: 25.20% [1245500/4942000] [252.0/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 16:23:01,941 - Train: 25.20% [1245600/4942000] [252.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 16:23:35,091 - Train: 25.21% [1245700/4942000] [252.1/1000.0] [batch_t 0.335 (0.331)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-09 16:24:08,126 - Train: 25.21% [1245800/4942000] [252.1/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 16:24:41,060 - Train: 25.21% [1245900/4942000] [252.1/1000.0] [batch_t 0.322 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 16:25:13,999 - Train: 25.21% [1246000/4942000] [252.1/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 16:25:47,270 - Train: 25.21% [1246100/4942000] [252.1/1000.0] [batch_t 0.336 (0.333)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-09 16:26:20,292 - Train: 25.22% [1246200/4942000] [252.2/1000.0] [batch_t 0.338 (0.330)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-09 16:26:53,407 - Train: 25.22% [1246300/4942000] [252.2/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 16:27:26,482 - Train: 25.22% [1246400/4942000] [252.2/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 16:27:59,426 - Train: 25.22% [1246500/4942000] [252.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 16:28:32,321 - Train: 25.22% [1246600/4942000] [252.2/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 16:29:06,283 - Train: 25.23% [1246700/4942000] [252.3/1000.0] [batch_t 0.324 (0.340)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 16:29:39,341 - Train: 25.23% [1246800/4942000] [252.3/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-09 16:30:12,878 - Train: 25.23% [1246900/4942000] [252.3/1000.0] [batch_t 0.324 (0.335)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 16:30:45,904 - Train: 25.23% [1247000/4942000] [252.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 16:31:19,004 - Train: 25.23% [1247100/4942000] [252.3/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 16:31:52,010 - Train: 25.24% [1247200/4942000] [252.4/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 16:32:25,023 - Train: 25.24% [1247300/4942000] [252.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 16:32:58,181 - Train: 25.24% [1247400/4942000] [252.4/1000.0] [batch_t 0.337 (0.331)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-09 16:33:31,363 - Train: 25.24% [1247500/4942000] [252.4/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 16:34:25,298 - Train: 25.24% [1247600/4942000] [252.4/1000.0] [batch_t 0.331 (0.539)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 16:35:12,705 - Train: 25.25% [1247700/4942000] [252.5/1000.0] [batch_t 0.328 (0.474)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 16:36:20,657 - Train: 25.25% [1247800/4942000] [252.5/1000.0] [batch_t 0.334 (0.679)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 16:38:03,861 - Train: 25.25% [1247900/4942000] [252.5/1000.0] [batch_t 0.330 (1.032)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 16:38:41,352 - Train: 25.25% [1248000/4942000] [252.5/1000.0] [batch_t 0.327 (0.375)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 16:39:16,383 - Train: 25.25% [1248100/4942000] [252.5/1000.0] [batch_t 0.330 (0.350)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 16:39:50,351 - Train: 25.26% [1248200/4942000] [252.6/1000.0] [batch_t 0.330 (0.340)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 16:40:23,443 - Train: 25.26% [1248300/4942000] [252.6/1000.0] [batch_t 0.337 (0.331)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-09 16:40:56,601 - Train: 25.26% [1248400/4942000] [252.6/1000.0] [batch_t 0.341 (0.331)] [data_t 0.002] [optim_t 0.339] [lr 0.005000] 2024-04-09 16:41:31,304 - Train: 25.26% [1248500/4942000] [252.6/1000.0] [batch_t 0.337 (0.347)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-09 16:42:06,080 - Train: 25.27% [1248600/4942000] [252.7/1000.0] [batch_t 2.048 (0.348)] [data_t 1.726] [optim_t 0.321] [lr 0.005000] 2024-04-09 16:42:39,555 - Train: 25.27% [1248700/4942000] [252.7/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 16:43:14,141 - Train: 25.27% [1248800/4942000] [252.7/1000.0] [batch_t 0.328 (0.346)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 16:43:47,133 - Train: 25.27% [1248900/4942000] [252.7/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 16:44:21,266 - Train: 25.27% [1249000/4942000] [252.7/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 16:44:54,226 - Train: 25.28% [1249100/4942000] [252.8/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 16:45:28,729 - Train: 25.28% [1249200/4942000] [252.8/1000.0] [batch_t 0.333 (0.345)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 16:46:01,736 - Train: 25.28% [1249300/4942000] [252.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 16:46:36,426 - Train: 25.28% [1249400/4942000] [252.8/1000.0] [batch_t 0.330 (0.347)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 16:47:20,744 - Train: 25.28% [1249500/4942000] [252.8/1000.0] [batch_t 0.334 (0.443)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-09 16:47:53,866 - Train: 25.29% [1249600/4942000] [252.9/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 16:48:27,020 - Train: 25.29% [1249700/4942000] [252.9/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 16:49:00,182 - Train: 25.29% [1249800/4942000] [252.9/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 16:49:33,162 - Train: 25.29% [1249900/4942000] [252.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 16:50:06,066 - Train: 25.29% [1250000/4942000] [252.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 16:50:38,954 - Train: 25.30% [1250100/4942000] [253.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 16:51:11,976 - Train: 25.30% [1250200/4942000] [253.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 16:51:44,865 - Train: 25.30% [1250300/4942000] [253.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 16:51:53,441 - ==> Total time: 6 days, 22:54:32 Eta: 20 days, 12:48:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 16:52:20,379 - Train: 25.30% [1250400/4942000] [253.0/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 16:52:53,372 - Train: 25.30% [1250500/4942000] [253.0/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 16:53:26,411 - Train: 25.31% [1250600/4942000] [253.1/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 16:53:59,555 - Train: 25.31% [1250700/4942000] [253.1/1000.0] [batch_t 0.336 (0.331)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-09 16:54:32,478 - Train: 25.31% [1250800/4942000] [253.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 16:55:05,503 - Train: 25.31% [1250900/4942000] [253.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 16:55:38,530 - Train: 25.31% [1251000/4942000] [253.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 16:56:11,482 - Train: 25.32% [1251100/4942000] [253.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 16:56:44,368 - Train: 25.32% [1251200/4942000] [253.2/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 16:57:17,302 - Train: 25.32% [1251300/4942000] [253.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 16:57:50,255 - Train: 25.32% [1251400/4942000] [253.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 16:58:23,271 - Train: 25.32% [1251500/4942000] [253.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 16:58:56,277 - Train: 25.33% [1251600/4942000] [253.3/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 16:59:29,179 - Train: 25.33% [1251700/4942000] [253.3/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 17:00:02,219 - Train: 25.33% [1251800/4942000] [253.3/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 17:00:38,245 - Train: 25.33% [1251900/4942000] [253.3/1000.0] [batch_t 0.330 (0.360)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 17:01:12,768 - Train: 25.33% [1252000/4942000] [253.3/1000.0] [batch_t 0.336 (0.345)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-09 17:01:45,685 - Train: 25.34% [1252100/4942000] [253.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 17:02:18,607 - Train: 25.34% [1252200/4942000] [253.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 17:02:51,499 - Train: 25.34% [1252300/4942000] [253.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 17:03:24,531 - Train: 25.34% [1252400/4942000] [253.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 17:03:57,472 - Train: 25.34% [1252500/4942000] [253.4/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 17:04:30,441 - Train: 25.35% [1252600/4942000] [253.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 17:05:03,359 - Train: 25.35% [1252700/4942000] [253.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 17:05:36,269 - Train: 25.35% [1252800/4942000] [253.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 17:06:09,271 - Train: 25.35% [1252900/4942000] [253.5/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 17:06:42,217 - Train: 25.35% [1253000/4942000] [253.5/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 17:07:15,280 - Train: 25.36% [1253100/4942000] [253.6/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 17:07:48,138 - Train: 25.36% [1253200/4942000] [253.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 17:08:21,128 - Train: 25.36% [1253300/4942000] [253.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 17:08:54,051 - Train: 25.36% [1253400/4942000] [253.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 17:09:26,903 - Train: 25.36% [1253500/4942000] [253.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 17:09:59,782 - Train: 25.37% [1253600/4942000] [253.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 17:10:35,569 - Train: 25.37% [1253700/4942000] [253.7/1000.0] [batch_t 0.325 (0.358)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 17:11:08,568 - Train: 25.37% [1253800/4942000] [253.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 17:11:41,491 - Train: 25.37% [1253900/4942000] [253.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 17:12:15,891 - Train: 25.37% [1254000/4942000] [253.7/1000.0] [batch_t 0.329 (0.344)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 17:12:50,935 - Train: 25.38% [1254100/4942000] [253.8/1000.0] [batch_t 0.330 (0.350)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 17:13:24,000 - Train: 25.38% [1254200/4942000] [253.8/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 17:13:57,118 - Train: 25.38% [1254300/4942000] [253.8/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 17:14:30,214 - Train: 25.38% [1254400/4942000] [253.8/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 17:15:04,412 - Train: 25.38% [1254500/4942000] [253.8/1000.0] [batch_t 0.331 (0.342)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 17:15:37,312 - Train: 25.39% [1254600/4942000] [253.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 17:16:10,893 - Train: 25.39% [1254700/4942000] [253.9/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 17:16:43,872 - Train: 25.39% [1254800/4942000] [253.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 17:17:16,851 - Train: 25.39% [1254900/4942000] [253.9/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 17:17:49,832 - Train: 25.39% [1255000/4942000] [253.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 17:18:22,826 - Train: 25.40% [1255100/4942000] [254.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 17:18:55,893 - Train: 25.40% [1255200/4942000] [254.0/1000.0] [batch_t 0.337 (0.331)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-09 17:19:18,648 - ==> Total time: 6 days, 23:21:57 Eta: 20 days, 11:33:19 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 17:19:31,669 - Train: 25.40% [1255300/4942000] [254.0/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 17:20:04,725 - Train: 25.40% [1255400/4942000] [254.0/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 17:20:37,730 - Train: 25.40% [1255500/4942000] [254.0/1000.0] [batch_t 0.336 (0.330)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-09 17:21:13,767 - Train: 25.41% [1255600/4942000] [254.1/1000.0] [batch_t 0.334 (0.360)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 17:21:46,679 - Train: 25.41% [1255700/4942000] [254.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 17:22:34,973 - Train: 25.41% [1255800/4942000] [254.1/1000.0] [batch_t 0.328 (0.483)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 17:23:08,479 - Train: 25.41% [1255900/4942000] [254.1/1000.0] [batch_t 0.323 (0.335)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 17:23:41,530 - Train: 25.41% [1256000/4942000] [254.1/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 17:24:24,458 - Train: 25.42% [1256100/4942000] [254.2/1000.0] [batch_t 0.325 (0.429)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 17:24:59,204 - Train: 25.42% [1256200/4942000] [254.2/1000.0] [batch_t 0.330 (0.347)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 17:25:36,419 - Train: 25.42% [1256300/4942000] [254.2/1000.0] [batch_t 0.328 (0.372)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 17:26:11,743 - Train: 25.42% [1256400/4942000] [254.2/1000.0] [batch_t 0.329 (0.353)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 17:26:48,533 - Train: 25.42% [1256500/4942000] [254.2/1000.0] [batch_t 0.333 (0.368)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 17:27:22,782 - Train: 25.43% [1256600/4942000] [254.3/1000.0] [batch_t 0.330 (0.342)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 17:27:55,863 - Train: 25.43% [1256700/4942000] [254.3/1000.0] [batch_t 0.336 (0.331)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-09 17:28:31,569 - Train: 25.43% [1256800/4942000] [254.3/1000.0] [batch_t 0.329 (0.357)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 17:29:04,483 - Train: 25.43% [1256900/4942000] [254.3/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 17:29:40,699 - Train: 25.44% [1257000/4942000] [254.4/1000.0] [batch_t 0.332 (0.362)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 17:30:15,996 - Train: 25.44% [1257100/4942000] [254.4/1000.0] [batch_t 0.330 (0.353)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 17:30:48,952 - Train: 25.44% [1257200/4942000] [254.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 17:31:22,597 - Train: 25.44% [1257300/4942000] [254.4/1000.0] [batch_t 0.323 (0.336)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 17:31:55,465 - Train: 25.44% [1257400/4942000] [254.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 17:32:29,264 - Train: 25.45% [1257500/4942000] [254.5/1000.0] [batch_t 0.331 (0.338)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 17:33:02,145 - Train: 25.45% [1257600/4942000] [254.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 17:33:36,265 - Train: 25.45% [1257700/4942000] [254.5/1000.0] [batch_t 0.331 (0.341)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 17:34:09,430 - Train: 25.45% [1257800/4942000] [254.5/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 17:34:54,145 - Train: 25.45% [1257900/4942000] [254.5/1000.0] [batch_t 0.334 (0.447)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 17:35:27,119 - Train: 25.46% [1258000/4942000] [254.6/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 17:36:00,078 - Train: 25.46% [1258100/4942000] [254.6/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 17:36:33,000 - Train: 25.46% [1258200/4942000] [254.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 17:37:06,023 - Train: 25.46% [1258300/4942000] [254.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 17:37:39,030 - Train: 25.46% [1258400/4942000] [254.6/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 17:38:12,106 - Train: 25.47% [1258500/4942000] [254.7/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 17:38:45,108 - Train: 25.47% [1258600/4942000] [254.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 17:39:18,042 - Train: 25.47% [1258700/4942000] [254.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 17:39:50,919 - Train: 25.47% [1258800/4942000] [254.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 17:40:23,991 - Train: 25.47% [1258900/4942000] [254.7/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 17:40:56,956 - Train: 25.48% [1259000/4942000] [254.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 17:41:32,147 - Train: 25.48% [1259100/4942000] [254.8/1000.0] [batch_t 0.320 (0.352)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-09 17:42:05,091 - Train: 25.48% [1259200/4942000] [254.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 17:42:38,044 - Train: 25.48% [1259300/4942000] [254.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 17:43:12,402 - Train: 25.48% [1259400/4942000] [254.8/1000.0] [batch_t 0.336 (0.343)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-09 17:43:46,930 - Train: 25.49% [1259500/4942000] [254.9/1000.0] [batch_t 0.320 (0.345)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-09 17:44:21,243 - Train: 25.49% [1259600/4942000] [254.9/1000.0] [batch_t 0.330 (0.343)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 17:44:54,180 - Train: 25.49% [1259700/4942000] [254.9/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 17:45:28,152 - Train: 25.49% [1259800/4942000] [254.9/1000.0] [batch_t 0.330 (0.340)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 17:46:01,117 - Train: 25.49% [1259900/4942000] [254.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 17:46:35,297 - Train: 25.50% [1260000/4942000] [255.0/1000.0] [batch_t 0.327 (0.342)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 17:47:09,412 - Train: 25.50% [1260100/4942000] [255.0/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 17:47:43,919 - Train: 25.50% [1260200/4942000] [255.0/1000.0] [batch_t 0.328 (0.345)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 17:47:47,208 - ==> Total time: 6 days, 23:50:26 Eta: 20 days, 10:21:28 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 17:48:20,944 - Train: 25.50% [1260300/4942000] [255.0/1000.0] [batch_t 0.328 (0.346)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 17:48:53,961 - Train: 25.50% [1260400/4942000] [255.0/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 17:49:27,365 - Train: 25.51% [1260500/4942000] [255.1/1000.0] [batch_t 0.330 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 17:50:00,402 - Train: 25.51% [1260600/4942000] [255.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 17:50:34,145 - Train: 25.51% [1260700/4942000] [255.1/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 17:51:08,459 - Train: 25.51% [1260800/4942000] [255.1/1000.0] [batch_t 0.331 (0.343)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 17:51:41,379 - Train: 25.51% [1260900/4942000] [255.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 17:52:14,587 - Train: 25.52% [1261000/4942000] [255.2/1000.0] [batch_t 0.332 (0.332)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 17:52:47,502 - Train: 25.52% [1261100/4942000] [255.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 17:53:20,546 - Train: 25.52% [1261200/4942000] [255.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 17:54:07,367 - Train: 25.52% [1261300/4942000] [255.2/1000.0] [batch_t 0.327 (0.468)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 17:54:40,247 - Train: 25.52% [1261400/4942000] [255.2/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 17:55:13,137 - Train: 25.53% [1261500/4942000] [255.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 17:55:46,020 - Train: 25.53% [1261600/4942000] [255.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 17:56:19,075 - Train: 25.53% [1261700/4942000] [255.3/1000.0] [batch_t 0.338 (0.330)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-09 17:56:52,051 - Train: 25.53% [1261800/4942000] [255.3/1000.0] [batch_t 0.336 (0.330)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-09 17:57:25,177 - Train: 25.53% [1261900/4942000] [255.3/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 17:57:58,111 - Train: 25.54% [1262000/4942000] [255.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 17:58:31,108 - Train: 25.54% [1262100/4942000] [255.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 17:59:04,032 - Train: 25.54% [1262200/4942000] [255.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 17:59:37,132 - Train: 25.54% [1262300/4942000] [255.4/1000.0] [batch_t 0.336 (0.331)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-09 18:00:10,333 - Train: 25.54% [1262400/4942000] [255.4/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 18:00:43,299 - Train: 25.55% [1262500/4942000] [255.5/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 18:01:16,394 - Train: 25.55% [1262600/4942000] [255.5/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 18:01:49,315 - Train: 25.55% [1262700/4942000] [255.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 18:02:22,229 - Train: 25.55% [1262800/4942000] [255.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 18:02:55,232 - Train: 25.55% [1262900/4942000] [255.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 18:03:28,120 - Train: 25.56% [1263000/4942000] [255.6/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 18:04:01,166 - Train: 25.56% [1263100/4942000] [255.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 18:04:34,916 - Train: 25.56% [1263200/4942000] [255.6/1000.0] [batch_t 0.326 (0.337)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 18:05:07,837 - Train: 25.56% [1263300/4942000] [255.6/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 18:05:41,129 - Train: 25.56% [1263400/4942000] [255.6/1000.0] [batch_t 0.331 (0.333)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 18:06:14,083 - Train: 25.57% [1263500/4942000] [255.7/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 18:06:47,046 - Train: 25.57% [1263600/4942000] [255.7/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 18:07:20,021 - Train: 25.57% [1263700/4942000] [255.7/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 18:07:52,984 - Train: 25.57% [1263800/4942000] [255.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 18:08:26,023 - Train: 25.57% [1263900/4942000] [255.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 18:08:58,903 - Train: 25.58% [1264000/4942000] [255.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 18:09:31,809 - Train: 25.58% [1264100/4942000] [255.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 18:10:04,646 - Train: 25.58% [1264200/4942000] [255.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 18:10:37,531 - Train: 25.58% [1264300/4942000] [255.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 18:11:11,240 - Train: 25.58% [1264400/4942000] [255.8/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 18:11:44,119 - Train: 25.59% [1264500/4942000] [255.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 18:12:17,106 - Train: 25.59% [1264600/4942000] [255.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 18:12:50,040 - Train: 25.59% [1264700/4942000] [255.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 18:13:23,852 - Train: 25.59% [1264800/4942000] [255.9/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 18:13:56,723 - Train: 25.59% [1264900/4942000] [255.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 18:14:29,660 - Train: 25.60% [1265000/4942000] [256.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 18:15:02,580 - Train: 25.60% [1265100/4942000] [256.0/1000.0] [batch_t 0.321 (0.329)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 18:15:19,677 - ==> Total time: 7 days, 0:17:58 Eta: 20 days, 9:07:15 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 18:15:37,400 - Train: 25.60% [1265200/4942000] [256.0/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 18:16:10,234 - Train: 25.60% [1265300/4942000] [256.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 18:16:43,150 - Train: 25.61% [1265400/4942000] [256.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 18:17:16,036 - Train: 25.61% [1265500/4942000] [256.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 18:17:48,905 - Train: 25.61% [1265600/4942000] [256.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 18:18:21,904 - Train: 25.61% [1265700/4942000] [256.1/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 18:18:54,759 - Train: 25.61% [1265800/4942000] [256.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 18:19:27,724 - Train: 25.62% [1265900/4942000] [256.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 18:20:00,548 - Train: 25.62% [1266000/4942000] [256.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 18:20:33,447 - Train: 25.62% [1266100/4942000] [256.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 18:21:06,315 - Train: 25.62% [1266200/4942000] [256.2/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 18:21:39,141 - Train: 25.62% [1266300/4942000] [256.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 18:22:11,970 - Train: 25.63% [1266400/4942000] [256.3/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 18:22:44,781 - Train: 25.63% [1266500/4942000] [256.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 18:23:17,624 - Train: 25.63% [1266600/4942000] [256.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 18:23:50,461 - Train: 25.63% [1266700/4942000] [256.3/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 18:24:23,315 - Train: 25.63% [1266800/4942000] [256.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 18:24:56,164 - Train: 25.64% [1266900/4942000] [256.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 18:25:29,004 - Train: 25.64% [1267000/4942000] [256.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 18:26:01,862 - Train: 25.64% [1267100/4942000] [256.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 18:26:34,647 - Train: 25.64% [1267200/4942000] [256.4/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 18:27:07,505 - Train: 25.64% [1267300/4942000] [256.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 18:27:40,342 - Train: 25.65% [1267400/4942000] [256.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 18:28:13,170 - Train: 25.65% [1267500/4942000] [256.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 18:28:46,052 - Train: 25.65% [1267600/4942000] [256.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 18:29:18,996 - Train: 25.65% [1267700/4942000] [256.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 18:29:51,866 - Train: 25.65% [1267800/4942000] [256.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 18:30:24,822 - Train: 25.66% [1267900/4942000] [256.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 18:30:57,684 - Train: 25.66% [1268000/4942000] [256.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 18:31:30,699 - Train: 25.66% [1268100/4942000] [256.6/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 18:32:03,618 - Train: 25.66% [1268200/4942000] [256.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 18:32:36,676 - Train: 25.66% [1268300/4942000] [256.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 18:33:09,740 - Train: 25.67% [1268400/4942000] [256.7/1000.0] [batch_t 0.325 (0.331)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 18:33:42,725 - Train: 25.67% [1268500/4942000] [256.7/1000.0] [batch_t 0.339 (0.330)] [data_t 0.003] [optim_t 0.336] [lr 0.005000] 2024-04-09 18:34:15,825 - Train: 25.67% [1268600/4942000] [256.7/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-09 18:34:48,747 - Train: 25.67% [1268700/4942000] [256.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 18:35:21,847 - Train: 25.67% [1268800/4942000] [256.7/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 18:35:54,769 - Train: 25.68% [1268900/4942000] [256.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 18:36:27,664 - Train: 25.68% [1269000/4942000] [256.8/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 18:37:00,561 - Train: 25.68% [1269100/4942000] [256.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 18:37:33,664 - Train: 25.68% [1269200/4942000] [256.8/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-09 18:38:06,721 - Train: 25.68% [1269300/4942000] [256.8/1000.0] [batch_t 0.337 (0.330)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-09 18:38:39,852 - Train: 25.69% [1269400/4942000] [256.9/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-09 18:39:12,955 - Train: 25.69% [1269500/4942000] [256.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-09 18:39:45,984 - Train: 25.69% [1269600/4942000] [256.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-09 18:40:19,083 - Train: 25.69% [1269700/4942000] [256.9/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-09 18:40:52,170 - Train: 25.69% [1269800/4942000] [256.9/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 18:41:25,192 - Train: 25.70% [1269900/4942000] [257.0/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 18:41:58,237 - Train: 25.70% [1270000/4942000] [257.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-09 18:42:29,387 - ==> Total time: 7 days, 0:45:08 Eta: 20 days, 7:52:18 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 18:42:34,251 - Train: 25.70% [1270100/4942000] [257.0/1000.0] [batch_t 0.330 (0.365)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-09 18:43:07,242 - Train: 25.70% [1270200/4942000] [257.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-09 18:43:40,336 - Train: 25.70% [1270300/4942000] [257.0/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-09 18:44:13,363 - Train: 25.71% [1270400/4942000] [257.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-09 18:44:46,447 - Train: 25.71% [1270500/4942000] [257.1/1000.0] [batch_t 0.327 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-09 18:45:19,495 - Train: 25.71% [1270600/4942000] [257.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-09 18:45:52,576 - Train: 25.71% [1270700/4942000] [257.1/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-09 18:46:25,625 - Train: 25.71% [1270800/4942000] [257.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 18:46:58,626 - Train: 25.72% [1270900/4942000] [257.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-09 18:47:31,651 - Train: 25.72% [1271000/4942000] [257.2/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 18:48:04,661 - Train: 25.72% [1271100/4942000] [257.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-09 18:48:37,688 - Train: 25.72% [1271200/4942000] [257.2/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 18:49:10,760 - Train: 25.72% [1271300/4942000] [257.2/1000.0] [batch_t 0.326 (0.331)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-09 18:49:43,778 - Train: 25.73% [1271400/4942000] [257.3/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-09 18:50:16,815 - Train: 25.73% [1271500/4942000] [257.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 18:50:50,006 - Train: 25.73% [1271600/4942000] [257.3/1000.0] [batch_t 0.333 (0.332)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-09 18:51:22,990 - Train: 25.73% [1271700/4942000] [257.3/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 18:51:55,931 - Train: 25.73% [1271800/4942000] [257.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-09 18:52:28,939 - Train: 25.74% [1271900/4942000] [257.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-09 18:53:01,935 - Train: 25.74% [1272000/4942000] [257.4/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-09 18:53:34,926 - Train: 25.74% [1272100/4942000] [257.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-09 18:54:07,938 - Train: 25.74% [1272200/4942000] [257.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-09 18:54:41,072 - Train: 25.74% [1272300/4942000] [257.4/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-09 18:55:14,107 - Train: 25.75% [1272400/4942000] [257.5/1000.0] [batch_t 0.324 (0.330)] [data_t 0.003] [optim_t 0.321] [lr 0.005000] 2024-04-09 18:55:47,120 - Train: 25.75% [1272500/4942000] [257.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-09 18:56:20,178 - Train: 25.75% [1272600/4942000] [257.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-09 18:56:53,226 - Train: 25.75% [1272700/4942000] [257.5/1000.0] [batch_t 0.337 (0.330)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-09 18:57:26,170 - Train: 25.75% [1272800/4942000] [257.5/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 18:57:59,117 - Train: 25.76% [1272900/4942000] [257.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 18:58:32,226 - Train: 25.76% [1273000/4942000] [257.6/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 18:59:05,212 - Train: 25.76% [1273100/4942000] [257.6/1000.0] [batch_t 0.336 (0.330)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-09 18:59:38,234 - Train: 25.76% [1273200/4942000] [257.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 19:00:11,188 - Train: 25.76% [1273300/4942000] [257.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 19:00:44,133 - Train: 25.77% [1273400/4942000] [257.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 19:01:17,110 - Train: 25.77% [1273500/4942000] [257.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 19:01:50,037 - Train: 25.77% [1273600/4942000] [257.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 19:02:22,970 - Train: 25.77% [1273700/4942000] [257.7/1000.0] [batch_t 0.337 (0.329)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-09 19:02:55,894 - Train: 25.77% [1273800/4942000] [257.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 19:03:28,842 - Train: 25.78% [1273900/4942000] [257.8/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 19:04:01,790 - Train: 25.78% [1274000/4942000] [257.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 19:04:34,721 - Train: 25.78% [1274100/4942000] [257.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 19:05:07,606 - Train: 25.78% [1274200/4942000] [257.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 19:05:40,546 - Train: 25.79% [1274300/4942000] [257.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 19:06:13,658 - Train: 25.79% [1274400/4942000] [257.9/1000.0] [batch_t 0.335 (0.331)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-09 19:06:57,935 - Train: 25.79% [1274500/4942000] [257.9/1000.0] [batch_t 0.329 (0.443)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 19:07:30,923 - Train: 25.79% [1274600/4942000] [257.9/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 19:08:03,893 - Train: 25.79% [1274700/4942000] [257.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 19:08:36,850 - Train: 25.80% [1274800/4942000] [258.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 19:09:09,815 - Train: 25.80% [1274900/4942000] [258.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 19:09:42,828 - Train: 25.80% [1275000/4942000] [258.0/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 19:09:54,707 - ==> Total time: 7 days, 1:12:33 Eta: 20 days, 6:38:27 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 19:10:19,028 - Train: 25.80% [1275100/4942000] [258.0/1000.0] [batch_t 0.330 (0.345)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 19:10:52,040 - Train: 25.80% [1275200/4942000] [258.0/1000.0] [batch_t 0.337 (0.330)] [data_t 0.003] [optim_t 0.335] [lr 0.005000] 2024-04-09 19:11:25,022 - Train: 25.81% [1275300/4942000] [258.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 19:11:57,951 - Train: 25.81% [1275400/4942000] [258.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 19:12:30,947 - Train: 25.81% [1275500/4942000] [258.1/1000.0] [batch_t 0.337 (0.330)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-09 19:13:03,933 - Train: 25.81% [1275600/4942000] [258.1/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 19:13:36,878 - Train: 25.81% [1275700/4942000] [258.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 19:14:09,965 - Train: 25.82% [1275800/4942000] [258.2/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 19:14:42,914 - Train: 25.82% [1275900/4942000] [258.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 19:15:15,861 - Train: 25.82% [1276000/4942000] [258.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 19:15:48,877 - Train: 25.82% [1276100/4942000] [258.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 19:16:21,888 - Train: 25.82% [1276200/4942000] [258.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 19:16:54,824 - Train: 25.83% [1276300/4942000] [258.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 19:17:28,394 - Train: 25.83% [1276400/4942000] [258.3/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 19:18:01,325 - Train: 25.83% [1276500/4942000] [258.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 19:18:34,246 - Train: 25.83% [1276600/4942000] [258.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 19:19:07,198 - Train: 25.83% [1276700/4942000] [258.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 19:19:40,111 - Train: 25.84% [1276800/4942000] [258.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 19:20:13,292 - Train: 25.84% [1276900/4942000] [258.4/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 19:20:46,219 - Train: 25.84% [1277000/4942000] [258.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 19:21:19,276 - Train: 25.84% [1277100/4942000] [258.4/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 19:21:52,408 - Train: 25.84% [1277200/4942000] [258.4/1000.0] [batch_t 0.322 (0.331)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 19:22:25,367 - Train: 25.85% [1277300/4942000] [258.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 19:22:58,308 - Train: 25.85% [1277400/4942000] [258.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 19:23:32,844 - Train: 25.85% [1277500/4942000] [258.5/1000.0] [batch_t 0.326 (0.345)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 19:24:05,929 - Train: 25.85% [1277600/4942000] [258.5/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 19:24:38,842 - Train: 25.85% [1277700/4942000] [258.5/1000.0] [batch_t 0.326 (0.329)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-09 19:25:11,908 - Train: 25.86% [1277800/4942000] [258.6/1000.0] [batch_t 0.333 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 19:25:44,864 - Train: 25.86% [1277900/4942000] [258.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 19:26:18,107 - Train: 25.86% [1278000/4942000] [258.6/1000.0] [batch_t 0.324 (0.332)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 19:26:51,172 - Train: 25.86% [1278100/4942000] [258.6/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 19:27:35,730 - Train: 25.86% [1278200/4942000] [258.6/1000.0] [batch_t 0.329 (0.445)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 19:28:08,807 - Train: 25.87% [1278300/4942000] [258.7/1000.0] [batch_t 0.325 (0.331)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 19:28:43,106 - Train: 25.87% [1278400/4942000] [258.7/1000.0] [batch_t 0.324 (0.343)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 19:29:16,128 - Train: 25.87% [1278500/4942000] [258.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 19:29:49,401 - Train: 25.87% [1278600/4942000] [258.7/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 19:30:22,468 - Train: 25.87% [1278700/4942000] [258.7/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 19:30:55,452 - Train: 25.88% [1278800/4942000] [258.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 19:31:28,373 - Train: 25.88% [1278900/4942000] [258.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 19:32:01,237 - Train: 25.88% [1279000/4942000] [258.8/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 19:32:34,172 - Train: 25.88% [1279100/4942000] [258.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 19:33:07,237 - Train: 25.88% [1279200/4942000] [258.8/1000.0] [batch_t 0.335 (0.331)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-09 19:33:40,176 - Train: 25.89% [1279300/4942000] [258.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 19:34:13,132 - Train: 25.89% [1279400/4942000] [258.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 19:34:46,206 - Train: 25.89% [1279500/4942000] [258.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 19:35:19,157 - Train: 25.89% [1279600/4942000] [258.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 19:35:52,113 - Train: 25.89% [1279700/4942000] [258.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 19:36:25,061 - Train: 25.90% [1279800/4942000] [259.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 19:36:58,017 - Train: 25.90% [1279900/4942000] [259.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 19:37:23,719 - ==> Total time: 7 days, 1:40:02 Eta: 20 days, 5:25:09 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 19:37:33,331 - Train: 25.90% [1280000/4942000] [259.0/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 19:38:06,271 - Train: 25.90% [1280100/4942000] [259.0/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 19:38:39,196 - Train: 25.90% [1280200/4942000] [259.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 19:39:12,118 - Train: 25.91% [1280300/4942000] [259.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 19:39:45,049 - Train: 25.91% [1280400/4942000] [259.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 19:40:18,024 - Train: 25.91% [1280500/4942000] [259.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 19:40:50,908 - Train: 25.91% [1280600/4942000] [259.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 19:41:23,880 - Train: 25.91% [1280700/4942000] [259.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 19:41:56,870 - Train: 25.92% [1280800/4942000] [259.2/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 19:42:29,879 - Train: 25.92% [1280900/4942000] [259.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 19:43:02,820 - Train: 25.92% [1281000/4942000] [259.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 19:43:35,792 - Train: 25.92% [1281100/4942000] [259.2/1000.0] [batch_t 0.325 (0.330)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-09 19:44:08,772 - Train: 25.92% [1281200/4942000] [259.2/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 19:44:41,710 - Train: 25.93% [1281300/4942000] [259.3/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-09 19:45:14,787 - Train: 25.93% [1281400/4942000] [259.3/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 19:45:47,709 - Train: 25.93% [1281500/4942000] [259.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 19:46:20,666 - Train: 25.93% [1281600/4942000] [259.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 19:46:53,604 - Train: 25.93% [1281700/4942000] [259.3/1000.0] [batch_t 0.334 (0.329)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-09 19:47:26,737 - Train: 25.94% [1281800/4942000] [259.4/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 19:47:59,689 - Train: 25.94% [1281900/4942000] [259.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 19:48:33,903 - Train: 25.94% [1282000/4942000] [259.4/1000.0] [batch_t 0.333 (0.342)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-09 19:49:07,623 - Train: 25.94% [1282100/4942000] [259.4/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 19:49:40,580 - Train: 25.94% [1282200/4942000] [259.4/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 19:50:13,632 - Train: 25.95% [1282300/4942000] [259.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 19:50:46,550 - Train: 25.95% [1282400/4942000] [259.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 19:51:20,667 - Train: 25.95% [1282500/4942000] [259.5/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 19:51:53,589 - Train: 25.95% [1282600/4942000] [259.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 19:52:27,786 - Train: 25.96% [1282700/4942000] [259.6/1000.0] [batch_t 0.330 (0.342)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 19:53:12,130 - Train: 25.96% [1282800/4942000] [259.6/1000.0] [batch_t 0.326 (0.443)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 19:53:45,066 - Train: 25.96% [1282900/4942000] [259.6/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 19:54:18,003 - Train: 25.96% [1283000/4942000] [259.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-09 19:54:50,931 - Train: 25.96% [1283100/4942000] [259.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 19:55:24,061 - Train: 25.97% [1283200/4942000] [259.7/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 19:55:56,944 - Train: 25.97% [1283300/4942000] [259.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 19:56:29,865 - Train: 25.97% [1283400/4942000] [259.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 19:57:02,781 - Train: 25.97% [1283500/4942000] [259.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 19:57:35,670 - Train: 25.97% [1283600/4942000] [259.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 19:58:08,595 - Train: 25.98% [1283700/4942000] [259.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 19:58:44,403 - Train: 25.98% [1283800/4942000] [259.8/1000.0] [batch_t 0.326 (0.358)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 19:59:17,323 - Train: 25.98% [1283900/4942000] [259.8/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 19:59:50,222 - Train: 25.98% [1284000/4942000] [259.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 20:00:23,122 - Train: 25.98% [1284100/4942000] [259.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-09 20:00:56,007 - Train: 25.99% [1284200/4942000] [259.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 20:01:29,002 - Train: 25.99% [1284300/4942000] [259.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 20:02:01,914 - Train: 25.99% [1284400/4942000] [259.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 20:02:34,845 - Train: 25.99% [1284500/4942000] [259.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-09 20:03:07,764 - Train: 25.99% [1284600/4942000] [259.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 20:03:40,718 - Train: 26.00% [1284700/4942000] [260.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-09 20:04:13,676 - Train: 26.00% [1284800/4942000] [260.0/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 20:04:46,583 - Train: 26.00% [1284900/4942000] [260.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 20:04:53,186 - ==> Total time: 7 days, 2:07:32 Eta: 20 days, 4:12:13 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 20:05:21,648 - Train: 26.00% [1285000/4942000] [260.0/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 20:05:54,523 - Train: 26.00% [1285100/4942000] [260.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 20:06:27,465 - Train: 26.01% [1285200/4942000] [260.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 20:07:00,381 - Train: 26.01% [1285300/4942000] [260.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 20:07:33,294 - Train: 26.01% [1285400/4942000] [260.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 20:08:06,197 - Train: 26.01% [1285500/4942000] [260.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 20:08:39,267 - Train: 26.01% [1285600/4942000] [260.1/1000.0] [batch_t 0.324 (0.331)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 20:09:12,218 - Train: 26.02% [1285700/4942000] [260.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 20:09:45,121 - Train: 26.02% [1285800/4942000] [260.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 20:10:18,034 - Train: 26.02% [1285900/4942000] [260.2/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 20:10:50,921 - Train: 26.02% [1286000/4942000] [260.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 20:11:23,864 - Train: 26.02% [1286100/4942000] [260.2/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 20:11:56,780 - Train: 26.03% [1286200/4942000] [260.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 20:12:29,682 - Train: 26.03% [1286300/4942000] [260.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 20:13:02,581 - Train: 26.03% [1286400/4942000] [260.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 20:13:35,479 - Train: 26.03% [1286500/4942000] [260.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 20:14:08,358 - Train: 26.03% [1286600/4942000] [260.3/1000.0] [batch_t 0.323 (0.329)] [data_t 0.003] [optim_t 0.321] [lr 0.005000] 2024-04-09 20:14:41,261 - Train: 26.04% [1286700/4942000] [260.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 20:15:14,182 - Train: 26.04% [1286800/4942000] [260.4/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 20:15:47,052 - Train: 26.04% [1286900/4942000] [260.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 20:16:20,026 - Train: 26.04% [1287000/4942000] [260.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-09 20:16:52,939 - Train: 26.04% [1287100/4942000] [260.4/1000.0] [batch_t 0.323 (0.329)] [data_t 0.003] [optim_t 0.320] [lr 0.005000] 2024-04-09 20:17:25,814 - Train: 26.05% [1287200/4942000] [260.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 20:17:58,684 - Train: 26.05% [1287300/4942000] [260.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 20:18:31,594 - Train: 26.05% [1287400/4942000] [260.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 20:19:04,461 - Train: 26.05% [1287500/4942000] [260.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 20:19:37,353 - Train: 26.05% [1287600/4942000] [260.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 20:20:10,252 - Train: 26.06% [1287700/4942000] [260.6/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 20:20:43,143 - Train: 26.06% [1287800/4942000] [260.6/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 20:21:15,999 - Train: 26.06% [1287900/4942000] [260.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 20:21:48,910 - Train: 26.06% [1288000/4942000] [260.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 20:22:21,775 - Train: 26.06% [1288100/4942000] [260.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 20:22:54,673 - Train: 26.07% [1288200/4942000] [260.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 20:23:27,562 - Train: 26.07% [1288300/4942000] [260.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 20:24:00,553 - Train: 26.07% [1288400/4942000] [260.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-09 20:24:33,591 - Train: 26.07% [1288500/4942000] [260.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 20:25:06,549 - Train: 26.07% [1288600/4942000] [260.7/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 20:25:39,494 - Train: 26.08% [1288700/4942000] [260.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 20:26:12,471 - Train: 26.08% [1288800/4942000] [260.8/1000.0] [batch_t 0.336 (0.330)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-09 20:26:45,448 - Train: 26.08% [1288900/4942000] [260.8/1000.0] [batch_t 0.338 (0.330)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-09 20:27:27,495 - Train: 26.08% [1289000/4942000] [260.8/1000.0] [batch_t 0.500 (0.420)] [data_t 0.175] [optim_t 0.325] [lr 0.005000] 2024-04-09 20:28:00,479 - Train: 26.08% [1289100/4942000] [260.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 20:28:46,052 - Train: 26.09% [1289200/4942000] [260.9/1000.0] [batch_t 1.525 (0.456)] [data_t 1.199] [optim_t 0.326] [lr 0.005000] 2024-04-09 20:29:32,144 - Train: 26.09% [1289300/4942000] [260.9/1000.0] [batch_t 0.325 (0.461)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 20:31:31,008 - Train: 26.09% [1289400/4942000] [260.9/1000.0] [batch_t 0.328 (1.189)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 20:32:32,918 - Train: 26.09% [1289500/4942000] [260.9/1000.0] [batch_t 1.264 (0.619)] [data_t 0.936] [optim_t 0.329] [lr 0.005000] 2024-04-09 20:33:46,612 - Train: 26.09% [1289600/4942000] [260.9/1000.0] [batch_t 0.329 (0.737)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 20:34:26,160 - Train: 26.10% [1289700/4942000] [261.0/1000.0] [batch_t 0.329 (0.395)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 20:34:59,969 - Train: 26.10% [1289800/4942000] [261.0/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 20:35:27,439 - ==> Total time: 7 days, 2:38:06 Eta: 20 days, 3:08:21 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 20:35:51,084 - Train: 26.10% [1289900/4942000] [261.0/1000.0] [batch_t 0.330 (0.539)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 20:36:31,590 - Train: 26.10% [1290000/4942000] [261.0/1000.0] [batch_t 0.328 (0.405)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 20:37:08,354 - Train: 26.10% [1290100/4942000] [261.0/1000.0] [batch_t 0.328 (0.368)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 20:37:43,554 - Train: 26.11% [1290200/4942000] [261.1/1000.0] [batch_t 0.328 (0.352)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 20:38:46,256 - Train: 26.11% [1290300/4942000] [261.1/1000.0] [batch_t 0.328 (0.627)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 20:39:33,020 - Train: 26.11% [1290400/4942000] [261.1/1000.0] [batch_t 0.329 (0.468)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 20:40:14,106 - Train: 26.11% [1290500/4942000] [261.1/1000.0] [batch_t 0.328 (0.411)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 20:40:47,147 - Train: 26.11% [1290600/4942000] [261.1/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 20:41:20,891 - Train: 26.12% [1290700/4942000] [261.2/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 20:41:53,788 - Train: 26.12% [1290800/4942000] [261.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 20:42:27,005 - Train: 26.12% [1290900/4942000] [261.2/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 20:42:59,910 - Train: 26.12% [1291000/4942000] [261.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 20:43:32,964 - Train: 26.13% [1291100/4942000] [261.3/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 20:44:05,965 - Train: 26.13% [1291200/4942000] [261.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 20:44:39,022 - Train: 26.13% [1291300/4942000] [261.3/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 20:45:11,968 - Train: 26.13% [1291400/4942000] [261.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 20:45:44,850 - Train: 26.13% [1291500/4942000] [261.3/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 20:46:17,803 - Train: 26.14% [1291600/4942000] [261.4/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 20:46:50,699 - Train: 26.14% [1291700/4942000] [261.4/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 20:47:23,686 - Train: 26.14% [1291800/4942000] [261.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 20:47:56,631 - Train: 26.14% [1291900/4942000] [261.4/1000.0] [batch_t 0.321 (0.329)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-09 20:48:29,639 - Train: 26.14% [1292000/4942000] [261.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 20:49:02,733 - Train: 26.15% [1292100/4942000] [261.5/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 20:49:35,677 - Train: 26.15% [1292200/4942000] [261.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 20:50:08,569 - Train: 26.15% [1292300/4942000] [261.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-09 20:50:41,571 - Train: 26.15% [1292400/4942000] [261.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 20:51:14,638 - Train: 26.15% [1292500/4942000] [261.5/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 20:51:47,700 - Train: 26.16% [1292600/4942000] [261.6/1000.0] [batch_t 0.321 (0.331)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-09 20:52:20,979 - Train: 26.16% [1292700/4942000] [261.6/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 20:52:53,916 - Train: 26.16% [1292800/4942000] [261.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 20:53:27,020 - Train: 26.16% [1292900/4942000] [261.6/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 20:53:59,927 - Train: 26.16% [1293000/4942000] [261.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 20:54:33,018 - Train: 26.17% [1293100/4942000] [261.7/1000.0] [batch_t 0.325 (0.331)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 20:55:06,046 - Train: 26.17% [1293200/4942000] [261.7/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 20:55:39,005 - Train: 26.17% [1293300/4942000] [261.7/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 20:56:11,955 - Train: 26.17% [1293400/4942000] [261.7/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 20:56:44,967 - Train: 26.17% [1293500/4942000] [261.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 20:57:17,928 - Train: 26.18% [1293600/4942000] [261.8/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 20:57:50,884 - Train: 26.18% [1293700/4942000] [261.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-09 20:58:23,822 - Train: 26.18% [1293800/4942000] [261.8/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 20:58:56,735 - Train: 26.18% [1293900/4942000] [261.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 20:59:30,045 - Train: 26.18% [1294000/4942000] [261.8/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 21:00:03,003 - Train: 26.19% [1294100/4942000] [261.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 21:00:37,614 - Train: 26.19% [1294200/4942000] [261.9/1000.0] [batch_t 0.329 (0.346)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 21:01:11,638 - Train: 26.19% [1294300/4942000] [261.9/1000.0] [batch_t 0.332 (0.340)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 21:01:44,583 - Train: 26.19% [1294400/4942000] [261.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 21:02:18,743 - Train: 26.19% [1294500/4942000] [261.9/1000.0] [batch_t 0.328 (0.342)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 21:02:51,640 - Train: 26.20% [1294600/4942000] [262.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 21:03:30,809 - Train: 26.20% [1294700/4942000] [262.0/1000.0] [batch_t 0.329 (0.392)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 21:04:05,556 - Train: 26.20% [1294800/4942000] [262.0/1000.0] [batch_t 0.331 (0.340)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 21:04:06,879 - ==> Total time: 7 days, 3:06:46 Eta: 20 days, 1:59:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 21:04:41,124 - Train: 26.20% [1294900/4942000] [262.0/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 21:05:14,991 - Train: 26.20% [1295000/4942000] [262.0/1000.0] [batch_t 0.333 (0.339)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 21:05:47,970 - Train: 26.21% [1295100/4942000] [262.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 21:06:20,887 - Train: 26.21% [1295200/4942000] [262.1/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 21:06:53,830 - Train: 26.21% [1295300/4942000] [262.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 21:07:27,030 - Train: 26.21% [1295400/4942000] [262.1/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 21:08:00,238 - Train: 26.21% [1295500/4942000] [262.1/1000.0] [batch_t 0.335 (0.332)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-09 21:08:33,237 - Train: 26.22% [1295600/4942000] [262.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 21:09:06,245 - Train: 26.22% [1295700/4942000] [262.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 21:09:39,196 - Train: 26.22% [1295800/4942000] [262.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 21:10:12,721 - Train: 26.22% [1295900/4942000] [262.2/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 21:10:45,747 - Train: 26.22% [1296000/4942000] [262.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 21:11:18,914 - Train: 26.23% [1296100/4942000] [262.3/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 21:11:52,101 - Train: 26.23% [1296200/4942000] [262.3/1000.0] [batch_t 0.326 (0.332)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 21:12:25,376 - Train: 26.23% [1296300/4942000] [262.3/1000.0] [batch_t 0.328 (0.333)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-09 21:12:58,388 - Train: 26.23% [1296400/4942000] [262.3/1000.0] [batch_t 0.337 (0.330)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-09 21:13:31,447 - Train: 26.23% [1296500/4942000] [262.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 21:14:04,451 - Train: 26.24% [1296600/4942000] [262.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 21:14:38,213 - Train: 26.24% [1296700/4942000] [262.4/1000.0] [batch_t 0.325 (0.337)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 21:15:11,206 - Train: 26.24% [1296800/4942000] [262.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 21:15:44,161 - Train: 26.24% [1296900/4942000] [262.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 21:16:17,320 - Train: 26.24% [1297000/4942000] [262.4/1000.0] [batch_t 0.336 (0.331)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-09 21:16:50,300 - Train: 26.25% [1297100/4942000] [262.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 21:17:23,389 - Train: 26.25% [1297200/4942000] [262.5/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 21:17:56,389 - Train: 26.25% [1297300/4942000] [262.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 21:18:29,377 - Train: 26.25% [1297400/4942000] [262.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 21:19:02,440 - Train: 26.25% [1297500/4942000] [262.5/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 21:19:35,477 - Train: 26.26% [1297600/4942000] [262.6/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 21:20:08,477 - Train: 26.26% [1297700/4942000] [262.6/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-09 21:20:41,439 - Train: 26.26% [1297800/4942000] [262.6/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 21:21:14,465 - Train: 26.26% [1297900/4942000] [262.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 21:21:47,469 - Train: 26.26% [1298000/4942000] [262.6/1000.0] [batch_t 0.323 (0.330)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 21:22:20,587 - Train: 26.27% [1298100/4942000] [262.7/1000.0] [batch_t 0.333 (0.331)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 21:22:53,576 - Train: 26.27% [1298200/4942000] [262.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-09 21:23:26,616 - Train: 26.27% [1298300/4942000] [262.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-09 21:23:59,946 - Train: 26.27% [1298400/4942000] [262.7/1000.0] [batch_t 0.332 (0.333)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 21:24:32,905 - Train: 26.27% [1298500/4942000] [262.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 21:25:05,945 - Train: 26.28% [1298600/4942000] [262.8/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 21:25:38,965 - Train: 26.28% [1298700/4942000] [262.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 21:26:11,980 - Train: 26.28% [1298800/4942000] [262.8/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-09 21:26:44,987 - Train: 26.28% [1298900/4942000] [262.8/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 21:27:18,001 - Train: 26.28% [1299000/4942000] [262.8/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-09 21:27:50,975 - Train: 26.29% [1299100/4942000] [262.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-09 21:28:23,982 - Train: 26.29% [1299200/4942000] [262.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 21:28:57,023 - Train: 26.29% [1299300/4942000] [262.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 21:29:30,013 - Train: 26.29% [1299400/4942000] [262.9/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 21:30:03,066 - Train: 26.30% [1299500/4942000] [263.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 21:30:36,025 - Train: 26.30% [1299600/4942000] [263.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 21:31:08,929 - Train: 26.30% [1299700/4942000] [263.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 21:31:24,115 - ==> Total time: 7 days, 3:34:03 Eta: 20 days, 0:46:50 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 21:31:44,793 - Train: 26.30% [1299800/4942000] [263.0/1000.0] [batch_t 0.322 (0.336)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 21:32:17,727 - Train: 26.30% [1299900/4942000] [263.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-09 21:32:50,770 - Train: 26.31% [1300000/4942000] [263.1/1000.0] [batch_t 0.336 (0.330)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-09 21:33:23,736 - Train: 26.31% [1300100/4942000] [263.1/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 21:33:56,659 - Train: 26.31% [1300200/4942000] [263.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 21:34:29,558 - Train: 26.31% [1300300/4942000] [263.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 21:35:02,448 - Train: 26.31% [1300400/4942000] [263.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 21:35:35,381 - Train: 26.32% [1300500/4942000] [263.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 21:36:08,890 - Train: 26.32% [1300600/4942000] [263.2/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 21:36:41,724 - Train: 26.32% [1300700/4942000] [263.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 21:37:14,582 - Train: 26.32% [1300800/4942000] [263.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 21:37:47,472 - Train: 26.32% [1300900/4942000] [263.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 21:38:20,704 - Train: 26.33% [1301000/4942000] [263.3/1000.0] [batch_t 0.325 (0.332)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 21:38:53,562 - Train: 26.33% [1301100/4942000] [263.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 21:39:26,561 - Train: 26.33% [1301200/4942000] [263.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 21:39:59,521 - Train: 26.33% [1301300/4942000] [263.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 21:40:32,458 - Train: 26.33% [1301400/4942000] [263.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 21:41:05,341 - Train: 26.34% [1301500/4942000] [263.4/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 21:41:38,256 - Train: 26.34% [1301600/4942000] [263.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 21:42:11,131 - Train: 26.34% [1301700/4942000] [263.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 21:42:44,038 - Train: 26.34% [1301800/4942000] [263.4/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 21:43:17,025 - Train: 26.34% [1301900/4942000] [263.4/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 21:43:49,841 - Train: 26.35% [1302000/4942000] [263.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 21:44:23,383 - Train: 26.35% [1302100/4942000] [263.5/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 21:44:56,217 - Train: 26.35% [1302200/4942000] [263.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 21:45:29,152 - Train: 26.35% [1302300/4942000] [263.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 21:46:01,979 - Train: 26.35% [1302400/4942000] [263.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 21:46:36,180 - Train: 26.36% [1302500/4942000] [263.6/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 21:47:10,321 - Train: 26.36% [1302600/4942000] [263.6/1000.0] [batch_t 0.326 (0.341)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 21:47:43,271 - Train: 26.36% [1302700/4942000] [263.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 21:48:16,462 - Train: 26.36% [1302800/4942000] [263.6/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 21:48:49,322 - Train: 26.36% [1302900/4942000] [263.6/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 21:49:22,684 - Train: 26.37% [1303000/4942000] [263.7/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 21:49:55,612 - Train: 26.37% [1303100/4942000] [263.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 21:50:28,872 - Train: 26.37% [1303200/4942000] [263.7/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 21:51:01,813 - Train: 26.37% [1303300/4942000] [263.7/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 21:51:35,229 - Train: 26.37% [1303400/4942000] [263.7/1000.0] [batch_t 0.335 (0.334)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-09 21:52:08,574 - Train: 26.38% [1303500/4942000] [263.8/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 21:52:41,497 - Train: 26.38% [1303600/4942000] [263.8/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 21:53:14,947 - Train: 26.38% [1303700/4942000] [263.8/1000.0] [batch_t 0.331 (0.334)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 21:53:47,894 - Train: 26.38% [1303800/4942000] [263.8/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 21:54:21,476 - Train: 26.38% [1303900/4942000] [263.8/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 21:54:54,446 - Train: 26.39% [1304000/4942000] [263.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 21:55:28,027 - Train: 26.39% [1304100/4942000] [263.9/1000.0] [batch_t 0.331 (0.336)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 21:56:08,229 - Train: 26.39% [1304200/4942000] [263.9/1000.0] [batch_t 0.329 (0.402)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 21:56:41,132 - Train: 26.39% [1304300/4942000] [263.9/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 21:57:14,226 - Train: 26.39% [1304400/4942000] [263.9/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 21:57:47,101 - Train: 26.40% [1304500/4942000] [264.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 21:58:20,082 - Train: 26.40% [1304600/4942000] [264.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 21:58:49,046 - ==> Total time: 7 days, 4:01:28 Eta: 19 days, 23:35:00 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 21:58:55,119 - Train: 26.40% [1304700/4942000] [264.0/1000.0] [batch_t 0.330 (0.340)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 21:59:27,990 - Train: 26.40% [1304800/4942000] [264.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 22:00:00,856 - Train: 26.40% [1304900/4942000] [264.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 22:00:33,725 - Train: 26.41% [1305000/4942000] [264.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 22:01:06,626 - Train: 26.41% [1305100/4942000] [264.1/1000.0] [batch_t 0.336 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-09 22:01:39,524 - Train: 26.41% [1305200/4942000] [264.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 22:02:12,383 - Train: 26.41% [1305300/4942000] [264.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 22:02:45,357 - Train: 26.41% [1305400/4942000] [264.1/1000.0] [batch_t 0.323 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 22:03:18,175 - Train: 26.42% [1305500/4942000] [264.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 22:03:51,014 - Train: 26.42% [1305600/4942000] [264.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 22:04:23,884 - Train: 26.42% [1305700/4942000] [264.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 22:04:56,736 - Train: 26.42% [1305800/4942000] [264.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 22:05:29,684 - Train: 26.42% [1305900/4942000] [264.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 22:06:02,508 - Train: 26.43% [1306000/4942000] [264.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 22:06:35,420 - Train: 26.43% [1306100/4942000] [264.3/1000.0] [batch_t 0.320 (0.329)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-09 22:07:09,929 - Train: 26.43% [1306200/4942000] [264.3/1000.0] [batch_t 0.333 (0.345)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 22:07:42,906 - Train: 26.43% [1306300/4942000] [264.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 22:08:15,877 - Train: 26.43% [1306400/4942000] [264.3/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 22:08:48,788 - Train: 26.44% [1306500/4942000] [264.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 22:09:21,710 - Train: 26.44% [1306600/4942000] [264.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 22:09:54,667 - Train: 26.44% [1306700/4942000] [264.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 22:10:27,780 - Train: 26.44% [1306800/4942000] [264.4/1000.0] [batch_t 0.494 (0.331)] [data_t 0.002] [optim_t 0.493] [lr 0.005000] 2024-04-09 22:11:00,673 - Train: 26.44% [1306900/4942000] [264.4/1000.0] [batch_t 0.336 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-09 22:11:33,567 - Train: 26.45% [1307000/4942000] [264.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 22:12:06,524 - Train: 26.45% [1307100/4942000] [264.5/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 22:12:39,419 - Train: 26.45% [1307200/4942000] [264.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 22:13:12,392 - Train: 26.45% [1307300/4942000] [264.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 22:13:45,376 - Train: 26.45% [1307400/4942000] [264.5/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 22:14:18,346 - Train: 26.46% [1307500/4942000] [264.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 22:14:51,254 - Train: 26.46% [1307600/4942000] [264.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 22:15:24,161 - Train: 26.46% [1307700/4942000] [264.6/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 22:15:57,067 - Train: 26.46% [1307800/4942000] [264.6/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 22:16:30,018 - Train: 26.46% [1307900/4942000] [264.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 22:17:02,912 - Train: 26.47% [1308000/4942000] [264.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 22:17:35,810 - Train: 26.47% [1308100/4942000] [264.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 22:18:08,689 - Train: 26.47% [1308200/4942000] [264.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 22:18:41,684 - Train: 26.47% [1308300/4942000] [264.7/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 22:19:14,625 - Train: 26.48% [1308400/4942000] [264.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 22:19:47,569 - Train: 26.48% [1308500/4942000] [264.8/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 22:20:20,490 - Train: 26.48% [1308600/4942000] [264.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 22:20:53,486 - Train: 26.48% [1308700/4942000] [264.8/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 22:21:26,428 - Train: 26.48% [1308800/4942000] [264.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 22:21:59,317 - Train: 26.49% [1308900/4942000] [264.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 22:22:32,277 - Train: 26.49% [1309000/4942000] [264.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 22:23:05,298 - Train: 26.49% [1309100/4942000] [264.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 22:23:38,212 - Train: 26.49% [1309200/4942000] [264.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 22:24:11,260 - Train: 26.49% [1309300/4942000] [264.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 22:24:44,188 - Train: 26.50% [1309400/4942000] [265.0/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 22:25:17,168 - Train: 26.50% [1309500/4942000] [265.0/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 22:25:50,179 - Train: 26.50% [1309600/4942000] [265.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 22:26:00,105 - ==> Total time: 7 days, 4:28:39 Eta: 19 days, 22:22:52 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 22:26:28,137 - Train: 26.50% [1309700/4942000] [265.0/1000.0] [batch_t 0.327 (0.361)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 22:27:01,128 - Train: 26.50% [1309800/4942000] [265.0/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 22:27:34,041 - Train: 26.51% [1309900/4942000] [265.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 22:28:06,923 - Train: 26.51% [1310000/4942000] [265.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 22:28:39,824 - Train: 26.51% [1310100/4942000] [265.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 22:29:12,706 - Train: 26.51% [1310200/4942000] [265.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 22:29:45,622 - Train: 26.51% [1310300/4942000] [265.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 22:30:18,546 - Train: 26.52% [1310400/4942000] [265.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 22:30:51,455 - Train: 26.52% [1310500/4942000] [265.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 22:31:24,342 - Train: 26.52% [1310600/4942000] [265.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 22:31:57,198 - Train: 26.52% [1310700/4942000] [265.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 22:32:30,070 - Train: 26.52% [1310800/4942000] [265.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 22:33:03,093 - Train: 26.53% [1310900/4942000] [265.3/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 22:33:36,022 - Train: 26.53% [1311000/4942000] [265.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 22:34:09,065 - Train: 26.53% [1311100/4942000] [265.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 22:34:41,955 - Train: 26.53% [1311200/4942000] [265.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 22:35:14,852 - Train: 26.53% [1311300/4942000] [265.3/1000.0] [batch_t 0.337 (0.329)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-09 22:35:47,750 - Train: 26.54% [1311400/4942000] [265.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 22:36:20,612 - Train: 26.54% [1311500/4942000] [265.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 22:36:53,495 - Train: 26.54% [1311600/4942000] [265.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 22:37:26,443 - Train: 26.54% [1311700/4942000] [265.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 22:37:59,459 - Train: 26.54% [1311800/4942000] [265.4/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 22:38:32,540 - Train: 26.55% [1311900/4942000] [265.5/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 22:39:05,509 - Train: 26.55% [1312000/4942000] [265.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 22:39:38,580 - Train: 26.55% [1312100/4942000] [265.5/1000.0] [batch_t 0.333 (0.331)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 22:40:11,606 - Train: 26.55% [1312200/4942000] [265.5/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 22:40:44,591 - Train: 26.55% [1312300/4942000] [265.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 22:41:17,575 - Train: 26.56% [1312400/4942000] [265.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 22:41:50,543 - Train: 26.56% [1312500/4942000] [265.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 22:42:23,763 - Train: 26.56% [1312600/4942000] [265.6/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 22:42:56,726 - Train: 26.56% [1312700/4942000] [265.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 22:43:29,737 - Train: 26.56% [1312800/4942000] [265.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 22:44:02,662 - Train: 26.57% [1312900/4942000] [265.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 22:44:35,633 - Train: 26.57% [1313000/4942000] [265.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 22:45:08,603 - Train: 26.57% [1313100/4942000] [265.7/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 22:45:41,501 - Train: 26.57% [1313200/4942000] [265.7/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 22:46:14,448 - Train: 26.57% [1313300/4942000] [265.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 22:46:47,448 - Train: 26.58% [1313400/4942000] [265.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 22:47:20,512 - Train: 26.58% [1313500/4942000] [265.8/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 22:47:53,545 - Train: 26.58% [1313600/4942000] [265.8/1000.0] [batch_t 0.342 (0.330)] [data_t 0.002] [optim_t 0.340] [lr 0.005000] 2024-04-09 22:48:26,983 - Train: 26.58% [1313700/4942000] [265.8/1000.0] [batch_t 0.331 (0.334)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 22:48:59,924 - Train: 26.58% [1313800/4942000] [265.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 22:49:32,906 - Train: 26.59% [1313900/4942000] [265.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 22:50:06,035 - Train: 26.59% [1314000/4942000] [265.9/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 22:50:39,013 - Train: 26.59% [1314100/4942000] [265.9/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 22:51:12,053 - Train: 26.59% [1314200/4942000] [265.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 22:51:45,013 - Train: 26.59% [1314300/4942000] [265.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 22:52:17,993 - Train: 26.60% [1314400/4942000] [266.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 22:52:50,970 - Train: 26.60% [1314500/4942000] [266.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 22:53:14,681 - ==> Total time: 7 days, 4:55:53 Eta: 19 days, 21:11:14 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 22:53:26,254 - Train: 26.60% [1314600/4942000] [266.0/1000.0] [batch_t 0.332 (0.333)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 22:53:59,175 - Train: 26.60% [1314700/4942000] [266.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 22:54:32,077 - Train: 26.60% [1314800/4942000] [266.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 22:55:05,059 - Train: 26.61% [1314900/4942000] [266.1/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 22:55:37,989 - Train: 26.61% [1315000/4942000] [266.1/1000.0] [batch_t 0.337 (0.329)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-09 22:56:10,889 - Train: 26.61% [1315100/4942000] [266.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 22:56:43,790 - Train: 26.61% [1315200/4942000] [266.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 22:57:16,814 - Train: 26.61% [1315300/4942000] [266.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 22:57:49,868 - Train: 26.62% [1315400/4942000] [266.2/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-09 22:58:22,742 - Train: 26.62% [1315500/4942000] [266.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 22:58:55,631 - Train: 26.62% [1315600/4942000] [266.2/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 22:59:28,539 - Train: 26.62% [1315700/4942000] [266.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 23:00:01,411 - Train: 26.62% [1315800/4942000] [266.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 23:00:34,331 - Train: 26.63% [1315900/4942000] [266.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 23:01:07,289 - Train: 26.63% [1316000/4942000] [266.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 23:01:40,188 - Train: 26.63% [1316100/4942000] [266.3/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 23:02:13,101 - Train: 26.63% [1316200/4942000] [266.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 23:02:45,981 - Train: 26.63% [1316300/4942000] [266.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 23:03:18,932 - Train: 26.64% [1316400/4942000] [266.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 23:03:51,850 - Train: 26.64% [1316500/4942000] [266.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 23:04:24,749 - Train: 26.64% [1316600/4942000] [266.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 23:04:57,648 - Train: 26.64% [1316700/4942000] [266.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 23:05:30,673 - Train: 26.65% [1316800/4942000] [266.5/1000.0] [batch_t 0.322 (0.330)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-09 23:06:03,604 - Train: 26.65% [1316900/4942000] [266.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 23:06:36,487 - Train: 26.65% [1317000/4942000] [266.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 23:07:09,343 - Train: 26.65% [1317100/4942000] [266.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 23:07:42,223 - Train: 26.65% [1317200/4942000] [266.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 23:08:15,171 - Train: 26.66% [1317300/4942000] [266.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 23:08:48,105 - Train: 26.66% [1317400/4942000] [266.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 23:09:20,998 - Train: 26.66% [1317500/4942000] [266.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 23:09:53,868 - Train: 26.66% [1317600/4942000] [266.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 23:10:26,748 - Train: 26.66% [1317700/4942000] [266.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 23:10:59,621 - Train: 26.67% [1317800/4942000] [266.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 23:11:32,547 - Train: 26.67% [1317900/4942000] [266.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 23:12:05,384 - Train: 26.67% [1318000/4942000] [266.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 23:12:38,290 - Train: 26.67% [1318100/4942000] [266.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 23:13:11,338 - Train: 26.67% [1318200/4942000] [266.7/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 23:13:44,206 - Train: 26.68% [1318300/4942000] [266.8/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 23:14:17,106 - Train: 26.68% [1318400/4942000] [266.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 23:14:50,174 - Train: 26.68% [1318500/4942000] [266.8/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 23:15:23,110 - Train: 26.68% [1318600/4942000] [266.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 23:15:56,061 - Train: 26.68% [1318700/4942000] [266.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 23:16:28,995 - Train: 26.69% [1318800/4942000] [266.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 23:17:01,910 - Train: 26.69% [1318900/4942000] [266.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 23:17:34,841 - Train: 26.69% [1319000/4942000] [266.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 23:18:07,734 - Train: 26.69% [1319100/4942000] [266.9/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 23:18:40,615 - Train: 26.69% [1319200/4942000] [266.9/1000.0] [batch_t 0.336 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-09 23:19:13,638 - Train: 26.70% [1319300/4942000] [267.0/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 23:19:46,554 - Train: 26.70% [1319400/4942000] [267.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 23:20:19,416 - Train: 26.70% [1319500/4942000] [267.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 23:20:24,019 - ==> Total time: 7 days, 5:23:03 Eta: 19 days, 19:59:41 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 23:20:54,458 - Train: 26.70% [1319600/4942000] [267.0/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 23:21:27,407 - Train: 26.70% [1319700/4942000] [267.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 23:22:00,300 - Train: 26.71% [1319800/4942000] [267.1/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-09 23:22:42,818 - Train: 26.71% [1319900/4942000] [267.1/1000.0] [batch_t 1.328 (0.425)] [data_t 1.001] [optim_t 0.327] [lr 0.005000] 2024-04-09 23:23:18,445 - Train: 26.71% [1320000/4942000] [267.1/1000.0] [batch_t 0.327 (0.356)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 23:23:51,370 - Train: 26.71% [1320100/4942000] [267.1/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-09 23:24:24,291 - Train: 26.71% [1320200/4942000] [267.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 23:24:57,169 - Train: 26.72% [1320300/4942000] [267.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 23:25:31,054 - Train: 26.72% [1320400/4942000] [267.2/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 23:26:04,261 - Train: 26.72% [1320500/4942000] [267.2/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 23:26:37,209 - Train: 26.72% [1320600/4942000] [267.2/1000.0] [batch_t 0.337 (0.329)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-09 23:27:10,089 - Train: 26.72% [1320700/4942000] [267.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 23:27:43,077 - Train: 26.73% [1320800/4942000] [267.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 23:28:17,956 - Train: 26.73% [1320900/4942000] [267.3/1000.0] [batch_t 0.327 (0.349)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 23:28:51,001 - Train: 26.73% [1321000/4942000] [267.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 23:29:23,904 - Train: 26.73% [1321100/4942000] [267.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 23:29:56,727 - Train: 26.73% [1321200/4942000] [267.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 23:30:29,622 - Train: 26.74% [1321300/4942000] [267.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 23:31:02,471 - Train: 26.74% [1321400/4942000] [267.4/1000.0] [batch_t 0.321 (0.328)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-09 23:31:35,513 - Train: 26.74% [1321500/4942000] [267.4/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 23:32:08,413 - Train: 26.74% [1321600/4942000] [267.4/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 23:32:41,301 - Train: 26.74% [1321700/4942000] [267.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 23:33:14,173 - Train: 26.75% [1321800/4942000] [267.5/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 23:33:47,148 - Train: 26.75% [1321900/4942000] [267.5/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 23:34:21,893 - Train: 26.75% [1322000/4942000] [267.5/1000.0] [batch_t 0.328 (0.347)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 23:34:54,794 - Train: 26.75% [1322100/4942000] [267.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 23:35:27,696 - Train: 26.75% [1322200/4942000] [267.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 23:36:00,559 - Train: 26.76% [1322300/4942000] [267.6/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 23:36:33,545 - Train: 26.76% [1322400/4942000] [267.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 23:37:06,479 - Train: 26.76% [1322500/4942000] [267.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 23:37:39,343 - Train: 26.76% [1322600/4942000] [267.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 23:38:12,309 - Train: 26.76% [1322700/4942000] [267.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 23:38:45,324 - Train: 26.77% [1322800/4942000] [267.7/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-09 23:39:18,224 - Train: 26.77% [1322900/4942000] [267.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 23:39:51,165 - Train: 26.77% [1323000/4942000] [267.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 23:40:25,074 - Train: 26.77% [1323100/4942000] [267.7/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 23:40:57,976 - Train: 26.77% [1323200/4942000] [267.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 23:41:32,072 - Train: 26.78% [1323300/4942000] [267.8/1000.0] [batch_t 0.331 (0.341)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 23:42:04,958 - Train: 26.78% [1323400/4942000] [267.8/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-09 23:42:37,899 - Train: 26.78% [1323500/4942000] [267.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 23:43:10,783 - Train: 26.78% [1323600/4942000] [267.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 23:43:43,619 - Train: 26.78% [1323700/4942000] [267.8/1000.0] [batch_t 0.339 (0.328)] [data_t 0.002] [optim_t 0.337] [lr 0.005000] 2024-04-09 23:44:17,376 - Train: 26.79% [1323800/4942000] [267.9/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 23:44:50,449 - Train: 26.79% [1323900/4942000] [267.9/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-09 23:45:23,611 - Train: 26.79% [1324000/4942000] [267.9/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 23:45:56,528 - Train: 26.79% [1324100/4942000] [267.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 23:46:29,411 - Train: 26.79% [1324200/4942000] [267.9/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-09 23:47:02,374 - Train: 26.80% [1324300/4942000] [268.0/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 23:47:35,333 - Train: 26.80% [1324400/4942000] [268.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 23:47:53,771 - ==> Total time: 7 days, 5:50:32 Eta: 19 days, 18:49:24 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-09 23:48:10,513 - Train: 26.80% [1324500/4942000] [268.0/1000.0] [batch_t 0.335 (0.333)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-09 23:48:44,242 - Train: 26.80% [1324600/4942000] [268.0/1000.0] [batch_t 0.758 (0.337)] [data_t 0.431] [optim_t 0.327] [lr 0.005000] 2024-04-09 23:49:18,396 - Train: 26.80% [1324700/4942000] [268.0/1000.0] [batch_t 0.326 (0.341)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 23:49:51,372 - Train: 26.81% [1324800/4942000] [268.1/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 23:50:25,224 - Train: 26.81% [1324900/4942000] [268.1/1000.0] [batch_t 0.332 (0.338)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-09 23:50:58,162 - Train: 26.81% [1325000/4942000] [268.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 23:51:31,715 - Train: 26.81% [1325100/4942000] [268.1/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 23:52:04,766 - Train: 26.82% [1325200/4942000] [268.2/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 23:52:37,625 - Train: 26.82% [1325300/4942000] [268.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 23:53:10,576 - Train: 26.82% [1325400/4942000] [268.2/1000.0] [batch_t 0.322 (0.329)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-09 23:53:43,451 - Train: 26.82% [1325500/4942000] [268.2/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 23:54:16,398 - Train: 26.82% [1325600/4942000] [268.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 23:54:49,347 - Train: 26.83% [1325700/4942000] [268.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 23:55:22,228 - Train: 26.83% [1325800/4942000] [268.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 23:55:55,212 - Train: 26.83% [1325900/4942000] [268.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-09 23:56:28,630 - Train: 26.83% [1326000/4942000] [268.3/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 23:57:01,580 - Train: 26.83% [1326100/4942000] [268.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 23:57:34,521 - Train: 26.84% [1326200/4942000] [268.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 23:58:07,387 - Train: 26.84% [1326300/4942000] [268.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-09 23:58:40,312 - Train: 26.84% [1326400/4942000] [268.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-09 23:59:13,224 - Train: 26.84% [1326500/4942000] [268.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-09 23:59:46,301 - Train: 26.84% [1326600/4942000] [268.4/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 00:00:19,214 - Train: 26.85% [1326700/4942000] [268.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 00:00:52,134 - Train: 26.85% [1326800/4942000] [268.5/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-10 00:01:25,035 - Train: 26.85% [1326900/4942000] [268.5/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 00:01:58,666 - Train: 26.85% [1327000/4942000] [268.5/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 00:02:31,570 - Train: 26.85% [1327100/4942000] [268.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 00:03:05,275 - Train: 26.86% [1327200/4942000] [268.6/1000.0] [batch_t 1.142 (0.337)] [data_t 0.815] [optim_t 0.326] [lr 0.005000] 2024-04-10 00:03:39,728 - Train: 26.86% [1327300/4942000] [268.6/1000.0] [batch_t 0.325 (0.344)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-10 00:04:12,994 - Train: 26.86% [1327400/4942000] [268.6/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 00:04:45,859 - Train: 26.86% [1327500/4942000] [268.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 00:05:18,736 - Train: 26.86% [1327600/4942000] [268.6/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 00:05:51,600 - Train: 26.87% [1327700/4942000] [268.7/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 00:06:24,593 - Train: 26.87% [1327800/4942000] [268.7/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 00:06:57,454 - Train: 26.87% [1327900/4942000] [268.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 00:07:31,016 - Train: 26.87% [1328000/4942000] [268.7/1000.0] [batch_t 0.335 (0.336)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-10 00:08:03,944 - Train: 26.87% [1328100/4942000] [268.7/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 00:08:37,728 - Train: 26.88% [1328200/4942000] [268.8/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 00:09:11,411 - Train: 26.88% [1328300/4942000] [268.8/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 00:09:44,353 - Train: 26.88% [1328400/4942000] [268.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 00:10:17,184 - Train: 26.88% [1328500/4942000] [268.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 00:10:50,048 - Train: 26.88% [1328600/4942000] [268.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 00:11:24,412 - Train: 26.89% [1328700/4942000] [268.9/1000.0] [batch_t 0.321 (0.344)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-10 00:11:57,363 - Train: 26.89% [1328800/4942000] [268.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 00:12:30,269 - Train: 26.89% [1328900/4942000] [268.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 00:13:03,293 - Train: 26.89% [1329000/4942000] [268.9/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 00:13:36,990 - Train: 26.89% [1329100/4942000] [268.9/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 00:14:09,890 - Train: 26.90% [1329200/4942000] [269.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 00:14:42,780 - Train: 26.90% [1329300/4942000] [269.0/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 00:15:16,614 - ==> Total time: 7 days, 6:17:55 Eta: 19 days, 17:39:07 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-10 00:15:19,481 - Train: 26.90% [1329400/4942000] [269.0/1000.0] [batch_t 0.325 (0.350)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-10 00:15:52,368 - Train: 26.90% [1329500/4942000] [269.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 00:16:26,101 - Train: 26.90% [1329600/4942000] [269.0/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 00:16:58,947 - Train: 26.91% [1329700/4942000] [269.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 00:17:31,795 - Train: 26.91% [1329800/4942000] [269.1/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-10 00:18:04,652 - Train: 26.91% [1329900/4942000] [269.1/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 00:18:37,533 - Train: 26.91% [1330000/4942000] [269.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 00:19:10,936 - Train: 26.91% [1330100/4942000] [269.1/1000.0] [batch_t 0.334 (0.334)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-10 00:19:43,831 - Train: 26.92% [1330200/4942000] [269.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 00:20:16,711 - Train: 26.92% [1330300/4942000] [269.2/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-10 00:20:49,586 - Train: 26.92% [1330400/4942000] [269.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 00:21:29,048 - Train: 26.92% [1330500/4942000] [269.2/1000.0] [batch_t 0.327 (0.395)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 00:22:01,946 - Train: 26.92% [1330600/4942000] [269.2/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-10 00:22:38,738 - Train: 26.93% [1330700/4942000] [269.3/1000.0] [batch_t 0.333 (0.368)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 00:23:12,826 - Train: 26.93% [1330800/4942000] [269.3/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 00:23:48,826 - Train: 26.93% [1330900/4942000] [269.3/1000.0] [batch_t 0.327 (0.360)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 00:24:22,425 - Train: 26.93% [1331000/4942000] [269.3/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 00:24:55,275 - Train: 26.93% [1331100/4942000] [269.3/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 00:25:29,387 - Train: 26.94% [1331200/4942000] [269.4/1000.0] [batch_t 0.326 (0.341)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 00:26:02,257 - Train: 26.94% [1331300/4942000] [269.4/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-10 00:26:35,838 - Train: 26.94% [1331400/4942000] [269.4/1000.0] [batch_t 0.325 (0.336)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-10 00:27:08,734 - Train: 26.94% [1331500/4942000] [269.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 00:27:41,593 - Train: 26.94% [1331600/4942000] [269.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-10 00:28:15,408 - Train: 26.95% [1331700/4942000] [269.5/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 00:28:48,246 - Train: 26.95% [1331800/4942000] [269.5/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-10 00:29:21,088 - Train: 26.95% [1331900/4942000] [269.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 00:29:53,944 - Train: 26.95% [1332000/4942000] [269.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 00:30:26,854 - Train: 26.95% [1332100/4942000] [269.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 00:30:59,740 - Train: 26.96% [1332200/4942000] [269.6/1000.0] [batch_t 0.321 (0.329)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-10 00:31:32,829 - Train: 26.96% [1332300/4942000] [269.6/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 00:32:05,758 - Train: 26.96% [1332400/4942000] [269.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 00:32:38,666 - Train: 26.96% [1332500/4942000] [269.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 00:33:11,604 - Train: 26.96% [1332600/4942000] [269.6/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-10 00:33:44,564 - Train: 26.97% [1332700/4942000] [269.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 00:34:17,527 - Train: 26.97% [1332800/4942000] [269.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 00:34:50,445 - Train: 26.97% [1332900/4942000] [269.7/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 00:35:23,489 - Train: 26.97% [1333000/4942000] [269.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 00:35:56,388 - Train: 26.97% [1333100/4942000] [269.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 00:36:29,418 - Train: 26.98% [1333200/4942000] [269.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 00:37:02,331 - Train: 26.98% [1333300/4942000] [269.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 00:37:35,427 - Train: 26.98% [1333400/4942000] [269.8/1000.0] [batch_t 0.325 (0.331)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-10 00:38:08,599 - Train: 26.98% [1333500/4942000] [269.8/1000.0] [batch_t 0.333 (0.332)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 00:38:41,636 - Train: 26.99% [1333600/4942000] [269.9/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 00:39:14,853 - Train: 26.99% [1333700/4942000] [269.9/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 00:39:48,071 - Train: 26.99% [1333800/4942000] [269.9/1000.0] [batch_t 0.332 (0.332)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-10 00:40:22,275 - Train: 26.99% [1333900/4942000] [269.9/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 00:40:55,297 - Train: 26.99% [1334000/4942000] [269.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 00:41:29,498 - Train: 27.00% [1334100/4942000] [270.0/1000.0] [batch_t 0.332 (0.342)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-10 00:42:02,617 - Train: 27.00% [1334200/4942000] [270.0/1000.0] [batch_t 0.333 (0.331)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 00:42:37,651 - Train: 27.00% [1334300/4942000] [270.0/1000.0] [batch_t 0.334 (0.350)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-10 00:42:50,905 - ==> Total time: 7 days, 6:45:30 Eta: 19 days, 16:29:41 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-10 00:43:13,256 - Train: 27.00% [1334400/4942000] [270.0/1000.0] [batch_t 0.338 (0.333)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-10 00:43:46,157 - Train: 27.00% [1334500/4942000] [270.0/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 00:44:19,987 - Train: 27.01% [1334600/4942000] [270.1/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 00:44:52,998 - Train: 27.01% [1334700/4942000] [270.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 00:45:26,889 - Train: 27.01% [1334800/4942000] [270.1/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 00:45:59,918 - Train: 27.01% [1334900/4942000] [270.1/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 00:46:33,401 - Train: 27.01% [1335000/4942000] [270.1/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 00:47:07,407 - Train: 27.02% [1335100/4942000] [270.2/1000.0] [batch_t 0.332 (0.340)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-10 00:47:40,506 - Train: 27.02% [1335200/4942000] [270.2/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 00:48:14,452 - Train: 27.02% [1335300/4942000] [270.2/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 00:48:47,421 - Train: 27.02% [1335400/4942000] [270.2/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 00:49:21,879 - Train: 27.02% [1335500/4942000] [270.2/1000.0] [batch_t 0.331 (0.344)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 00:49:54,904 - Train: 27.03% [1335600/4942000] [270.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 00:50:28,648 - Train: 27.03% [1335700/4942000] [270.3/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 00:51:01,631 - Train: 27.03% [1335800/4942000] [270.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 00:51:35,011 - Train: 27.03% [1335900/4942000] [270.3/1000.0] [batch_t 0.332 (0.334)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-10 00:52:08,324 - Train: 27.03% [1336000/4942000] [270.3/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 00:52:41,309 - Train: 27.04% [1336100/4942000] [270.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 00:53:14,271 - Train: 27.04% [1336200/4942000] [270.4/1000.0] [batch_t 0.323 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-10 00:53:47,243 - Train: 27.04% [1336300/4942000] [270.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 00:54:20,488 - Train: 27.04% [1336400/4942000] [270.4/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 00:54:53,676 - Train: 27.04% [1336500/4942000] [270.4/1000.0] [batch_t 0.326 (0.332)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 00:55:27,384 - Train: 27.05% [1336600/4942000] [270.5/1000.0] [batch_t 0.331 (0.337)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 00:56:00,353 - Train: 27.05% [1336700/4942000] [270.5/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-10 00:56:34,178 - Train: 27.05% [1336800/4942000] [270.5/1000.0] [batch_t 0.332 (0.338)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 00:57:07,168 - Train: 27.05% [1336900/4942000] [270.5/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 00:57:40,107 - Train: 27.05% [1337000/4942000] [270.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 00:58:14,205 - Train: 27.06% [1337100/4942000] [270.6/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 00:58:47,182 - Train: 27.06% [1337200/4942000] [270.6/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-10 00:59:21,165 - Train: 27.06% [1337300/4942000] [270.6/1000.0] [batch_t 0.330 (0.340)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 00:59:54,217 - Train: 27.06% [1337400/4942000] [270.6/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-10 01:00:27,183 - Train: 27.06% [1337500/4942000] [270.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 01:01:00,175 - Train: 27.07% [1337600/4942000] [270.7/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-10 01:01:33,168 - Train: 27.07% [1337700/4942000] [270.7/1000.0] [batch_t 0.336 (0.330)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-10 01:02:06,055 - Train: 27.07% [1337800/4942000] [270.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 01:02:39,065 - Train: 27.07% [1337900/4942000] [270.7/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 01:03:12,621 - Train: 27.07% [1338000/4942000] [270.7/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 01:03:45,650 - Train: 27.08% [1338100/4942000] [270.8/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-10 01:04:18,567 - Train: 27.08% [1338200/4942000] [270.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-10 01:04:51,667 - Train: 27.08% [1338300/4942000] [270.8/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 01:05:24,740 - Train: 27.08% [1338400/4942000] [270.8/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 01:05:57,625 - Train: 27.08% [1338500/4942000] [270.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 01:06:30,688 - Train: 27.09% [1338600/4942000] [270.9/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 01:07:03,687 - Train: 27.09% [1338700/4942000] [270.9/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-10 01:07:36,713 - Train: 27.09% [1338800/4942000] [270.9/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-10 01:08:10,029 - Train: 27.09% [1338900/4942000] [270.9/1000.0] [batch_t 0.332 (0.333)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-10 01:08:42,983 - Train: 27.09% [1339000/4942000] [270.9/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 01:09:16,099 - Train: 27.10% [1339100/4942000] [271.0/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 01:09:49,153 - Train: 27.10% [1339200/4942000] [271.0/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 01:10:16,149 - ==> Total time: 7 days, 7:12:55 Eta: 19 days, 15:20:08 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-10 01:10:24,262 - Train: 27.10% [1339300/4942000] [271.0/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 01:10:57,226 - Train: 27.10% [1339400/4942000] [271.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 01:11:30,175 - Train: 27.10% [1339500/4942000] [271.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 01:12:03,084 - Train: 27.11% [1339600/4942000] [271.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 01:12:36,144 - Train: 27.11% [1339700/4942000] [271.1/1000.0] [batch_t 0.337 (0.330)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-10 01:13:09,198 - Train: 27.11% [1339800/4942000] [271.1/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 01:13:42,233 - Train: 27.11% [1339900/4942000] [271.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 01:14:15,285 - Train: 27.11% [1340000/4942000] [271.1/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 01:14:48,190 - Train: 27.12% [1340100/4942000] [271.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 01:15:21,174 - Train: 27.12% [1340200/4942000] [271.2/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 01:15:54,198 - Train: 27.12% [1340300/4942000] [271.2/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 01:16:27,218 - Train: 27.12% [1340400/4942000] [271.2/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 01:17:00,273 - Train: 27.12% [1340500/4942000] [271.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 01:17:33,657 - Train: 27.13% [1340600/4942000] [271.3/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 01:18:06,710 - Train: 27.13% [1340700/4942000] [271.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 01:18:39,942 - Train: 27.13% [1340800/4942000] [271.3/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 01:19:12,990 - Train: 27.13% [1340900/4942000] [271.3/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 01:19:46,067 - Train: 27.13% [1341000/4942000] [271.3/1000.0] [batch_t 0.335 (0.331)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-10 01:20:19,186 - Train: 27.14% [1341100/4942000] [271.4/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 01:20:52,235 - Train: 27.14% [1341200/4942000] [271.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 01:21:25,205 - Train: 27.14% [1341300/4942000] [271.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 01:21:58,235 - Train: 27.14% [1341400/4942000] [271.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 01:22:36,369 - Train: 27.14% [1341500/4942000] [271.4/1000.0] [batch_t 0.327 (0.381)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 01:23:11,771 - Train: 27.15% [1341600/4942000] [271.5/1000.0] [batch_t 0.329 (0.354)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 01:23:44,734 - Train: 27.15% [1341700/4942000] [271.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 01:24:17,730 - Train: 27.15% [1341800/4942000] [271.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 01:24:50,704 - Train: 27.15% [1341900/4942000] [271.5/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 01:25:23,665 - Train: 27.15% [1342000/4942000] [271.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 01:25:56,613 - Train: 27.16% [1342100/4942000] [271.6/1000.0] [batch_t 0.336 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-10 01:26:29,694 - Train: 27.16% [1342200/4942000] [271.6/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-10 01:27:02,685 - Train: 27.16% [1342300/4942000] [271.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 01:27:35,698 - Train: 27.16% [1342400/4942000] [271.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 01:28:08,682 - Train: 27.17% [1342500/4942000] [271.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 01:28:41,702 - Train: 27.17% [1342600/4942000] [271.7/1000.0] [batch_t 0.337 (0.330)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-10 01:29:14,742 - Train: 27.17% [1342700/4942000] [271.7/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 01:29:47,734 - Train: 27.17% [1342800/4942000] [271.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 01:30:20,704 - Train: 27.17% [1342900/4942000] [271.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 01:30:53,850 - Train: 27.18% [1343000/4942000] [271.8/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 01:31:26,776 - Train: 27.18% [1343100/4942000] [271.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 01:31:59,741 - Train: 27.18% [1343200/4942000] [271.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 01:32:32,662 - Train: 27.18% [1343300/4942000] [271.8/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 01:33:05,626 - Train: 27.18% [1343400/4942000] [271.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 01:33:38,544 - Train: 27.19% [1343500/4942000] [271.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 01:34:11,467 - Train: 27.19% [1343600/4942000] [271.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 01:34:44,596 - Train: 27.19% [1343700/4942000] [271.9/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 01:35:17,538 - Train: 27.19% [1343800/4942000] [271.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 01:35:50,555 - Train: 27.19% [1343900/4942000] [271.9/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-10 01:36:23,430 - Train: 27.20% [1344000/4942000] [272.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 01:36:56,480 - Train: 27.20% [1344100/4942000] [272.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 01:37:29,477 - Train: 27.20% [1344200/4942000] [272.0/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-10 01:37:37,386 - ==> Total time: 7 days, 7:40:16 Eta: 19 days, 14:10:44 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-10 01:38:04,393 - Train: 27.20% [1344300/4942000] [272.0/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 01:38:38,425 - Train: 27.20% [1344400/4942000] [272.0/1000.0] [batch_t 0.333 (0.340)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 01:39:11,401 - Train: 27.21% [1344500/4942000] [272.1/1000.0] [batch_t 0.321 (0.330)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-10 01:39:44,323 - Train: 27.21% [1344600/4942000] [272.1/1000.0] [batch_t 0.336 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-10 01:40:17,309 - Train: 27.21% [1344700/4942000] [272.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 01:40:50,247 - Train: 27.21% [1344800/4942000] [272.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 01:41:23,615 - Train: 27.21% [1344900/4942000] [272.1/1000.0] [batch_t 0.325 (0.334)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-10 01:41:56,775 - Train: 27.22% [1345000/4942000] [272.2/1000.0] [batch_t 0.324 (0.332)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-10 01:42:29,719 - Train: 27.22% [1345100/4942000] [272.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 01:43:02,639 - Train: 27.22% [1345200/4942000] [272.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 01:43:35,541 - Train: 27.22% [1345300/4942000] [272.2/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-10 01:44:08,534 - Train: 27.22% [1345400/4942000] [272.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 01:44:41,538 - Train: 27.23% [1345500/4942000] [272.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 01:45:14,423 - Train: 27.23% [1345600/4942000] [272.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 01:45:47,317 - Train: 27.23% [1345700/4942000] [272.3/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-10 01:46:20,298 - Train: 27.23% [1345800/4942000] [272.3/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 01:46:53,204 - Train: 27.23% [1345900/4942000] [272.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 01:47:26,083 - Train: 27.24% [1346000/4942000] [272.4/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 01:47:59,027 - Train: 27.24% [1346100/4942000] [272.4/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-10 01:48:32,063 - Train: 27.24% [1346200/4942000] [272.4/1000.0] [batch_t 0.334 (0.330)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-10 01:49:04,931 - Train: 27.24% [1346300/4942000] [272.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 01:49:37,973 - Train: 27.24% [1346400/4942000] [272.4/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 01:50:10,897 - Train: 27.25% [1346500/4942000] [272.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 01:50:43,820 - Train: 27.25% [1346600/4942000] [272.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 01:51:16,703 - Train: 27.25% [1346700/4942000] [272.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 01:51:49,663 - Train: 27.25% [1346800/4942000] [272.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 01:52:22,524 - Train: 27.25% [1346900/4942000] [272.5/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-10 01:52:55,621 - Train: 27.26% [1347000/4942000] [272.6/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 01:53:28,504 - Train: 27.26% [1347100/4942000] [272.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 01:54:01,416 - Train: 27.26% [1347200/4942000] [272.6/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-10 01:54:34,385 - Train: 27.26% [1347300/4942000] [272.6/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-10 01:55:07,328 - Train: 27.26% [1347400/4942000] [272.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 01:55:40,173 - Train: 27.27% [1347500/4942000] [272.7/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 01:56:13,140 - Train: 27.27% [1347600/4942000] [272.7/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 01:56:46,009 - Train: 27.27% [1347700/4942000] [272.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 01:57:19,030 - Train: 27.27% [1347800/4942000] [272.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 01:57:51,902 - Train: 27.27% [1347900/4942000] [272.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 01:58:24,771 - Train: 27.28% [1348000/4942000] [272.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 01:58:57,737 - Train: 27.28% [1348100/4942000] [272.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 01:59:30,711 - Train: 27.28% [1348200/4942000] [272.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 02:00:03,540 - Train: 27.28% [1348300/4942000] [272.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 02:00:36,494 - Train: 27.28% [1348400/4942000] [272.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 02:01:09,440 - Train: 27.29% [1348500/4942000] [272.9/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 02:01:42,389 - Train: 27.29% [1348600/4942000] [272.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 02:02:15,320 - Train: 27.29% [1348700/4942000] [272.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 02:02:48,289 - Train: 27.29% [1348800/4942000] [272.9/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 02:03:21,161 - Train: 27.29% [1348900/4942000] [272.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 02:03:54,137 - Train: 27.30% [1349000/4942000] [273.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 02:04:26,958 - Train: 27.30% [1349100/4942000] [273.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 02:04:48,617 - ==> Total time: 7 days, 8:07:27 Eta: 19 days, 13:01:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-10 02:05:01,804 - Train: 27.30% [1349200/4942000] [273.0/1000.0] [batch_t 0.333 (0.331)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-10 02:05:34,788 - Train: 27.30% [1349300/4942000] [273.0/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 02:06:07,801 - Train: 27.30% [1349400/4942000] [273.0/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-10 02:06:40,838 - Train: 27.31% [1349500/4942000] [273.1/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-10 02:07:13,893 - Train: 27.31% [1349600/4942000] [273.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 02:07:47,034 - Train: 27.31% [1349700/4942000] [273.1/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 02:08:20,046 - Train: 27.31% [1349800/4942000] [273.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 02:08:53,059 - Train: 27.31% [1349900/4942000] [273.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 02:09:26,116 - Train: 27.32% [1350000/4942000] [273.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 02:09:59,234 - Train: 27.32% [1350100/4942000] [273.2/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 02:10:32,255 - Train: 27.32% [1350200/4942000] [273.2/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-10 02:11:05,194 - Train: 27.32% [1350300/4942000] [273.2/1000.0] [batch_t 0.337 (0.329)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-10 02:11:38,111 - Train: 27.32% [1350400/4942000] [273.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 02:12:11,032 - Train: 27.33% [1350500/4942000] [273.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 02:12:44,310 - Train: 27.33% [1350600/4942000] [273.3/1000.0] [batch_t 0.333 (0.333)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 02:13:17,438 - Train: 27.33% [1350700/4942000] [273.3/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 02:13:50,448 - Train: 27.33% [1350800/4942000] [273.3/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-10 02:14:23,474 - Train: 27.34% [1350900/4942000] [273.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 02:14:56,514 - Train: 27.34% [1351000/4942000] [273.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 02:15:29,483 - Train: 27.34% [1351100/4942000] [273.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 02:16:02,446 - Train: 27.34% [1351200/4942000] [273.4/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 02:16:35,444 - Train: 27.34% [1351300/4942000] [273.4/1000.0] [batch_t 0.323 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-10 02:17:08,438 - Train: 27.35% [1351400/4942000] [273.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 02:17:41,484 - Train: 27.35% [1351500/4942000] [273.5/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-10 02:18:14,541 - Train: 27.35% [1351600/4942000] [273.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 02:18:47,539 - Train: 27.35% [1351700/4942000] [273.5/1000.0] [batch_t 0.337 (0.330)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-10 02:19:20,570 - Train: 27.35% [1351800/4942000] [273.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 02:19:53,550 - Train: 27.36% [1351900/4942000] [273.6/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 02:20:26,567 - Train: 27.36% [1352000/4942000] [273.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 02:20:59,654 - Train: 27.36% [1352100/4942000] [273.6/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 02:21:32,604 - Train: 27.36% [1352200/4942000] [273.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 02:22:05,542 - Train: 27.36% [1352300/4942000] [273.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 02:22:38,505 - Train: 27.37% [1352400/4942000] [273.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 02:23:11,547 - Train: 27.37% [1352500/4942000] [273.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 02:23:44,650 - Train: 27.37% [1352600/4942000] [273.7/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 02:24:17,643 - Train: 27.37% [1352700/4942000] [273.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 02:24:50,679 - Train: 27.37% [1352800/4942000] [273.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 02:25:23,756 - Train: 27.38% [1352900/4942000] [273.8/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 02:25:56,736 - Train: 27.38% [1353000/4942000] [273.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 02:26:29,747 - Train: 27.38% [1353100/4942000] [273.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 02:27:02,732 - Train: 27.38% [1353200/4942000] [273.8/1000.0] [batch_t 0.336 (0.330)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-10 02:27:35,719 - Train: 27.38% [1353300/4942000] [273.8/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-10 02:28:08,748 - Train: 27.39% [1353400/4942000] [273.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 02:28:41,848 - Train: 27.39% [1353500/4942000] [273.9/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 02:29:14,787 - Train: 27.39% [1353600/4942000] [273.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 02:29:47,930 - Train: 27.39% [1353700/4942000] [273.9/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 02:30:20,896 - Train: 27.39% [1353800/4942000] [273.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 02:30:53,972 - Train: 27.40% [1353900/4942000] [274.0/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-10 02:31:27,018 - Train: 27.40% [1354000/4942000] [274.0/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 02:32:00,037 - Train: 27.40% [1354100/4942000] [274.0/1000.0] [batch_t 0.336 (0.330)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-10 02:32:02,691 - ==> Total time: 7 days, 8:34:41 Eta: 19 days, 11:52:04 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-10 02:32:36,882 - Train: 27.40% [1354200/4942000] [274.0/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 02:33:09,865 - Train: 27.40% [1354300/4942000] [274.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 02:33:42,869 - Train: 27.41% [1354400/4942000] [274.1/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 02:34:15,843 - Train: 27.41% [1354500/4942000] [274.1/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 02:34:48,740 - Train: 27.41% [1354600/4942000] [274.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 02:35:21,757 - Train: 27.41% [1354700/4942000] [274.1/1000.0] [batch_t 0.336 (0.330)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-10 02:35:54,904 - Train: 27.41% [1354800/4942000] [274.1/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 02:36:27,960 - Train: 27.42% [1354900/4942000] [274.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 02:37:00,895 - Train: 27.42% [1355000/4942000] [274.2/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-10 02:37:33,892 - Train: 27.42% [1355100/4942000] [274.2/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-10 02:38:06,771 - Train: 27.42% [1355200/4942000] [274.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 02:38:39,705 - Train: 27.42% [1355300/4942000] [274.2/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 02:39:12,774 - Train: 27.43% [1355400/4942000] [274.3/1000.0] [batch_t 0.337 (0.331)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-10 02:39:45,791 - Train: 27.43% [1355500/4942000] [274.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 02:40:18,830 - Train: 27.43% [1355600/4942000] [274.3/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 02:40:51,758 - Train: 27.43% [1355700/4942000] [274.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 02:41:24,735 - Train: 27.43% [1355800/4942000] [274.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 02:41:57,696 - Train: 27.44% [1355900/4942000] [274.4/1000.0] [batch_t 0.339 (0.330)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-10 02:42:30,695 - Train: 27.44% [1356000/4942000] [274.4/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-10 02:43:03,614 - Train: 27.44% [1356100/4942000] [274.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 02:43:36,626 - Train: 27.44% [1356200/4942000] [274.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 02:44:09,711 - Train: 27.44% [1356300/4942000] [274.4/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 02:44:42,631 - Train: 27.45% [1356400/4942000] [274.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 02:45:15,575 - Train: 27.45% [1356500/4942000] [274.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 02:45:48,418 - Train: 27.45% [1356600/4942000] [274.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 02:46:21,459 - Train: 27.45% [1356700/4942000] [274.5/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 02:46:54,403 - Train: 27.45% [1356800/4942000] [274.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 02:47:27,375 - Train: 27.46% [1356900/4942000] [274.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 02:48:00,462 - Train: 27.46% [1357000/4942000] [274.6/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 02:48:33,443 - Train: 27.46% [1357100/4942000] [274.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 02:49:06,366 - Train: 27.46% [1357200/4942000] [274.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 02:49:39,288 - Train: 27.46% [1357300/4942000] [274.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 02:50:12,278 - Train: 27.47% [1357400/4942000] [274.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 02:50:45,225 - Train: 27.47% [1357500/4942000] [274.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 02:51:18,163 - Train: 27.47% [1357600/4942000] [274.7/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-10 02:51:51,222 - Train: 27.47% [1357700/4942000] [274.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 02:52:24,129 - Train: 27.47% [1357800/4942000] [274.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-10 02:52:57,092 - Train: 27.48% [1357900/4942000] [274.8/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-10 02:53:30,264 - Train: 27.48% [1358000/4942000] [274.8/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 02:54:03,261 - Train: 27.48% [1358100/4942000] [274.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 02:54:36,220 - Train: 27.48% [1358200/4942000] [274.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 02:55:09,255 - Train: 27.48% [1358300/4942000] [274.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 02:55:42,169 - Train: 27.49% [1358400/4942000] [274.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 02:56:15,226 - Train: 27.49% [1358500/4942000] [274.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 02:56:48,113 - Train: 27.49% [1358600/4942000] [274.9/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 02:57:21,043 - Train: 27.49% [1358700/4942000] [274.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 02:57:53,977 - Train: 27.49% [1358800/4942000] [274.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 02:58:26,884 - Train: 27.50% [1358900/4942000] [275.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 02:58:59,811 - Train: 27.50% [1359000/4942000] [275.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 02:59:16,277 - ==> Total time: 7 days, 9:01:55 Eta: 19 days, 10:43:15 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-10 02:59:34,986 - Train: 27.50% [1359100/4942000] [275.0/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 03:00:08,773 - Train: 27.50% [1359200/4942000] [275.0/1000.0] [batch_t 0.334 (0.338)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-10 03:00:41,670 - Train: 27.51% [1359300/4942000] [275.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 03:01:14,622 - Train: 27.51% [1359400/4942000] [275.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 03:01:47,543 - Train: 27.51% [1359500/4942000] [275.1/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-10 03:02:20,548 - Train: 27.51% [1359600/4942000] [275.1/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-10 03:02:53,640 - Train: 27.51% [1359700/4942000] [275.1/1000.0] [batch_t 0.336 (0.331)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-10 03:03:26,733 - Train: 27.52% [1359800/4942000] [275.2/1000.0] [batch_t 0.337 (0.331)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-10 03:03:59,732 - Train: 27.52% [1359900/4942000] [275.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 03:04:32,666 - Train: 27.52% [1360000/4942000] [275.2/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-10 03:05:05,613 - Train: 27.52% [1360100/4942000] [275.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 03:05:38,510 - Train: 27.52% [1360200/4942000] [275.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 03:06:12,107 - Train: 27.53% [1360300/4942000] [275.3/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 03:06:45,157 - Train: 27.53% [1360400/4942000] [275.3/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-10 03:07:18,335 - Train: 27.53% [1360500/4942000] [275.3/1000.0] [batch_t 0.335 (0.332)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-10 03:07:51,270 - Train: 27.53% [1360600/4942000] [275.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 03:08:24,266 - Train: 27.53% [1360700/4942000] [275.3/1000.0] [batch_t 0.340 (0.330)] [data_t 0.002] [optim_t 0.338] [lr 0.005000] 2024-04-10 03:08:57,269 - Train: 27.54% [1360800/4942000] [275.4/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 03:09:30,204 - Train: 27.54% [1360900/4942000] [275.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 03:10:03,071 - Train: 27.54% [1361000/4942000] [275.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 03:10:36,015 - Train: 27.54% [1361100/4942000] [275.4/1000.0] [batch_t 0.339 (0.329)] [data_t 0.002] [optim_t 0.337] [lr 0.005000] 2024-04-10 03:11:08,904 - Train: 27.54% [1361200/4942000] [275.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 03:11:41,788 - Train: 27.55% [1361300/4942000] [275.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 03:12:14,754 - Train: 27.55% [1361400/4942000] [275.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 03:12:47,724 - Train: 27.55% [1361500/4942000] [275.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 03:13:20,773 - Train: 27.55% [1361600/4942000] [275.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 03:13:53,721 - Train: 27.55% [1361700/4942000] [275.5/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 03:14:27,180 - Train: 27.56% [1361800/4942000] [275.6/1000.0] [batch_t 0.330 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 03:15:00,161 - Train: 27.56% [1361900/4942000] [275.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 03:15:33,298 - Train: 27.56% [1362000/4942000] [275.6/1000.0] [batch_t 0.335 (0.331)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-10 03:16:06,435 - Train: 27.56% [1362100/4942000] [275.6/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 03:16:39,305 - Train: 27.56% [1362200/4942000] [275.6/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-10 03:17:12,225 - Train: 27.57% [1362300/4942000] [275.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 03:17:45,259 - Train: 27.57% [1362400/4942000] [275.7/1000.0] [batch_t 0.337 (0.330)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-10 03:18:18,228 - Train: 27.57% [1362500/4942000] [275.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 03:18:51,140 - Train: 27.57% [1362600/4942000] [275.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 03:19:24,114 - Train: 27.57% [1362700/4942000] [275.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 03:19:57,233 - Train: 27.58% [1362800/4942000] [275.8/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 03:20:30,143 - Train: 27.58% [1362900/4942000] [275.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 03:21:03,116 - Train: 27.58% [1363000/4942000] [275.8/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-10 03:21:36,083 - Train: 27.58% [1363100/4942000] [275.8/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 03:22:08,996 - Train: 27.58% [1363200/4942000] [275.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 03:22:41,984 - Train: 27.59% [1363300/4942000] [275.9/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 03:23:15,203 - Train: 27.59% [1363400/4942000] [275.9/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 03:23:48,279 - Train: 27.59% [1363500/4942000] [275.9/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 03:24:21,258 - Train: 27.59% [1363600/4942000] [275.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 03:24:54,188 - Train: 27.59% [1363700/4942000] [275.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 03:25:27,209 - Train: 27.60% [1363800/4942000] [276.0/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-10 03:26:00,143 - Train: 27.60% [1363900/4942000] [276.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 03:26:30,546 - ==> Total time: 7 days, 9:29:09 Eta: 19 days, 9:34:45 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-10 03:26:35,135 - Train: 27.60% [1364000/4942000] [276.0/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 03:27:08,066 - Train: 27.60% [1364100/4942000] [276.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 03:27:41,049 - Train: 27.60% [1364200/4942000] [276.0/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 03:28:14,004 - Train: 27.61% [1364300/4942000] [276.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 03:28:46,998 - Train: 27.61% [1364400/4942000] [276.1/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 03:29:19,999 - Train: 27.61% [1364500/4942000] [276.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 03:29:53,138 - Train: 27.61% [1364600/4942000] [276.1/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 03:30:26,176 - Train: 27.61% [1364700/4942000] [276.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 03:30:59,341 - Train: 27.62% [1364800/4942000] [276.2/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 03:31:32,266 - Train: 27.62% [1364900/4942000] [276.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 03:32:05,180 - Train: 27.62% [1365000/4942000] [276.2/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-10 03:32:38,212 - Train: 27.62% [1365100/4942000] [276.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 03:33:11,167 - Train: 27.62% [1365200/4942000] [276.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 03:33:44,123 - Train: 27.63% [1365300/4942000] [276.3/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-10 03:34:16,986 - Train: 27.63% [1365400/4942000] [276.3/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-10 03:34:49,948 - Train: 27.63% [1365500/4942000] [276.3/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 03:35:22,841 - Train: 27.63% [1365600/4942000] [276.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 03:35:55,690 - Train: 27.63% [1365700/4942000] [276.3/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-10 03:36:28,699 - Train: 27.64% [1365800/4942000] [276.4/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 03:37:01,642 - Train: 27.64% [1365900/4942000] [276.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 03:37:34,575 - Train: 27.64% [1366000/4942000] [276.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 03:38:07,592 - Train: 27.64% [1366100/4942000] [276.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 03:38:40,525 - Train: 27.64% [1366200/4942000] [276.4/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 03:39:13,535 - Train: 27.65% [1366300/4942000] [276.5/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 03:39:46,535 - Train: 27.65% [1366400/4942000] [276.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 03:40:19,792 - Train: 27.65% [1366500/4942000] [276.5/1000.0] [batch_t 0.326 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 03:40:52,670 - Train: 27.65% [1366600/4942000] [276.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 03:41:25,684 - Train: 27.65% [1366700/4942000] [276.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 03:41:58,601 - Train: 27.66% [1366800/4942000] [276.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 03:42:31,529 - Train: 27.66% [1366900/4942000] [276.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-10 03:43:05,041 - Train: 27.66% [1367000/4942000] [276.6/1000.0] [batch_t 0.891 (0.335)] [data_t 0.564] [optim_t 0.327] [lr 0.005000] 2024-04-10 03:43:38,269 - Train: 27.66% [1367100/4942000] [276.6/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 03:44:11,265 - Train: 27.66% [1367200/4942000] [276.6/1000.0] [batch_t 0.336 (0.330)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-10 03:44:44,198 - Train: 27.67% [1367300/4942000] [276.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 03:45:17,727 - Train: 27.67% [1367400/4942000] [276.7/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 03:45:50,806 - Train: 27.67% [1367500/4942000] [276.7/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 03:46:23,806 - Train: 27.67% [1367600/4942000] [276.7/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-10 03:46:56,948 - Train: 27.68% [1367700/4942000] [276.8/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 03:47:30,813 - Train: 27.68% [1367800/4942000] [276.8/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 03:48:03,732 - Train: 27.68% [1367900/4942000] [276.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 03:48:37,048 - Train: 27.68% [1368000/4942000] [276.8/1000.0] [batch_t 0.336 (0.333)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-10 03:49:10,181 - Train: 27.68% [1368100/4942000] [276.8/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 03:49:43,093 - Train: 27.69% [1368200/4942000] [276.9/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 03:50:17,628 - Train: 27.69% [1368300/4942000] [276.9/1000.0] [batch_t 0.323 (0.345)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-10 03:50:50,513 - Train: 27.69% [1368400/4942000] [276.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 03:51:24,397 - Train: 27.69% [1368500/4942000] [276.9/1000.0] [batch_t 0.333 (0.339)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 03:51:57,473 - Train: 27.69% [1368600/4942000] [276.9/1000.0] [batch_t 0.325 (0.331)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-10 03:52:31,648 - Train: 27.70% [1368700/4942000] [277.0/1000.0] [batch_t 0.331 (0.342)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 03:53:04,667 - Train: 27.70% [1368800/4942000] [277.0/1000.0] [batch_t 0.404 (0.330)] [data_t 0.078] [optim_t 0.327] [lr 0.005000] 2024-04-10 03:53:37,606 - Train: 27.70% [1368900/4942000] [277.0/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 03:53:48,785 - ==> Total time: 7 days, 9:56:27 Eta: 19 days, 8:26:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-10 03:54:13,886 - Train: 27.70% [1369000/4942000] [277.0/1000.0] [batch_t 0.331 (0.351)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 03:54:46,843 - Train: 27.70% [1369100/4942000] [277.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 03:55:21,206 - Train: 27.71% [1369200/4942000] [277.1/1000.0] [batch_t 0.336 (0.344)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-10 03:55:54,165 - Train: 27.71% [1369300/4942000] [277.1/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 03:56:27,510 - Train: 27.71% [1369400/4942000] [277.1/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 03:57:00,519 - Train: 27.71% [1369500/4942000] [277.1/1000.0] [batch_t 0.337 (0.330)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-10 03:57:34,883 - Train: 27.71% [1369600/4942000] [277.1/1000.0] [batch_t 0.327 (0.344)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 03:58:10,497 - Train: 27.72% [1369700/4942000] [277.2/1000.0] [batch_t 0.329 (0.356)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 03:58:43,554 - Train: 27.72% [1369800/4942000] [277.2/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-10 03:59:16,671 - Train: 27.72% [1369900/4942000] [277.2/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 03:59:49,700 - Train: 27.72% [1370000/4942000] [277.2/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 04:00:22,970 - Train: 27.72% [1370100/4942000] [277.2/1000.0] [batch_t 0.331 (0.333)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 04:00:55,970 - Train: 27.73% [1370200/4942000] [277.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 04:01:30,574 - Train: 27.73% [1370300/4942000] [277.3/1000.0] [batch_t 0.330 (0.346)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 04:02:03,909 - Train: 27.73% [1370400/4942000] [277.3/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 04:02:37,335 - Train: 27.73% [1370500/4942000] [277.3/1000.0] [batch_t 0.334 (0.334)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-10 04:03:11,408 - Train: 27.73% [1370600/4942000] [277.3/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 04:03:44,404 - Train: 27.74% [1370700/4942000] [277.4/1000.0] [batch_t 0.326 (0.330)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-10 04:04:17,943 - Train: 27.74% [1370800/4942000] [277.4/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 04:04:50,982 - Train: 27.74% [1370900/4942000] [277.4/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 04:05:24,207 - Train: 27.74% [1371000/4942000] [277.4/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 04:05:57,167 - Train: 27.74% [1371100/4942000] [277.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 04:06:30,565 - Train: 27.75% [1371200/4942000] [277.5/1000.0] [batch_t 0.330 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 04:07:03,562 - Train: 27.75% [1371300/4942000] [277.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 04:07:36,785 - Train: 27.75% [1371400/4942000] [277.5/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 04:08:10,154 - Train: 27.75% [1371500/4942000] [277.5/1000.0] [batch_t 0.335 (0.334)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-10 04:08:43,190 - Train: 27.75% [1371600/4942000] [277.5/1000.0] [batch_t 0.336 (0.330)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-10 04:09:16,247 - Train: 27.76% [1371700/4942000] [277.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 04:09:49,291 - Train: 27.76% [1371800/4942000] [277.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 04:10:22,477 - Train: 27.76% [1371900/4942000] [277.6/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 04:10:55,574 - Train: 27.76% [1372000/4942000] [277.6/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 04:11:28,582 - Train: 27.76% [1372100/4942000] [277.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 04:12:01,529 - Train: 27.77% [1372200/4942000] [277.7/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-10 04:12:34,530 - Train: 27.77% [1372300/4942000] [277.7/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 04:13:07,599 - Train: 27.77% [1372400/4942000] [277.7/1000.0] [batch_t 0.333 (0.331)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 04:13:40,637 - Train: 27.77% [1372500/4942000] [277.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 04:14:13,685 - Train: 27.77% [1372600/4942000] [277.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 04:14:46,796 - Train: 27.78% [1372700/4942000] [277.8/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 04:15:19,777 - Train: 27.78% [1372800/4942000] [277.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 04:15:52,800 - Train: 27.78% [1372900/4942000] [277.8/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 04:16:25,845 - Train: 27.78% [1373000/4942000] [277.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 04:16:58,779 - Train: 27.78% [1373100/4942000] [277.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 04:17:31,816 - Train: 27.79% [1373200/4942000] [277.9/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 04:18:05,023 - Train: 27.79% [1373300/4942000] [277.9/1000.0] [batch_t 0.333 (0.332)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 04:18:38,008 - Train: 27.79% [1373400/4942000] [277.9/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-10 04:19:10,989 - Train: 27.79% [1373500/4942000] [277.9/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 04:19:44,059 - Train: 27.79% [1373600/4942000] [277.9/1000.0] [batch_t 0.334 (0.331)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-10 04:20:17,569 - Train: 27.80% [1373700/4942000] [278.0/1000.0] [batch_t 0.342 (0.335)] [data_t 0.002] [optim_t 0.340] [lr 0.005000] 2024-04-10 04:20:50,569 - Train: 27.80% [1373800/4942000] [278.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 04:21:16,628 - ==> Total time: 7 days, 10:23:55 Eta: 19 days, 7:19:24 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-10 04:21:26,930 - Train: 27.80% [1373900/4942000] [278.0/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 04:21:59,790 - Train: 27.80% [1374000/4942000] [278.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 04:22:32,742 - Train: 27.80% [1374100/4942000] [278.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 04:23:05,627 - Train: 27.81% [1374200/4942000] [278.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 04:23:38,593 - Train: 27.81% [1374300/4942000] [278.1/1000.0] [batch_t 0.337 (0.330)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-10 04:24:11,613 - Train: 27.81% [1374400/4942000] [278.1/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 04:24:44,726 - Train: 27.81% [1374500/4942000] [278.1/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 04:25:17,993 - Train: 27.81% [1374600/4942000] [278.1/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 04:25:50,950 - Train: 27.82% [1374700/4942000] [278.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 04:26:24,345 - Train: 27.82% [1374800/4942000] [278.2/1000.0] [batch_t 0.326 (0.334)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 04:26:57,330 - Train: 27.82% [1374900/4942000] [278.2/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 04:27:30,649 - Train: 27.82% [1375000/4942000] [278.2/1000.0] [batch_t 0.331 (0.333)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 04:28:03,590 - Train: 27.82% [1375100/4942000] [278.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 04:28:36,540 - Train: 27.83% [1375200/4942000] [278.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 04:29:09,833 - Train: 27.83% [1375300/4942000] [278.3/1000.0] [batch_t 0.326 (0.333)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 04:29:42,884 - Train: 27.83% [1375400/4942000] [278.3/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 04:30:16,073 - Train: 27.83% [1375500/4942000] [278.3/1000.0] [batch_t 0.332 (0.332)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-10 04:30:49,137 - Train: 27.83% [1375600/4942000] [278.3/1000.0] [batch_t 0.325 (0.331)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-10 04:31:22,113 - Train: 27.84% [1375700/4942000] [278.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 04:31:55,252 - Train: 27.84% [1375800/4942000] [278.4/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 04:32:28,415 - Train: 27.84% [1375900/4942000] [278.4/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 04:33:01,450 - Train: 27.84% [1376000/4942000] [278.4/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-10 04:33:34,544 - Train: 27.85% [1376100/4942000] [278.5/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 04:34:07,579 - Train: 27.85% [1376200/4942000] [278.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 04:34:40,567 - Train: 27.85% [1376300/4942000] [278.5/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-10 04:35:13,635 - Train: 27.85% [1376400/4942000] [278.5/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 04:35:46,653 - Train: 27.85% [1376500/4942000] [278.5/1000.0] [batch_t 0.323 (0.330)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-10 04:36:19,662 - Train: 27.86% [1376600/4942000] [278.6/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 04:36:52,613 - Train: 27.86% [1376700/4942000] [278.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 04:37:25,577 - Train: 27.86% [1376800/4942000] [278.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 04:37:58,531 - Train: 27.86% [1376900/4942000] [278.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 04:38:31,464 - Train: 27.86% [1377000/4942000] [278.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 04:39:04,398 - Train: 27.87% [1377100/4942000] [278.7/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-10 04:39:37,326 - Train: 27.87% [1377200/4942000] [278.7/1000.0] [batch_t 0.336 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-10 04:40:10,305 - Train: 27.87% [1377300/4942000] [278.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 04:40:43,273 - Train: 27.87% [1377400/4942000] [278.7/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 04:41:16,397 - Train: 27.87% [1377500/4942000] [278.7/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 04:41:49,378 - Train: 27.88% [1377600/4942000] [278.8/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-10 04:42:22,343 - Train: 27.88% [1377700/4942000] [278.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 04:42:55,207 - Train: 27.88% [1377800/4942000] [278.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 04:43:28,115 - Train: 27.88% [1377900/4942000] [278.8/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 04:44:00,968 - Train: 27.88% [1378000/4942000] [278.8/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-10 04:44:33,896 - Train: 27.89% [1378100/4942000] [278.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 04:45:06,950 - Train: 27.89% [1378200/4942000] [278.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 04:45:39,936 - Train: 27.89% [1378300/4942000] [278.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 04:46:13,061 - Train: 27.89% [1378400/4942000] [278.9/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 04:46:46,125 - Train: 27.89% [1378500/4942000] [278.9/1000.0] [batch_t 0.339 (0.331)] [data_t 0.002] [optim_t 0.337] [lr 0.005000] 2024-04-10 04:47:19,160 - Train: 27.90% [1378600/4942000] [279.0/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-10 04:47:52,280 - Train: 27.90% [1378700/4942000] [279.0/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 04:48:25,340 - Train: 27.90% [1378800/4942000] [279.0/1000.0] [batch_t 0.338 (0.330)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-10 04:48:31,312 - ==> Total time: 7 days, 10:51:10 Eta: 19 days, 6:11:49 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-10 04:49:00,899 - Train: 27.90% [1378900/4942000] [279.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 04:49:33,827 - Train: 27.90% [1379000/4942000] [279.0/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-10 04:50:06,730 - Train: 27.91% [1379100/4942000] [279.1/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 04:50:39,808 - Train: 27.91% [1379200/4942000] [279.1/1000.0] [batch_t 0.338 (0.331)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-10 04:51:12,772 - Train: 27.91% [1379300/4942000] [279.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 04:51:45,787 - Train: 27.91% [1379400/4942000] [279.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 04:52:18,745 - Train: 27.91% [1379500/4942000] [279.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 04:52:51,664 - Train: 27.92% [1379600/4942000] [279.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 04:53:24,632 - Train: 27.92% [1379700/4942000] [279.2/1000.0] [batch_t 0.336 (0.330)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-10 04:53:57,583 - Train: 27.92% [1379800/4942000] [279.2/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-10 04:54:30,723 - Train: 27.92% [1379900/4942000] [279.2/1000.0] [batch_t 0.335 (0.331)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-10 04:55:03,744 - Train: 27.92% [1380000/4942000] [279.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 04:55:36,725 - Train: 27.93% [1380100/4942000] [279.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 04:56:09,651 - Train: 27.93% [1380200/4942000] [279.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 04:56:42,754 - Train: 27.93% [1380300/4942000] [279.3/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 04:57:15,670 - Train: 27.93% [1380400/4942000] [279.3/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-10 04:57:48,642 - Train: 27.93% [1380500/4942000] [279.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 04:58:21,625 - Train: 27.94% [1380600/4942000] [279.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 04:58:54,585 - Train: 27.94% [1380700/4942000] [279.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 04:59:27,538 - Train: 27.94% [1380800/4942000] [279.4/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-10 05:00:00,505 - Train: 27.94% [1380900/4942000] [279.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 05:00:33,502 - Train: 27.94% [1381000/4942000] [279.4/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-10 05:01:06,456 - Train: 27.95% [1381100/4942000] [279.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 05:01:39,440 - Train: 27.95% [1381200/4942000] [279.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 05:02:12,384 - Train: 27.95% [1381300/4942000] [279.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 05:02:45,307 - Train: 27.95% [1381400/4942000] [279.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 05:03:18,313 - Train: 27.95% [1381500/4942000] [279.5/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-10 05:03:51,272 - Train: 27.96% [1381600/4942000] [279.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 05:04:25,577 - Train: 27.96% [1381700/4942000] [279.6/1000.0] [batch_t 0.337 (0.343)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-10 05:04:58,717 - Train: 27.96% [1381800/4942000] [279.6/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 05:05:31,724 - Train: 27.96% [1381900/4942000] [279.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 05:06:04,735 - Train: 27.96% [1382000/4942000] [279.6/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 05:06:37,761 - Train: 27.97% [1382100/4942000] [279.7/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 05:07:10,660 - Train: 27.97% [1382200/4942000] [279.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 05:07:43,579 - Train: 27.97% [1382300/4942000] [279.7/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 05:08:16,467 - Train: 27.97% [1382400/4942000] [279.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 05:08:49,443 - Train: 27.97% [1382500/4942000] [279.7/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 05:09:22,416 - Train: 27.98% [1382600/4942000] [279.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 05:09:55,454 - Train: 27.98% [1382700/4942000] [279.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 05:10:28,421 - Train: 27.98% [1382800/4942000] [279.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 05:11:01,428 - Train: 27.98% [1382900/4942000] [279.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 05:11:34,378 - Train: 27.98% [1383000/4942000] [279.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 05:12:07,366 - Train: 27.99% [1383100/4942000] [279.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 05:12:40,351 - Train: 27.99% [1383200/4942000] [279.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 05:13:13,269 - Train: 27.99% [1383300/4942000] [279.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 05:13:46,244 - Train: 27.99% [1383400/4942000] [279.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 05:14:19,251 - Train: 27.99% [1383500/4942000] [279.9/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 05:14:52,263 - Train: 28.00% [1383600/4942000] [280.0/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-10 05:15:25,179 - Train: 28.00% [1383700/4942000] [280.0/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-10 05:15:44,921 - ==> Total time: 7 days, 11:18:24 Eta: 19 days, 5:04:27 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-10 05:16:00,362 - Train: 28.00% [1383800/4942000] [280.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 05:16:33,260 - Train: 28.00% [1383900/4942000] [280.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 05:17:06,208 - Train: 28.00% [1384000/4942000] [280.0/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-10 05:17:39,153 - Train: 28.01% [1384100/4942000] [280.1/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-10 05:18:11,997 - Train: 28.01% [1384200/4942000] [280.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 05:18:44,963 - Train: 28.01% [1384300/4942000] [280.1/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-10 05:19:17,859 - Train: 28.01% [1384400/4942000] [280.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 05:19:50,959 - Train: 28.01% [1384500/4942000] [280.1/1000.0] [batch_t 0.339 (0.331)] [data_t 0.002] [optim_t 0.337] [lr 0.005000] 2024-04-10 05:20:23,967 - Train: 28.02% [1384600/4942000] [280.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 05:20:56,994 - Train: 28.02% [1384700/4942000] [280.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 05:21:29,916 - Train: 28.02% [1384800/4942000] [280.2/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 05:22:02,864 - Train: 28.02% [1384900/4942000] [280.2/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 05:22:35,795 - Train: 28.03% [1385000/4942000] [280.3/1000.0] [batch_t 0.336 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-10 05:23:08,757 - Train: 28.03% [1385100/4942000] [280.3/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-10 05:23:41,754 - Train: 28.03% [1385200/4942000] [280.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 05:24:14,768 - Train: 28.03% [1385300/4942000] [280.3/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-10 05:24:47,901 - Train: 28.03% [1385400/4942000] [280.3/1000.0] [batch_t 0.336 (0.331)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-10 05:25:20,990 - Train: 28.04% [1385500/4942000] [280.4/1000.0] [batch_t 0.333 (0.331)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 05:25:54,040 - Train: 28.04% [1385600/4942000] [280.4/1000.0] [batch_t 0.338 (0.330)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-10 05:26:27,060 - Train: 28.04% [1385700/4942000] [280.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 05:27:00,061 - Train: 28.04% [1385800/4942000] [280.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 05:27:33,237 - Train: 28.04% [1385900/4942000] [280.4/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 05:28:06,468 - Train: 28.05% [1386000/4942000] [280.5/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 05:28:39,527 - Train: 28.05% [1386100/4942000] [280.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 05:29:12,524 - Train: 28.05% [1386200/4942000] [280.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 05:29:45,558 - Train: 28.05% [1386300/4942000] [280.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 05:30:18,634 - Train: 28.05% [1386400/4942000] [280.5/1000.0] [batch_t 0.339 (0.331)] [data_t 0.002] [optim_t 0.337] [lr 0.005000] 2024-04-10 05:30:51,645 - Train: 28.06% [1386500/4942000] [280.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 05:31:24,727 - Train: 28.06% [1386600/4942000] [280.6/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 05:31:57,686 - Train: 28.06% [1386700/4942000] [280.6/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-10 05:32:30,686 - Train: 28.06% [1386800/4942000] [280.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-10 05:33:03,723 - Train: 28.06% [1386900/4942000] [280.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-10 05:33:36,656 - Train: 28.07% [1387000/4942000] [280.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 05:34:09,652 - Train: 28.07% [1387100/4942000] [280.7/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-10 05:34:42,656 - Train: 28.07% [1387200/4942000] [280.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 05:35:15,633 - Train: 28.07% [1387300/4942000] [280.7/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-10 05:35:48,659 - Train: 28.07% [1387400/4942000] [280.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 05:36:21,643 - Train: 28.08% [1387500/4942000] [280.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 05:36:54,711 - Train: 28.08% [1387600/4942000] [280.8/1000.0] [batch_t 0.335 (0.331)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-10 05:37:27,696 - Train: 28.08% [1387700/4942000] [280.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 05:38:00,605 - Train: 28.08% [1387800/4942000] [280.8/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 05:38:33,560 - Train: 28.08% [1387900/4942000] [280.8/1000.0] [batch_t 0.342 (0.329)] [data_t 0.002] [optim_t 0.340] [lr 0.005000] 2024-04-10 05:39:06,623 - Train: 28.09% [1388000/4942000] [280.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 05:39:39,692 - Train: 28.09% [1388100/4942000] [280.9/1000.0] [batch_t 0.323 (0.331)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-10 05:40:12,721 - Train: 28.09% [1388200/4942000] [280.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 05:40:45,660 - Train: 28.09% [1388300/4942000] [280.9/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-10 05:41:18,773 - Train: 28.09% [1388400/4942000] [280.9/1000.0] [batch_t 0.342 (0.331)] [data_t 0.002] [optim_t 0.340] [lr 0.005000] 2024-04-10 05:41:51,723 - Train: 28.10% [1388500/4942000] [281.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 05:42:24,767 - Train: 28.10% [1388600/4942000] [281.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 05:42:57,677 - Train: 28.10% [1388700/4942000] [281.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-10 05:42:58,334 - ==> Total time: 7 days, 11:45:37 Eta: 19 days, 3:57:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco2_nhcs57c1d4_20240402-175720' 2024-04-10 05:43:32,912 - Train: 28.10% [1388800/4942000] [281.0/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-10 05:44:05,902 - Train: 28.10% [1388900/4942000] [281.0/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 05:44:38,933 - Train: 28.11% [1389000/4942000] [281.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 05:45:11,924 - Train: 28.11% [1389100/4942000] [281.1/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 05:45:44,888 - Train: 28.11% [1389200/4942000] [281.1/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 05:46:17,862 - Train: 28.11% [1389300/4942000] [281.1/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-10 05:46:50,845 - Train: 28.11% [1389400/4942000] [281.1/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 05:47:23,863 - Train: 28.12% [1389500/4942000] [281.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-10 05:47:56,930 - Train: 28.12% [1389600/4942000] [281.2/1000.0] [batch_t 0.334 (0.331)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-10 05:48:29,986 - Train: 28.12% [1389700/4942000] [281.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-10 05:49:03,084 - Train: 28.12% [1389800/4942000] [281.2/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 05:49:36,158 - Train: 28.12% [1389900/4942000] [281.2/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-10 05:50:09,270 - Train: 28.13% [1390000/4942000] [281.3/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-10 05:50:47,682 - Train: 28.13% [1390100/4942000] [281.3/1000.0] [batch_t 0.324 (0.384)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-10 05:55:56,268 - Train: 28.13% [1390200/4942000] [281.3/1000.0] [batch_t 0.327 (3.086)] [data_t 0.002] [optim_t 0.325] [lr 0.005000]