2024-04-02 23:26:01,543 - ==> Logging on master GPU: 0 2024-04-02 23:26:01,543 - ==> Running Trainer: MAMBAADTrainer 2024-04-02 23:26:01,544 - ==> Using GPU: [0] for Training 2024-04-02 23:26:01,544 - ==> Building model 2024-04-02 23:26:02,079 - Loaded from checkpoint 'model/pretrain/resnet34-43635321.pth' 2024-04-02 23:26:20,415 - ------------------------------------ MAMBAAD ------------------------------------ | module | #parameters or shape | #flops | |:------------------------------------------------------------|:-----------------------|:-------------| | model | 23.099M | 7.537G | | net_t | 8.17M | 3.955G | | net_t.conv1 | 9.408K | 0.154G | | net_t.conv1.weight | (64, 3, 7, 7) | | | net_t.bn1 | 0.128K | 2.097M | | net_t.bn1.weight | (64,) | | | net_t.bn1.bias | (64,) | | | net_t.layer1 | 0.222M | 0.909G | | net_t.layer1.0 | 73.984K | 0.303G | | net_t.layer1.0.conv1 | 36.864K | 0.151G | | net_t.layer1.0.conv1.weight | (64, 64, 3, 3) | | | net_t.layer1.0.bn1 | 0.128K | 0.524M | | net_t.layer1.0.bn1.weight | (64,) | | | net_t.layer1.0.bn1.bias | (64,) | | | net_t.layer1.0.conv2 | 36.864K | 0.151G | | net_t.layer1.0.conv2.weight | (64, 64, 3, 3) | | | net_t.layer1.0.bn2 | 0.128K | 0.524M | | net_t.layer1.0.bn2.weight | (64,) | | | net_t.layer1.0.bn2.bias | (64,) | | | net_t.layer1.1 | 73.984K | 0.303G | | net_t.layer1.1.conv1 | 36.864K | 0.151G | | net_t.layer1.1.conv1.weight | (64, 64, 3, 3) | | | net_t.layer1.1.bn1 | 0.128K | 0.524M | | net_t.layer1.1.bn1.weight | (64,) | | | net_t.layer1.1.bn1.bias | (64,) | | | net_t.layer1.1.conv2 | 36.864K | 0.151G | | net_t.layer1.1.conv2.weight | (64, 64, 3, 3) | | | net_t.layer1.1.bn2 | 0.128K | 0.524M | | net_t.layer1.1.bn2.weight | (64,) | | | net_t.layer1.1.bn2.bias | (64,) | | | net_t.layer1.2 | 73.984K | 0.303G | | net_t.layer1.2.conv1 | 36.864K | 0.151G | | net_t.layer1.2.conv1.weight | (64, 64, 3, 3) | | | net_t.layer1.2.bn1 | 0.128K | 0.524M | | net_t.layer1.2.bn1.weight | (64,) | | | net_t.layer1.2.bn1.bias | (64,) | | | net_t.layer1.2.conv2 | 36.864K | 0.151G | | net_t.layer1.2.conv2.weight | (64, 64, 3, 3) | | | net_t.layer1.2.bn2 | 0.128K | 0.524M | | net_t.layer1.2.bn2.weight | (64,) | | | net_t.layer1.2.bn2.bias | (64,) | | | net_t.layer2 | 1.116M | 1.143G | | net_t.layer2.0 | 0.23M | 0.236G | | net_t.layer2.0.conv1 | 73.728K | 75.497M | | net_t.layer2.0.conv1.weight | (128, 64, 3, 3) | | | net_t.layer2.0.bn1 | 0.256K | 0.262M | | net_t.layer2.0.bn1.weight | (128,) | | | net_t.layer2.0.bn1.bias | (128,) | | | net_t.layer2.0.conv2 | 0.147M | 0.151G | | net_t.layer2.0.conv2.weight | (128, 128, 3, 3) | | | net_t.layer2.0.bn2 | 0.256K | 0.262M | | net_t.layer2.0.bn2.weight | (128,) | | | net_t.layer2.0.bn2.bias | (128,) | | | net_t.layer2.0.downsample | 8.448K | 8.651M | | net_t.layer2.0.downsample.0 | 8.192K | 8.389M | | net_t.layer2.0.downsample.1 | 0.256K | 0.262M | | net_t.layer2.1 | 0.295M | 0.303G | | net_t.layer2.1.conv1 | 0.147M | 0.151G | | net_t.layer2.1.conv1.weight | (128, 128, 3, 3) | | | net_t.layer2.1.bn1 | 0.256K | 0.262M | | net_t.layer2.1.bn1.weight | (128,) | | | net_t.layer2.1.bn1.bias | (128,) | | | net_t.layer2.1.conv2 | 0.147M | 0.151G | | net_t.layer2.1.conv2.weight | (128, 128, 3, 3) | | | net_t.layer2.1.bn2 | 0.256K | 0.262M | | net_t.layer2.1.bn2.weight | (128,) | | | net_t.layer2.1.bn2.bias | (128,) | | | net_t.layer2.2 | 0.295M | 0.303G | | net_t.layer2.2.conv1 | 0.147M | 0.151G | | net_t.layer2.2.conv1.weight | (128, 128, 3, 3) | | | net_t.layer2.2.bn1 | 0.256K | 0.262M | | net_t.layer2.2.bn1.weight | (128,) | | | net_t.layer2.2.bn1.bias | (128,) | | | net_t.layer2.2.conv2 | 0.147M | 0.151G | | net_t.layer2.2.conv2.weight | (128, 128, 3, 3) | | | net_t.layer2.2.bn2 | 0.256K | 0.262M | | net_t.layer2.2.bn2.weight | (128,) | | | net_t.layer2.2.bn2.bias | (128,) | | | net_t.layer2.3 | 0.295M | 0.303G | | net_t.layer2.3.conv1 | 0.147M | 0.151G | | net_t.layer2.3.conv1.weight | (128, 128, 3, 3) | | | net_t.layer2.3.bn1 | 0.256K | 0.262M | | net_t.layer2.3.bn1.weight | (128,) | | | net_t.layer2.3.bn1.bias | (128,) | | | net_t.layer2.3.conv2 | 0.147M | 0.151G | | net_t.layer2.3.conv2.weight | (128, 128, 3, 3) | | | net_t.layer2.3.bn2 | 0.256K | 0.262M | | net_t.layer2.3.bn2.weight | (128,) | | | net_t.layer2.3.bn2.bias | (128,) | | | net_t.layer3 | 6.822M | 1.747G | | net_t.layer3.0 | 0.919M | 0.235G | | net_t.layer3.0.conv1 | 0.295M | 75.497M | | net_t.layer3.0.conv1.weight | (256, 128, 3, 3) | | | net_t.layer3.0.bn1 | 0.512K | 0.131M | | net_t.layer3.0.bn1.weight | (256,) | | | net_t.layer3.0.bn1.bias | (256,) | | | net_t.layer3.0.conv2 | 0.59M | 0.151G | | net_t.layer3.0.conv2.weight | (256, 256, 3, 3) | | | net_t.layer3.0.bn2 | 0.512K | 0.131M | | net_t.layer3.0.bn2.weight | (256,) | | | net_t.layer3.0.bn2.bias | (256,) | | | net_t.layer3.0.downsample | 33.28K | 8.52M | | net_t.layer3.0.downsample.0 | 32.768K | 8.389M | | net_t.layer3.0.downsample.1 | 0.512K | 0.131M | | net_t.layer3.1 | 1.181M | 0.302G | | net_t.layer3.1.conv1 | 0.59M | 0.151G | | net_t.layer3.1.conv1.weight | (256, 256, 3, 3) | | | net_t.layer3.1.bn1 | 0.512K | 0.131M | | net_t.layer3.1.bn1.weight | (256,) | | | net_t.layer3.1.bn1.bias | (256,) | | | net_t.layer3.1.conv2 | 0.59M | 0.151G | | net_t.layer3.1.conv2.weight | (256, 256, 3, 3) | | | net_t.layer3.1.bn2 | 0.512K | 0.131M | | net_t.layer3.1.bn2.weight | (256,) | | | net_t.layer3.1.bn2.bias | (256,) | | | net_t.layer3.2 | 1.181M | 0.302G | | net_t.layer3.2.conv1 | 0.59M | 0.151G | | net_t.layer3.2.conv1.weight | (256, 256, 3, 3) | | | net_t.layer3.2.bn1 | 0.512K | 0.131M | | net_t.layer3.2.bn1.weight | (256,) | | | net_t.layer3.2.bn1.bias | (256,) | | | net_t.layer3.2.conv2 | 0.59M | 0.151G | | net_t.layer3.2.conv2.weight | (256, 256, 3, 3) | | | net_t.layer3.2.bn2 | 0.512K | 0.131M | | net_t.layer3.2.bn2.weight | (256,) | | | net_t.layer3.2.bn2.bias | (256,) | | | net_t.layer3.3 | 1.181M | 0.302G | | net_t.layer3.3.conv1 | 0.59M | 0.151G | | net_t.layer3.3.conv1.weight | (256, 256, 3, 3) | | | net_t.layer3.3.bn1 | 0.512K | 0.131M | | net_t.layer3.3.bn1.weight | (256,) | | | net_t.layer3.3.bn1.bias | (256,) | | | net_t.layer3.3.conv2 | 0.59M | 0.151G | | net_t.layer3.3.conv2.weight | (256, 256, 3, 3) | | | net_t.layer3.3.bn2 | 0.512K | 0.131M | | net_t.layer3.3.bn2.weight | (256,) | | | net_t.layer3.3.bn2.bias | (256,) | | | net_t.layer3.4 | 1.181M | 0.302G | | net_t.layer3.4.conv1 | 0.59M | 0.151G | | net_t.layer3.4.conv1.weight | (256, 256, 3, 3) | | | net_t.layer3.4.bn1 | 0.512K | 0.131M | | net_t.layer3.4.bn1.weight | (256,) | | | net_t.layer3.4.bn1.bias | (256,) | | | net_t.layer3.4.conv2 | 0.59M | 0.151G | | net_t.layer3.4.conv2.weight | (256, 256, 3, 3) | | | net_t.layer3.4.bn2 | 0.512K | 0.131M | | net_t.layer3.4.bn2.weight | (256,) | | | net_t.layer3.4.bn2.bias | (256,) | | | net_t.layer3.5 | 1.181M | 0.302G | | net_t.layer3.5.conv1 | 0.59M | 0.151G | | net_t.layer3.5.conv1.weight | (256, 256, 3, 3) | | | net_t.layer3.5.bn1 | 0.512K | 0.131M | | net_t.layer3.5.bn1.weight | (256,) | | | net_t.layer3.5.bn1.bias | (256,) | | | net_t.layer3.5.conv2 | 0.59M | 0.151G | | net_t.layer3.5.conv2.weight | (256, 256, 3, 3) | | | net_t.layer3.5.bn2 | 0.512K | 0.131M | | net_t.layer3.5.bn2.weight | (256,) | | | net_t.layer3.5.bn2.bias | (256,) | | | mff_oce | 1.458M | 0.269G | | mff_oce.bn_layer | 0.94M | 66.47M | | mff_oce.bn_layer.0 | 0.379M | 30.622M | | mff_oce.bn_layer.0.conv1 | 32.768K | 8.389M | | mff_oce.bn_layer.0.conv1.weight | (128, 256, 1, 1) | | | mff_oce.bn_layer.0.bn1 | 0.256K | 65.536K | | mff_oce.bn_layer.0.bn1.weight | (128,) | | | mff_oce.bn_layer.0.bn1.bias | (128,) | | | mff_oce.bn_layer.0.conv2 | 0.147M | 9.437M | | mff_oce.bn_layer.0.conv2.weight | (128, 128, 3, 3) | | | mff_oce.bn_layer.0.bn2 | 0.256K | 16.384K | | mff_oce.bn_layer.0.bn2.weight | (128,) | | | mff_oce.bn_layer.0.bn2.bias | (128,) | | | mff_oce.bn_layer.0.conv3 | 65.536K | 4.194M | | mff_oce.bn_layer.0.conv3.weight | (512, 128, 1, 1) | | | mff_oce.bn_layer.0.bn3 | 1.024K | 65.536K | | mff_oce.bn_layer.0.bn3.weight | (512,) | | | mff_oce.bn_layer.0.bn3.bias | (512,) | | | mff_oce.bn_layer.0.downsample | 0.132M | 8.454M | | mff_oce.bn_layer.0.downsample.0 | 0.131M | 8.389M | | mff_oce.bn_layer.0.downsample.1 | 1.024K | 65.536K | | mff_oce.bn_layer.1 | 0.28M | 17.924M | | mff_oce.bn_layer.1.conv1 | 65.536K | 4.194M | | mff_oce.bn_layer.1.conv1.weight | (128, 512, 1, 1) | | | mff_oce.bn_layer.1.bn1 | 0.256K | 16.384K | | mff_oce.bn_layer.1.bn1.weight | (128,) | | | mff_oce.bn_layer.1.bn1.bias | (128,) | | | mff_oce.bn_layer.1.conv2 | 0.147M | 9.437M | | mff_oce.bn_layer.1.conv2.weight | (128, 128, 3, 3) | | | mff_oce.bn_layer.1.bn2 | 0.256K | 16.384K | | mff_oce.bn_layer.1.bn2.weight | (128,) | | | mff_oce.bn_layer.1.bn2.bias | (128,) | | | mff_oce.bn_layer.1.conv3 | 65.536K | 4.194M | | mff_oce.bn_layer.1.conv3.weight | (512, 128, 1, 1) | | | mff_oce.bn_layer.1.bn3 | 1.024K | 65.536K | | mff_oce.bn_layer.1.bn3.weight | (512,) | | | mff_oce.bn_layer.1.bn3.bias | (512,) | | | mff_oce.bn_layer.2 | 0.28M | 17.924M | | mff_oce.bn_layer.2.conv1 | 65.536K | 4.194M | | mff_oce.bn_layer.2.conv1.weight | (128, 512, 1, 1) | | | mff_oce.bn_layer.2.bn1 | 0.256K | 16.384K | | mff_oce.bn_layer.2.bn1.weight | (128,) | | | mff_oce.bn_layer.2.bn1.bias | (128,) | | | mff_oce.bn_layer.2.conv2 | 0.147M | 9.437M | | mff_oce.bn_layer.2.conv2.weight | (128, 128, 3, 3) | | | mff_oce.bn_layer.2.bn2 | 0.256K | 16.384K | | mff_oce.bn_layer.2.bn2.weight | (128,) | | | mff_oce.bn_layer.2.bn2.bias | (128,) | | | mff_oce.bn_layer.2.conv3 | 65.536K | 4.194M | | mff_oce.bn_layer.2.conv3.weight | (512, 128, 1, 1) | | | mff_oce.bn_layer.2.bn3 | 1.024K | 65.536K | | mff_oce.bn_layer.2.bn3.weight | (512,) | | | mff_oce.bn_layer.2.bn3.bias | (512,) | | | mff_oce.conv1 | 73.728K | 75.497M | | mff_oce.conv1.weight | (128, 64, 3, 3) | | | mff_oce.bn1 | 0.256K | 0.262M | | mff_oce.bn1.weight | (128,) | | | mff_oce.bn1.bias | (128,) | | | mff_oce.conv2 | 0.295M | 75.497M | | mff_oce.conv2.weight | (256, 128, 3, 3) | | | mff_oce.bn2 | 0.512K | 0.131M | | mff_oce.bn2.weight | (256,) | | | mff_oce.bn2.bias | (256,) | | | mff_oce.conv21 | 16.512K | 16.777M | | mff_oce.conv21.weight | (128, 128, 1, 1) | | | mff_oce.conv21.bias | (128,) | | | mff_oce.bn21 | 0.256K | 0.262M | | mff_oce.bn21.weight | (128,) | | | mff_oce.bn21.bias | (128,) | | | mff_oce.conv31 | 65.792K | 16.777M | | mff_oce.conv31.weight | (256, 256, 1, 1) | | | mff_oce.conv31.bias | (256,) | | | mff_oce.bn31 | 0.512K | 0.131M | | mff_oce.bn31.weight | (256,) | | | mff_oce.bn31.bias | (256,) | | | mff_oce.convf | 65.792K | 16.777M | | mff_oce.convf.weight | (256, 256, 1, 1) | | | mff_oce.convf.bias | (256,) | | | mff_oce.bnf | 0.512K | 0.131M | | mff_oce.bnf.weight | (256,) | | | mff_oce.bnf.bias | (256,) | | | net_s.layers_up | 13.471M | 3.313G | | net_s.layers_up.0.blocks.0 | 8.04M | 0.501G | | net_s.layers_up.0.blocks.0.smm_blocks | 6.16M | 0.381G | | net_s.layers_up.0.blocks.0.smm_blocks.0 | 2.053M | 0.127G | | net_s.layers_up.0.blocks.0.smm_blocks.0.ln_1 | 1.024K | 0.164M | | net_s.layers_up.0.blocks.0.smm_blocks.0.self_attention | 2.052M | 0.127G | | net_s.layers_up.0.blocks.0.smm_blocks.1 | 2.053M | 0.127G | | net_s.layers_up.0.blocks.0.smm_blocks.1.ln_1 | 1.024K | 0.164M | | net_s.layers_up.0.blocks.0.smm_blocks.1.self_attention | 2.052M | 0.127G | | net_s.layers_up.0.blocks.0.smm_blocks.2 | 2.053M | 0.127G | | net_s.layers_up.0.blocks.0.smm_blocks.2.ln_1 | 1.024K | 0.164M | | net_s.layers_up.0.blocks.0.smm_blocks.2.self_attention | 2.052M | 0.127G | | net_s.layers_up.0.blocks.0.conv1b3 | 0.263M | 16.908M | | net_s.layers_up.0.blocks.0.conv1b3.0 | 0.263M | 16.777M | | net_s.layers_up.0.blocks.0.conv1b3.0.weight | (512, 512, 1, 1) | | | net_s.layers_up.0.blocks.0.conv1b3.0.bias | (512,) | | | net_s.layers_up.0.blocks.0.conv1b3.1 | | 0.131M | | net_s.layers_up.0.blocks.0.conv1a3 | 0.263M | 16.908M | | net_s.layers_up.0.blocks.0.conv1a3.0 | 0.263M | 16.777M | | net_s.layers_up.0.blocks.0.conv1a3.0.weight | (512, 512, 1, 1) | | | net_s.layers_up.0.blocks.0.conv1a3.0.bias | (512,) | | | net_s.layers_up.0.blocks.0.conv1a3.1 | | 0.131M | | net_s.layers_up.0.blocks.0.conv1b5 | 0.263M | 16.908M | | net_s.layers_up.0.blocks.0.conv1b5.0 | 0.263M | 16.777M | | net_s.layers_up.0.blocks.0.conv1b5.0.weight | (512, 512, 1, 1) | | | net_s.layers_up.0.blocks.0.conv1b5.0.bias | (512,) | | | net_s.layers_up.0.blocks.0.conv1b5.1 | | 0.131M | | net_s.layers_up.0.blocks.0.conv1a5 | 0.263M | 16.908M | | net_s.layers_up.0.blocks.0.conv1a5.0 | 0.263M | 16.777M | | net_s.layers_up.0.blocks.0.conv1a5.0.weight | (512, 512, 1, 1) | | | net_s.layers_up.0.blocks.0.conv1a5.0.bias | (512,) | | | net_s.layers_up.0.blocks.0.conv1a5.1 | | 0.131M | | net_s.layers_up.0.blocks.0.conv33.0 | 4.608K | | | net_s.layers_up.0.blocks.0.conv33.0.weight | (512, 1, 3, 3) | | | net_s.layers_up.0.blocks.0.conv55 | 12.8K | 0.95M | | net_s.layers_up.0.blocks.0.conv55.0 | 12.8K | 0.819M | | net_s.layers_up.0.blocks.0.conv55.0.weight | (512, 1, 5, 5) | | | net_s.layers_up.0.blocks.0.conv55.1 | | 0.131M | | net_s.layers_up.0.blocks.0.conv77 | 25.088K | 1.737M | | net_s.layers_up.0.blocks.0.conv77.0 | 25.088K | 1.606M | | net_s.layers_up.0.blocks.0.conv77.0.weight | (512, 1, 7, 7) | | | net_s.layers_up.0.blocks.0.conv77.1 | | 0.131M | | net_s.layers_up.0.blocks.0.finalconv11 | 0.787M | 50.332M | | net_s.layers_up.0.blocks.0.finalconv11.weight | (512, 1536, 1, 1) | | | net_s.layers_up.0.blocks.0.finalconv11.bias | (512,) | | | net_s.layers_up.1 | 3.761M | 0.827G | | net_s.layers_up.1.blocks | 3.236M | 0.793G | | net_s.layers_up.1.blocks.0 | 1.618M | 0.397G | | net_s.layers_up.1.blocks.0.smm_blocks | 1.137M | 0.273G | | net_s.layers_up.1.blocks.0.conv1b3 | 65.792K | 17.039M | | net_s.layers_up.1.blocks.0.conv1a3 | 65.792K | 17.039M | | net_s.layers_up.1.blocks.0.conv1b5 | 65.792K | 17.039M | | net_s.layers_up.1.blocks.0.conv1a5 | 65.792K | 17.039M | | net_s.layers_up.1.blocks.0.conv33.0 | 2.304K | | | net_s.layers_up.1.blocks.0.conv55 | 6.4K | 1.901M | | net_s.layers_up.1.blocks.0.conv77 | 12.544K | 3.473M | | net_s.layers_up.1.blocks.0.finalconv11 | 0.197M | 50.332M | | net_s.layers_up.1.blocks.1 | 1.618M | 0.397G | | net_s.layers_up.1.blocks.1.smm_blocks | 1.137M | 0.273G | | net_s.layers_up.1.blocks.1.conv1b3 | 65.792K | 17.039M | | net_s.layers_up.1.blocks.1.conv1a3 | 65.792K | 17.039M | | net_s.layers_up.1.blocks.1.conv1b5 | 65.792K | 17.039M | | net_s.layers_up.1.blocks.1.conv1a5 | 65.792K | 17.039M | | net_s.layers_up.1.blocks.1.conv33.0 | 2.304K | | | net_s.layers_up.1.blocks.1.conv55 | 6.4K | 1.901M | | net_s.layers_up.1.blocks.1.conv77 | 12.544K | 3.473M | | net_s.layers_up.1.blocks.1.finalconv11 | 0.197M | 50.332M | | net_s.layers_up.1.upsample | 0.525M | 33.882M | | net_s.layers_up.1.upsample.expand | 0.524M | 33.554M | | net_s.layers_up.1.upsample.expand.weight | (1024, 512) | | | net_s.layers_up.1.upsample.norm | 0.512K | 0.328M | | net_s.layers_up.1.upsample.norm.weight | (256,) | | | net_s.layers_up.1.upsample.norm.bias | (256,) | | | net_s.layers_up.2 | 1.411M | 1.227G | | net_s.layers_up.2.blocks | 1.279M | 1.193G | | net_s.layers_up.2.blocks.0 | 0.64M | 0.596G | | net_s.layers_up.2.blocks.0.smm_blocks | 0.514M | 0.466G | | net_s.layers_up.2.blocks.0.conv1b3 | 16.512K | 17.302M | | net_s.layers_up.2.blocks.0.conv1a3 | 16.512K | 17.302M | | net_s.layers_up.2.blocks.0.conv1b5 | 16.512K | 17.302M | | net_s.layers_up.2.blocks.0.conv1a5 | 16.512K | 17.302M | | net_s.layers_up.2.blocks.0.conv33.0 | 1.152K | | | net_s.layers_up.2.blocks.0.conv55 | 3.2K | 3.801M | | net_s.layers_up.2.blocks.0.conv77 | 6.272K | 6.947M | | net_s.layers_up.2.blocks.0.finalconv11 | 49.28K | 50.332M | | net_s.layers_up.2.blocks.1 | 0.64M | 0.596G | | net_s.layers_up.2.blocks.1.smm_blocks | 0.514M | 0.466G | | net_s.layers_up.2.blocks.1.conv1b3 | 16.512K | 17.302M | | net_s.layers_up.2.blocks.1.conv1a3 | 16.512K | 17.302M | | net_s.layers_up.2.blocks.1.conv1b5 | 16.512K | 17.302M | | net_s.layers_up.2.blocks.1.conv1a5 | 16.512K | 17.302M | | net_s.layers_up.2.blocks.1.conv33.0 | 1.152K | | | net_s.layers_up.2.blocks.1.conv55 | 3.2K | 3.801M | | net_s.layers_up.2.blocks.1.conv77 | 6.272K | 6.947M | | net_s.layers_up.2.blocks.1.finalconv11 | 49.28K | 50.332M | | net_s.layers_up.2.upsample | 0.131M | 34.21M | | net_s.layers_up.2.upsample.expand | 0.131M | 33.554M | | net_s.layers_up.2.upsample.expand.weight | (512, 256) | | | net_s.layers_up.2.upsample.norm | 0.256K | 0.655M | | net_s.layers_up.2.upsample.norm.weight | (128,) | | | net_s.layers_up.2.upsample.norm.bias | (128,) | | | net_s.layers_up.3 | 0.26M | 0.758G | | net_s.layers_up.3.blocks.0 | 0.227M | 0.723G | | net_s.layers_up.3.blocks.0.smm_blocks | 0.192M | 0.58G | | net_s.layers_up.3.blocks.0.smm_blocks.0 | 64.128K | 0.193G | | net_s.layers_up.3.blocks.0.smm_blocks.1 | 64.128K | 0.193G | | net_s.layers_up.3.blocks.0.smm_blocks.2 | 64.128K | 0.193G | | net_s.layers_up.3.blocks.0.conv1b3 | 4.16K | 17.826M | | net_s.layers_up.3.blocks.0.conv1b3.0 | 4.16K | 16.777M | | net_s.layers_up.3.blocks.0.conv1b3.1 | | 1.049M | | net_s.layers_up.3.blocks.0.conv1a3 | 4.16K | 17.826M | | net_s.layers_up.3.blocks.0.conv1a3.0 | 4.16K | 16.777M | | net_s.layers_up.3.blocks.0.conv1a3.1 | | 1.049M | | net_s.layers_up.3.blocks.0.conv1b5 | 4.16K | 17.826M | | net_s.layers_up.3.blocks.0.conv1b5.0 | 4.16K | 16.777M | | net_s.layers_up.3.blocks.0.conv1b5.1 | | 1.049M | | net_s.layers_up.3.blocks.0.conv1a5 | 4.16K | 17.826M | | net_s.layers_up.3.blocks.0.conv1a5.0 | 4.16K | 16.777M | | net_s.layers_up.3.blocks.0.conv1a5.1 | | 1.049M | | net_s.layers_up.3.blocks.0.conv33.0 | 0.576K | | | net_s.layers_up.3.blocks.0.conv33.0.weight | (64, 1, 3, 3) | | | net_s.layers_up.3.blocks.0.conv55 | 1.6K | 7.602M | | net_s.layers_up.3.blocks.0.conv55.0 | 1.6K | 6.554M | | net_s.layers_up.3.blocks.0.conv55.1 | | 1.049M | | net_s.layers_up.3.blocks.0.conv77 | 3.136K | 13.894M | | net_s.layers_up.3.blocks.0.conv77.0 | 3.136K | 12.845M | | net_s.layers_up.3.blocks.0.conv77.1 | | 1.049M | | net_s.layers_up.3.blocks.0.finalconv11 | 12.352K | 50.332M | | net_s.layers_up.3.blocks.0.finalconv11.weight | (64, 192, 1, 1) | | | net_s.layers_up.3.blocks.0.finalconv11.bias | (64,) | | | net_s.layers_up.3.upsample | 32.896K | 34.865M | | net_s.layers_up.3.upsample.expand | 32.768K | 33.554M | | net_s.layers_up.3.upsample.expand.weight | (256, 128) | | | net_s.layers_up.3.upsample.norm | 0.128K | 1.311M | | net_s.layers_up.3.upsample.norm.weight | (64,) | | | net_s.layers_up.3.upsample.norm.bias | (64,) | | --------------------------------------------------------------------------------- 2024-04-02 23:26:20,416 - ==> Creating optimizer 2024-04-02 23:26:20,427 - ==> Loading dataset: DefaultAD 2024-04-02 23:26:20,971 - ==> ********** cfg ********** epoch_full : 1000 fvcore_b : 1 fvcore_c : 3 metrics : ['mAUROC_sp_max', 'mAP_sp_max', 'mF1_max_sp_max', 'mAUROC_px', 'mAP_px', 'mF1_max_px', 'mAUPRO_px', 'mF1_px_0.2_0.8_0.1', 'mAcc_px_0.2_0.8_0.1', 'mIoU_px_0.2_0.8_0.1', 'mIoU_max_px'] evaluator.kwargs : {'metrics': ['mAUROC_sp_max', 'mAP_sp_max', 'mF1_max_sp_max', 'mAUROC_px', 'mAP_px', 'mF1_max_px', 'mAUPRO_px', 'mF1_px_0.2_0.8_0.1', 'mAcc_px_0.2_0.8_0.1', 'mIoU_px_0.2_0.8_0.1', 'mIoU_max_px'], 'pooling_ks': None, 'max_step_aupro': 100} optim.lr : 0.005 optim.kwargs : {'name': 'adamw', 'betas': (0.9, 0.999), 'eps': 1e-08, 'weight_decay': 0.0001, 'amsgrad': False} trainer.name : MAMBAADTrainer trainer.checkpoint : runs trainer.logdir_sub : trainer.resume_dir : trainer.cuda_deterministic : False trainer.epoch_full : 1000 trainer.scheduler_kwargs : {'name': 'step', 'lr_noise': None, 'noise_pct': 0.67, 'noise_std': 1.0, 'noise_seed': 42, 'lr_min': 5e-05, 'warmup_lr': 5e-06, 'warmup_iters': -1, 'cooldown_iters': 0, 'warmup_epochs': 0, 'cooldown_epochs': 0, 'use_iters': True, 'patience_iters': 0, 'patience_epochs': 0, 'decay_iters': 0, 'decay_epochs': 800, 'cycle_decay': 0.1, 'decay_rate': 0.1} trainer.mixup_kwargs : {'mixup_alpha': 0.8, 'cutmix_alpha': 1.0, 'cutmix_minmax': None, 'prob': 0.0, 'switch_prob': 0.5, 'mode': 'batch', 'correct_lam': True, 'label_smoothing': 0.1} trainer.test_start_epoch : 1000 trainer.test_per_epoch : 50 trainer.find_unused_parameters : False trainer.sync_BN : apex trainer.dist_BN : trainer.scaler : none trainer.data.batch_size : 16 trainer.data.batch_size_per_gpu : 16 trainer.data.batch_size_test : 16 trainer.data.batch_size_per_gpu_test : 16 trainer.data.num_workers_per_gpu : 4 trainer.data.drop_last : True trainer.data.pin_memory : True trainer.data.persistent_workers : False trainer.data.num_workers : 4 trainer.iter : 0 trainer.epoch : 0 trainer.iter_full : 4848000 trainer.metric_recorder : {'mAUROC_sp_max_coco': [], 'mAP_sp_max_coco': [], 'mF1_max_sp_max_coco': [], 'mAUROC_px_coco': [], 'mAP_px_coco': [], 'mF1_max_px_coco': [], 'mAUPRO_px_coco': [], 'mF1_px_0.2_0.8_0.1_coco': [], 'mAcc_px_0.2_0.8_0.1_coco': [], 'mIoU_px_0.2_0.8_0.1_coco': [], 'mIoU_max_px_coco': []} loss.loss_terms : [{'type': 'L2Loss', 'name': 'pixel', 'lam': 5.0}] loss.clip_grad : 5.0 loss.create_graph : False loss.retain_graph : False adv : False logging.log_terms_train : [{'name': 'batch_t', 'fmt': ':>5.3f', 'add_name': 'avg'}, {'name': 'data_t', 'fmt': ':>5.3f'}, {'name': 'optim_t', 'fmt': ':>5.3f'}, {'name': 'lr', 'fmt': ':>7.6f'}, {'name': 'cos', 'suffixes': [''], 'fmt': ':>5.3f', 'add_name': 'avg'}] logging.log_terms_test : [{'name': 'batch_t', 'fmt': ':>5.3f', 'add_name': 'avg'}, {'name': 'cos', 'suffixes': [''], 'fmt': ':>5.3f', 'add_name': 'avg'}] logging.train_reset_log_per : 100 logging.train_log_per : 100 logging.test_log_per : 50 data.sampler : naive data.loader_type : pil data.loader_type_target : pil_L data.type : DefaultAD data.root : data/coco data.meta : meta_20_3.json data.cls_names : ['coco'] data.train_transforms : [{'type': 'Resize', 'size': (256, 256), 'interpolation': }, {'type': 'CenterCrop', 'size': (256, 256)}, {'type': 'ToTensor'}, {'type': 'Normalize', 'mean': (0.485, 0.456, 0.406), 'std': (0.229, 0.224, 0.225), 'inplace': True}] data.test_transforms : [{'type': 'Resize', 'size': (256, 256), 'interpolation': }, {'type': 'CenterCrop', 'size': (256, 256)}, {'type': 'ToTensor'}, {'type': 'Normalize', 'mean': (0.485, 0.456, 0.406), 'std': (0.229, 0.224, 0.225), 'inplace': True}] data.target_transforms : [{'type': 'Resize', 'size': (256, 256), 'interpolation': }, {'type': 'CenterCrop', 'size': (256, 256)}, {'type': 'ToTensor'}] data.train_size : 4848 data.test_size : 310 data.train_length : 77580 data.test_length : 4952 model_t.name : timm_resnet34 model_t.kwargs : {'pretrained': False, 'checkpoint_path': 'model/pretrain/resnet34-43635321.pth', 'strict': False, 'features_only': True, 'out_indices': [1, 2, 3]} model_s : {'depths_decoder': [3, 4, 6, 3], 'scan_type': 'hilbert', 'num_direction': 4} model.name : mambaadhcs57c1 model.kwargs : {'pretrained': False, 'checkpoint_path': '', 'strict': True, 'model_t': Namespace(name='timm_resnet34', kwargs={'pretrained': False, 'checkpoint_path': 'model/pretrain/resnet34-43635321.pth', 'strict': False, 'features_only': True, 'out_indices': [1, 2, 3]}), 'model_s': {'depths_decoder': [3, 4, 6, 3], 'scan_type': 'hilbert', 'num_direction': 4}} seed : 42 size : 256 warmup_epochs : 0 test_start_epoch : 1000 test_per_epoch : 50 batch_train : 16 batch_test_per : 16 lr : 0.005 weight_decay : 0.0001 cfg_path : configs.mambaad.mambaad_coco3_nhcs57c1d4 mode : train sleep : -1 memory : -1 dist_url : env:// logger_rank : 0 opts : [] command : python3 -m torch.distributed.launch --nproc_per_node=$nproc_per_node --nnodes=$nnodes --node_rank=$node_rank --master_addr=$master_addr --master_port=$master_port --use_env run.py -c configs.mambaad.mambaad_coco3_nhcs57c1d4 -m train --sleep -1 --memory -1 --dist_url env:// --logger_rank 0 task_start_time : 10065453.407139895 dist : False world_size : 1 rank : 0 local_rank : 0 ngpus_per_node : 1 nnodes : 1 master : True logdir : runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601 logger.filters : [] logger.name : root logger.level : 20 logger.parent : None logger.propagate : True logger.disabled : False logdir_train : runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601/show_train logdir_test : runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601/show_test 2024-04-02 23:26:20,971 - ==> Starting training with 1 nodes x 1 GPUs 2024-04-02 23:27:44,982 - Train: 0.00% [100/4848000] [0.0/1000.0] [batch_t 0.759 (0.837)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-02 23:29:00,905 - Train: 0.00% [200/4848000] [0.0/1000.0] [batch_t 0.761 (0.759)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-02 23:30:17,079 - Train: 0.01% [300/4848000] [0.1/1000.0] [batch_t 0.761 (0.762)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-02 23:31:32,535 - Train: 0.01% [400/4848000] [0.1/1000.0] [batch_t 0.763 (0.754)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-02 23:32:50,999 - Train: 0.01% [500/4848000] [0.1/1000.0] [batch_t 0.757 (0.784)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-02 23:34:06,470 - Train: 0.01% [600/4848000] [0.1/1000.0] [batch_t 0.762 (0.755)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-02 23:35:22,647 - Train: 0.01% [700/4848000] [0.1/1000.0] [batch_t 0.762 (0.762)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-02 23:36:38,206 - Train: 0.02% [800/4848000] [0.2/1000.0] [batch_t 0.766 (0.755)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-02 23:37:53,757 - Train: 0.02% [900/4848000] [0.2/1000.0] [batch_t 0.748 (0.755)] [data_t 0.002] [optim_t 0.745] [lr 0.005000] 2024-04-02 23:39:12,739 - Train: 0.02% [1000/4848000] [0.2/1000.0] [batch_t 0.744 (0.790)] [data_t 0.002] [optim_t 0.741] [lr 0.005000] 2024-04-02 23:40:29,831 - Train: 0.02% [1100/4848000] [0.2/1000.0] [batch_t 0.767 (0.771)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-02 23:41:46,456 - Train: 0.02% [1200/4848000] [0.2/1000.0] [batch_t 0.762 (0.766)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-02 23:43:03,322 - Train: 0.03% [1300/4848000] [0.3/1000.0] [batch_t 0.753 (0.769)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-02 23:44:24,036 - Train: 0.03% [1400/4848000] [0.3/1000.0] [batch_t 0.749 (0.807)] [data_t 0.002] [optim_t 0.747] [lr 0.005000] 2024-04-02 23:45:39,545 - Train: 0.03% [1500/4848000] [0.3/1000.0] [batch_t 0.753 (0.755)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-02 23:46:56,101 - Train: 0.03% [1600/4848000] [0.3/1000.0] [batch_t 0.758 (0.765)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-02 23:48:11,618 - Train: 0.04% [1700/4848000] [0.4/1000.0] [batch_t 0.752 (0.755)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-02 23:49:27,008 - Train: 0.04% [1800/4848000] [0.4/1000.0] [batch_t 0.766 (0.754)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-02 23:50:42,489 - Train: 0.04% [1900/4848000] [0.4/1000.0] [batch_t 0.761 (0.755)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-02 23:51:57,932 - Train: 0.04% [2000/4848000] [0.4/1000.0] [batch_t 0.761 (0.754)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-02 23:53:13,430 - Train: 0.04% [2100/4848000] [0.4/1000.0] [batch_t 0.762 (0.755)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-02 23:54:28,977 - Train: 0.05% [2200/4848000] [0.5/1000.0] [batch_t 0.758 (0.755)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-02 23:55:44,368 - Train: 0.05% [2300/4848000] [0.5/1000.0] [batch_t 0.748 (0.754)] [data_t 0.002] [optim_t 0.746] [lr 0.005000] 2024-04-02 23:56:59,919 - Train: 0.05% [2400/4848000] [0.5/1000.0] [batch_t 0.762 (0.755)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-02 23:58:15,348 - Train: 0.05% [2500/4848000] [0.5/1000.0] [batch_t 0.768 (0.754)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-02 23:59:30,774 - Train: 0.05% [2600/4848000] [0.5/1000.0] [batch_t 0.748 (0.754)] [data_t 0.002] [optim_t 0.745] [lr 0.005000] 2024-04-03 00:00:46,336 - Train: 0.06% [2700/4848000] [0.6/1000.0] [batch_t 0.757 (0.756)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-03 00:02:01,841 - Train: 0.06% [2800/4848000] [0.6/1000.0] [batch_t 0.753 (0.755)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-03 00:03:17,431 - Train: 0.06% [2900/4848000] [0.6/1000.0] [batch_t 0.753 (0.756)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-03 00:04:32,935 - Train: 0.06% [3000/4848000] [0.6/1000.0] [batch_t 0.764 (0.755)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-03 00:05:48,486 - Train: 0.06% [3100/4848000] [0.6/1000.0] [batch_t 0.754 (0.755)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-03 00:07:03,996 - Train: 0.07% [3200/4848000] [0.7/1000.0] [batch_t 0.758 (0.755)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-03 00:08:19,656 - Train: 0.07% [3300/4848000] [0.7/1000.0] [batch_t 0.761 (0.757)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-03 00:09:35,192 - Train: 0.07% [3400/4848000] [0.7/1000.0] [batch_t 0.752 (0.755)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-03 00:10:50,637 - Train: 0.07% [3500/4848000] [0.7/1000.0] [batch_t 0.749 (0.754)] [data_t 0.002] [optim_t 0.747] [lr 0.005000] 2024-04-03 00:12:06,327 - Train: 0.07% [3600/4848000] [0.7/1000.0] [batch_t 0.754 (0.757)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-03 00:13:21,878 - Train: 0.08% [3700/4848000] [0.8/1000.0] [batch_t 0.754 (0.755)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-03 00:14:37,444 - Train: 0.08% [3800/4848000] [0.8/1000.0] [batch_t 0.752 (0.756)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-03 00:15:52,940 - Train: 0.08% [3900/4848000] [0.8/1000.0] [batch_t 0.759 (0.755)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-03 00:17:08,398 - Train: 0.08% [4000/4848000] [0.8/1000.0] [batch_t 0.757 (0.754)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-03 00:18:23,984 - Train: 0.08% [4100/4848000] [0.8/1000.0] [batch_t 0.755 (0.756)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-03 00:19:39,520 - Train: 0.09% [4200/4848000] [0.9/1000.0] [batch_t 0.763 (0.755)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-03 00:20:55,196 - Train: 0.09% [4300/4848000] [0.9/1000.0] [batch_t 0.763 (0.757)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-03 00:22:10,829 - Train: 0.09% [4400/4848000] [0.9/1000.0] [batch_t 0.754 (0.756)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-03 00:23:26,467 - Train: 0.09% [4500/4848000] [0.9/1000.0] [batch_t 0.761 (0.756)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-03 00:24:42,144 - Train: 0.09% [4600/4848000] [0.9/1000.0] [batch_t 0.754 (0.757)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-03 00:25:57,757 - Train: 0.10% [4700/4848000] [1.0/1000.0] [batch_t 0.753 (0.756)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-03 00:27:13,338 - Train: 0.10% [4800/4848000] [1.0/1000.0] [batch_t 0.767 (0.756)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-03 00:27:49,114 - ==> Total time: 1:01:47 Eta: 42 days, 20:51:46 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 00:28:09,456 - Train: 0.10% [4900/4848000] [1.0/1000.0] [batch_t 0.330 (0.354)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 00:28:42,244 - Train: 0.10% [5000/4848000] [1.0/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 00:29:17,367 - Train: 0.11% [5100/4848000] [1.1/1000.0] [batch_t 0.323 (0.351)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 00:29:50,155 - Train: 0.11% [5200/4848000] [1.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 00:30:25,234 - Train: 0.11% [5300/4848000] [1.1/1000.0] [batch_t 0.331 (0.351)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 00:31:02,249 - Train: 0.11% [5400/4848000] [1.1/1000.0] [batch_t 0.332 (0.370)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 00:31:37,195 - Train: 0.11% [5500/4848000] [1.1/1000.0] [batch_t 0.329 (0.349)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 00:32:11,199 - Train: 0.12% [5600/4848000] [1.2/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 00:32:44,198 - Train: 0.12% [5700/4848000] [1.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 00:33:19,475 - Train: 0.12% [5800/4848000] [1.2/1000.0] [batch_t 0.326 (0.353)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 00:33:52,263 - Train: 0.12% [5900/4848000] [1.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 00:34:25,755 - Train: 0.12% [6000/4848000] [1.2/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 00:34:58,505 - Train: 0.13% [6100/4848000] [1.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 00:35:34,760 - Train: 0.13% [6200/4848000] [1.3/1000.0] [batch_t 0.337 (0.362)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-03 00:36:10,410 - Train: 0.13% [6300/4848000] [1.3/1000.0] [batch_t 0.328 (0.356)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 00:36:43,227 - Train: 0.13% [6400/4848000] [1.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 00:37:18,157 - Train: 0.13% [6500/4848000] [1.3/1000.0] [batch_t 0.330 (0.349)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 00:37:51,013 - Train: 0.14% [6600/4848000] [1.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 00:38:26,149 - Train: 0.14% [6700/4848000] [1.4/1000.0] [batch_t 0.327 (0.351)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 00:38:58,919 - Train: 0.14% [6800/4848000] [1.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 00:39:34,703 - Train: 0.14% [6900/4848000] [1.4/1000.0] [batch_t 0.330 (0.358)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 00:40:07,564 - Train: 0.14% [7000/4848000] [1.4/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 00:40:40,499 - Train: 0.15% [7100/4848000] [1.5/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 00:41:17,321 - Train: 0.15% [7200/4848000] [1.5/1000.0] [batch_t 0.329 (0.368)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 00:41:50,133 - Train: 0.15% [7300/4848000] [1.5/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 00:42:26,274 - Train: 0.15% [7400/4848000] [1.5/1000.0] [batch_t 0.328 (0.361)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 00:42:59,030 - Train: 0.15% [7500/4848000] [1.5/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 00:43:34,126 - Train: 0.16% [7600/4848000] [1.6/1000.0] [batch_t 0.333 (0.351)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 00:44:10,669 - Train: 0.16% [7700/4848000] [1.6/1000.0] [batch_t 0.425 (0.365)] [data_t 0.099] [optim_t 0.326] [lr 0.005000] 2024-04-03 00:44:43,424 - Train: 0.16% [7800/4848000] [1.6/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 00:45:20,013 - Train: 0.16% [7900/4848000] [1.6/1000.0] [batch_t 0.329 (0.366)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 00:45:52,753 - Train: 0.17% [8000/4848000] [1.7/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 00:46:25,528 - Train: 0.17% [8100/4848000] [1.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 00:46:58,300 - Train: 0.17% [8200/4848000] [1.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 00:47:33,935 - Train: 0.17% [8300/4848000] [1.7/1000.0] [batch_t 0.329 (0.356)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 00:48:07,880 - Train: 0.17% [8400/4848000] [1.7/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 00:48:40,719 - Train: 0.18% [8500/4848000] [1.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 00:49:15,842 - Train: 0.18% [8600/4848000] [1.8/1000.0] [batch_t 0.326 (0.351)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 00:49:48,625 - Train: 0.18% [8700/4848000] [1.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 00:50:23,890 - Train: 0.18% [8800/4848000] [1.8/1000.0] [batch_t 0.329 (0.353)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 00:50:56,634 - Train: 0.18% [8900/4848000] [1.8/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 00:51:29,856 - Train: 0.19% [9000/4848000] [1.9/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 00:52:02,581 - Train: 0.19% [9100/4848000] [1.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 00:52:35,937 - Train: 0.19% [9200/4848000] [1.9/1000.0] [batch_t 0.326 (0.333)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 00:53:08,974 - Train: 0.19% [9300/4848000] [1.9/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 00:53:41,773 - Train: 0.19% [9400/4848000] [1.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 00:54:14,781 - Train: 0.20% [9500/4848000] [2.0/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 00:54:47,508 - Train: 0.20% [9600/4848000] [2.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 00:55:18,981 - ==> Total time: 1:29:17 Eta: 30 days, 22:36:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 00:55:21,897 - Train: 0.20% [9700/4848000] [2.0/1000.0] [batch_t 0.332 (0.368)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 00:55:54,709 - Train: 0.20% [9800/4848000] [2.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 00:56:28,746 - Train: 0.20% [9900/4848000] [2.0/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 00:57:01,504 - Train: 0.21% [10000/4848000] [2.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 00:57:36,644 - Train: 0.21% [10100/4848000] [2.1/1000.0] [batch_t 0.327 (0.351)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 00:58:12,216 - Train: 0.21% [10200/4848000] [2.1/1000.0] [batch_t 0.330 (0.356)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 00:58:45,100 - Train: 0.21% [10300/4848000] [2.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 00:59:18,905 - Train: 0.21% [10400/4848000] [2.1/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 00:59:51,640 - Train: 0.22% [10500/4848000] [2.2/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 01:00:25,416 - Train: 0.22% [10600/4848000] [2.2/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 01:00:58,192 - Train: 0.22% [10700/4848000] [2.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 01:01:31,978 - Train: 0.22% [10800/4848000] [2.2/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 01:02:06,599 - Train: 0.22% [10900/4848000] [2.2/1000.0] [batch_t 2.225 (0.346)] [data_t 1.902] [optim_t 0.323] [lr 0.005000] 2024-04-03 01:02:39,337 - Train: 0.23% [11000/4848000] [2.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 01:03:14,581 - Train: 0.23% [11100/4848000] [2.3/1000.0] [batch_t 0.324 (0.352)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 01:03:47,277 - Train: 0.23% [11200/4848000] [2.3/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 01:04:21,111 - Train: 0.23% [11300/4848000] [2.3/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 01:04:53,834 - Train: 0.24% [11400/4848000] [2.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 01:05:29,621 - Train: 0.24% [11500/4848000] [2.4/1000.0] [batch_t 0.329 (0.358)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 01:06:02,429 - Train: 0.24% [11600/4848000] [2.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 01:06:38,011 - Train: 0.24% [11700/4848000] [2.4/1000.0] [batch_t 0.327 (0.356)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 01:07:13,867 - Train: 0.24% [11800/4848000] [2.4/1000.0] [batch_t 0.326 (0.358)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 01:07:46,647 - Train: 0.25% [11900/4848000] [2.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 01:08:20,017 - Train: 0.25% [12000/4848000] [2.5/1000.0] [batch_t 0.332 (0.334)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 01:08:52,865 - Train: 0.25% [12100/4848000] [2.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 01:09:26,885 - Train: 0.25% [12200/4848000] [2.5/1000.0] [batch_t 0.324 (0.340)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 01:09:59,681 - Train: 0.25% [12300/4848000] [2.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 01:10:34,401 - Train: 0.26% [12400/4848000] [2.6/1000.0] [batch_t 0.334 (0.347)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 01:11:09,229 - Train: 0.26% [12500/4848000] [2.6/1000.0] [batch_t 0.327 (0.348)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 01:11:42,124 - Train: 0.26% [12600/4848000] [2.6/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 01:12:17,711 - Train: 0.26% [12700/4848000] [2.6/1000.0] [batch_t 0.328 (0.356)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 01:12:50,717 - Train: 0.26% [12800/4848000] [2.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 01:13:24,961 - Train: 0.27% [12900/4848000] [2.7/1000.0] [batch_t 0.327 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 01:13:57,763 - Train: 0.27% [13000/4848000] [2.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 01:14:32,618 - Train: 0.27% [13100/4848000] [2.7/1000.0] [batch_t 0.329 (0.348)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 01:15:09,330 - Train: 0.27% [13200/4848000] [2.7/1000.0] [batch_t 0.369 (0.367)] [data_t 0.042] [optim_t 0.328] [lr 0.005000] 2024-04-03 01:15:42,111 - Train: 0.27% [13300/4848000] [2.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 01:16:17,678 - Train: 0.28% [13400/4848000] [2.8/1000.0] [batch_t 0.328 (0.356)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 01:16:50,453 - Train: 0.28% [13500/4848000] [2.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 01:17:24,008 - Train: 0.28% [13600/4848000] [2.8/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 01:17:56,762 - Train: 0.28% [13700/4848000] [2.8/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 01:18:31,803 - Train: 0.28% [13800/4848000] [2.8/1000.0] [batch_t 0.329 (0.350)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 01:19:05,178 - Train: 0.29% [13900/4848000] [2.9/1000.0] [batch_t 0.849 (0.334)] [data_t 0.516] [optim_t 0.333] [lr 0.005000] 2024-04-03 01:19:37,982 - Train: 0.29% [14000/4848000] [2.9/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 01:20:10,798 - Train: 0.29% [14100/4848000] [2.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 01:20:43,707 - Train: 0.29% [14200/4848000] [2.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 01:21:18,907 - Train: 0.29% [14300/4848000] [2.9/1000.0] [batch_t 0.330 (0.352)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 01:21:51,608 - Train: 0.30% [14400/4848000] [3.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 01:22:27,130 - Train: 0.30% [14500/4848000] [3.0/1000.0] [batch_t 0.328 (0.355)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 01:22:41,533 - ==> Total time: 1:56:40 Eta: 26 days, 22:12:24 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 01:23:02,180 - Train: 0.30% [14600/4848000] [3.0/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 01:23:38,180 - Train: 0.30% [14700/4848000] [3.0/1000.0] [batch_t 0.330 (0.360)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 01:24:13,458 - Train: 0.31% [14800/4848000] [3.1/1000.0] [batch_t 0.328 (0.353)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 01:24:46,253 - Train: 0.31% [14900/4848000] [3.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 01:25:22,472 - Train: 0.31% [15000/4848000] [3.1/1000.0] [batch_t 0.332 (0.362)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 01:25:55,248 - Train: 0.31% [15100/4848000] [3.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 01:26:28,580 - Train: 0.31% [15200/4848000] [3.1/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 01:27:01,407 - Train: 0.32% [15300/4848000] [3.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 01:27:37,384 - Train: 0.32% [15400/4848000] [3.2/1000.0] [batch_t 0.328 (0.360)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 01:28:12,875 - Train: 0.32% [15500/4848000] [3.2/1000.0] [batch_t 0.330 (0.355)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 01:28:45,638 - Train: 0.32% [15600/4848000] [3.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 01:29:19,330 - Train: 0.32% [15700/4848000] [3.2/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 01:29:52,072 - Train: 0.33% [15800/4848000] [3.3/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 01:30:26,016 - Train: 0.33% [15900/4848000] [3.3/1000.0] [batch_t 0.325 (0.339)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 01:30:58,780 - Train: 0.33% [16000/4848000] [3.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 01:31:32,363 - Train: 0.33% [16100/4848000] [3.3/1000.0] [batch_t 0.332 (0.336)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 01:32:08,482 - Train: 0.33% [16200/4848000] [3.3/1000.0] [batch_t 1.847 (0.361)] [data_t 1.524] [optim_t 0.323] [lr 0.005000] 2024-04-03 01:32:41,742 - Train: 0.34% [16300/4848000] [3.4/1000.0] [batch_t 0.331 (0.333)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 01:33:17,145 - Train: 0.34% [16400/4848000] [3.4/1000.0] [batch_t 0.327 (0.354)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 01:33:49,999 - Train: 0.34% [16500/4848000] [3.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 01:34:23,497 - Train: 0.34% [16600/4848000] [3.4/1000.0] [batch_t 0.333 (0.335)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 01:34:56,321 - Train: 0.34% [16700/4848000] [3.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 01:35:29,769 - Train: 0.35% [16800/4848000] [3.5/1000.0] [batch_t 0.324 (0.334)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 01:36:02,641 - Train: 0.35% [16900/4848000] [3.5/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 01:36:37,190 - Train: 0.35% [17000/4848000] [3.5/1000.0] [batch_t 0.329 (0.345)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 01:37:10,070 - Train: 0.35% [17100/4848000] [3.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 01:37:42,934 - Train: 0.35% [17200/4848000] [3.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 01:38:16,491 - Train: 0.36% [17300/4848000] [3.6/1000.0] [batch_t 0.326 (0.335)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 01:38:49,262 - Train: 0.36% [17400/4848000] [3.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 01:39:24,785 - Train: 0.36% [17500/4848000] [3.6/1000.0] [batch_t 0.326 (0.355)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 01:39:57,669 - Train: 0.36% [17600/4848000] [3.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 01:40:32,691 - Train: 0.37% [17700/4848000] [3.7/1000.0] [batch_t 0.335 (0.350)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 01:41:08,332 - Train: 0.37% [17800/4848000] [3.7/1000.0] [batch_t 0.330 (0.356)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 01:41:41,164 - Train: 0.37% [17900/4848000] [3.7/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 01:42:13,958 - Train: 0.37% [18000/4848000] [3.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 01:42:46,676 - Train: 0.37% [18100/4848000] [3.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 01:43:21,540 - Train: 0.38% [18200/4848000] [3.8/1000.0] [batch_t 0.328 (0.349)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 01:43:54,297 - Train: 0.38% [18300/4848000] [3.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 01:44:27,925 - Train: 0.38% [18400/4848000] [3.8/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 01:45:00,730 - Train: 0.38% [18500/4848000] [3.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 01:45:33,945 - Train: 0.38% [18600/4848000] [3.8/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 01:46:07,997 - Train: 0.39% [18700/4848000] [3.9/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 01:46:40,782 - Train: 0.39% [18800/4848000] [3.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 01:47:16,036 - Train: 0.39% [18900/4848000] [3.9/1000.0] [batch_t 0.328 (0.352)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 01:47:48,840 - Train: 0.39% [19000/4848000] [3.9/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 01:48:21,600 - Train: 0.39% [19100/4848000] [3.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 01:48:54,334 - Train: 0.40% [19200/4848000] [4.0/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 01:49:28,325 - Train: 0.40% [19300/4848000] [4.0/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 01:49:58,480 - ==> Total time: 2:23:56 Eta: 24 days, 21:23:27 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 01:50:05,109 - Train: 0.40% [19400/4848000] [4.0/1000.0] [batch_t 0.329 (0.573)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 01:50:37,954 - Train: 0.40% [19500/4848000] [4.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 01:51:13,407 - Train: 0.40% [19600/4848000] [4.0/1000.0] [batch_t 0.325 (0.354)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 01:51:46,348 - Train: 0.41% [19700/4848000] [4.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 01:52:19,537 - Train: 0.41% [19800/4848000] [4.1/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 01:52:52,396 - Train: 0.41% [19900/4848000] [4.1/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 01:53:26,069 - Train: 0.41% [20000/4848000] [4.1/1000.0] [batch_t 0.324 (0.337)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 01:53:58,916 - Train: 0.41% [20100/4848000] [4.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 01:54:32,313 - Train: 0.42% [20200/4848000] [4.2/1000.0] [batch_t 0.335 (0.334)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 01:55:05,066 - Train: 0.42% [20300/4848000] [4.2/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 01:55:40,566 - Train: 0.42% [20400/4848000] [4.2/1000.0] [batch_t 0.324 (0.355)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 01:56:14,258 - Train: 0.42% [20500/4848000] [4.2/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 01:56:47,016 - Train: 0.42% [20600/4848000] [4.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 01:57:19,823 - Train: 0.43% [20700/4848000] [4.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 01:57:52,588 - Train: 0.43% [20800/4848000] [4.3/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 01:58:25,499 - Train: 0.43% [20900/4848000] [4.3/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 01:58:58,256 - Train: 0.43% [21000/4848000] [4.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 01:59:32,215 - Train: 0.44% [21100/4848000] [4.4/1000.0] [batch_t 0.325 (0.339)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 02:00:06,318 - Train: 0.44% [21200/4848000] [4.4/1000.0] [batch_t 0.411 (0.341)] [data_t 0.079] [optim_t 0.332] [lr 0.005000] 2024-04-03 02:00:39,512 - Train: 0.44% [21300/4848000] [4.4/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 02:01:14,539 - Train: 0.44% [21400/4848000] [4.4/1000.0] [batch_t 0.328 (0.350)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 02:01:47,339 - Train: 0.44% [21500/4848000] [4.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 02:02:20,700 - Train: 0.45% [21600/4848000] [4.5/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 02:02:53,491 - Train: 0.45% [21700/4848000] [4.5/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 02:03:26,269 - Train: 0.45% [21800/4848000] [4.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 02:03:59,088 - Train: 0.45% [21900/4848000] [4.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 02:04:32,777 - Train: 0.45% [22000/4848000] [4.5/1000.0] [batch_t 0.325 (0.337)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 02:05:06,393 - Train: 0.46% [22100/4848000] [4.6/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 02:05:39,223 - Train: 0.46% [22200/4848000] [4.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 02:06:12,352 - Train: 0.46% [22300/4848000] [4.6/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 02:06:45,121 - Train: 0.46% [22400/4848000] [4.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 02:07:18,485 - Train: 0.46% [22500/4848000] [4.6/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 02:07:51,257 - Train: 0.47% [22600/4848000] [4.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 02:08:24,132 - Train: 0.47% [22700/4848000] [4.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 02:08:56,855 - Train: 0.47% [22800/4848000] [4.7/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 02:09:30,691 - Train: 0.47% [22900/4848000] [4.7/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 02:10:03,522 - Train: 0.47% [23000/4848000] [4.7/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 02:10:39,967 - Train: 0.48% [23100/4848000] [4.8/1000.0] [batch_t 0.325 (0.364)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 02:11:14,909 - Train: 0.48% [23200/4848000] [4.8/1000.0] [batch_t 0.326 (0.349)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 02:11:47,767 - Train: 0.48% [23300/4848000] [4.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 02:12:21,217 - Train: 0.48% [23400/4848000] [4.8/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 02:12:54,023 - Train: 0.48% [23500/4848000] [4.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 02:13:27,200 - Train: 0.49% [23600/4848000] [4.9/1000.0] [batch_t 0.324 (0.332)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 02:13:59,950 - Train: 0.49% [23700/4848000] [4.9/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 02:14:33,669 - Train: 0.49% [23800/4848000] [4.9/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 02:15:06,404 - Train: 0.49% [23900/4848000] [4.9/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 02:15:39,151 - Train: 0.50% [24000/4848000] [5.0/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 02:16:12,419 - Train: 0.50% [24100/4848000] [5.0/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 02:16:45,193 - Train: 0.50% [24200/4848000] [5.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 02:16:58,291 - ==> Total time: 2:50:56 Eta: 23 days, 14:58:21 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 02:17:20,984 - Train: 0.50% [24300/4848000] [5.0/1000.0] [batch_t 0.330 (0.352)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 02:17:53,758 - Train: 0.50% [24400/4848000] [5.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 02:18:27,333 - Train: 0.51% [24500/4848000] [5.1/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 02:19:00,020 - Train: 0.51% [24600/4848000] [5.1/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 02:19:32,832 - Train: 0.51% [24700/4848000] [5.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 02:20:05,654 - Train: 0.51% [24800/4848000] [5.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 02:20:38,469 - Train: 0.51% [24900/4848000] [5.1/1000.0] [batch_t 0.321 (0.328)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-03 02:21:11,291 - Train: 0.52% [25000/4848000] [5.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 02:21:44,076 - Train: 0.52% [25100/4848000] [5.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 02:22:17,626 - Train: 0.52% [25200/4848000] [5.2/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 02:22:50,435 - Train: 0.52% [25300/4848000] [5.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 02:23:24,851 - Train: 0.52% [25400/4848000] [5.2/1000.0] [batch_t 0.328 (0.344)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 02:23:57,708 - Train: 0.53% [25500/4848000] [5.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 02:24:32,791 - Train: 0.53% [25600/4848000] [5.3/1000.0] [batch_t 0.329 (0.351)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 02:25:05,528 - Train: 0.53% [25700/4848000] [5.3/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-03 02:25:38,270 - Train: 0.53% [25800/4848000] [5.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 02:26:11,578 - Train: 0.53% [25900/4848000] [5.3/1000.0] [batch_t 0.326 (0.333)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 02:26:44,337 - Train: 0.54% [26000/4848000] [5.4/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 02:27:19,320 - Train: 0.54% [26100/4848000] [5.4/1000.0] [batch_t 0.328 (0.350)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 02:27:52,105 - Train: 0.54% [26200/4848000] [5.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 02:28:27,356 - Train: 0.54% [26300/4848000] [5.4/1000.0] [batch_t 0.333 (0.352)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 02:29:00,086 - Train: 0.54% [26400/4848000] [5.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 02:29:34,096 - Train: 0.55% [26500/4848000] [5.5/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 02:30:09,313 - Train: 0.55% [26600/4848000] [5.5/1000.0] [batch_t 0.330 (0.352)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 02:30:42,125 - Train: 0.55% [26700/4848000] [5.5/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 02:31:15,532 - Train: 0.55% [26800/4848000] [5.5/1000.0] [batch_t 0.331 (0.334)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 02:31:48,484 - Train: 0.55% [26900/4848000] [5.5/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 02:32:24,115 - Train: 0.56% [27000/4848000] [5.6/1000.0] [batch_t 0.326 (0.356)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 02:32:56,879 - Train: 0.56% [27100/4848000] [5.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 02:33:32,244 - Train: 0.56% [27200/4848000] [5.6/1000.0] [batch_t 0.338 (0.354)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-03 02:34:05,215 - Train: 0.56% [27300/4848000] [5.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 02:34:38,123 - Train: 0.57% [27400/4848000] [5.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 02:35:13,011 - Train: 0.57% [27500/4848000] [5.7/1000.0] [batch_t 0.328 (0.349)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 02:35:45,868 - Train: 0.57% [27600/4848000] [5.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 02:36:20,499 - Train: 0.57% [27700/4848000] [5.7/1000.0] [batch_t 0.334 (0.346)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 02:36:53,330 - Train: 0.57% [27800/4848000] [5.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 02:37:27,777 - Train: 0.58% [27900/4848000] [5.8/1000.0] [batch_t 0.329 (0.344)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 02:38:00,591 - Train: 0.58% [28000/4848000] [5.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 02:38:34,295 - Train: 0.58% [28100/4848000] [5.8/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 02:39:09,090 - Train: 0.58% [28200/4848000] [5.8/1000.0] [batch_t 0.330 (0.348)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 02:39:42,049 - Train: 0.58% [28300/4848000] [5.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 02:40:15,973 - Train: 0.59% [28400/4848000] [5.9/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 02:40:48,795 - Train: 0.59% [28500/4848000] [5.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 02:41:21,618 - Train: 0.59% [28600/4848000] [5.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 02:41:54,401 - Train: 0.59% [28700/4848000] [5.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 02:42:27,233 - Train: 0.59% [28800/4848000] [5.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 02:42:59,991 - Train: 0.60% [28900/4848000] [6.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 02:43:35,287 - Train: 0.60% [29000/4848000] [6.0/1000.0] [batch_t 0.329 (0.353)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 02:44:04,186 - ==> Total time: 3:18:02 Eta: 22 days, 18:49:24 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 02:44:11,956 - Train: 0.60% [29100/4848000] [6.0/1000.0] [batch_t 0.326 (0.453)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 02:44:44,702 - Train: 0.60% [29200/4848000] [6.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 02:45:20,173 - Train: 0.60% [29300/4848000] [6.0/1000.0] [batch_t 0.327 (0.355)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 02:45:52,952 - Train: 0.61% [29400/4848000] [6.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 02:46:26,954 - Train: 0.61% [29500/4848000] [6.1/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 02:47:00,015 - Train: 0.61% [29600/4848000] [6.1/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 02:47:32,774 - Train: 0.61% [29700/4848000] [6.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 02:48:07,798 - Train: 0.61% [29800/4848000] [6.1/1000.0] [batch_t 0.328 (0.350)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 02:48:40,566 - Train: 0.62% [29900/4848000] [6.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 02:49:13,308 - Train: 0.62% [30000/4848000] [6.2/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 02:49:46,202 - Train: 0.62% [30100/4848000] [6.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 02:50:19,000 - Train: 0.62% [30200/4848000] [6.2/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 02:50:51,825 - Train: 0.62% [30300/4848000] [6.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 02:51:26,788 - Train: 0.63% [30400/4848000] [6.3/1000.0] [batch_t 0.330 (0.350)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 02:51:59,587 - Train: 0.63% [30500/4848000] [6.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 02:52:32,465 - Train: 0.63% [30600/4848000] [6.3/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 02:53:05,977 - Train: 0.63% [30700/4848000] [6.3/1000.0] [batch_t 0.326 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 02:53:38,792 - Train: 0.64% [30800/4848000] [6.4/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 02:54:12,613 - Train: 0.64% [30900/4848000] [6.4/1000.0] [batch_t 0.334 (0.338)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 02:54:45,426 - Train: 0.64% [31000/4848000] [6.4/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 02:55:18,622 - Train: 0.64% [31100/4848000] [6.4/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 02:55:51,438 - Train: 0.64% [31200/4848000] [6.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 02:56:25,279 - Train: 0.65% [31300/4848000] [6.5/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 02:56:58,121 - Train: 0.65% [31400/4848000] [6.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 02:57:31,261 - Train: 0.65% [31500/4848000] [6.5/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 02:58:04,013 - Train: 0.65% [31600/4848000] [6.5/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 02:58:37,472 - Train: 0.65% [31700/4848000] [6.5/1000.0] [batch_t 0.330 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 02:59:11,227 - Train: 0.66% [31800/4848000] [6.6/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 02:59:44,033 - Train: 0.66% [31900/4848000] [6.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 03:00:17,621 - Train: 0.66% [32000/4848000] [6.6/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 03:00:50,435 - Train: 0.66% [32100/4848000] [6.6/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 03:01:23,229 - Train: 0.66% [32200/4848000] [6.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 03:01:56,088 - Train: 0.67% [32300/4848000] [6.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 03:02:28,945 - Train: 0.67% [32400/4848000] [6.7/1000.0] [batch_t 0.337 (0.328)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-03 03:03:01,820 - Train: 0.67% [32500/4848000] [6.7/1000.0] [batch_t 0.336 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-03 03:03:35,369 - Train: 0.67% [32600/4848000] [6.7/1000.0] [batch_t 0.326 (0.335)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 03:04:08,581 - Train: 0.67% [32700/4848000] [6.7/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 03:04:41,443 - Train: 0.68% [32800/4848000] [6.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 03:05:15,492 - Train: 0.68% [32900/4848000] [6.8/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 03:05:48,273 - Train: 0.68% [33000/4848000] [6.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 03:06:21,873 - Train: 0.68% [33100/4848000] [6.8/1000.0] [batch_t 0.332 (0.336)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 03:06:54,738 - Train: 0.68% [33200/4848000] [6.8/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 03:07:28,361 - Train: 0.69% [33300/4848000] [6.9/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 03:08:01,173 - Train: 0.69% [33400/4848000] [6.9/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 03:08:35,009 - Train: 0.69% [33500/4848000] [6.9/1000.0] [batch_t 0.333 (0.338)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 03:09:08,386 - Train: 0.69% [33600/4848000] [6.9/1000.0] [batch_t 0.333 (0.334)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 03:09:41,170 - Train: 0.70% [33700/4848000] [7.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 03:10:15,298 - Train: 0.70% [33800/4848000] [7.0/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 03:10:48,059 - Train: 0.70% [33900/4848000] [7.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 03:10:59,839 - ==> Total time: 3:44:58 Eta: 22 days, 3:53:55 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 03:11:23,166 - Train: 0.70% [34000/4848000] [7.0/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 03:11:55,952 - Train: 0.70% [34100/4848000] [7.0/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 03:12:29,866 - Train: 0.71% [34200/4848000] [7.1/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 03:13:02,658 - Train: 0.71% [34300/4848000] [7.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 03:13:35,422 - Train: 0.71% [34400/4848000] [7.1/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 03:14:09,744 - Train: 0.71% [34500/4848000] [7.1/1000.0] [batch_t 0.328 (0.343)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 03:14:42,484 - Train: 0.71% [34600/4848000] [7.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 03:15:16,109 - Train: 0.72% [34700/4848000] [7.2/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 03:15:48,983 - Train: 0.72% [34800/4848000] [7.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 03:16:23,844 - Train: 0.72% [34900/4848000] [7.2/1000.0] [batch_t 0.328 (0.348)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 03:16:56,567 - Train: 0.72% [35000/4848000] [7.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 03:17:30,559 - Train: 0.72% [35100/4848000] [7.2/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 03:18:03,394 - Train: 0.73% [35200/4848000] [7.3/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 03:18:38,231 - Train: 0.73% [35300/4848000] [7.3/1000.0] [batch_t 0.328 (0.348)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 03:19:12,205 - Train: 0.73% [35400/4848000] [7.3/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 03:19:45,009 - Train: 0.73% [35500/4848000] [7.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 03:20:17,823 - Train: 0.73% [35600/4848000] [7.3/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 03:20:50,627 - Train: 0.74% [35700/4848000] [7.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 03:21:24,459 - Train: 0.74% [35800/4848000] [7.4/1000.0] [batch_t 0.344 (0.338)] [data_t 0.002] [optim_t 0.342] [lr 0.005000] 2024-04-03 03:21:57,338 - Train: 0.74% [35900/4848000] [7.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 03:22:30,813 - Train: 0.74% [36000/4848000] [7.4/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 03:23:03,612 - Train: 0.74% [36100/4848000] [7.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 03:23:36,854 - Train: 0.75% [36200/4848000] [7.5/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 03:24:09,594 - Train: 0.75% [36300/4848000] [7.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 03:24:42,361 - Train: 0.75% [36400/4848000] [7.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 03:25:18,051 - Train: 0.75% [36500/4848000] [7.5/1000.0] [batch_t 0.328 (0.357)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 03:25:50,827 - Train: 0.75% [36600/4848000] [7.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 03:26:23,604 - Train: 0.76% [36700/4848000] [7.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 03:26:56,505 - Train: 0.76% [36800/4848000] [7.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 03:27:29,604 - Train: 0.76% [36900/4848000] [7.6/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 03:28:02,421 - Train: 0.76% [37000/4848000] [7.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 03:28:37,087 - Train: 0.77% [37100/4848000] [7.7/1000.0] [batch_t 0.330 (0.347)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 03:29:10,378 - Train: 0.77% [37200/4848000] [7.7/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 03:29:43,147 - Train: 0.77% [37300/4848000] [7.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 03:30:18,722 - Train: 0.77% [37400/4848000] [7.7/1000.0] [batch_t 0.330 (0.356)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 03:30:51,469 - Train: 0.77% [37500/4848000] [7.7/1000.0] [batch_t 0.335 (0.327)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 03:31:26,447 - Train: 0.78% [37600/4848000] [7.8/1000.0] [batch_t 0.334 (0.350)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 03:31:59,236 - Train: 0.78% [37700/4848000] [7.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 03:32:34,774 - Train: 0.78% [37800/4848000] [7.8/1000.0] [batch_t 0.329 (0.355)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 03:33:09,601 - Train: 0.78% [37900/4848000] [7.8/1000.0] [batch_t 0.328 (0.348)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 03:33:42,426 - Train: 0.78% [38000/4848000] [7.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 03:34:15,213 - Train: 0.79% [38100/4848000] [7.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 03:34:47,983 - Train: 0.79% [38200/4848000] [7.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 03:35:22,073 - Train: 0.79% [38300/4848000] [7.9/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 03:35:54,860 - Train: 0.79% [38400/4848000] [7.9/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 03:36:29,361 - Train: 0.79% [38500/4848000] [7.9/1000.0] [batch_t 0.330 (0.345)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 03:37:02,072 - Train: 0.80% [38600/4848000] [8.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 03:37:37,279 - Train: 0.80% [38700/4848000] [8.0/1000.0] [batch_t 0.327 (0.352)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 03:38:04,766 - ==> Total time: 4:12:03 Eta: 21 days, 16:54:44 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 03:38:13,942 - Train: 0.80% [38800/4848000] [8.0/1000.0] [batch_t 0.327 (0.382)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 03:38:46,713 - Train: 0.80% [38900/4848000] [8.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 03:39:22,084 - Train: 0.80% [39000/4848000] [8.0/1000.0] [batch_t 0.335 (0.354)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 03:39:54,869 - Train: 0.81% [39100/4848000] [8.1/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 03:40:30,866 - Train: 0.81% [39200/4848000] [8.1/1000.0] [batch_t 0.326 (0.360)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 03:41:03,614 - Train: 0.81% [39300/4848000] [8.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 03:41:38,419 - Train: 0.81% [39400/4848000] [8.1/1000.0] [batch_t 0.327 (0.348)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 03:42:12,573 - Train: 0.81% [39500/4848000] [8.1/1000.0] [batch_t 0.325 (0.341)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 03:42:45,569 - Train: 0.82% [39600/4848000] [8.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 03:43:19,047 - Train: 0.82% [39700/4848000] [8.2/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 03:43:51,842 - Train: 0.82% [39800/4848000] [8.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 03:44:25,655 - Train: 0.82% [39900/4848000] [8.2/1000.0] [batch_t 0.323 (0.338)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 03:44:58,441 - Train: 0.83% [40000/4848000] [8.3/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 03:45:32,018 - Train: 0.83% [40100/4848000] [8.3/1000.0] [batch_t 0.332 (0.336)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 03:46:07,565 - Train: 0.83% [40200/4848000] [8.3/1000.0] [batch_t 3.023 (0.355)] [data_t 2.696] [optim_t 0.327] [lr 0.005000] 2024-04-03 03:46:40,467 - Train: 0.83% [40300/4848000] [8.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 03:47:14,192 - Train: 0.83% [40400/4848000] [8.3/1000.0] [batch_t 0.335 (0.337)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 03:47:47,049 - Train: 0.84% [40500/4848000] [8.4/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 03:48:20,699 - Train: 0.84% [40600/4848000] [8.4/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 03:48:53,491 - Train: 0.84% [40700/4848000] [8.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 03:49:26,367 - Train: 0.84% [40800/4848000] [8.4/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 03:49:59,138 - Train: 0.84% [40900/4848000] [8.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 03:50:34,212 - Train: 0.85% [41000/4848000] [8.5/1000.0] [batch_t 0.328 (0.351)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 03:51:07,983 - Train: 0.85% [41100/4848000] [8.5/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 03:51:40,743 - Train: 0.85% [41200/4848000] [8.5/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 03:52:16,342 - Train: 0.85% [41300/4848000] [8.5/1000.0] [batch_t 0.328 (0.356)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 03:52:49,186 - Train: 0.85% [41400/4848000] [8.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 03:53:23,784 - Train: 0.86% [41500/4848000] [8.6/1000.0] [batch_t 0.324 (0.346)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 03:53:56,576 - Train: 0.86% [41600/4848000] [8.6/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 03:54:31,577 - Train: 0.86% [41700/4848000] [8.6/1000.0] [batch_t 0.327 (0.350)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 03:55:04,400 - Train: 0.86% [41800/4848000] [8.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 03:55:37,500 - Train: 0.86% [41900/4848000] [8.6/1000.0] [batch_t 0.337 (0.331)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-03 03:56:12,517 - Train: 0.87% [42000/4848000] [8.7/1000.0] [batch_t 0.326 (0.350)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 03:56:45,435 - Train: 0.87% [42100/4848000] [8.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 03:57:20,711 - Train: 0.87% [42200/4848000] [8.7/1000.0] [batch_t 0.329 (0.353)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 03:57:53,452 - Train: 0.87% [42300/4848000] [8.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 03:58:26,324 - Train: 0.87% [42400/4848000] [8.7/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 03:58:59,175 - Train: 0.88% [42500/4848000] [8.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 03:59:33,160 - Train: 0.88% [42600/4848000] [8.8/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 04:00:08,291 - Train: 0.88% [42700/4848000] [8.8/1000.0] [batch_t 0.329 (0.351)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 04:00:41,110 - Train: 0.88% [42800/4848000] [8.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 04:01:13,898 - Train: 0.88% [42900/4848000] [8.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 04:01:46,654 - Train: 0.89% [43000/4848000] [8.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 04:02:20,126 - Train: 0.89% [43100/4848000] [8.9/1000.0] [batch_t 0.326 (0.335)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 04:02:52,868 - Train: 0.89% [43200/4848000] [8.9/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 04:03:26,621 - Train: 0.89% [43300/4848000] [8.9/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 04:03:59,451 - Train: 0.90% [43400/4848000] [9.0/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 04:04:34,570 - Train: 0.90% [43500/4848000] [9.0/1000.0] [batch_t 0.332 (0.351)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 04:05:07,350 - Train: 0.90% [43600/4848000] [9.0/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 04:05:17,844 - ==> Total time: 4:39:16 Eta: 21 days, 8:30:59 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 04:05:41,593 - Train: 0.90% [43700/4848000] [9.0/1000.0] [batch_t 0.336 (0.331)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-03 04:06:15,045 - Train: 0.90% [43800/4848000] [9.0/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 04:06:48,001 - Train: 0.91% [43900/4848000] [9.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 04:07:20,757 - Train: 0.91% [44000/4848000] [9.1/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 04:07:53,555 - Train: 0.91% [44100/4848000] [9.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 04:08:30,396 - Train: 0.91% [44200/4848000] [9.1/1000.0] [batch_t 0.332 (0.368)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 04:09:03,186 - Train: 0.91% [44300/4848000] [9.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 04:09:37,522 - Train: 0.92% [44400/4848000] [9.2/1000.0] [batch_t 0.323 (0.343)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 04:10:12,216 - Train: 0.92% [44500/4848000] [9.2/1000.0] [batch_t 0.327 (0.347)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 04:10:45,008 - Train: 0.92% [44600/4848000] [9.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 04:11:19,413 - Train: 0.92% [44700/4848000] [9.2/1000.0] [batch_t 0.331 (0.344)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 04:11:52,132 - Train: 0.92% [44800/4848000] [9.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 04:12:26,938 - Train: 0.93% [44900/4848000] [9.3/1000.0] [batch_t 0.328 (0.348)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 04:12:59,704 - Train: 0.93% [45000/4848000] [9.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 04:13:32,576 - Train: 0.93% [45100/4848000] [9.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 04:14:05,919 - Train: 0.93% [45200/4848000] [9.3/1000.0] [batch_t 0.325 (0.333)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 04:14:38,695 - Train: 0.93% [45300/4848000] [9.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 04:15:12,203 - Train: 0.94% [45400/4848000] [9.4/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 04:15:44,942 - Train: 0.94% [45500/4848000] [9.4/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 04:16:18,945 - Train: 0.94% [45600/4848000] [9.4/1000.0] [batch_t 0.326 (0.340)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 04:16:51,672 - Train: 0.94% [45700/4848000] [9.4/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 04:17:25,689 - Train: 0.94% [45800/4848000] [9.4/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 04:17:58,577 - Train: 0.95% [45900/4848000] [9.5/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 04:18:31,441 - Train: 0.95% [46000/4848000] [9.5/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 04:19:04,237 - Train: 0.95% [46100/4848000] [9.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 04:19:40,802 - Train: 0.95% [46200/4848000] [9.5/1000.0] [batch_t 0.332 (0.366)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 04:20:14,405 - Train: 0.96% [46300/4848000] [9.6/1000.0] [batch_t 0.331 (0.336)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 04:20:47,252 - Train: 0.96% [46400/4848000] [9.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 04:21:20,000 - Train: 0.96% [46500/4848000] [9.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 04:21:52,781 - Train: 0.96% [46600/4848000] [9.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 04:22:26,773 - Train: 0.96% [46700/4848000] [9.6/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 04:22:59,587 - Train: 0.97% [46800/4848000] [9.7/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 04:23:33,703 - Train: 0.97% [46900/4848000] [9.7/1000.0] [batch_t 0.325 (0.341)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 04:24:09,541 - Train: 0.97% [47000/4848000] [9.7/1000.0] [batch_t 0.334 (0.358)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 04:24:42,367 - Train: 0.97% [47100/4848000] [9.7/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 04:25:18,735 - Train: 0.97% [47200/4848000] [9.7/1000.0] [batch_t 0.330 (0.364)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 04:25:51,502 - Train: 0.98% [47300/4848000] [9.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 04:26:27,163 - Train: 0.98% [47400/4848000] [9.8/1000.0] [batch_t 0.325 (0.357)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 04:26:59,939 - Train: 0.98% [47500/4848000] [9.8/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 04:27:35,518 - Train: 0.98% [47600/4848000] [9.8/1000.0] [batch_t 0.331 (0.356)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 04:28:10,403 - Train: 0.98% [47700/4848000] [9.8/1000.0] [batch_t 0.327 (0.349)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 04:28:43,215 - Train: 0.99% [47800/4848000] [9.9/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 04:29:17,796 - Train: 0.99% [47900/4848000] [9.9/1000.0] [batch_t 0.328 (0.346)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 04:29:50,546 - Train: 0.99% [48000/4848000] [9.9/1000.0] [batch_t 0.335 (0.327)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 04:30:28,223 - Train: 0.99% [48100/4848000] [9.9/1000.0] [batch_t 0.331 (0.377)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 04:31:01,022 - Train: 0.99% [48200/4848000] [9.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 04:31:35,685 - Train: 1.00% [48300/4848000] [10.0/1000.0] [batch_t 0.329 (0.347)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 04:32:11,748 - Train: 1.00% [48400/4848000] [10.0/1000.0] [batch_t 0.330 (0.361)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 04:32:38,025 - ==> Total time: 5:06:36 Eta: 21 days, 1:54:15 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 04:32:46,290 - Train: 1.00% [48500/4848000] [10.0/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 04:33:20,167 - Train: 1.00% [48600/4848000] [10.0/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 04:33:53,016 - Train: 1.00% [48700/4848000] [10.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 04:34:26,903 - Train: 1.01% [48800/4848000] [10.1/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 04:34:59,741 - Train: 1.01% [48900/4848000] [10.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 04:35:34,433 - Train: 1.01% [49000/4848000] [10.1/1000.0] [batch_t 0.330 (0.347)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 04:36:08,367 - Train: 1.01% [49100/4848000] [10.1/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 04:36:41,156 - Train: 1.01% [49200/4848000] [10.1/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 04:37:15,795 - Train: 1.02% [49300/4848000] [10.2/1000.0] [batch_t 0.328 (0.346)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 04:37:48,766 - Train: 1.02% [49400/4848000] [10.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 04:38:21,763 - Train: 1.02% [49500/4848000] [10.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 04:38:54,490 - Train: 1.02% [49600/4848000] [10.2/1000.0] [batch_t 0.334 (0.327)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 04:39:30,011 - Train: 1.03% [49700/4848000] [10.3/1000.0] [batch_t 0.329 (0.355)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 04:40:02,787 - Train: 1.03% [49800/4848000] [10.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 04:40:38,439 - Train: 1.03% [49900/4848000] [10.3/1000.0] [batch_t 0.325 (0.356)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 04:41:11,280 - Train: 1.03% [50000/4848000] [10.3/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 04:41:44,050 - Train: 1.03% [50100/4848000] [10.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 04:42:18,226 - Train: 1.04% [50200/4848000] [10.4/1000.0] [batch_t 0.330 (0.342)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 04:42:50,994 - Train: 1.04% [50300/4848000] [10.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 04:43:24,698 - Train: 1.04% [50400/4848000] [10.4/1000.0] [batch_t 0.333 (0.337)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 04:43:57,517 - Train: 1.04% [50500/4848000] [10.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 04:44:30,798 - Train: 1.04% [50600/4848000] [10.4/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 04:45:03,535 - Train: 1.05% [50700/4848000] [10.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 04:45:37,463 - Train: 1.05% [50800/4848000] [10.5/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 04:46:12,612 - Train: 1.05% [50900/4848000] [10.5/1000.0] [batch_t 0.329 (0.351)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 04:46:45,431 - Train: 1.05% [51000/4848000] [10.5/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 04:47:18,951 - Train: 1.05% [51100/4848000] [10.5/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 04:47:51,771 - Train: 1.06% [51200/4848000] [10.6/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 04:48:25,452 - Train: 1.06% [51300/4848000] [10.6/1000.0] [batch_t 0.334 (0.337)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 04:48:58,239 - Train: 1.06% [51400/4848000] [10.6/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 04:49:31,009 - Train: 1.06% [51500/4848000] [10.6/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 04:50:03,768 - Train: 1.06% [51600/4848000] [10.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 04:50:39,471 - Train: 1.07% [51700/4848000] [10.7/1000.0] [batch_t 0.329 (0.357)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 04:51:15,096 - Train: 1.07% [51800/4848000] [10.7/1000.0] [batch_t 0.325 (0.356)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 04:51:47,821 - Train: 1.07% [51900/4848000] [10.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 04:52:21,677 - Train: 1.07% [52000/4848000] [10.7/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 04:52:54,552 - Train: 1.07% [52100/4848000] [10.7/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 04:53:29,622 - Train: 1.08% [52200/4848000] [10.8/1000.0] [batch_t 0.330 (0.351)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 04:54:02,491 - Train: 1.08% [52300/4848000] [10.8/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 04:54:37,692 - Train: 1.08% [52400/4848000] [10.8/1000.0] [batch_t 0.333 (0.352)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 04:55:12,958 - Train: 1.08% [52500/4848000] [10.8/1000.0] [batch_t 0.331 (0.353)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 04:55:45,784 - Train: 1.08% [52600/4848000] [10.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 04:56:18,592 - Train: 1.09% [52700/4848000] [10.9/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 04:56:51,368 - Train: 1.09% [52800/4848000] [10.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 04:57:26,345 - Train: 1.09% [52900/4848000] [10.9/1000.0] [batch_t 0.329 (0.350)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 04:57:59,142 - Train: 1.09% [53000/4848000] [10.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 04:58:34,479 - Train: 1.10% [53100/4848000] [11.0/1000.0] [batch_t 0.326 (0.353)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 04:59:07,971 - Train: 1.10% [53200/4848000] [11.0/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 04:59:40,780 - Train: 1.10% [53300/4848000] [11.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 04:59:49,949 - ==> Total time: 5:33:48 Eta: 20 days, 20:12:19 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 05:00:16,676 - Train: 1.10% [53400/4848000] [11.0/1000.0] [batch_t 0.331 (0.348)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 05:00:49,457 - Train: 1.10% [53500/4848000] [11.0/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 05:01:24,194 - Train: 1.11% [53600/4848000] [11.1/1000.0] [batch_t 0.327 (0.347)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 05:01:56,976 - Train: 1.11% [53700/4848000] [11.1/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 05:02:31,799 - Train: 1.11% [53800/4848000] [11.1/1000.0] [batch_t 0.329 (0.348)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 05:03:04,574 - Train: 1.11% [53900/4848000] [11.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 05:03:38,259 - Train: 1.11% [54000/4848000] [11.1/1000.0] [batch_t 0.326 (0.337)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 05:04:11,423 - Train: 1.12% [54100/4848000] [11.2/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 05:04:44,223 - Train: 1.12% [54200/4848000] [11.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 05:05:16,958 - Train: 1.12% [54300/4848000] [11.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 05:05:49,721 - Train: 1.12% [54400/4848000] [11.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 05:06:23,140 - Train: 1.12% [54500/4848000] [11.2/1000.0] [batch_t 0.335 (0.334)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 05:06:55,908 - Train: 1.13% [54600/4848000] [11.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 05:07:29,797 - Train: 1.13% [54700/4848000] [11.3/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 05:08:02,584 - Train: 1.13% [54800/4848000] [11.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 05:08:36,355 - Train: 1.13% [54900/4848000] [11.3/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 05:09:11,795 - Train: 1.13% [55000/4848000] [11.3/1000.0] [batch_t 0.325 (0.354)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 05:09:44,667 - Train: 1.14% [55100/4848000] [11.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 05:10:18,907 - Train: 1.14% [55200/4848000] [11.4/1000.0] [batch_t 0.334 (0.342)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 05:10:51,689 - Train: 1.14% [55300/4848000] [11.4/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 05:11:26,939 - Train: 1.14% [55400/4848000] [11.4/1000.0] [batch_t 0.331 (0.352)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 05:11:59,719 - Train: 1.14% [55500/4848000] [11.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 05:12:34,132 - Train: 1.15% [55600/4848000] [11.5/1000.0] [batch_t 0.329 (0.344)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 05:13:08,795 - Train: 1.15% [55700/4848000] [11.5/1000.0] [batch_t 0.330 (0.347)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 05:13:41,618 - Train: 1.15% [55800/4848000] [11.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 05:14:14,387 - Train: 1.15% [55900/4848000] [11.5/1000.0] [batch_t 0.321 (0.328)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-03 05:14:47,189 - Train: 1.16% [56000/4848000] [11.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 05:15:20,747 - Train: 1.16% [56100/4848000] [11.6/1000.0] [batch_t 0.333 (0.335)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 05:15:53,508 - Train: 1.16% [56200/4848000] [11.6/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 05:16:27,021 - Train: 1.16% [56300/4848000] [11.6/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 05:16:59,761 - Train: 1.16% [56400/4848000] [11.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 05:17:33,927 - Train: 1.17% [56500/4848000] [11.7/1000.0] [batch_t 0.330 (0.342)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 05:18:09,054 - Train: 1.17% [56600/4848000] [11.7/1000.0] [batch_t 0.328 (0.351)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 05:18:41,957 - Train: 1.17% [56700/4848000] [11.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 05:19:15,699 - Train: 1.17% [56800/4848000] [11.7/1000.0] [batch_t 0.325 (0.337)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 05:19:48,621 - Train: 1.17% [56900/4848000] [11.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 05:20:23,924 - Train: 1.18% [57000/4848000] [11.8/1000.0] [batch_t 0.324 (0.353)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 05:20:56,793 - Train: 1.18% [57100/4848000] [11.8/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 05:21:30,317 - Train: 1.18% [57200/4848000] [11.8/1000.0] [batch_t 0.326 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 05:22:03,109 - Train: 1.18% [57300/4848000] [11.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 05:22:37,051 - Train: 1.18% [57400/4848000] [11.8/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 05:23:11,115 - Train: 1.19% [57500/4848000] [11.9/1000.0] [batch_t 0.323 (0.341)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 05:23:43,825 - Train: 1.19% [57600/4848000] [11.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 05:24:18,213 - Train: 1.19% [57700/4848000] [11.9/1000.0] [batch_t 0.326 (0.344)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 05:24:51,015 - Train: 1.19% [57800/4848000] [11.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 05:25:23,943 - Train: 1.19% [57900/4848000] [11.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 05:25:56,697 - Train: 1.20% [58000/4848000] [12.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 05:26:29,436 - Train: 1.20% [58100/4848000] [12.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 05:26:54,326 - ==> Total time: 6:00:52 Eta: 20 days, 15:12:29 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 05:27:04,248 - Train: 1.20% [58200/4848000] [12.0/1000.0] [batch_t 0.331 (0.344)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 05:27:37,006 - Train: 1.20% [58300/4848000] [12.0/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 05:28:09,828 - Train: 1.20% [58400/4848000] [12.0/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 05:28:42,594 - Train: 1.21% [58500/4848000] [12.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 05:29:16,730 - Train: 1.21% [58600/4848000] [12.1/1000.0] [batch_t 0.323 (0.341)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 05:29:49,458 - Train: 1.21% [58700/4848000] [12.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 05:30:22,195 - Train: 1.21% [58800/4848000] [12.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 05:30:55,073 - Train: 1.21% [58900/4848000] [12.1/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 05:31:27,935 - Train: 1.22% [59000/4848000] [12.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 05:32:00,812 - Train: 1.22% [59100/4848000] [12.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 05:32:33,721 - Train: 1.22% [59200/4848000] [12.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 05:33:06,817 - Train: 1.22% [59300/4848000] [12.2/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 05:33:39,590 - Train: 1.23% [59400/4848000] [12.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 05:34:12,804 - Train: 1.23% [59500/4848000] [12.3/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 05:34:45,561 - Train: 1.23% [59600/4848000] [12.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 05:35:21,710 - Train: 1.23% [59700/4848000] [12.3/1000.0] [batch_t 0.325 (0.361)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 05:35:54,519 - Train: 1.23% [59800/4848000] [12.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 05:36:29,978 - Train: 1.24% [59900/4848000] [12.4/1000.0] [batch_t 0.323 (0.354)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 05:37:02,746 - Train: 1.24% [60000/4848000] [12.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 05:37:36,245 - Train: 1.24% [60100/4848000] [12.4/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 05:38:08,991 - Train: 1.24% [60200/4848000] [12.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 05:38:41,798 - Train: 1.24% [60300/4848000] [12.4/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 05:39:17,504 - Train: 1.25% [60400/4848000] [12.5/1000.0] [batch_t 0.330 (0.357)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 05:39:50,264 - Train: 1.25% [60500/4848000] [12.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 05:40:25,438 - Train: 1.25% [60600/4848000] [12.5/1000.0] [batch_t 0.323 (0.352)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 05:40:58,336 - Train: 1.25% [60700/4848000] [12.5/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 05:41:33,813 - Train: 1.25% [60800/4848000] [12.5/1000.0] [batch_t 0.330 (0.355)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 05:42:07,143 - Train: 1.26% [60900/4848000] [12.6/1000.0] [batch_t 0.332 (0.333)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 05:42:39,945 - Train: 1.26% [61000/4848000] [12.6/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 05:43:16,192 - Train: 1.26% [61100/4848000] [12.6/1000.0] [batch_t 0.329 (0.362)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 05:43:48,970 - Train: 1.26% [61200/4848000] [12.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 05:44:24,355 - Train: 1.26% [61300/4848000] [12.6/1000.0] [batch_t 0.330 (0.354)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 05:44:57,249 - Train: 1.27% [61400/4848000] [12.7/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 05:45:32,626 - Train: 1.27% [61500/4848000] [12.7/1000.0] [batch_t 0.328 (0.354)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 05:46:06,442 - Train: 1.27% [61600/4848000] [12.7/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 05:46:39,278 - Train: 1.27% [61700/4848000] [12.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 05:47:12,155 - Train: 1.27% [61800/4848000] [12.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 05:47:44,988 - Train: 1.28% [61900/4848000] [12.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 05:48:18,719 - Train: 1.28% [62000/4848000] [12.8/1000.0] [batch_t 0.334 (0.337)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 05:48:51,668 - Train: 1.28% [62100/4848000] [12.8/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 05:49:24,423 - Train: 1.28% [62200/4848000] [12.8/1000.0] [batch_t 0.335 (0.327)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 05:49:57,218 - Train: 1.29% [62300/4848000] [12.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 05:50:32,670 - Train: 1.29% [62400/4848000] [12.9/1000.0] [batch_t 0.328 (0.354)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 05:51:05,505 - Train: 1.29% [62500/4848000] [12.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 05:51:38,280 - Train: 1.29% [62600/4848000] [12.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 05:52:12,090 - Train: 1.29% [62700/4848000] [12.9/1000.0] [batch_t 0.323 (0.338)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 05:52:44,934 - Train: 1.30% [62800/4848000] [13.0/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 05:53:18,586 - Train: 1.30% [62900/4848000] [13.0/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 05:53:51,343 - Train: 1.30% [63000/4848000] [13.0/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 05:53:59,220 - ==> Total time: 6:27:57 Eta: 20 days, 10:55:16 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 05:54:25,744 - Train: 1.30% [63100/4848000] [13.0/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 05:54:58,538 - Train: 1.30% [63200/4848000] [13.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 05:55:31,347 - Train: 1.31% [63300/4848000] [13.1/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 05:56:04,139 - Train: 1.31% [63400/4848000] [13.1/1000.0] [batch_t 0.336 (0.328)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-03 05:56:37,044 - Train: 1.31% [63500/4848000] [13.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 05:57:11,029 - Train: 1.31% [63600/4848000] [13.1/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 05:57:43,893 - Train: 1.31% [63700/4848000] [13.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 05:58:19,299 - Train: 1.32% [63800/4848000] [13.2/1000.0] [batch_t 0.328 (0.354)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 05:58:52,039 - Train: 1.32% [63900/4848000] [13.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 05:59:25,783 - Train: 1.32% [64000/4848000] [13.2/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 05:59:58,505 - Train: 1.32% [64100/4848000] [13.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 06:00:31,753 - Train: 1.32% [64200/4848000] [13.2/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 06:01:04,495 - Train: 1.33% [64300/4848000] [13.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 06:01:37,226 - Train: 1.33% [64400/4848000] [13.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 06:02:10,006 - Train: 1.33% [64500/4848000] [13.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 06:02:42,795 - Train: 1.33% [64600/4848000] [13.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 06:03:15,575 - Train: 1.33% [64700/4848000] [13.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 06:03:48,319 - Train: 1.34% [64800/4848000] [13.4/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 06:04:23,423 - Train: 1.34% [64900/4848000] [13.4/1000.0] [batch_t 0.327 (0.351)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 06:04:56,195 - Train: 1.34% [65000/4848000] [13.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 06:05:31,036 - Train: 1.34% [65100/4848000] [13.4/1000.0] [batch_t 0.330 (0.348)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 06:06:03,898 - Train: 1.34% [65200/4848000] [13.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 06:06:39,711 - Train: 1.35% [65300/4848000] [13.5/1000.0] [batch_t 0.329 (0.358)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 06:07:12,405 - Train: 1.35% [65400/4848000] [13.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 06:07:45,130 - Train: 1.35% [65500/4848000] [13.5/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 06:08:19,896 - Train: 1.35% [65600/4848000] [13.5/1000.0] [batch_t 0.327 (0.348)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 06:08:52,707 - Train: 1.36% [65700/4848000] [13.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 06:09:29,491 - Train: 1.36% [65800/4848000] [13.6/1000.0] [batch_t 0.326 (0.368)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 06:10:02,311 - Train: 1.36% [65900/4848000] [13.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 06:10:37,534 - Train: 1.36% [66000/4848000] [13.6/1000.0] [batch_t 0.330 (0.352)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 06:11:11,863 - Train: 1.36% [66100/4848000] [13.6/1000.0] [batch_t 0.327 (0.343)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 06:11:44,638 - Train: 1.37% [66200/4848000] [13.7/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 06:12:19,150 - Train: 1.37% [66300/4848000] [13.7/1000.0] [batch_t 0.327 (0.345)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 06:12:51,943 - Train: 1.37% [66400/4848000] [13.7/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 06:13:26,512 - Train: 1.37% [66500/4848000] [13.7/1000.0] [batch_t 0.328 (0.346)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 06:13:59,300 - Train: 1.37% [66600/4848000] [13.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 06:14:34,117 - Train: 1.38% [66700/4848000] [13.8/1000.0] [batch_t 0.329 (0.348)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 06:15:06,883 - Train: 1.38% [66800/4848000] [13.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 06:15:39,600 - Train: 1.38% [66900/4848000] [13.8/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 06:16:14,262 - Train: 1.38% [67000/4848000] [13.8/1000.0] [batch_t 0.329 (0.347)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 06:16:47,024 - Train: 1.38% [67100/4848000] [13.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 06:17:20,871 - Train: 1.39% [67200/4848000] [13.9/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 06:17:53,666 - Train: 1.39% [67300/4848000] [13.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 06:18:27,298 - Train: 1.39% [67400/4848000] [13.9/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 06:19:00,148 - Train: 1.39% [67500/4848000] [13.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 06:19:32,953 - Train: 1.39% [67600/4848000] [13.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 06:20:07,779 - Train: 1.40% [67700/4848000] [14.0/1000.0] [batch_t 0.323 (0.348)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 06:20:40,749 - Train: 1.40% [67800/4848000] [14.0/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 06:21:04,474 - ==> Total time: 6:55:02 Eta: 20 days, 7:11:20 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 06:21:15,843 - Train: 1.40% [67900/4848000] [14.0/1000.0] [batch_t 0.326 (0.335)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 06:21:48,637 - Train: 1.40% [68000/4848000] [14.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 06:22:22,386 - Train: 1.40% [68100/4848000] [14.0/1000.0] [batch_t 0.336 (0.337)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-03 06:22:55,250 - Train: 1.41% [68200/4848000] [14.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 06:23:28,084 - Train: 1.41% [68300/4848000] [14.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 06:24:01,024 - Train: 1.41% [68400/4848000] [14.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-03 06:24:33,947 - Train: 1.41% [68500/4848000] [14.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-03 06:25:07,310 - Train: 1.42% [68600/4848000] [14.2/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 06:25:40,193 - Train: 1.42% [68700/4848000] [14.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-03 06:26:13,052 - Train: 1.42% [68800/4848000] [14.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-03 06:26:45,897 - Train: 1.42% [68900/4848000] [14.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 06:27:19,564 - Train: 1.42% [69000/4848000] [14.2/1000.0] [batch_t 0.330 (0.337)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-03 06:27:52,600 - Train: 1.43% [69100/4848000] [14.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-03 06:28:26,183 - Train: 1.43% [69200/4848000] [14.3/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 06:28:58,942 - Train: 1.43% [69300/4848000] [14.3/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 06:29:31,747 - Train: 1.43% [69400/4848000] [14.3/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 06:30:04,454 - Train: 1.43% [69500/4848000] [14.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 06:30:39,762 - Train: 1.44% [69600/4848000] [14.4/1000.0] [batch_t 0.324 (0.353)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 06:31:13,713 - Train: 1.44% [69700/4848000] [14.4/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 06:31:46,462 - Train: 1.44% [69800/4848000] [14.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 06:32:20,379 - Train: 1.44% [69900/4848000] [14.4/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 06:32:53,128 - Train: 1.44% [70000/4848000] [14.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 06:33:28,486 - Train: 1.45% [70100/4848000] [14.5/1000.0] [batch_t 0.330 (0.353)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 06:34:01,183 - Train: 1.45% [70200/4848000] [14.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 06:34:33,933 - Train: 1.45% [70300/4848000] [14.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 06:35:07,197 - Train: 1.45% [70400/4848000] [14.5/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 06:35:40,086 - Train: 1.45% [70500/4848000] [14.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 06:36:13,580 - Train: 1.46% [70600/4848000] [14.6/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 06:36:46,349 - Train: 1.46% [70700/4848000] [14.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 06:37:21,954 - Train: 1.46% [70800/4848000] [14.6/1000.0] [batch_t 0.328 (0.356)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 06:37:54,782 - Train: 1.46% [70900/4848000] [14.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 06:38:27,640 - Train: 1.46% [71000/4848000] [14.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 06:39:00,418 - Train: 1.47% [71100/4848000] [14.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 06:39:35,952 - Train: 1.47% [71200/4848000] [14.7/1000.0] [batch_t 0.330 (0.355)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 06:40:09,047 - Train: 1.47% [71300/4848000] [14.7/1000.0] [batch_t 0.335 (0.331)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 06:40:41,849 - Train: 1.47% [71400/4848000] [14.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 06:41:14,682 - Train: 1.47% [71500/4848000] [14.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 06:41:47,421 - Train: 1.48% [71600/4848000] [14.8/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 06:42:20,216 - Train: 1.48% [71700/4848000] [14.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 06:42:52,999 - Train: 1.48% [71800/4848000] [14.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 06:43:26,599 - Train: 1.48% [71900/4848000] [14.8/1000.0] [batch_t 0.335 (0.336)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 06:43:59,473 - Train: 1.49% [72000/4848000] [14.9/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 06:44:34,860 - Train: 1.49% [72100/4848000] [14.9/1000.0] [batch_t 0.327 (0.354)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 06:45:08,550 - Train: 1.49% [72200/4848000] [14.9/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 06:45:41,354 - Train: 1.49% [72300/4848000] [14.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 06:46:16,826 - Train: 1.49% [72400/4848000] [14.9/1000.0] [batch_t 0.326 (0.355)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 06:46:49,575 - Train: 1.50% [72500/4848000] [15.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 06:47:26,070 - Train: 1.50% [72600/4848000] [15.0/1000.0] [batch_t 0.329 (0.365)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 06:47:58,766 - Train: 1.50% [72700/4848000] [15.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 06:48:07,757 - ==> Total time: 7:22:06 Eta: 20 days, 3:51:30 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 06:48:35,690 - Train: 1.50% [72800/4848000] [15.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 06:49:09,439 - Train: 1.50% [72900/4848000] [15.0/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 06:49:42,140 - Train: 1.51% [73000/4848000] [15.1/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 06:50:15,720 - Train: 1.51% [73100/4848000] [15.1/1000.0] [batch_t 0.331 (0.336)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 06:50:48,538 - Train: 1.51% [73200/4848000] [15.1/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 06:51:22,424 - Train: 1.51% [73300/4848000] [15.1/1000.0] [batch_t 0.325 (0.339)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 06:51:55,194 - Train: 1.51% [73400/4848000] [15.1/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 06:52:29,902 - Train: 1.52% [73500/4848000] [15.2/1000.0] [batch_t 0.328 (0.347)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 06:53:02,656 - Train: 1.52% [73600/4848000] [15.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 06:53:36,720 - Train: 1.52% [73700/4848000] [15.2/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 06:54:09,584 - Train: 1.52% [73800/4848000] [15.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 06:54:42,335 - Train: 1.52% [73900/4848000] [15.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 06:55:15,714 - Train: 1.53% [74000/4848000] [15.3/1000.0] [batch_t 0.333 (0.334)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 06:55:48,448 - Train: 1.53% [74100/4848000] [15.3/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 06:56:21,216 - Train: 1.53% [74200/4848000] [15.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 06:56:54,048 - Train: 1.53% [74300/4848000] [15.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 06:57:26,897 - Train: 1.53% [74400/4848000] [15.3/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 06:57:59,703 - Train: 1.54% [74500/4848000] [15.4/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 06:58:32,517 - Train: 1.54% [74600/4848000] [15.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 06:59:05,337 - Train: 1.54% [74700/4848000] [15.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 06:59:38,284 - Train: 1.54% [74800/4848000] [15.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 07:00:12,607 - Train: 1.54% [74900/4848000] [15.4/1000.0] [batch_t 0.328 (0.343)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 07:00:45,377 - Train: 1.55% [75000/4848000] [15.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 07:01:19,592 - Train: 1.55% [75100/4848000] [15.5/1000.0] [batch_t 0.330 (0.342)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 07:01:52,419 - Train: 1.55% [75200/4848000] [15.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 07:02:25,194 - Train: 1.55% [75300/4848000] [15.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 07:02:58,029 - Train: 1.56% [75400/4848000] [15.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 07:03:30,874 - Train: 1.56% [75500/4848000] [15.6/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 07:04:03,680 - Train: 1.56% [75600/4848000] [15.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 07:04:37,504 - Train: 1.56% [75700/4848000] [15.6/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 07:05:11,386 - Train: 1.56% [75800/4848000] [15.6/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 07:05:44,131 - Train: 1.57% [75900/4848000] [15.7/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 07:06:17,823 - Train: 1.57% [76000/4848000] [15.7/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 07:06:50,590 - Train: 1.57% [76100/4848000] [15.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 07:07:23,499 - Train: 1.57% [76200/4848000] [15.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 07:07:56,219 - Train: 1.57% [76300/4848000] [15.7/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 07:08:31,454 - Train: 1.58% [76400/4848000] [15.8/1000.0] [batch_t 0.328 (0.352)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 07:09:04,266 - Train: 1.58% [76500/4848000] [15.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 07:09:38,394 - Train: 1.58% [76600/4848000] [15.8/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 07:10:11,198 - Train: 1.58% [76700/4848000] [15.8/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 07:10:43,971 - Train: 1.58% [76800/4848000] [15.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 07:11:17,485 - Train: 1.59% [76900/4848000] [15.9/1000.0] [batch_t 0.332 (0.335)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 07:11:50,246 - Train: 1.59% [77000/4848000] [15.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 07:12:22,990 - Train: 1.59% [77100/4848000] [15.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 07:12:55,795 - Train: 1.59% [77200/4848000] [15.9/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 07:13:30,313 - Train: 1.59% [77300/4848000] [15.9/1000.0] [batch_t 0.328 (0.345)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 07:14:03,154 - Train: 1.60% [77400/4848000] [16.0/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 07:14:38,326 - Train: 1.60% [77500/4848000] [16.0/1000.0] [batch_t 0.328 (0.351)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 07:15:00,568 - ==> Total time: 7:48:59 Eta: 20 days, 0:42:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 07:15:16,839 - Train: 1.60% [77600/4848000] [16.0/1000.0] [batch_t 0.328 (0.459)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 07:15:49,621 - Train: 1.60% [77700/4848000] [16.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 07:16:22,441 - Train: 1.60% [77800/4848000] [16.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 07:16:55,226 - Train: 1.61% [77900/4848000] [16.1/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 07:17:28,696 - Train: 1.61% [78000/4848000] [16.1/1000.0] [batch_t 0.325 (0.335)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 07:18:01,446 - Train: 1.61% [78100/4848000] [16.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 07:18:34,275 - Train: 1.61% [78200/4848000] [16.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 07:19:07,044 - Train: 1.62% [78300/4848000] [16.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 07:19:39,793 - Train: 1.62% [78400/4848000] [16.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 07:20:12,608 - Train: 1.62% [78500/4848000] [16.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 07:20:45,362 - Train: 1.62% [78600/4848000] [16.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 07:21:18,085 - Train: 1.62% [78700/4848000] [16.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 07:21:50,810 - Train: 1.63% [78800/4848000] [16.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 07:22:23,543 - Train: 1.63% [78900/4848000] [16.3/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 07:22:56,445 - Train: 1.63% [79000/4848000] [16.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 07:23:29,205 - Train: 1.63% [79100/4848000] [16.3/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 07:24:01,973 - Train: 1.63% [79200/4848000] [16.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 07:24:36,250 - Train: 1.64% [79300/4848000] [16.4/1000.0] [batch_t 0.330 (0.343)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 07:25:09,078 - Train: 1.64% [79400/4848000] [16.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 07:25:41,859 - Train: 1.64% [79500/4848000] [16.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 07:26:14,581 - Train: 1.64% [79600/4848000] [16.4/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 07:26:47,295 - Train: 1.64% [79700/4848000] [16.4/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 07:27:20,123 - Train: 1.65% [79800/4848000] [16.5/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 07:27:52,897 - Train: 1.65% [79900/4848000] [16.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 07:28:26,422 - Train: 1.65% [80000/4848000] [16.5/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 07:28:59,170 - Train: 1.65% [80100/4848000] [16.5/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 07:29:35,328 - Train: 1.65% [80200/4848000] [16.5/1000.0] [batch_t 0.331 (0.361)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 07:30:08,730 - Train: 1.66% [80300/4848000] [16.6/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 07:30:41,648 - Train: 1.66% [80400/4848000] [16.6/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 07:31:16,151 - Train: 1.66% [80500/4848000] [16.6/1000.0] [batch_t 0.326 (0.345)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 07:31:48,964 - Train: 1.66% [80600/4848000] [16.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 07:32:24,277 - Train: 1.66% [80700/4848000] [16.6/1000.0] [batch_t 0.329 (0.353)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 07:32:57,097 - Train: 1.67% [80800/4848000] [16.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 07:33:29,864 - Train: 1.67% [80900/4848000] [16.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 07:34:02,714 - Train: 1.67% [81000/4848000] [16.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 07:34:35,589 - Train: 1.67% [81100/4848000] [16.7/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 07:35:08,359 - Train: 1.67% [81200/4848000] [16.7/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 07:35:41,154 - Train: 1.68% [81300/4848000] [16.8/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 07:36:13,921 - Train: 1.68% [81400/4848000] [16.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 07:36:46,607 - Train: 1.68% [81500/4848000] [16.8/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 07:37:19,367 - Train: 1.68% [81600/4848000] [16.8/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 07:37:52,094 - Train: 1.69% [81700/4848000] [16.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 07:38:26,059 - Train: 1.69% [81800/4848000] [16.9/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 07:38:58,845 - Train: 1.69% [81900/4848000] [16.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 07:39:34,357 - Train: 1.69% [82000/4848000] [16.9/1000.0] [batch_t 0.327 (0.355)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 07:40:07,139 - Train: 1.69% [82100/4848000] [16.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 07:40:39,922 - Train: 1.70% [82200/4848000] [17.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 07:41:12,740 - Train: 1.70% [82300/4848000] [17.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 07:41:45,506 - Train: 1.70% [82400/4848000] [17.0/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 07:41:50,769 - ==> Total time: 8:15:49 Eta: 19 days, 21:50:07 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 07:42:19,899 - Train: 1.70% [82500/4848000] [17.0/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 07:42:52,754 - Train: 1.70% [82600/4848000] [17.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 07:43:26,139 - Train: 1.71% [82700/4848000] [17.1/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 07:43:58,971 - Train: 1.71% [82800/4848000] [17.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 07:44:33,729 - Train: 1.71% [82900/4848000] [17.1/1000.0] [batch_t 0.329 (0.347)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 07:45:07,611 - Train: 1.71% [83000/4848000] [17.1/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 07:45:40,524 - Train: 1.71% [83100/4848000] [17.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 07:46:14,606 - Train: 1.72% [83200/4848000] [17.2/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 07:46:47,502 - Train: 1.72% [83300/4848000] [17.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 07:47:20,407 - Train: 1.72% [83400/4848000] [17.2/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 07:47:53,277 - Train: 1.72% [83500/4848000] [17.2/1000.0] [batch_t 0.338 (0.329)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-03 07:48:28,258 - Train: 1.72% [83600/4848000] [17.2/1000.0] [batch_t 0.330 (0.350)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 07:49:01,154 - Train: 1.73% [83700/4848000] [17.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 07:49:36,533 - Train: 1.73% [83800/4848000] [17.3/1000.0] [batch_t 0.328 (0.354)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 07:50:09,290 - Train: 1.73% [83900/4848000] [17.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 07:50:41,998 - Train: 1.73% [84000/4848000] [17.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 07:51:15,982 - Train: 1.73% [84100/4848000] [17.3/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 07:51:48,733 - Train: 1.74% [84200/4848000] [17.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 07:52:21,499 - Train: 1.74% [84300/4848000] [17.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 07:52:54,269 - Train: 1.74% [84400/4848000] [17.4/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 07:53:29,680 - Train: 1.74% [84500/4848000] [17.4/1000.0] [batch_t 0.324 (0.354)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 07:54:02,521 - Train: 1.75% [84600/4848000] [17.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 07:54:35,278 - Train: 1.75% [84700/4848000] [17.5/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 07:55:08,071 - Train: 1.75% [84800/4848000] [17.5/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 07:55:40,828 - Train: 1.75% [84900/4848000] [17.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 07:56:15,019 - Train: 1.75% [85000/4848000] [17.5/1000.0] [batch_t 0.325 (0.342)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 07:56:47,859 - Train: 1.76% [85100/4848000] [17.6/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 07:57:20,696 - Train: 1.76% [85200/4848000] [17.6/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 07:57:53,642 - Train: 1.76% [85300/4848000] [17.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 07:58:26,750 - Train: 1.76% [85400/4848000] [17.6/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 07:58:59,591 - Train: 1.76% [85500/4848000] [17.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 07:59:34,292 - Train: 1.77% [85600/4848000] [17.7/1000.0] [batch_t 0.331 (0.347)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 08:00:09,240 - Train: 1.77% [85700/4848000] [17.7/1000.0] [batch_t 0.329 (0.349)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:00:50,234 - Train: 1.77% [85800/4848000] [17.7/1000.0] [batch_t 0.328 (0.410)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 08:01:25,447 - Train: 1.77% [85900/4848000] [17.7/1000.0] [batch_t 0.328 (0.352)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 08:01:58,205 - Train: 1.77% [86000/4848000] [17.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:02:34,649 - Train: 1.78% [86100/4848000] [17.8/1000.0] [batch_t 0.329 (0.364)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:03:13,050 - Train: 1.78% [86200/4848000] [17.8/1000.0] [batch_t 0.337 (0.384)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-03 08:03:45,868 - Train: 1.78% [86300/4848000] [17.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 08:04:19,645 - Train: 1.78% [86400/4848000] [17.8/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 08:04:52,361 - Train: 1.78% [86500/4848000] [17.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:05:26,180 - Train: 1.79% [86600/4848000] [17.9/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:05:58,935 - Train: 1.79% [86700/4848000] [17.9/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 08:06:33,289 - Train: 1.79% [86800/4848000] [17.9/1000.0] [batch_t 0.327 (0.343)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 08:07:07,715 - Train: 1.79% [86900/4848000] [17.9/1000.0] [batch_t 0.329 (0.344)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:07:40,442 - Train: 1.79% [87000/4848000] [17.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 08:08:18,027 - Train: 1.80% [87100/4848000] [18.0/1000.0] [batch_t 0.333 (0.376)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 08:08:50,753 - Train: 1.80% [87200/4848000] [18.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 08:09:12,446 - ==> Total time: 8:43:10 Eta: 19 days, 19:42:30 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 08:09:26,387 - Train: 1.80% [87300/4848000] [18.0/1000.0] [batch_t 0.330 (0.347)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 08:09:59,300 - Train: 1.80% [87400/4848000] [18.0/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 08:10:34,292 - Train: 1.80% [87500/4848000] [18.0/1000.0] [batch_t 0.329 (0.350)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:11:07,013 - Train: 1.81% [87600/4848000] [18.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 08:11:39,755 - Train: 1.81% [87700/4848000] [18.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 08:12:12,451 - Train: 1.81% [87800/4848000] [18.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:12:45,152 - Train: 1.81% [87900/4848000] [18.1/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-03 08:13:20,326 - Train: 1.82% [88000/4848000] [18.2/1000.0] [batch_t 0.329 (0.352)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:13:53,006 - Train: 1.82% [88100/4848000] [18.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 08:14:26,391 - Train: 1.82% [88200/4848000] [18.2/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:14:59,136 - Train: 1.82% [88300/4848000] [18.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 08:15:34,023 - Train: 1.82% [88400/4848000] [18.2/1000.0] [batch_t 0.330 (0.349)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 08:16:06,740 - Train: 1.83% [88500/4848000] [18.3/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 08:16:39,469 - Train: 1.83% [88600/4848000] [18.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:17:12,283 - Train: 1.83% [88700/4848000] [18.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:17:45,028 - Train: 1.83% [88800/4848000] [18.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:18:18,701 - Train: 1.83% [88900/4848000] [18.3/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 08:18:51,399 - Train: 1.84% [89000/4848000] [18.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 08:19:25,146 - Train: 1.84% [89100/4848000] [18.4/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:19:57,913 - Train: 1.84% [89200/4848000] [18.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:20:32,517 - Train: 1.84% [89300/4848000] [18.4/1000.0] [batch_t 0.329 (0.346)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:21:05,867 - Train: 1.84% [89400/4848000] [18.4/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 08:21:38,714 - Train: 1.85% [89500/4848000] [18.5/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 08:22:13,840 - Train: 1.85% [89600/4848000] [18.5/1000.0] [batch_t 0.329 (0.351)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 08:22:46,530 - Train: 1.85% [89700/4848000] [18.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:23:19,242 - Train: 1.85% [89800/4848000] [18.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 08:23:51,995 - Train: 1.85% [89900/4848000] [18.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 08:24:24,757 - Train: 1.86% [90000/4848000] [18.6/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 08:24:57,491 - Train: 1.86% [90100/4848000] [18.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 08:25:31,271 - Train: 1.86% [90200/4848000] [18.6/1000.0] [batch_t 0.332 (0.338)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 08:26:04,112 - Train: 1.86% [90300/4848000] [18.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 08:26:36,963 - Train: 1.86% [90400/4848000] [18.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:27:10,336 - Train: 1.87% [90500/4848000] [18.7/1000.0] [batch_t 0.325 (0.334)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 08:27:43,095 - Train: 1.87% [90600/4848000] [18.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:28:16,813 - Train: 1.87% [90700/4848000] [18.7/1000.0] [batch_t 0.335 (0.337)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 08:28:49,565 - Train: 1.87% [90800/4848000] [18.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 08:29:22,752 - Train: 1.88% [90900/4848000] [18.8/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 08:29:55,516 - Train: 1.88% [91000/4848000] [18.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 08:30:28,613 - Train: 1.88% [91100/4848000] [18.8/1000.0] [batch_t 0.325 (0.331)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 08:31:01,432 - Train: 1.88% [91200/4848000] [18.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:31:34,461 - Train: 1.88% [91300/4848000] [18.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:32:09,505 - Train: 1.89% [91400/4848000] [18.9/1000.0] [batch_t 0.326 (0.350)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 08:32:42,285 - Train: 1.89% [91500/4848000] [18.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:33:16,263 - Train: 1.89% [91600/4848000] [18.9/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 08:33:49,052 - Train: 1.89% [91700/4848000] [18.9/1000.0] [batch_t 0.336 (0.328)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-03 08:34:22,874 - Train: 1.89% [91800/4848000] [18.9/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:34:55,686 - Train: 1.90% [91900/4848000] [19.0/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 08:35:29,435 - Train: 1.90% [92000/4848000] [19.0/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 08:36:02,145 - Train: 1.90% [92100/4848000] [19.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 08:36:09,436 - ==> Total time: 9:10:07 Eta: 19 days, 17:24:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 08:36:40,263 - Train: 1.90% [92200/4848000] [19.0/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 08:37:15,798 - Train: 1.90% [92300/4848000] [19.0/1000.0] [batch_t 0.325 (0.355)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 08:37:48,565 - Train: 1.91% [92400/4848000] [19.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 08:38:21,293 - Train: 1.91% [92500/4848000] [19.1/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 08:38:54,071 - Train: 1.91% [92600/4848000] [19.1/1000.0] [batch_t 0.336 (0.328)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-03 08:39:27,680 - Train: 1.91% [92700/4848000] [19.1/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 08:40:00,509 - Train: 1.91% [92800/4848000] [19.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 08:40:33,324 - Train: 1.92% [92900/4848000] [19.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 08:41:07,148 - Train: 1.92% [93000/4848000] [19.2/1000.0] [batch_t 0.324 (0.338)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 08:41:40,048 - Train: 1.92% [93100/4848000] [19.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:42:13,969 - Train: 1.92% [93200/4848000] [19.2/1000.0] [batch_t 0.324 (0.339)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 08:42:46,647 - Train: 1.92% [93300/4848000] [19.2/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 08:43:21,339 - Train: 1.93% [93400/4848000] [19.3/1000.0] [batch_t 0.329 (0.347)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:43:54,107 - Train: 1.93% [93500/4848000] [19.3/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 08:44:28,913 - Train: 1.93% [93600/4848000] [19.3/1000.0] [batch_t 0.323 (0.348)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 08:45:01,657 - Train: 1.93% [93700/4848000] [19.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 08:45:36,401 - Train: 1.93% [93800/4848000] [19.3/1000.0] [batch_t 0.327 (0.347)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 08:46:10,678 - Train: 1.94% [93900/4848000] [19.4/1000.0] [batch_t 0.327 (0.343)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 08:46:43,504 - Train: 1.94% [94000/4848000] [19.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 08:47:16,370 - Train: 1.94% [94100/4848000] [19.4/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 08:47:49,156 - Train: 1.94% [94200/4848000] [19.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:48:24,469 - Train: 1.95% [94300/4848000] [19.5/1000.0] [batch_t 0.328 (0.353)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 08:48:57,180 - Train: 1.95% [94400/4848000] [19.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:49:30,032 - Train: 1.95% [94500/4848000] [19.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:50:02,826 - Train: 1.95% [94600/4848000] [19.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:50:35,658 - Train: 1.95% [94700/4848000] [19.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 08:51:08,920 - Train: 1.96% [94800/4848000] [19.6/1000.0] [batch_t 0.332 (0.333)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 08:51:41,602 - Train: 1.96% [94900/4848000] [19.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:52:14,877 - Train: 1.96% [95000/4848000] [19.6/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 08:52:47,588 - Train: 1.96% [95100/4848000] [19.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:53:21,866 - Train: 1.96% [95200/4848000] [19.6/1000.0] [batch_t 0.329 (0.343)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:53:54,764 - Train: 1.97% [95300/4848000] [19.7/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-03 08:54:28,757 - Train: 1.97% [95400/4848000] [19.7/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:55:01,485 - Train: 1.97% [95500/4848000] [19.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 08:55:35,457 - Train: 1.97% [95600/4848000] [19.7/1000.0] [batch_t 0.325 (0.340)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 08:56:08,187 - Train: 1.97% [95700/4848000] [19.7/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 08:56:40,991 - Train: 1.98% [95800/4848000] [19.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 08:57:16,608 - Train: 1.98% [95900/4848000] [19.8/1000.0] [batch_t 0.329 (0.356)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 08:57:49,584 - Train: 1.98% [96000/4848000] [19.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:58:23,490 - Train: 1.98% [96100/4848000] [19.8/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 08:58:56,303 - Train: 1.98% [96200/4848000] [19.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 08:59:31,026 - Train: 1.99% [96300/4848000] [19.9/1000.0] [batch_t 0.327 (0.347)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 09:00:03,868 - Train: 1.99% [96400/4848000] [19.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 09:00:38,010 - Train: 1.99% [96500/4848000] [19.9/1000.0] [batch_t 0.326 (0.341)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 09:01:10,905 - Train: 1.99% [96600/4848000] [19.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 09:01:43,617 - Train: 1.99% [96700/4848000] [19.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 09:02:16,883 - Train: 2.00% [96800/4848000] [20.0/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 09:02:49,578 - Train: 2.00% [96900/4848000] [20.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 09:03:09,279 - ==> Total time: 9:37:07 Eta: 19 days, 15:19:21 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 09:03:24,118 - Train: 2.00% [97000/4848000] [20.0/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 09:03:56,902 - Train: 2.00% [97100/4848000] [20.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 09:04:29,727 - Train: 2.00% [97200/4848000] [20.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 09:05:02,598 - Train: 2.01% [97300/4848000] [20.1/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 09:05:36,427 - Train: 2.01% [97400/4848000] [20.1/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 09:06:09,195 - Train: 2.01% [97500/4848000] [20.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 09:06:41,970 - Train: 2.01% [97600/4848000] [20.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 09:07:16,493 - Train: 2.02% [97700/4848000] [20.2/1000.0] [batch_t 0.327 (0.345)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 09:07:49,251 - Train: 2.02% [97800/4848000] [20.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 09:08:22,040 - Train: 2.02% [97900/4848000] [20.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 09:08:54,752 - Train: 2.02% [98000/4848000] [20.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:09:30,559 - Train: 2.02% [98100/4848000] [20.2/1000.0] [batch_t 0.327 (0.358)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 09:10:03,317 - Train: 2.03% [98200/4848000] [20.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 09:10:36,976 - Train: 2.03% [98300/4848000] [20.3/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 09:11:11,002 - Train: 2.03% [98400/4848000] [20.3/1000.0] [batch_t 0.330 (0.340)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:11:43,686 - Train: 2.03% [98500/4848000] [20.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 09:12:18,309 - Train: 2.03% [98600/4848000] [20.3/1000.0] [batch_t 0.323 (0.346)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 09:12:50,990 - Train: 2.04% [98700/4848000] [20.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 09:13:24,888 - Train: 2.04% [98800/4848000] [20.4/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 09:13:57,657 - Train: 2.04% [98900/4848000] [20.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:14:32,146 - Train: 2.04% [99000/4848000] [20.4/1000.0] [batch_t 0.325 (0.345)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 09:15:04,895 - Train: 2.04% [99100/4848000] [20.4/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 09:15:38,771 - Train: 2.05% [99200/4848000] [20.5/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 09:16:11,451 - Train: 2.05% [99300/4848000] [20.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 09:16:44,150 - Train: 2.05% [99400/4848000] [20.5/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 09:17:19,414 - Train: 2.05% [99500/4848000] [20.5/1000.0] [batch_t 0.327 (0.353)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 09:17:52,146 - Train: 2.05% [99600/4848000] [20.5/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 09:18:24,927 - Train: 2.06% [99700/4848000] [20.6/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 09:18:57,631 - Train: 2.06% [99800/4848000] [20.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 09:19:31,606 - Train: 2.06% [99900/4848000] [20.6/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 09:20:04,314 - Train: 2.06% [100000/4848000] [20.6/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 09:20:38,157 - Train: 2.06% [100100/4848000] [20.6/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 09:21:12,010 - Train: 2.07% [100200/4848000] [20.7/1000.0] [batch_t 0.332 (0.338)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 09:21:44,787 - Train: 2.07% [100300/4848000] [20.7/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 09:22:18,255 - Train: 2.07% [100400/4848000] [20.7/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 09:22:50,959 - Train: 2.07% [100500/4848000] [20.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 09:23:24,348 - Train: 2.08% [100600/4848000] [20.8/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 09:23:57,130 - Train: 2.08% [100700/4848000] [20.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 09:24:32,635 - Train: 2.08% [100800/4848000] [20.8/1000.0] [batch_t 0.334 (0.355)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 09:25:05,372 - Train: 2.08% [100900/4848000] [20.8/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:25:38,054 - Train: 2.08% [101000/4848000] [20.8/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 09:26:10,710 - Train: 2.09% [101100/4848000] [20.9/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 09:26:43,408 - Train: 2.09% [101200/4848000] [20.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 09:27:18,738 - Train: 2.09% [101300/4848000] [20.9/1000.0] [batch_t 0.327 (0.353)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 09:27:51,530 - Train: 2.09% [101400/4848000] [20.9/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 09:28:24,658 - Train: 2.09% [101500/4848000] [20.9/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 09:28:57,344 - Train: 2.10% [101600/4848000] [21.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 09:29:32,424 - Train: 2.10% [101700/4848000] [21.0/1000.0] [batch_t 0.329 (0.351)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:30:05,157 - Train: 2.10% [101800/4848000] [21.0/1000.0] [batch_t 0.334 (0.327)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 09:30:07,771 - ==> Total time: 10:04:06 Eta: 19 days, 13:22:46 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 09:30:39,576 - Train: 2.10% [101900/4848000] [21.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:31:13,394 - Train: 2.10% [102000/4848000] [21.0/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 09:31:46,184 - Train: 2.11% [102100/4848000] [21.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 09:32:19,524 - Train: 2.11% [102200/4848000] [21.1/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 09:32:52,216 - Train: 2.11% [102300/4848000] [21.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 09:33:25,259 - Train: 2.11% [102400/4848000] [21.1/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 09:33:57,946 - Train: 2.11% [102500/4848000] [21.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:34:31,314 - Train: 2.12% [102600/4848000] [21.2/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:35:04,019 - Train: 2.12% [102700/4848000] [21.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 09:35:38,894 - Train: 2.12% [102800/4848000] [21.2/1000.0] [batch_t 0.322 (0.349)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 09:36:11,602 - Train: 2.12% [102900/4848000] [21.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 09:36:44,428 - Train: 2.12% [103000/4848000] [21.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 09:37:18,577 - Train: 2.13% [103100/4848000] [21.3/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 09:37:51,286 - Train: 2.13% [103200/4848000] [21.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 09:38:25,420 - Train: 2.13% [103300/4848000] [21.3/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 09:38:58,154 - Train: 2.13% [103400/4848000] [21.3/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 09:39:32,036 - Train: 2.13% [103500/4848000] [21.3/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 09:40:04,721 - Train: 2.14% [103600/4848000] [21.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:40:37,629 - Train: 2.14% [103700/4848000] [21.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 09:41:11,489 - Train: 2.14% [103800/4848000] [21.4/1000.0] [batch_t 0.331 (0.339)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 09:41:44,236 - Train: 2.14% [103900/4848000] [21.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:42:19,620 - Train: 2.15% [104000/4848000] [21.5/1000.0] [batch_t 0.329 (0.354)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 09:42:52,326 - Train: 2.15% [104100/4848000] [21.5/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 09:43:26,079 - Train: 2.15% [104200/4848000] [21.5/1000.0] [batch_t 0.325 (0.337)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 09:43:58,740 - Train: 2.15% [104300/4848000] [21.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 09:44:32,776 - Train: 2.15% [104400/4848000] [21.5/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 09:45:07,450 - Train: 2.16% [104500/4848000] [21.6/1000.0] [batch_t 0.328 (0.347)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 09:45:40,361 - Train: 2.16% [104600/4848000] [21.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:46:15,602 - Train: 2.16% [104700/4848000] [21.6/1000.0] [batch_t 0.323 (0.352)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 09:46:48,286 - Train: 2.16% [104800/4848000] [21.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 09:47:21,938 - Train: 2.16% [104900/4848000] [21.6/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 09:47:54,759 - Train: 2.17% [105000/4848000] [21.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 09:48:28,255 - Train: 2.17% [105100/4848000] [21.7/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:49:00,964 - Train: 2.17% [105200/4848000] [21.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 09:49:34,454 - Train: 2.17% [105300/4848000] [21.7/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 09:50:10,306 - Train: 2.17% [105400/4848000] [21.7/1000.0] [batch_t 0.325 (0.358)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 09:50:43,065 - Train: 2.18% [105500/4848000] [21.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 09:51:16,422 - Train: 2.18% [105600/4848000] [21.8/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 09:51:49,174 - Train: 2.18% [105700/4848000] [21.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 09:52:21,840 - Train: 2.18% [105800/4848000] [21.8/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 09:52:54,645 - Train: 2.18% [105900/4848000] [21.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:53:27,480 - Train: 2.19% [106000/4848000] [21.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 09:54:00,184 - Train: 2.19% [106100/4848000] [21.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 09:54:32,931 - Train: 2.19% [106200/4848000] [21.9/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:55:07,081 - Train: 2.19% [106300/4848000] [21.9/1000.0] [batch_t 0.326 (0.341)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 09:55:39,825 - Train: 2.19% [106400/4848000] [21.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 09:56:15,672 - Train: 2.20% [106500/4848000] [22.0/1000.0] [batch_t 0.327 (0.358)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 09:56:48,416 - Train: 2.20% [106600/4848000] [22.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 09:57:08,032 - ==> Total time: 10:31:06 Eta: 19 days, 11:35:39 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 09:57:23,861 - Train: 2.20% [106700/4848000] [22.0/1000.0] [batch_t 0.332 (0.333)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 09:57:56,605 - Train: 2.20% [106800/4848000] [22.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 09:58:29,640 - Train: 2.21% [106900/4848000] [22.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 09:59:02,327 - Train: 2.21% [107000/4848000] [22.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 09:59:36,576 - Train: 2.21% [107100/4848000] [22.1/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:00:10,752 - Train: 2.21% [107200/4848000] [22.1/1000.0] [batch_t 0.326 (0.342)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 10:00:43,458 - Train: 2.21% [107300/4848000] [22.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 10:01:16,174 - Train: 2.22% [107400/4848000] [22.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:01:48,895 - Train: 2.22% [107500/4848000] [22.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:02:22,575 - Train: 2.22% [107600/4848000] [22.2/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 10:02:55,212 - Train: 2.22% [107700/4848000] [22.2/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 10:03:29,439 - Train: 2.22% [107800/4848000] [22.2/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:04:02,141 - Train: 2.23% [107900/4848000] [22.3/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 10:04:38,068 - Train: 2.23% [108000/4848000] [22.3/1000.0] [batch_t 0.330 (0.359)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 10:05:13,377 - Train: 2.23% [108100/4848000] [22.3/1000.0] [batch_t 0.327 (0.353)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 10:05:46,105 - Train: 2.23% [108200/4848000] [22.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:06:21,915 - Train: 2.23% [108300/4848000] [22.3/1000.0] [batch_t 0.328 (0.358)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:06:54,607 - Train: 2.24% [108400/4848000] [22.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 10:07:30,575 - Train: 2.24% [108500/4848000] [22.4/1000.0] [batch_t 0.328 (0.360)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 10:08:03,428 - Train: 2.24% [108600/4848000] [22.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 10:08:42,976 - Train: 2.24% [108700/4848000] [22.4/1000.0] [batch_t 0.329 (0.395)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:09:17,917 - Train: 2.24% [108800/4848000] [22.4/1000.0] [batch_t 0.331 (0.349)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 10:09:50,619 - Train: 2.25% [108900/4848000] [22.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:10:24,537 - Train: 2.25% [109000/4848000] [22.5/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:10:57,259 - Train: 2.25% [109100/4848000] [22.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 10:11:32,579 - Train: 2.25% [109200/4848000] [22.5/1000.0] [batch_t 0.329 (0.353)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:12:08,983 - Train: 2.25% [109300/4848000] [22.5/1000.0] [batch_t 0.329 (0.364)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:12:44,515 - Train: 2.26% [109400/4848000] [22.6/1000.0] [batch_t 0.327 (0.355)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 10:13:22,314 - Train: 2.26% [109500/4848000] [22.6/1000.0] [batch_t 0.329 (0.378)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:13:55,471 - Train: 2.26% [109600/4848000] [22.6/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 10:14:33,211 - Train: 2.26% [109700/4848000] [22.6/1000.0] [batch_t 0.328 (0.377)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 10:15:37,866 - Train: 2.26% [109800/4848000] [22.6/1000.0] [batch_t 0.330 (0.646)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 10:16:23,091 - Train: 2.27% [109900/4848000] [22.7/1000.0] [batch_t 0.329 (0.452)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:16:59,266 - Train: 2.27% [110000/4848000] [22.7/1000.0] [batch_t 0.531 (0.362)] [data_t 0.208] [optim_t 0.324] [lr 0.005000] 2024-04-03 10:17:54,306 - Train: 2.27% [110100/4848000] [22.7/1000.0] [batch_t 0.328 (0.550)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 10:18:41,020 - Train: 2.27% [110200/4848000] [22.7/1000.0] [batch_t 0.330 (0.467)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 10:19:19,465 - Train: 2.28% [110300/4848000] [22.8/1000.0] [batch_t 0.330 (0.384)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 10:20:43,238 - Train: 2.28% [110400/4848000] [22.8/1000.0] [batch_t 0.328 (0.838)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 10:21:33,888 - Train: 2.28% [110500/4848000] [22.8/1000.0] [batch_t 0.329 (0.506)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:22:44,199 - Train: 2.28% [110600/4848000] [22.8/1000.0] [batch_t 0.329 (0.703)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:23:47,739 - Train: 2.28% [110700/4848000] [22.8/1000.0] [batch_t 0.792 (0.635)] [data_t 0.466] [optim_t 0.326] [lr 0.005000] 2024-04-03 10:25:09,635 - Train: 2.29% [110800/4848000] [22.9/1000.0] [batch_t 0.778 (0.819)] [data_t 0.449] [optim_t 0.329] [lr 0.005000] 2024-04-03 10:27:44,046 - Train: 2.29% [110900/4848000] [22.9/1000.0] [batch_t 2.352 (1.544)] [data_t 2.027] [optim_t 0.325] [lr 0.005000] 2024-04-03 10:29:49,404 - Train: 2.29% [111000/4848000] [22.9/1000.0] [batch_t 0.326 (1.253)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 10:30:52,547 - Train: 2.29% [111100/4848000] [22.9/1000.0] [batch_t 0.332 (0.631)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 10:31:56,577 - Train: 2.29% [111200/4848000] [22.9/1000.0] [batch_t 0.324 (0.640)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 10:33:27,566 - Train: 2.30% [111300/4848000] [23.0/1000.0] [batch_t 0.329 (0.910)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:34:53,972 - Train: 2.30% [111400/4848000] [23.0/1000.0] [batch_t 0.937 (0.864)] [data_t 0.613] [optim_t 0.324] [lr 0.005000] 2024-04-03 10:36:14,000 - Train: 2.30% [111500/4848000] [23.0/1000.0] [batch_t 0.329 (0.800)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:36:15,308 - ==> Total time: 11:10:13 Eta: 19 days, 18:30:12 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 10:36:54,314 - Train: 2.30% [111600/4848000] [23.0/1000.0] [batch_t 0.329 (0.384)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:37:31,414 - Train: 2.30% [111700/4848000] [23.0/1000.0] [batch_t 0.329 (0.371)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:38:04,149 - Train: 2.31% [111800/4848000] [23.1/1000.0] [batch_t 0.334 (0.327)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 10:38:39,832 - Train: 2.31% [111900/4848000] [23.1/1000.0] [batch_t 0.327 (0.357)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 10:39:14,678 - Train: 2.31% [112000/4848000] [23.1/1000.0] [batch_t 0.327 (0.348)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 10:39:47,337 - Train: 2.31% [112100/4848000] [23.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 10:40:32,300 - Train: 2.31% [112200/4848000] [23.1/1000.0] [batch_t 0.347 (0.450)] [data_t 0.020] [optim_t 0.328] [lr 0.005000] 2024-04-03 10:41:09,745 - Train: 2.32% [112300/4848000] [23.2/1000.0] [batch_t 0.327 (0.374)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 10:41:45,719 - Train: 2.32% [112400/4848000] [23.2/1000.0] [batch_t 0.325 (0.359)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 10:42:22,371 - Train: 2.32% [112500/4848000] [23.2/1000.0] [batch_t 0.330 (0.366)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 10:42:57,870 - Train: 2.32% [112600/4848000] [23.2/1000.0] [batch_t 0.328 (0.355)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 10:43:51,496 - Train: 2.32% [112700/4848000] [23.2/1000.0] [batch_t 0.329 (0.536)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:44:29,490 - Train: 2.33% [112800/4848000] [23.3/1000.0] [batch_t 0.327 (0.380)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 10:45:02,274 - Train: 2.33% [112900/4848000] [23.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 10:45:38,755 - Train: 2.33% [113000/4848000] [23.3/1000.0] [batch_t 0.329 (0.365)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:46:20,730 - Train: 2.33% [113100/4848000] [23.3/1000.0] [batch_t 0.328 (0.420)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 10:47:38,183 - Train: 2.33% [113200/4848000] [23.3/1000.0] [batch_t 0.468 (0.774)] [data_t 0.141] [optim_t 0.326] [lr 0.005000] 2024-04-03 10:49:01,234 - Train: 2.34% [113300/4848000] [23.4/1000.0] [batch_t 2.118 (0.830)] [data_t 1.790] [optim_t 0.328] [lr 0.005000] 2024-04-03 10:49:56,102 - Train: 2.34% [113400/4848000] [23.4/1000.0] [batch_t 0.330 (0.549)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 10:50:35,159 - Train: 2.34% [113500/4848000] [23.4/1000.0] [batch_t 0.328 (0.390)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 10:51:10,739 - Train: 2.34% [113600/4848000] [23.4/1000.0] [batch_t 0.329 (0.356)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:51:43,447 - Train: 2.35% [113700/4848000] [23.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 10:52:18,853 - Train: 2.35% [113800/4848000] [23.5/1000.0] [batch_t 0.329 (0.354)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:52:54,914 - Train: 2.35% [113900/4848000] [23.5/1000.0] [batch_t 0.328 (0.361)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:53:38,782 - Train: 2.35% [114000/4848000] [23.5/1000.0] [batch_t 0.329 (0.439)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:54:15,640 - Train: 2.35% [114100/4848000] [23.5/1000.0] [batch_t 0.325 (0.368)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 10:54:48,360 - Train: 2.36% [114200/4848000] [23.6/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 10:55:24,024 - Train: 2.36% [114300/4848000] [23.6/1000.0] [batch_t 0.328 (0.357)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 10:55:56,856 - Train: 2.36% [114400/4848000] [23.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 10:56:37,285 - Train: 2.36% [114500/4848000] [23.6/1000.0] [batch_t 0.336 (0.404)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-03 10:57:20,458 - Train: 2.36% [114600/4848000] [23.6/1000.0] [batch_t 0.329 (0.432)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 10:57:53,386 - Train: 2.37% [114700/4848000] [23.7/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 10:58:29,700 - Train: 2.37% [114800/4848000] [23.7/1000.0] [batch_t 0.330 (0.363)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 10:59:02,382 - Train: 2.37% [114900/4848000] [23.7/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 10:59:38,973 - Train: 2.37% [115000/4848000] [23.7/1000.0] [batch_t 0.328 (0.366)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:00:15,991 - Train: 2.37% [115100/4848000] [23.7/1000.0] [batch_t 0.330 (0.370)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 11:00:48,680 - Train: 2.38% [115200/4848000] [23.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:01:24,205 - Train: 2.38% [115300/4848000] [23.8/1000.0] [batch_t 0.329 (0.355)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:01:56,967 - Train: 2.38% [115400/4848000] [23.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 11:02:31,712 - Train: 2.38% [115500/4848000] [23.8/1000.0] [batch_t 0.328 (0.347)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:03:04,857 - Train: 2.38% [115600/4848000] [23.8/1000.0] [batch_t 0.755 (0.331)] [data_t 0.427] [optim_t 0.328] [lr 0.005000] 2024-04-03 11:03:39,912 - Train: 2.39% [115700/4848000] [23.9/1000.0] [batch_t 0.325 (0.350)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 11:04:14,360 - Train: 2.39% [115800/4848000] [23.9/1000.0] [batch_t 0.329 (0.344)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 11:04:47,284 - Train: 2.39% [115900/4848000] [23.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:05:22,040 - Train: 2.39% [116000/4848000] [23.9/1000.0] [batch_t 0.327 (0.347)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 11:05:55,762 - Train: 2.39% [116100/4848000] [23.9/1000.0] [batch_t 0.326 (0.337)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 11:06:36,450 - Train: 2.40% [116200/4848000] [24.0/1000.0] [batch_t 0.329 (0.407)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:07:14,368 - Train: 2.40% [116300/4848000] [24.0/1000.0] [batch_t 0.329 (0.379)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:07:31,395 - ==> Total time: 11:41:29 Eta: 19 days, 19:27:35 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 11:07:49,913 - Train: 2.40% [116400/4848000] [24.0/1000.0] [batch_t 0.330 (0.350)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 11:08:25,289 - Train: 2.40% [116500/4848000] [24.0/1000.0] [batch_t 0.327 (0.354)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 11:08:57,984 - Train: 2.41% [116600/4848000] [24.1/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 11:09:33,350 - Train: 2.41% [116700/4848000] [24.1/1000.0] [batch_t 0.330 (0.354)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 11:10:07,856 - Train: 2.41% [116800/4848000] [24.1/1000.0] [batch_t 0.328 (0.345)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:10:40,559 - Train: 2.41% [116900/4848000] [24.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 11:11:17,495 - Train: 2.41% [117000/4848000] [24.1/1000.0] [batch_t 0.328 (0.369)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 11:11:50,242 - Train: 2.42% [117100/4848000] [24.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 11:12:29,981 - Train: 2.42% [117200/4848000] [24.2/1000.0] [batch_t 0.329 (0.397)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:13:02,686 - Train: 2.42% [117300/4848000] [24.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 11:13:38,386 - Train: 2.42% [117400/4848000] [24.2/1000.0] [batch_t 0.330 (0.357)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 11:14:13,963 - Train: 2.42% [117500/4848000] [24.2/1000.0] [batch_t 0.332 (0.356)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 11:14:46,686 - Train: 2.43% [117600/4848000] [24.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 11:15:23,016 - Train: 2.43% [117700/4848000] [24.3/1000.0] [batch_t 0.330 (0.363)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 11:15:55,806 - Train: 2.43% [117800/4848000] [24.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 11:16:31,550 - Train: 2.43% [117900/4848000] [24.3/1000.0] [batch_t 0.325 (0.357)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 11:17:04,211 - Train: 2.43% [118000/4848000] [24.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 11:17:40,247 - Train: 2.44% [118100/4848000] [24.4/1000.0] [batch_t 0.323 (0.360)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 11:18:14,509 - Train: 2.44% [118200/4848000] [24.4/1000.0] [batch_t 0.326 (0.343)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 11:18:47,217 - Train: 2.44% [118300/4848000] [24.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:19:21,603 - Train: 2.44% [118400/4848000] [24.4/1000.0] [batch_t 0.326 (0.344)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 11:19:54,280 - Train: 2.44% [118500/4848000] [24.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 11:20:30,508 - Train: 2.45% [118600/4848000] [24.5/1000.0] [batch_t 0.329 (0.362)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 11:21:03,274 - Train: 2.45% [118700/4848000] [24.5/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 11:21:41,415 - Train: 2.45% [118800/4848000] [24.5/1000.0] [batch_t 0.325 (0.381)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 11:22:17,800 - Train: 2.45% [118900/4848000] [24.5/1000.0] [batch_t 0.328 (0.364)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 11:22:50,512 - Train: 2.45% [119000/4848000] [24.5/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 11:23:25,955 - Train: 2.46% [119100/4848000] [24.6/1000.0] [batch_t 0.328 (0.354)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:23:58,675 - Train: 2.46% [119200/4848000] [24.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 11:24:34,638 - Train: 2.46% [119300/4848000] [24.6/1000.0] [batch_t 0.332 (0.360)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 11:25:09,065 - Train: 2.46% [119400/4848000] [24.6/1000.0] [batch_t 0.331 (0.344)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 11:25:41,800 - Train: 2.46% [119500/4848000] [24.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 11:26:16,674 - Train: 2.47% [119600/4848000] [24.7/1000.0] [batch_t 0.329 (0.349)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:26:49,361 - Train: 2.47% [119700/4848000] [24.7/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 11:27:24,655 - Train: 2.47% [119800/4848000] [24.7/1000.0] [batch_t 0.329 (0.353)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 11:27:57,372 - Train: 2.47% [119900/4848000] [24.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 11:28:32,113 - Train: 2.48% [120000/4848000] [24.8/1000.0] [batch_t 0.331 (0.347)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 11:29:08,572 - Train: 2.48% [120100/4848000] [24.8/1000.0] [batch_t 3.405 (0.365)] [data_t 3.076] [optim_t 0.329] [lr 0.005000] 2024-04-03 11:29:41,272 - Train: 2.48% [120200/4848000] [24.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:30:16,476 - Train: 2.48% [120300/4848000] [24.8/1000.0] [batch_t 0.329 (0.352)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:30:49,170 - Train: 2.48% [120400/4848000] [24.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 11:31:24,237 - Train: 2.49% [120500/4848000] [24.9/1000.0] [batch_t 0.327 (0.351)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 11:31:56,965 - Train: 2.49% [120600/4848000] [24.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 11:32:31,936 - Train: 2.49% [120700/4848000] [24.9/1000.0] [batch_t 0.325 (0.350)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 11:33:04,651 - Train: 2.49% [120800/4848000] [24.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:33:40,757 - Train: 2.49% [120900/4848000] [24.9/1000.0] [batch_t 0.327 (0.361)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 11:34:18,250 - Train: 2.50% [121000/4848000] [25.0/1000.0] [batch_t 0.328 (0.375)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 11:34:51,004 - Train: 2.50% [121100/4848000] [25.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:35:27,143 - Train: 2.50% [121200/4848000] [25.0/1000.0] [batch_t 0.329 (0.361)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:35:27,147 - ==> Total time: 12:09:25 Eta: 19 days, 18:07:40 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 11:36:01,825 - Train: 2.50% [121300/4848000] [25.0/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 11:36:35,480 - Train: 2.50% [121400/4848000] [25.0/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:37:17,470 - Train: 2.51% [121500/4848000] [25.1/1000.0] [batch_t 0.326 (0.420)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 11:37:50,182 - Train: 2.51% [121600/4848000] [25.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:38:27,993 - Train: 2.51% [121700/4848000] [25.1/1000.0] [batch_t 0.331 (0.378)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 11:39:21,032 - Train: 2.51% [121800/4848000] [25.1/1000.0] [batch_t 1.020 (0.530)] [data_t 0.693] [optim_t 0.327] [lr 0.005000] 2024-04-03 11:40:34,310 - Train: 2.51% [121900/4848000] [25.1/1000.0] [batch_t 0.405 (0.733)] [data_t 0.081] [optim_t 0.324] [lr 0.005000] 2024-04-03 11:41:16,748 - Train: 2.52% [122000/4848000] [25.2/1000.0] [batch_t 0.325 (0.424)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 11:57:40,119 - Train: 2.52% [122100/4848000] [25.2/1000.0] [batch_t 0.761 (9.833)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-03 11:59:03,308 - Train: 2.52% [122200/4848000] [25.2/1000.0] [batch_t 0.754 (0.832)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-03 12:00:34,885 - Train: 2.52% [122300/4848000] [25.2/1000.0] [batch_t 0.763 (0.916)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-03 12:01:50,334 - Train: 2.52% [122400/4848000] [25.2/1000.0] [batch_t 0.754 (0.754)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-03 12:03:05,852 - Train: 2.53% [122500/4848000] [25.3/1000.0] [batch_t 0.749 (0.755)] [data_t 0.002] [optim_t 0.747] [lr 0.005000] 2024-04-03 12:04:21,414 - Train: 2.53% [122600/4848000] [25.3/1000.0] [batch_t 0.757 (0.756)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-03 12:05:36,973 - Train: 2.53% [122700/4848000] [25.3/1000.0] [batch_t 0.758 (0.756)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-03 12:06:57,816 - Train: 2.53% [122800/4848000] [25.3/1000.0] [batch_t 0.762 (0.808)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-03 12:08:13,609 - Train: 2.54% [122900/4848000] [25.4/1000.0] [batch_t 0.763 (0.758)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-03 12:09:31,197 - Train: 2.54% [123000/4848000] [25.4/1000.0] [batch_t 0.758 (0.776)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-03 12:10:46,691 - Train: 2.54% [123100/4848000] [25.4/1000.0] [batch_t 0.758 (0.755)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-03 12:12:02,200 - Train: 2.54% [123200/4848000] [25.4/1000.0] [batch_t 0.752 (0.755)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-03 12:13:17,759 - Train: 2.54% [123300/4848000] [25.4/1000.0] [batch_t 0.758 (0.755)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-03 12:14:33,265 - Train: 2.55% [123400/4848000] [25.5/1000.0] [batch_t 0.758 (0.755)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-03 12:15:48,738 - Train: 2.55% [123500/4848000] [25.5/1000.0] [batch_t 0.753 (0.755)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-03 12:17:04,224 - Train: 2.55% [123600/4848000] [25.5/1000.0] [batch_t 0.762 (0.755)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-03 12:18:19,795 - Train: 2.55% [123700/4848000] [25.5/1000.0] [batch_t 0.763 (0.756)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-03 12:19:35,311 - Train: 2.55% [123800/4848000] [25.5/1000.0] [batch_t 0.759 (0.755)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-03 12:20:50,808 - Train: 2.56% [123900/4848000] [25.6/1000.0] [batch_t 0.767 (0.755)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-03 12:22:06,314 - Train: 2.56% [124000/4848000] [25.6/1000.0] [batch_t 0.752 (0.755)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-03 12:23:21,843 - Train: 2.56% [124100/4848000] [25.6/1000.0] [batch_t 0.759 (0.755)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-03 12:24:37,370 - Train: 2.56% [124200/4848000] [25.6/1000.0] [batch_t 0.743 (0.755)] [data_t 0.002] [optim_t 0.741] [lr 0.005000] 2024-04-03 12:25:52,876 - Train: 2.56% [124300/4848000] [25.6/1000.0] [batch_t 0.756 (0.755)] [data_t 0.002] [optim_t 0.754] [lr 0.005000] 2024-04-03 12:27:08,372 - Train: 2.57% [124400/4848000] [25.7/1000.0] [batch_t 0.763 (0.755)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-03 12:28:23,931 - Train: 2.57% [124500/4848000] [25.7/1000.0] [batch_t 0.749 (0.755)] [data_t 0.002] [optim_t 0.747] [lr 0.005000] 2024-04-03 12:29:01,434 - Train: 2.57% [124600/4848000] [25.7/1000.0] [batch_t 0.325 (0.375)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 12:29:34,203 - Train: 2.57% [124700/4848000] [25.7/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 12:30:06,962 - Train: 2.57% [124800/4848000] [25.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:30:39,679 - Train: 2.58% [124900/4848000] [25.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:31:15,591 - Train: 2.58% [125000/4848000] [25.8/1000.0] [batch_t 0.329 (0.359)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:31:48,290 - Train: 2.58% [125100/4848000] [25.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 12:32:21,150 - Train: 2.58% [125200/4848000] [25.8/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 12:32:53,943 - Train: 2.58% [125300/4848000] [25.8/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 12:33:26,668 - Train: 2.59% [125400/4848000] [25.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 12:33:59,376 - Train: 2.59% [125500/4848000] [25.9/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 12:34:32,104 - Train: 2.59% [125600/4848000] [25.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 12:35:06,265 - Train: 2.59% [125700/4848000] [25.9/1000.0] [batch_t 1.742 (0.342)] [data_t 1.414] [optim_t 0.328] [lr 0.005000] 2024-04-03 12:35:40,451 - Train: 2.59% [125800/4848000] [25.9/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:36:14,567 - Train: 2.60% [125900/4848000] [26.0/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:36:47,263 - Train: 2.60% [126000/4848000] [26.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:37:02,961 - ==> Total time: 13:11:01 Eta: 20 days, 13:52:59 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 12:37:22,728 - Train: 2.60% [126100/4848000] [26.0/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 12:37:55,409 - Train: 2.60% [126200/4848000] [26.0/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 12:38:28,130 - Train: 2.61% [126300/4848000] [26.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:39:00,836 - Train: 2.61% [126400/4848000] [26.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 12:39:33,782 - Train: 2.61% [126500/4848000] [26.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:40:06,460 - Train: 2.61% [126600/4848000] [26.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:40:39,145 - Train: 2.61% [126700/4848000] [26.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 12:41:14,268 - Train: 2.62% [126800/4848000] [26.2/1000.0] [batch_t 0.326 (0.351)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 12:41:46,999 - Train: 2.62% [126900/4848000] [26.2/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 12:42:19,689 - Train: 2.62% [127000/4848000] [26.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 12:42:52,559 - Train: 2.62% [127100/4848000] [26.2/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 12:43:25,452 - Train: 2.62% [127200/4848000] [26.2/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 12:43:58,124 - Train: 2.63% [127300/4848000] [26.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 12:44:31,933 - Train: 2.63% [127400/4848000] [26.3/1000.0] [batch_t 0.325 (0.338)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 12:45:04,850 - Train: 2.63% [127500/4848000] [26.3/1000.0] [batch_t 0.453 (0.329)] [data_t 0.127] [optim_t 0.325] [lr 0.005000] 2024-04-03 12:45:38,042 - Train: 2.63% [127600/4848000] [26.3/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 12:46:10,793 - Train: 2.63% [127700/4848000] [26.3/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 12:46:43,529 - Train: 2.64% [127800/4848000] [26.4/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 12:47:17,035 - Train: 2.64% [127900/4848000] [26.4/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 12:47:49,677 - Train: 2.64% [128000/4848000] [26.4/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 12:48:22,995 - Train: 2.64% [128100/4848000] [26.4/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 12:48:55,759 - Train: 2.64% [128200/4848000] [26.4/1000.0] [batch_t 0.346 (0.328)] [data_t 0.002] [optim_t 0.344] [lr 0.005000] 2024-04-03 12:49:29,846 - Train: 2.65% [128300/4848000] [26.5/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 12:50:02,555 - Train: 2.65% [128400/4848000] [26.5/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 12:50:36,163 - Train: 2.65% [128500/4848000] [26.5/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 12:51:10,182 - Train: 2.65% [128600/4848000] [26.5/1000.0] [batch_t 0.324 (0.340)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 12:51:42,888 - Train: 2.65% [128700/4848000] [26.5/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 12:52:15,574 - Train: 2.66% [128800/4848000] [26.6/1000.0] [batch_t 0.335 (0.327)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 12:52:48,364 - Train: 2.66% [128900/4848000] [26.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:53:21,051 - Train: 2.66% [129000/4848000] [26.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 12:53:53,723 - Train: 2.66% [129100/4848000] [26.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:54:27,801 - Train: 2.67% [129200/4848000] [26.7/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:55:00,523 - Train: 2.67% [129300/4848000] [26.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:55:34,037 - Train: 2.67% [129400/4848000] [26.7/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:56:06,893 - Train: 2.67% [129500/4848000] [26.7/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 12:56:39,554 - Train: 2.67% [129600/4848000] [26.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:57:12,266 - Train: 2.68% [129700/4848000] [26.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 12:57:44,949 - Train: 2.68% [129800/4848000] [26.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 12:58:18,412 - Train: 2.68% [129900/4848000] [26.8/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 12:58:51,201 - Train: 2.68% [130000/4848000] [26.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 12:59:24,701 - Train: 2.68% [130100/4848000] [26.8/1000.0] [batch_t 0.333 (0.335)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 12:59:57,415 - Train: 2.69% [130200/4848000] [26.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:00:33,014 - Train: 2.69% [130300/4848000] [26.9/1000.0] [batch_t 0.326 (0.356)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 13:01:06,297 - Train: 2.69% [130400/4848000] [26.9/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:01:39,041 - Train: 2.69% [130500/4848000] [26.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 13:02:12,704 - Train: 2.69% [130600/4848000] [26.9/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:02:45,424 - Train: 2.70% [130700/4848000] [27.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:03:18,412 - Train: 2.70% [130800/4848000] [27.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:03:49,843 - ==> Total time: 13:37:48 Eta: 20 days, 11:11:17 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 13:03:53,023 - Train: 2.70% [130900/4848000] [27.0/1000.0] [batch_t 0.324 (0.460)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 13:04:26,447 - Train: 2.70% [131000/4848000] [27.0/1000.0] [batch_t 0.336 (0.334)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-03 13:04:59,160 - Train: 2.70% [131100/4848000] [27.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:05:33,490 - Train: 2.71% [131200/4848000] [27.1/1000.0] [batch_t 0.329 (0.343)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:06:06,415 - Train: 2.71% [131300/4848000] [27.1/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 13:06:39,147 - Train: 2.71% [131400/4848000] [27.1/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 13:07:14,419 - Train: 2.71% [131500/4848000] [27.1/1000.0] [batch_t 0.330 (0.353)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 13:07:47,124 - Train: 2.71% [131600/4848000] [27.1/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 13:08:20,272 - Train: 2.72% [131700/4848000] [27.2/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 13:08:53,007 - Train: 2.72% [131800/4848000] [27.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:09:28,245 - Train: 2.72% [131900/4848000] [27.2/1000.0] [batch_t 0.332 (0.352)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 13:10:00,996 - Train: 2.72% [132000/4848000] [27.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 13:10:34,541 - Train: 2.72% [132100/4848000] [27.2/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:11:08,198 - Train: 2.73% [132200/4848000] [27.3/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:11:40,893 - Train: 2.73% [132300/4848000] [27.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:12:14,484 - Train: 2.73% [132400/4848000] [27.3/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:12:47,176 - Train: 2.73% [132500/4848000] [27.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:13:20,860 - Train: 2.74% [132600/4848000] [27.4/1000.0] [batch_t 0.325 (0.337)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 13:13:53,677 - Train: 2.74% [132700/4848000] [27.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:14:26,569 - Train: 2.74% [132800/4848000] [27.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 13:14:59,281 - Train: 2.74% [132900/4848000] [27.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:15:32,520 - Train: 2.74% [133000/4848000] [27.4/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 13:16:05,263 - Train: 2.75% [133100/4848000] [27.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 13:16:38,014 - Train: 2.75% [133200/4848000] [27.5/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 13:17:10,732 - Train: 2.75% [133300/4848000] [27.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:17:43,403 - Train: 2.75% [133400/4848000] [27.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:18:16,123 - Train: 2.75% [133500/4848000] [27.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:18:48,847 - Train: 2.76% [133600/4848000] [27.6/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 13:19:21,620 - Train: 2.76% [133700/4848000] [27.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 13:19:54,416 - Train: 2.76% [133800/4848000] [27.6/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 13:20:27,215 - Train: 2.76% [133900/4848000] [27.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 13:20:59,919 - Train: 2.76% [134000/4848000] [27.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:21:32,648 - Train: 2.77% [134100/4848000] [27.7/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 13:22:05,495 - Train: 2.77% [134200/4848000] [27.7/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 13:22:38,248 - Train: 2.77% [134300/4848000] [27.7/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 13:23:10,998 - Train: 2.77% [134400/4848000] [27.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:23:43,823 - Train: 2.77% [134500/4848000] [27.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:24:16,548 - Train: 2.78% [134600/4848000] [27.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:24:49,222 - Train: 2.78% [134700/4848000] [27.8/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 13:25:21,920 - Train: 2.78% [134800/4848000] [27.8/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 13:25:54,686 - Train: 2.78% [134900/4848000] [27.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:26:27,504 - Train: 2.78% [135000/4848000] [27.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 13:27:00,338 - Train: 2.79% [135100/4848000] [27.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 13:27:33,057 - Train: 2.79% [135200/4848000] [27.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:28:06,326 - Train: 2.79% [135300/4848000] [27.9/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:28:39,011 - Train: 2.79% [135400/4848000] [27.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:29:11,926 - Train: 2.79% [135500/4848000] [27.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:29:44,886 - Train: 2.80% [135600/4848000] [28.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:30:18,229 - Train: 2.80% [135700/4848000] [28.0/1000.0] [batch_t 0.326 (0.333)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 13:30:32,643 - ==> Total time: 14:04:31 Eta: 20 days, 8:36:52 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 13:30:52,918 - Train: 2.80% [135800/4848000] [28.0/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:31:25,596 - Train: 2.80% [135900/4848000] [28.0/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 13:31:58,276 - Train: 2.81% [136000/4848000] [28.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:32:30,959 - Train: 2.81% [136100/4848000] [28.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:33:03,703 - Train: 2.81% [136200/4848000] [28.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 13:33:36,438 - Train: 2.81% [136300/4848000] [28.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:34:09,188 - Train: 2.81% [136400/4848000] [28.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:34:41,862 - Train: 2.82% [136500/4848000] [28.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:35:15,850 - Train: 2.82% [136600/4848000] [28.2/1000.0] [batch_t 0.330 (0.340)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 13:35:48,575 - Train: 2.82% [136700/4848000] [28.2/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 13:36:21,267 - Train: 2.82% [136800/4848000] [28.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 13:36:54,039 - Train: 2.82% [136900/4848000] [28.2/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 13:37:27,462 - Train: 2.83% [137000/4848000] [28.3/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:38:00,140 - Train: 2.83% [137100/4848000] [28.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:38:32,843 - Train: 2.83% [137200/4848000] [28.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:39:06,295 - Train: 2.83% [137300/4848000] [28.3/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 13:39:39,037 - Train: 2.83% [137400/4848000] [28.3/1000.0] [batch_t 0.335 (0.327)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 13:40:12,288 - Train: 2.84% [137500/4848000] [28.4/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 13:40:44,974 - Train: 2.84% [137600/4848000] [28.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:41:17,678 - Train: 2.84% [137700/4848000] [28.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:41:50,410 - Train: 2.84% [137800/4848000] [28.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:42:23,128 - Train: 2.84% [137900/4848000] [28.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 13:42:55,849 - Train: 2.85% [138000/4848000] [28.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:43:28,630 - Train: 2.85% [138100/4848000] [28.5/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 13:44:01,356 - Train: 2.85% [138200/4848000] [28.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:44:34,240 - Train: 2.85% [138300/4848000] [28.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:45:07,113 - Train: 2.85% [138400/4848000] [28.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:45:39,773 - Train: 2.86% [138500/4848000] [28.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 13:46:12,506 - Train: 2.86% [138600/4848000] [28.6/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 13:46:45,205 - Train: 2.86% [138700/4848000] [28.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 13:47:17,964 - Train: 2.86% [138800/4848000] [28.6/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 13:47:50,692 - Train: 2.87% [138900/4848000] [28.7/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 13:48:23,392 - Train: 2.87% [139000/4848000] [28.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:48:56,094 - Train: 2.87% [139100/4848000] [28.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 13:49:29,510 - Train: 2.87% [139200/4848000] [28.7/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:50:02,243 - Train: 2.87% [139300/4848000] [28.7/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 13:50:36,150 - Train: 2.88% [139400/4848000] [28.8/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:51:08,865 - Train: 2.88% [139500/4848000] [28.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 13:51:44,397 - Train: 2.88% [139600/4848000] [28.8/1000.0] [batch_t 0.329 (0.355)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:52:18,603 - Train: 2.88% [139700/4848000] [28.8/1000.0] [batch_t 0.327 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:52:51,508 - Train: 2.88% [139800/4848000] [28.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:53:24,182 - Train: 2.89% [139900/4848000] [28.9/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 13:53:56,893 - Train: 2.89% [140000/4848000] [28.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:54:30,625 - Train: 2.89% [140100/4848000] [28.9/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 13:55:03,409 - Train: 2.89% [140200/4848000] [28.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 13:55:36,091 - Train: 2.89% [140300/4848000] [28.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:56:09,917 - Train: 2.90% [140400/4848000] [29.0/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 13:56:42,672 - Train: 2.90% [140500/4848000] [29.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:57:12,762 - ==> Total time: 14:31:11 Eta: 20 days, 6:09:46 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 13:57:16,815 - Train: 2.90% [140600/4848000] [29.0/1000.0] [batch_t 0.329 (0.345)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:57:49,490 - Train: 2.90% [140700/4848000] [29.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:58:22,201 - Train: 2.90% [140800/4848000] [29.0/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 13:58:54,878 - Train: 2.91% [140900/4848000] [29.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 13:59:27,570 - Train: 2.91% [141000/4848000] [29.1/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 14:00:00,282 - Train: 2.91% [141100/4848000] [29.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 14:00:33,613 - Train: 2.91% [141200/4848000] [29.1/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 14:01:06,809 - Train: 2.91% [141300/4848000] [29.1/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 14:01:39,520 - Train: 2.92% [141400/4848000] [29.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 14:02:12,816 - Train: 2.92% [141500/4848000] [29.2/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 14:02:45,596 - Train: 2.92% [141600/4848000] [29.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 14:03:18,282 - Train: 2.92% [141700/4848000] [29.2/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 14:03:51,048 - Train: 2.92% [141800/4848000] [29.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 14:04:23,795 - Train: 2.93% [141900/4848000] [29.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 14:04:56,501 - Train: 2.93% [142000/4848000] [29.3/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 14:05:29,161 - Train: 2.93% [142100/4848000] [29.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 14:06:01,883 - Train: 2.93% [142200/4848000] [29.3/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 14:06:35,691 - Train: 2.94% [142300/4848000] [29.4/1000.0] [batch_t 0.336 (0.338)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-03 14:07:09,068 - Train: 2.94% [142400/4848000] [29.4/1000.0] [batch_t 0.326 (0.334)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 14:07:41,774 - Train: 2.94% [142500/4848000] [29.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 14:08:14,995 - Train: 2.94% [142600/4848000] [29.4/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 14:08:47,672 - Train: 2.94% [142700/4848000] [29.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 14:09:20,403 - Train: 2.95% [142800/4848000] [29.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 14:09:53,171 - Train: 2.95% [142900/4848000] [29.5/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 14:10:25,905 - Train: 2.95% [143000/4848000] [29.5/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 14:10:58,649 - Train: 2.95% [143100/4848000] [29.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 14:11:32,163 - Train: 2.95% [143200/4848000] [29.5/1000.0] [batch_t 0.324 (0.335)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 14:12:05,678 - Train: 2.96% [143300/4848000] [29.6/1000.0] [batch_t 0.345 (0.335)] [data_t 0.022] [optim_t 0.324] [lr 0.005000] 2024-04-03 14:12:38,391 - Train: 2.96% [143400/4848000] [29.6/1000.0] [batch_t 0.335 (0.327)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 14:13:11,129 - Train: 2.96% [143500/4848000] [29.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 14:13:43,963 - Train: 2.96% [143600/4848000] [29.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 14:14:20,499 - Train: 2.96% [143700/4848000] [29.6/1000.0] [batch_t 0.325 (0.365)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 14:14:53,224 - Train: 2.97% [143800/4848000] [29.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 14:15:27,008 - Train: 2.97% [143900/4848000] [29.7/1000.0] [batch_t 0.324 (0.338)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 14:20:49,597 - Train: 2.97% [144000/4848000] [29.7/1000.0] [batch_t 0.757 (3.226)] [data_t 0.002] [optim_t 0.754] [lr 0.005000] 2024-04-03 14:22:05,128 - Train: 2.97% [144100/4848000] [29.7/1000.0] [batch_t 0.763 (0.755)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-03 14:23:20,704 - Train: 2.97% [144200/4848000] [29.7/1000.0] [batch_t 0.765 (0.756)] [data_t 0.002] [optim_t 0.763] [lr 0.005000] 2024-04-03 14:24:36,244 - Train: 2.98% [144300/4848000] [29.8/1000.0] [batch_t 0.763 (0.755)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-03 14:25:52,049 - Train: 2.98% [144400/4848000] [29.8/1000.0] [batch_t 0.765 (0.758)] [data_t 0.002] [optim_t 0.763] [lr 0.005000] 2024-04-03 14:27:07,762 - Train: 2.98% [144500/4848000] [29.8/1000.0] [batch_t 0.760 (0.757)] [data_t 0.002] [optim_t 0.758] [lr 0.005000] 2024-04-03 14:28:23,366 - Train: 2.98% [144600/4848000] [29.8/1000.0] [batch_t 0.754 (0.756)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-03 14:29:38,901 - Train: 2.98% [144700/4848000] [29.8/1000.0] [batch_t 0.759 (0.755)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-03 14:31:01,015 - Train: 2.99% [144800/4848000] [29.9/1000.0] [batch_t 2.694 (0.821)] [data_t 1.940] [optim_t 0.754] [lr 0.005000] 2024-04-03 14:41:21,621 - Train: 2.99% [144900/4848000] [29.9/1000.0] [batch_t 0.753 (6.206)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-03 14:42:39,059 - Train: 2.99% [145000/4848000] [29.9/1000.0] [batch_t 0.754 (0.774)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-03 14:43:54,641 - Train: 2.99% [145100/4848000] [29.9/1000.0] [batch_t 0.757 (0.756)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-03 14:45:10,319 - Train: 3.00% [145200/4848000] [30.0/1000.0] [batch_t 0.759 (0.757)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-03 14:46:25,873 - Train: 3.00% [145300/4848000] [30.0/1000.0] [batch_t 0.749 (0.755)] [data_t 0.002] [optim_t 0.747] [lr 0.005000] 2024-04-03 14:47:41,490 - Train: 3.00% [145400/4848000] [30.0/1000.0] [batch_t 0.764 (0.756)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-03 14:48:11,723 - ==> Total time: 15:22:10 Eta: 20 days, 16:56:50 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 14:49:00,165 - Train: 3.00% [145500/4848000] [30.0/1000.0] [batch_t 0.753 (0.780)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-03 14:50:15,688 - Train: 3.00% [145600/4848000] [30.0/1000.0] [batch_t 0.763 (0.755)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-03 14:51:31,151 - Train: 3.01% [145700/4848000] [30.1/1000.0] [batch_t 0.755 (0.755)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-03 14:52:46,660 - Train: 3.01% [145800/4848000] [30.1/1000.0] [batch_t 0.753 (0.755)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-03 14:54:02,251 - Train: 3.01% [145900/4848000] [30.1/1000.0] [batch_t 0.749 (0.756)] [data_t 0.002] [optim_t 0.747] [lr 0.005000] 2024-04-03 14:55:17,827 - Train: 3.01% [146000/4848000] [30.1/1000.0] [batch_t 0.754 (0.756)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-03 14:56:33,409 - Train: 3.01% [146100/4848000] [30.1/1000.0] [batch_t 0.764 (0.756)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-03 14:57:49,042 - Train: 3.02% [146200/4848000] [30.2/1000.0] [batch_t 0.753 (0.756)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-03 14:59:04,471 - Train: 3.02% [146300/4848000] [30.2/1000.0] [batch_t 0.741 (0.754)] [data_t 0.002] [optim_t 0.739] [lr 0.005000] 2024-04-03 15:00:20,062 - Train: 3.02% [146400/4848000] [30.2/1000.0] [batch_t 0.755 (0.756)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-03 15:01:35,606 - Train: 3.02% [146500/4848000] [30.2/1000.0] [batch_t 0.750 (0.755)] [data_t 0.002] [optim_t 0.748] [lr 0.005000] 2024-04-03 15:02:51,158 - Train: 3.02% [146600/4848000] [30.2/1000.0] [batch_t 0.764 (0.755)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-03 15:04:06,799 - Train: 3.03% [146700/4848000] [30.3/1000.0] [batch_t 0.755 (0.756)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-03 15:05:22,352 - Train: 3.03% [146800/4848000] [30.3/1000.0] [batch_t 0.758 (0.755)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-03 15:06:37,971 - Train: 3.03% [146900/4848000] [30.3/1000.0] [batch_t 0.763 (0.756)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-03 15:07:53,586 - Train: 3.03% [147000/4848000] [30.3/1000.0] [batch_t 0.750 (0.756)] [data_t 0.002] [optim_t 0.748] [lr 0.005000] 2024-04-03 15:09:09,022 - Train: 3.03% [147100/4848000] [30.3/1000.0] [batch_t 0.763 (0.754)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-03 15:10:24,563 - Train: 3.04% [147200/4848000] [30.4/1000.0] [batch_t 0.758 (0.755)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-03 15:11:40,183 - Train: 3.04% [147300/4848000] [30.4/1000.0] [batch_t 0.758 (0.756)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-03 15:12:55,724 - Train: 3.04% [147400/4848000] [30.4/1000.0] [batch_t 0.763 (0.755)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-03 15:14:12,819 - Train: 3.04% [147500/4848000] [30.4/1000.0] [batch_t 0.757 (0.771)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-03 15:15:28,258 - Train: 3.04% [147600/4848000] [30.4/1000.0] [batch_t 0.760 (0.754)] [data_t 0.002] [optim_t 0.758] [lr 0.005000] 2024-04-03 15:16:43,861 - Train: 3.05% [147700/4848000] [30.5/1000.0] [batch_t 0.765 (0.756)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-03 15:17:59,560 - Train: 3.05% [147800/4848000] [30.5/1000.0] [batch_t 0.760 (0.757)] [data_t 0.002] [optim_t 0.758] [lr 0.005000] 2024-04-03 15:19:15,034 - Train: 3.05% [147900/4848000] [30.5/1000.0] [batch_t 0.764 (0.755)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-03 15:20:30,631 - Train: 3.05% [148000/4848000] [30.5/1000.0] [batch_t 0.761 (0.756)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-03 15:21:46,236 - Train: 3.05% [148100/4848000] [30.5/1000.0] [batch_t 0.738 (0.756)] [data_t 0.002] [optim_t 0.736] [lr 0.005000] 2024-04-03 15:23:01,800 - Train: 3.06% [148200/4848000] [30.6/1000.0] [batch_t 0.758 (0.756)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-03 15:24:17,318 - Train: 3.06% [148300/4848000] [30.6/1000.0] [batch_t 0.760 (0.755)] [data_t 0.002] [optim_t 0.758] [lr 0.005000] 2024-04-03 15:25:32,744 - Train: 3.06% [148400/4848000] [30.6/1000.0] [batch_t 0.755 (0.754)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-03 15:26:48,306 - Train: 3.06% [148500/4848000] [30.6/1000.0] [batch_t 0.765 (0.756)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-03 15:28:03,931 - Train: 3.07% [148600/4848000] [30.7/1000.0] [batch_t 0.761 (0.756)] [data_t 0.002] [optim_t 0.758] [lr 0.005000] 2024-04-03 15:28:56,345 - Train: 3.07% [148700/4848000] [30.7/1000.0] [batch_t 0.330 (0.524)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 15:29:30,520 - Train: 3.07% [148800/4848000] [30.7/1000.0] [batch_t 0.330 (0.342)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 15:30:03,279 - Train: 3.07% [148900/4848000] [30.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 15:30:36,829 - Train: 3.07% [149000/4848000] [30.7/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 15:31:10,007 - Train: 3.08% [149100/4848000] [30.8/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 15:31:42,760 - Train: 3.08% [149200/4848000] [30.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 15:32:16,280 - Train: 3.08% [149300/4848000] [30.8/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 15:32:48,977 - Train: 3.08% [149400/4848000] [30.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 15:33:22,830 - Train: 3.08% [149500/4848000] [30.8/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 15:33:55,529 - Train: 3.09% [149600/4848000] [30.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 15:34:29,137 - Train: 3.09% [149700/4848000] [30.9/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 15:35:01,902 - Train: 3.09% [149800/4848000] [30.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 15:35:35,277 - Train: 3.09% [149900/4848000] [30.9/1000.0] [batch_t 0.325 (0.334)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 15:36:09,471 - Train: 3.09% [150000/4848000] [30.9/1000.0] [batch_t 0.328 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 15:36:42,138 - Train: 3.10% [150100/4848000] [31.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 15:37:14,886 - Train: 3.10% [150200/4848000] [31.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 15:37:43,730 - ==> Total time: 16:11:42 Eta: 21 days, 2:13:34 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 15:37:49,309 - Train: 3.10% [150300/4848000] [31.0/1000.0] [batch_t 0.327 (0.343)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 15:38:22,083 - Train: 3.10% [150400/4848000] [31.0/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 15:38:54,813 - Train: 3.10% [150500/4848000] [31.0/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 15:39:27,482 - Train: 3.11% [150600/4848000] [31.1/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 15:40:00,161 - Train: 3.11% [150700/4848000] [31.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 15:40:32,895 - Train: 3.11% [150800/4848000] [31.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 15:41:05,663 - Train: 3.11% [150900/4848000] [31.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 15:41:38,545 - Train: 3.11% [151000/4848000] [31.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 15:42:11,314 - Train: 3.12% [151100/4848000] [31.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 15:42:44,032 - Train: 3.12% [151200/4848000] [31.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 15:43:16,740 - Train: 3.12% [151300/4848000] [31.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 15:43:49,414 - Train: 3.12% [151400/4848000] [31.2/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 15:44:22,103 - Train: 3.12% [151500/4848000] [31.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 15:44:54,920 - Train: 3.13% [151600/4848000] [31.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 15:45:27,688 - Train: 3.13% [151700/4848000] [31.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 15:46:00,372 - Train: 3.13% [151800/4848000] [31.3/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 15:46:33,067 - Train: 3.13% [151900/4848000] [31.3/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 15:47:05,770 - Train: 3.14% [152000/4848000] [31.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 15:47:38,493 - Train: 3.14% [152100/4848000] [31.4/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 15:48:11,970 - Train: 3.14% [152200/4848000] [31.4/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 15:48:44,687 - Train: 3.14% [152300/4848000] [31.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 15:49:19,120 - Train: 3.14% [152400/4848000] [31.4/1000.0] [batch_t 0.328 (0.344)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 15:49:51,915 - Train: 3.15% [152500/4848000] [31.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 15:50:26,324 - Train: 3.15% [152600/4848000] [31.5/1000.0] [batch_t 0.326 (0.344)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 15:50:59,018 - Train: 3.15% [152700/4848000] [31.5/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 15:51:33,563 - Train: 3.15% [152800/4848000] [31.5/1000.0] [batch_t 0.324 (0.345)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 15:52:06,748 - Train: 3.15% [152900/4848000] [31.5/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 15:52:39,381 - Train: 3.16% [153000/4848000] [31.6/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 15:53:14,520 - Train: 3.16% [153100/4848000] [31.6/1000.0] [batch_t 0.327 (0.351)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 15:53:51,217 - Train: 3.16% [153200/4848000] [31.6/1000.0] [batch_t 0.327 (0.367)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 15:54:30,913 - Train: 3.16% [153300/4848000] [31.6/1000.0] [batch_t 0.328 (0.397)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 15:55:03,649 - Train: 3.16% [153400/4848000] [31.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 15:55:39,432 - Train: 3.17% [153500/4848000] [31.7/1000.0] [batch_t 0.324 (0.358)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 15:56:12,126 - Train: 3.17% [153600/4848000] [31.7/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 15:56:44,817 - Train: 3.17% [153700/4848000] [31.7/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 15:57:17,541 - Train: 3.17% [153800/4848000] [31.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 15:57:50,446 - Train: 3.17% [153900/4848000] [31.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 15:58:25,211 - Train: 3.18% [154000/4848000] [31.8/1000.0] [batch_t 0.330 (0.348)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 15:58:58,001 - Train: 3.18% [154100/4848000] [31.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 15:59:31,323 - Train: 3.18% [154200/4848000] [31.8/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:00:04,009 - Train: 3.18% [154300/4848000] [31.8/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:00:37,058 - Train: 3.18% [154400/4848000] [31.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:01:11,274 - Train: 3.19% [154500/4848000] [31.9/1000.0] [batch_t 0.326 (0.342)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 16:01:44,031 - Train: 3.19% [154600/4848000] [31.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:02:17,385 - Train: 3.19% [154700/4848000] [31.9/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 16:02:50,060 - Train: 3.19% [154800/4848000] [31.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:03:22,725 - Train: 3.20% [154900/4848000] [32.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 16:03:55,424 - Train: 3.20% [155000/4848000] [32.0/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 16:04:29,053 - Train: 3.20% [155100/4848000] [32.0/1000.0] [batch_t 0.326 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 16:04:40,829 - ==> Total time: 16:38:39 Eta: 20 days, 23:29:19 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 16:05:03,660 - Train: 3.20% [155200/4848000] [32.0/1000.0] [batch_t 0.326 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 16:05:38,390 - Train: 3.20% [155300/4848000] [32.0/1000.0] [batch_t 0.327 (0.347)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:06:11,196 - Train: 3.21% [155400/4848000] [32.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 16:06:43,869 - Train: 3.21% [155500/4848000] [32.1/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 16:07:17,964 - Train: 3.21% [155600/4848000] [32.1/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:07:50,710 - Train: 3.21% [155700/4848000] [32.1/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:08:24,222 - Train: 3.21% [155800/4848000] [32.1/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:08:56,903 - Train: 3.22% [155900/4848000] [32.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 16:09:31,544 - Train: 3.22% [156000/4848000] [32.2/1000.0] [batch_t 0.326 (0.346)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 16:10:04,247 - Train: 3.22% [156100/4848000] [32.2/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 16:10:38,876 - Train: 3.22% [156200/4848000] [32.2/1000.0] [batch_t 0.327 (0.346)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 16:11:12,628 - Train: 3.22% [156300/4848000] [32.2/1000.0] [batch_t 0.324 (0.337)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 16:11:45,411 - Train: 3.23% [156400/4848000] [32.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:12:18,747 - Train: 3.23% [156500/4848000] [32.3/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:12:51,437 - Train: 3.23% [156600/4848000] [32.3/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:13:25,480 - Train: 3.23% [156700/4848000] [32.3/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:13:58,181 - Train: 3.23% [156800/4848000] [32.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:14:30,889 - Train: 3.24% [156900/4848000] [32.4/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 16:15:03,609 - Train: 3.24% [157000/4848000] [32.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:15:37,539 - Train: 3.24% [157100/4848000] [32.4/1000.0] [batch_t 0.325 (0.339)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 16:16:11,407 - Train: 3.24% [157200/4848000] [32.4/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:16:44,097 - Train: 3.24% [157300/4848000] [32.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 16:17:22,326 - Train: 3.25% [157400/4848000] [32.5/1000.0] [batch_t 0.325 (0.382)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 16:17:55,163 - Train: 3.25% [157500/4848000] [32.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:18:29,387 - Train: 3.25% [157600/4848000] [32.5/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:19:02,095 - Train: 3.25% [157700/4848000] [32.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:19:36,091 - Train: 3.25% [157800/4848000] [32.5/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:20:09,216 - Train: 3.26% [157900/4848000] [32.6/1000.0] [batch_t 0.325 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 16:20:41,932 - Train: 3.26% [158000/4848000] [32.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 16:21:14,819 - Train: 3.26% [158100/4848000] [32.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:21:47,627 - Train: 3.26% [158200/4848000] [32.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:22:21,018 - Train: 3.27% [158300/4848000] [32.7/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:22:53,714 - Train: 3.27% [158400/4848000] [32.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:23:27,299 - Train: 3.27% [158500/4848000] [32.7/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:24:00,007 - Train: 3.27% [158600/4848000] [32.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:24:36,869 - Train: 3.27% [158700/4848000] [32.7/1000.0] [batch_t 0.327 (0.369)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:25:10,259 - Train: 3.28% [158800/4848000] [32.8/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 16:25:42,990 - Train: 3.28% [158900/4848000] [32.8/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:26:16,864 - Train: 3.28% [159000/4848000] [32.8/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:26:49,569 - Train: 3.28% [159100/4848000] [32.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:27:24,044 - Train: 3.28% [159200/4848000] [32.8/1000.0] [batch_t 0.335 (0.345)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 16:27:56,783 - Train: 3.29% [159300/4848000] [32.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:28:33,079 - Train: 3.29% [159400/4848000] [32.9/1000.0] [batch_t 0.328 (0.363)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:29:08,244 - Train: 3.29% [159500/4848000] [32.9/1000.0] [batch_t 0.326 (0.352)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 16:29:41,071 - Train: 3.29% [159600/4848000] [32.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:30:17,312 - Train: 3.29% [159700/4848000] [32.9/1000.0] [batch_t 0.327 (0.362)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 16:30:50,026 - Train: 3.30% [159800/4848000] [33.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:31:24,091 - Train: 3.30% [159900/4848000] [33.0/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 16:31:51,551 - ==> Total time: 17:05:50 Eta: 20 days, 21:00:03 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 16:31:58,282 - Train: 3.30% [160000/4848000] [33.0/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 16:32:34,184 - Train: 3.30% [160100/4848000] [33.0/1000.0] [batch_t 0.328 (0.359)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:33:09,856 - Train: 3.30% [160200/4848000] [33.0/1000.0] [batch_t 0.328 (0.357)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:33:42,545 - Train: 3.31% [160300/4848000] [33.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:34:17,699 - Train: 3.31% [160400/4848000] [33.1/1000.0] [batch_t 0.332 (0.351)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 16:34:50,489 - Train: 3.31% [160500/4848000] [33.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:35:26,102 - Train: 3.31% [160600/4848000] [33.1/1000.0] [batch_t 0.330 (0.356)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:35:58,842 - Train: 3.31% [160700/4848000] [33.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:36:34,384 - Train: 3.32% [160800/4848000] [33.2/1000.0] [batch_t 0.327 (0.355)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 16:37:09,673 - Train: 3.32% [160900/4848000] [33.2/1000.0] [batch_t 0.328 (0.353)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:37:42,941 - Train: 3.32% [161000/4848000] [33.2/1000.0] [batch_t 0.906 (0.333)] [data_t 0.581] [optim_t 0.325] [lr 0.005000] 2024-04-03 16:38:18,918 - Train: 3.32% [161100/4848000] [33.2/1000.0] [batch_t 0.330 (0.360)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:38:51,648 - Train: 3.33% [161200/4848000] [33.3/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 16:39:26,495 - Train: 3.33% [161300/4848000] [33.3/1000.0] [batch_t 0.329 (0.348)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:39:59,223 - Train: 3.33% [161400/4848000] [33.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:40:33,519 - Train: 3.33% [161500/4848000] [33.3/1000.0] [batch_t 0.324 (0.343)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 16:41:07,628 - Train: 3.33% [161600/4848000] [33.3/1000.0] [batch_t 0.325 (0.341)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 16:41:40,347 - Train: 3.34% [161700/4848000] [33.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:42:16,927 - Train: 3.34% [161800/4848000] [33.4/1000.0] [batch_t 0.329 (0.366)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:42:49,561 - Train: 3.34% [161900/4848000] [33.4/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 16:43:23,421 - Train: 3.34% [162000/4848000] [33.4/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 16:43:56,118 - Train: 3.34% [162100/4848000] [33.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:44:31,243 - Train: 3.35% [162200/4848000] [33.5/1000.0] [batch_t 0.326 (0.351)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 16:45:03,995 - Train: 3.35% [162300/4848000] [33.5/1000.0] [batch_t 0.335 (0.327)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 16:45:38,099 - Train: 3.35% [162400/4848000] [33.5/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:46:12,667 - Train: 3.35% [162500/4848000] [33.5/1000.0] [batch_t 0.331 (0.346)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 16:46:45,330 - Train: 3.35% [162600/4848000] [33.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 16:47:19,234 - Train: 3.36% [162700/4848000] [33.6/1000.0] [batch_t 0.331 (0.339)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 16:47:51,984 - Train: 3.36% [162800/4848000] [33.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:48:27,719 - Train: 3.36% [162900/4848000] [33.6/1000.0] [batch_t 0.329 (0.357)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:49:00,458 - Train: 3.36% [163000/4848000] [33.6/1000.0] [batch_t 0.337 (0.327)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-03 16:49:37,896 - Train: 3.36% [163100/4848000] [33.6/1000.0] [batch_t 0.325 (0.374)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 16:50:12,899 - Train: 3.37% [163200/4848000] [33.7/1000.0] [batch_t 0.327 (0.350)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:50:45,621 - Train: 3.37% [163300/4848000] [33.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:51:21,499 - Train: 3.37% [163400/4848000] [33.7/1000.0] [batch_t 0.326 (0.359)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 16:51:54,236 - Train: 3.37% [163500/4848000] [33.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:52:30,933 - Train: 3.37% [163600/4848000] [33.7/1000.0] [batch_t 0.328 (0.367)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:53:03,597 - Train: 3.38% [163700/4848000] [33.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 16:53:38,587 - Train: 3.38% [163800/4848000] [33.8/1000.0] [batch_t 0.329 (0.350)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:54:15,015 - Train: 3.38% [163900/4848000] [33.8/1000.0] [batch_t 0.331 (0.364)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 16:54:47,692 - Train: 3.38% [164000/4848000] [33.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 16:55:24,806 - Train: 3.38% [164100/4848000] [33.8/1000.0] [batch_t 0.329 (0.371)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:55:57,882 - Train: 3.39% [164200/4848000] [33.9/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:56:33,477 - Train: 3.39% [164300/4848000] [33.9/1000.0] [batch_t 0.327 (0.356)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 16:57:09,727 - Train: 3.39% [164400/4848000] [33.9/1000.0] [batch_t 0.470 (0.362)] [data_t 0.142] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:57:42,402 - Train: 3.39% [164500/4848000] [33.9/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 16:58:18,448 - Train: 3.40% [164600/4848000] [34.0/1000.0] [batch_t 0.328 (0.360)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 16:58:51,216 - Train: 3.40% [164700/4848000] [34.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 16:59:26,571 - Train: 3.40% [164800/4848000] [34.0/1000.0] [batch_t 0.324 (0.353)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 16:59:37,047 - ==> Total time: 17:33:35 Eta: 20 days, 18:54:25 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 17:00:00,975 - Train: 3.40% [164900/4848000] [34.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:00:35,691 - Train: 3.40% [165000/4848000] [34.0/1000.0] [batch_t 0.327 (0.347)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 17:01:10,889 - Train: 3.41% [165100/4848000] [34.1/1000.0] [batch_t 0.324 (0.352)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 17:01:43,804 - Train: 3.41% [165200/4848000] [34.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:02:18,658 - Train: 3.41% [165300/4848000] [34.1/1000.0] [batch_t 0.336 (0.348)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-03 17:02:51,357 - Train: 3.41% [165400/4848000] [34.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:03:26,970 - Train: 3.41% [165500/4848000] [34.1/1000.0] [batch_t 0.329 (0.356)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:03:59,663 - Train: 3.42% [165600/4848000] [34.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 17:04:35,774 - Train: 3.42% [165700/4848000] [34.2/1000.0] [batch_t 0.329 (0.361)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:05:10,562 - Train: 3.42% [165800/4848000] [34.2/1000.0] [batch_t 0.337 (0.348)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-03 17:05:43,285 - Train: 3.42% [165900/4848000] [34.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 17:06:22,295 - Train: 3.42% [166000/4848000] [34.2/1000.0] [batch_t 0.326 (0.390)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 17:06:54,974 - Train: 3.43% [166100/4848000] [34.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:07:28,810 - Train: 3.43% [166200/4848000] [34.3/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:08:01,565 - Train: 3.43% [166300/4848000] [34.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:08:36,942 - Train: 3.43% [166400/4848000] [34.3/1000.0] [batch_t 0.329 (0.354)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:09:11,884 - Train: 3.43% [166500/4848000] [34.3/1000.0] [batch_t 0.329 (0.349)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:09:44,627 - Train: 3.44% [166600/4848000] [34.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:10:21,421 - Train: 3.44% [166700/4848000] [34.4/1000.0] [batch_t 0.323 (0.368)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 17:10:54,086 - Train: 3.44% [166800/4848000] [34.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:11:28,852 - Train: 3.44% [166900/4848000] [34.4/1000.0] [batch_t 0.330 (0.348)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:12:01,612 - Train: 3.44% [167000/4848000] [34.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:12:38,404 - Train: 3.45% [167100/4848000] [34.5/1000.0] [batch_t 0.328 (0.368)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:13:12,775 - Train: 3.45% [167200/4848000] [34.5/1000.0] [batch_t 0.320 (0.344)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-03 17:13:45,423 - Train: 3.45% [167300/4848000] [34.5/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 17:14:20,701 - Train: 3.45% [167400/4848000] [34.5/1000.0] [batch_t 0.328 (0.353)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:14:53,400 - Train: 3.46% [167500/4848000] [34.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:15:30,356 - Train: 3.46% [167600/4848000] [34.6/1000.0] [batch_t 0.327 (0.369)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 17:16:03,099 - Train: 3.46% [167700/4848000] [34.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 17:16:37,982 - Train: 3.46% [167800/4848000] [34.6/1000.0] [batch_t 0.328 (0.349)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:17:13,570 - Train: 3.46% [167900/4848000] [34.6/1000.0] [batch_t 0.325 (0.356)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 17:17:52,382 - Train: 3.47% [168000/4848000] [34.7/1000.0] [batch_t 0.329 (0.388)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:18:28,844 - Train: 3.47% [168100/4848000] [34.7/1000.0] [batch_t 0.325 (0.365)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 17:19:01,598 - Train: 3.47% [168200/4848000] [34.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:19:37,328 - Train: 3.47% [168300/4848000] [34.7/1000.0] [batch_t 0.332 (0.357)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 17:20:14,737 - Train: 3.47% [168400/4848000] [34.7/1000.0] [batch_t 0.325 (0.374)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 17:20:47,441 - Train: 3.48% [168500/4848000] [34.8/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 17:21:23,151 - Train: 3.48% [168600/4848000] [34.8/1000.0] [batch_t 0.323 (0.357)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 17:21:55,835 - Train: 3.48% [168700/4848000] [34.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 17:22:31,704 - Train: 3.48% [168800/4848000] [34.8/1000.0] [batch_t 0.330 (0.359)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:23:04,418 - Train: 3.48% [168900/4848000] [34.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 17:23:40,750 - Train: 3.49% [169000/4848000] [34.9/1000.0] [batch_t 0.325 (0.363)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 17:24:31,767 - Train: 3.49% [169100/4848000] [34.9/1000.0] [batch_t 0.328 (0.510)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:25:05,444 - Train: 3.49% [169200/4848000] [34.9/1000.0] [batch_t 1.284 (0.337)] [data_t 0.957] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:25:38,167 - Train: 3.49% [169300/4848000] [34.9/1000.0] [batch_t 0.330 (0.327)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:26:12,679 - Train: 3.49% [169400/4848000] [34.9/1000.0] [batch_t 0.330 (0.345)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:26:45,473 - Train: 3.50% [169500/4848000] [35.0/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 17:27:22,902 - Train: 3.50% [169600/4848000] [35.0/1000.0] [batch_t 0.325 (0.374)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 17:27:49,063 - ==> Total time: 18:01:47 Eta: 20 days, 17:06:34 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 17:27:58,150 - Train: 3.50% [169700/4848000] [35.0/1000.0] [batch_t 0.328 (0.368)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:28:31,706 - Train: 3.50% [169800/4848000] [35.0/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:29:05,845 - Train: 3.50% [169900/4848000] [35.0/1000.0] [batch_t 1.745 (0.341)] [data_t 1.416] [optim_t 0.329] [lr 0.005000] 2024-04-03 17:29:38,533 - Train: 3.51% [170000/4848000] [35.1/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 17:30:12,516 - Train: 3.51% [170100/4848000] [35.1/1000.0] [batch_t 0.331 (0.340)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 17:30:45,210 - Train: 3.51% [170200/4848000] [35.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:31:18,127 - Train: 3.51% [170300/4848000] [35.1/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 17:31:50,864 - Train: 3.51% [170400/4848000] [35.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 17:32:24,623 - Train: 3.52% [170500/4848000] [35.2/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:32:57,356 - Train: 3.52% [170600/4848000] [35.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 17:33:31,372 - Train: 3.52% [170700/4848000] [35.2/1000.0] [batch_t 0.332 (0.340)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 17:34:04,464 - Train: 3.52% [170800/4848000] [35.2/1000.0] [batch_t 0.588 (0.331)] [data_t 0.262] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:34:37,838 - Train: 3.53% [170900/4848000] [35.3/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 17:35:10,768 - Train: 3.53% [171000/4848000] [35.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:35:43,413 - Train: 3.53% [171100/4848000] [35.3/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:36:17,166 - Train: 3.53% [171200/4848000] [35.3/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:36:49,859 - Train: 3.53% [171300/4848000] [35.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:37:24,866 - Train: 3.54% [171400/4848000] [35.4/1000.0] [batch_t 0.327 (0.350)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 17:37:57,599 - Train: 3.54% [171500/4848000] [35.4/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 17:38:31,055 - Train: 3.54% [171600/4848000] [35.4/1000.0] [batch_t 0.332 (0.334)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 17:39:03,788 - Train: 3.54% [171700/4848000] [35.4/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:39:38,195 - Train: 3.54% [171800/4848000] [35.4/1000.0] [batch_t 0.330 (0.344)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:40:12,031 - Train: 3.55% [171900/4848000] [35.5/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:40:44,777 - Train: 3.55% [172000/4848000] [35.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:41:17,919 - Train: 3.55% [172100/4848000] [35.5/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:42:11,552 - Train: 3.55% [172200/4848000] [35.5/1000.0] [batch_t 1.932 (0.536)] [data_t 1.604] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:43:09,323 - Train: 3.55% [172300/4848000] [35.5/1000.0] [batch_t 0.326 (0.578)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 17:43:54,527 - Train: 3.56% [172400/4848000] [35.6/1000.0] [batch_t 0.323 (0.452)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 17:44:31,963 - Train: 3.56% [172500/4848000] [35.6/1000.0] [batch_t 0.331 (0.374)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 17:45:05,879 - Train: 3.56% [172600/4848000] [35.6/1000.0] [batch_t 1.536 (0.339)] [data_t 1.213] [optim_t 0.323] [lr 0.005000] 2024-04-03 17:45:38,586 - Train: 3.56% [172700/4848000] [35.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 17:46:16,999 - Train: 3.56% [172800/4848000] [35.6/1000.0] [batch_t 0.326 (0.384)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 17:46:52,985 - Train: 3.57% [172900/4848000] [35.7/1000.0] [batch_t 0.328 (0.360)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:48:00,245 - Train: 3.57% [173000/4848000] [35.7/1000.0] [batch_t 0.327 (0.673)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:48:35,934 - Train: 3.57% [173100/4848000] [35.7/1000.0] [batch_t 0.328 (0.357)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:49:27,132 - Train: 3.57% [173200/4848000] [35.7/1000.0] [batch_t 0.329 (0.512)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:50:13,644 - Train: 3.57% [173300/4848000] [35.7/1000.0] [batch_t 0.328 (0.465)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:51:26,204 - Train: 3.58% [173400/4848000] [35.8/1000.0] [batch_t 0.327 (0.725)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 17:52:38,840 - Train: 3.58% [173500/4848000] [35.8/1000.0] [batch_t 0.328 (0.726)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:53:21,686 - Train: 3.58% [173600/4848000] [35.8/1000.0] [batch_t 0.365 (0.428)] [data_t 0.034] [optim_t 0.331] [lr 0.005000] 2024-04-03 17:53:54,545 - Train: 3.58% [173700/4848000] [35.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:54:33,331 - Train: 3.58% [173800/4848000] [35.8/1000.0] [batch_t 0.329 (0.388)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:55:32,262 - Train: 3.59% [173900/4848000] [35.9/1000.0] [batch_t 0.328 (0.589)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:56:07,135 - Train: 3.59% [174000/4848000] [35.9/1000.0] [batch_t 0.328 (0.349)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:56:39,882 - Train: 3.59% [174100/4848000] [35.9/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 17:57:15,985 - Train: 3.59% [174200/4848000] [35.9/1000.0] [batch_t 0.329 (0.361)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 17:57:51,580 - Train: 3.60% [174300/4848000] [36.0/1000.0] [batch_t 0.330 (0.356)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 17:58:34,629 - Train: 3.60% [174400/4848000] [36.0/1000.0] [batch_t 0.328 (0.430)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 17:59:21,722 - Train: 3.60% [174500/4848000] [36.0/1000.0] [batch_t 0.335 (0.471)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 17:59:39,727 - ==> Total time: 18:33:38 Eta: 20 days, 17:00:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 18:00:30,243 - Train: 3.60% [174600/4848000] [36.0/1000.0] [batch_t 0.322 (0.668)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-03 18:01:31,730 - Train: 3.60% [174700/4848000] [36.0/1000.0] [batch_t 0.328 (0.613)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:02:12,763 - Train: 3.61% [174800/4848000] [36.1/1000.0] [batch_t 0.327 (0.410)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 18:02:45,443 - Train: 3.61% [174900/4848000] [36.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:03:20,942 - Train: 3.61% [175000/4848000] [36.1/1000.0] [batch_t 0.331 (0.355)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 18:03:53,787 - Train: 3.61% [175100/4848000] [36.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:04:27,809 - Train: 3.61% [175200/4848000] [36.1/1000.0] [batch_t 0.323 (0.340)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 18:05:00,530 - Train: 3.62% [175300/4848000] [36.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:05:34,481 - Train: 3.62% [175400/4848000] [36.2/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:06:08,449 - Train: 3.62% [175500/4848000] [36.2/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:06:41,118 - Train: 3.62% [175600/4848000] [36.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 18:07:13,771 - Train: 3.62% [175700/4848000] [36.2/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:07:46,443 - Train: 3.63% [175800/4848000] [36.3/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 18:08:19,189 - Train: 3.63% [175900/4848000] [36.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 18:08:51,980 - Train: 3.63% [176000/4848000] [36.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:09:25,746 - Train: 3.63% [176100/4848000] [36.3/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:09:58,462 - Train: 3.63% [176200/4848000] [36.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:10:32,175 - Train: 3.64% [176300/4848000] [36.4/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:11:04,866 - Train: 3.64% [176400/4848000] [36.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:11:37,828 - Train: 3.64% [176500/4848000] [36.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:12:10,534 - Train: 3.64% [176600/4848000] [36.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:12:43,277 - Train: 3.64% [176700/4848000] [36.4/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 18:13:16,008 - Train: 3.65% [176800/4848000] [36.5/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 18:13:48,723 - Train: 3.65% [176900/4848000] [36.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:14:22,524 - Train: 3.65% [177000/4848000] [36.5/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 18:14:55,226 - Train: 3.65% [177100/4848000] [36.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:15:28,901 - Train: 3.66% [177200/4848000] [36.6/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:16:01,632 - Train: 3.66% [177300/4848000] [36.6/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 18:16:35,593 - Train: 3.66% [177400/4848000] [36.6/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:17:09,000 - Train: 3.66% [177500/4848000] [36.6/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:17:41,675 - Train: 3.66% [177600/4848000] [36.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 18:18:14,409 - Train: 3.67% [177700/4848000] [36.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 18:18:47,134 - Train: 3.67% [177800/4848000] [36.7/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 18:19:20,243 - Train: 3.67% [177900/4848000] [36.7/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 18:19:53,043 - Train: 3.67% [178000/4848000] [36.7/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 18:20:25,715 - Train: 3.67% [178100/4848000] [36.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 18:20:58,358 - Train: 3.68% [178200/4848000] [36.8/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:21:31,485 - Train: 3.68% [178300/4848000] [36.8/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 18:22:04,172 - Train: 3.68% [178400/4848000] [36.8/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 18:22:36,850 - Train: 3.68% [178500/4848000] [36.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:23:09,607 - Train: 3.68% [178600/4848000] [36.8/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 18:23:42,295 - Train: 3.69% [178700/4848000] [36.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:24:15,007 - Train: 3.69% [178800/4848000] [36.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:24:47,683 - Train: 3.69% [178900/4848000] [36.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:25:20,369 - Train: 3.69% [179000/4848000] [36.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:25:53,074 - Train: 3.69% [179100/4848000] [36.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:26:25,756 - Train: 3.70% [179200/4848000] [37.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 18:26:58,436 - Train: 3.70% [179300/4848000] [37.0/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 18:27:24,164 - ==> Total time: 19:01:22 Eta: 20 days, 15:06:40 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 18:27:33,514 - Train: 3.70% [179400/4848000] [37.0/1000.0] [batch_t 0.325 (0.337)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 18:28:06,205 - Train: 3.70% [179500/4848000] [37.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:28:38,931 - Train: 3.70% [179600/4848000] [37.0/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 18:29:11,621 - Train: 3.71% [179700/4848000] [37.1/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 18:29:44,376 - Train: 3.71% [179800/4848000] [37.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:30:17,159 - Train: 3.71% [179900/4848000] [37.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:30:49,857 - Train: 3.71% [180000/4848000] [37.1/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 18:31:22,550 - Train: 3.71% [180100/4848000] [37.1/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 18:31:55,330 - Train: 3.72% [180200/4848000] [37.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:32:31,677 - Train: 3.72% [180300/4848000] [37.2/1000.0] [batch_t 0.328 (0.363)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:33:04,389 - Train: 3.72% [180400/4848000] [37.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:33:37,126 - Train: 3.72% [180500/4848000] [37.2/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 18:34:10,656 - Train: 3.73% [180600/4848000] [37.3/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:34:43,470 - Train: 3.73% [180700/4848000] [37.3/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 18:35:18,504 - Train: 3.73% [180800/4848000] [37.3/1000.0] [batch_t 0.328 (0.350)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:35:51,183 - Train: 3.73% [180900/4848000] [37.3/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 18:36:25,840 - Train: 3.73% [181000/4848000] [37.3/1000.0] [batch_t 0.328 (0.346)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:36:58,518 - Train: 3.74% [181100/4848000] [37.4/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 18:37:33,554 - Train: 3.74% [181200/4848000] [37.4/1000.0] [batch_t 0.329 (0.350)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:38:09,095 - Train: 3.74% [181300/4848000] [37.4/1000.0] [batch_t 0.332 (0.355)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 18:38:41,830 - Train: 3.74% [181400/4848000] [37.4/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 18:39:14,572 - Train: 3.74% [181500/4848000] [37.4/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-03 18:39:47,317 - Train: 3.75% [181600/4848000] [37.5/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 18:40:20,044 - Train: 3.75% [181700/4848000] [37.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:40:52,707 - Train: 3.75% [181800/4848000] [37.5/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 18:41:25,405 - Train: 3.75% [181900/4848000] [37.5/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 18:41:58,134 - Train: 3.75% [182000/4848000] [37.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 18:42:30,994 - Train: 3.76% [182100/4848000] [37.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:43:03,675 - Train: 3.76% [182200/4848000] [37.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:43:36,352 - Train: 3.76% [182300/4848000] [37.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 18:44:10,984 - Train: 3.76% [182400/4848000] [37.6/1000.0] [batch_t 0.328 (0.346)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:44:43,680 - Train: 3.76% [182500/4848000] [37.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:45:18,635 - Train: 3.77% [182600/4848000] [37.7/1000.0] [batch_t 0.327 (0.349)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 18:45:51,374 - Train: 3.77% [182700/4848000] [37.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 18:46:25,178 - Train: 3.77% [182800/4848000] [37.7/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 18:46:57,862 - Train: 3.77% [182900/4848000] [37.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:47:31,292 - Train: 3.77% [183000/4848000] [37.7/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:48:03,977 - Train: 3.78% [183100/4848000] [37.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:48:36,697 - Train: 3.78% [183200/4848000] [37.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 18:49:09,532 - Train: 3.78% [183300/4848000] [37.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 18:49:42,228 - Train: 3.78% [183400/4848000] [37.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:50:14,895 - Train: 3.79% [183500/4848000] [37.9/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 18:50:47,676 - Train: 3.79% [183600/4848000] [37.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 18:51:20,372 - Train: 3.79% [183700/4848000] [37.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:51:53,066 - Train: 3.79% [183800/4848000] [37.9/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 18:52:25,735 - Train: 3.79% [183900/4848000] [37.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 18:52:58,446 - Train: 3.80% [184000/4848000] [38.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:53:31,177 - Train: 3.80% [184100/4848000] [38.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:54:03,884 - Train: 3.80% [184200/4848000] [38.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:54:11,732 - ==> Total time: 19:28:10 Eta: 20 days, 12:53:09 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 18:54:37,851 - Train: 3.80% [184300/4848000] [38.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 18:55:10,535 - Train: 3.80% [184400/4848000] [38.0/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 18:55:43,261 - Train: 3.81% [184500/4848000] [38.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 18:56:16,020 - Train: 3.81% [184600/4848000] [38.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 18:56:48,759 - Train: 3.81% [184700/4848000] [38.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 18:57:21,482 - Train: 3.81% [184800/4848000] [38.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:57:54,366 - Train: 3.81% [184900/4848000] [38.1/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 18:58:27,484 - Train: 3.82% [185000/4848000] [38.2/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 18:59:00,151 - Train: 3.82% [185100/4848000] [38.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 18:59:32,894 - Train: 3.82% [185200/4848000] [38.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:00:05,666 - Train: 3.82% [185300/4848000] [38.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:00:38,334 - Train: 3.82% [185400/4848000] [38.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 19:01:11,054 - Train: 3.83% [185500/4848000] [38.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:01:43,770 - Train: 3.83% [185600/4848000] [38.3/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:02:16,539 - Train: 3.83% [185700/4848000] [38.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:02:49,310 - Train: 3.83% [185800/4848000] [38.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:03:22,024 - Train: 3.83% [185900/4848000] [38.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:03:54,700 - Train: 3.84% [186000/4848000] [38.4/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 19:04:27,369 - Train: 3.84% [186100/4848000] [38.4/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 19:05:00,067 - Train: 3.84% [186200/4848000] [38.4/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 19:05:32,951 - Train: 3.84% [186300/4848000] [38.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:06:05,669 - Train: 3.84% [186400/4848000] [38.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:06:38,373 - Train: 3.85% [186500/4848000] [38.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:07:11,139 - Train: 3.85% [186600/4848000] [38.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 19:07:43,841 - Train: 3.85% [186700/4848000] [38.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:08:16,586 - Train: 3.85% [186800/4848000] [38.5/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 19:08:49,319 - Train: 3.86% [186900/4848000] [38.6/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:09:22,175 - Train: 3.86% [187000/4848000] [38.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:09:54,876 - Train: 3.86% [187100/4848000] [38.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 19:10:27,575 - Train: 3.86% [187200/4848000] [38.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:11:00,293 - Train: 3.86% [187300/4848000] [38.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:11:33,838 - Train: 3.87% [187400/4848000] [38.7/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:12:07,072 - Train: 3.87% [187500/4848000] [38.7/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:12:39,837 - Train: 3.87% [187600/4848000] [38.7/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 19:13:12,583 - Train: 3.87% [187700/4848000] [38.7/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:13:59,314 - Train: 3.87% [187800/4848000] [38.7/1000.0] [batch_t 0.328 (0.467)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:14:36,619 - Train: 3.88% [187900/4848000] [38.8/1000.0] [batch_t 0.325 (0.373)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 19:15:11,990 - Train: 3.88% [188000/4848000] [38.8/1000.0] [batch_t 0.328 (0.354)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:15:44,771 - Train: 3.88% [188100/4848000] [38.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:16:20,377 - Train: 3.88% [188200/4848000] [38.8/1000.0] [batch_t 0.327 (0.356)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:16:53,121 - Train: 3.88% [188300/4848000] [38.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:17:26,677 - Train: 3.89% [188400/4848000] [38.9/1000.0] [batch_t 0.331 (0.335)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 19:17:59,408 - Train: 3.89% [188500/4848000] [38.9/1000.0] [batch_t 0.333 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-03 19:18:32,162 - Train: 3.89% [188600/4848000] [38.9/1000.0] [batch_t 0.334 (0.327)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 19:19:04,914 - Train: 3.89% [188700/4848000] [38.9/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 19:19:37,681 - Train: 3.89% [188800/4848000] [38.9/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 19:20:10,451 - Train: 3.90% [188900/4848000] [39.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 19:20:43,160 - Train: 3.90% [189000/4848000] [39.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:21:06,732 - ==> Total time: 19:55:05 Eta: 20 days, 10:48:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 19:21:17,475 - Train: 3.90% [189100/4848000] [39.0/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:21:50,164 - Train: 3.90% [189200/4848000] [39.0/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 19:22:22,848 - Train: 3.90% [189300/4848000] [39.0/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 19:22:55,547 - Train: 3.91% [189400/4848000] [39.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 19:23:28,244 - Train: 3.91% [189500/4848000] [39.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:24:00,934 - Train: 3.91% [189600/4848000] [39.1/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 19:24:33,609 - Train: 3.91% [189700/4848000] [39.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 19:25:06,285 - Train: 3.92% [189800/4848000] [39.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:25:38,956 - Train: 3.92% [189900/4848000] [39.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:26:11,639 - Train: 3.92% [190000/4848000] [39.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:26:44,337 - Train: 3.92% [190100/4848000] [39.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:27:17,012 - Train: 3.92% [190200/4848000] [39.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 19:27:49,665 - Train: 3.93% [190300/4848000] [39.3/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 19:28:22,369 - Train: 3.93% [190400/4848000] [39.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:28:55,040 - Train: 3.93% [190500/4848000] [39.3/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 19:29:27,864 - Train: 3.93% [190600/4848000] [39.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:30:00,526 - Train: 3.93% [190700/4848000] [39.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:30:33,213 - Train: 3.94% [190800/4848000] [39.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:31:05,870 - Train: 3.94% [190900/4848000] [39.4/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 19:31:38,541 - Train: 3.94% [191000/4848000] [39.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:32:12,184 - Train: 3.94% [191100/4848000] [39.4/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:32:44,839 - Train: 3.94% [191200/4848000] [39.4/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:33:18,869 - Train: 3.95% [191300/4848000] [39.5/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 19:33:51,551 - Train: 3.95% [191400/4848000] [39.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:34:25,021 - Train: 3.95% [191500/4848000] [39.5/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:34:57,736 - Train: 3.95% [191600/4848000] [39.5/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 19:35:30,452 - Train: 3.95% [191700/4848000] [39.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 19:36:03,159 - Train: 3.96% [191800/4848000] [39.6/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:36:35,842 - Train: 3.96% [191900/4848000] [39.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:37:09,609 - Train: 3.96% [192000/4848000] [39.6/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:37:42,295 - Train: 3.96% [192100/4848000] [39.6/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 19:38:15,259 - Train: 3.96% [192200/4848000] [39.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:38:47,945 - Train: 3.97% [192300/4848000] [39.7/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 19:39:21,156 - Train: 3.97% [192400/4848000] [39.7/1000.0] [batch_t 0.326 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 19:39:53,837 - Train: 3.97% [192500/4848000] [39.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:40:26,511 - Train: 3.97% [192600/4848000] [39.7/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:40:59,147 - Train: 3.97% [192700/4848000] [39.7/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:41:31,817 - Train: 3.98% [192800/4848000] [39.8/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 19:42:04,501 - Train: 3.98% [192900/4848000] [39.8/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:42:37,206 - Train: 3.98% [193000/4848000] [39.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 19:43:09,887 - Train: 3.98% [193100/4848000] [39.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:43:42,554 - Train: 3.99% [193200/4848000] [39.9/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-03 19:44:15,284 - Train: 3.99% [193300/4848000] [39.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:44:47,935 - Train: 3.99% [193400/4848000] [39.9/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:45:20,810 - Train: 3.99% [193500/4848000] [39.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:45:53,521 - Train: 3.99% [193600/4848000] [39.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:46:26,524 - Train: 4.00% [193700/4848000] [40.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:46:59,244 - Train: 4.00% [193800/4848000] [40.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 19:47:31,928 - Train: 4.00% [193900/4848000] [40.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:47:38,472 - ==> Total time: 20:21:36 Eta: 20 days, 8:38:47 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 19:48:06,132 - Train: 4.00% [194000/4848000] [40.0/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 19:48:38,809 - Train: 4.00% [194100/4848000] [40.0/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 19:49:11,484 - Train: 4.01% [194200/4848000] [40.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:49:44,172 - Train: 4.01% [194300/4848000] [40.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:50:16,916 - Train: 4.01% [194400/4848000] [40.1/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 19:50:49,633 - Train: 4.01% [194500/4848000] [40.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 19:51:22,306 - Train: 4.01% [194600/4848000] [40.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:51:54,960 - Train: 4.02% [194700/4848000] [40.2/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:52:27,758 - Train: 4.02% [194800/4848000] [40.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 19:53:00,474 - Train: 4.02% [194900/4848000] [40.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:53:33,174 - Train: 4.02% [195000/4848000] [40.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:54:05,859 - Train: 4.02% [195100/4848000] [40.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 19:54:38,526 - Train: 4.03% [195200/4848000] [40.3/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 19:55:11,263 - Train: 4.03% [195300/4848000] [40.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 19:55:43,918 - Train: 4.03% [195400/4848000] [40.3/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 19:56:17,669 - Train: 4.03% [195500/4848000] [40.3/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:56:50,341 - Train: 4.03% [195600/4848000] [40.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 19:57:23,028 - Train: 4.04% [195700/4848000] [40.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:57:55,720 - Train: 4.04% [195800/4848000] [40.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 19:58:28,430 - Train: 4.04% [195900/4848000] [40.4/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 19:59:01,121 - Train: 4.04% [196000/4848000] [40.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 19:59:35,477 - Train: 4.04% [196100/4848000] [40.4/1000.0] [batch_t 0.328 (0.343)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:00:09,994 - Train: 4.05% [196200/4848000] [40.5/1000.0] [batch_t 0.331 (0.345)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 20:00:42,665 - Train: 4.05% [196300/4848000] [40.5/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 20:01:15,333 - Train: 4.05% [196400/4848000] [40.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:01:48,057 - Train: 4.05% [196500/4848000] [40.5/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:02:20,813 - Train: 4.06% [196600/4848000] [40.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:02:53,484 - Train: 4.06% [196700/4848000] [40.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:03:26,190 - Train: 4.06% [196800/4848000] [40.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:03:58,880 - Train: 4.06% [196900/4848000] [40.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:04:31,645 - Train: 4.06% [197000/4848000] [40.6/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 20:05:04,381 - Train: 4.07% [197100/4848000] [40.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:05:37,101 - Train: 4.07% [197200/4848000] [40.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:06:09,802 - Train: 4.07% [197300/4848000] [40.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:06:42,454 - Train: 4.07% [197400/4848000] [40.7/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:07:15,137 - Train: 4.07% [197500/4848000] [40.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:07:47,983 - Train: 4.08% [197600/4848000] [40.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:08:20,652 - Train: 4.08% [197700/4848000] [40.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:08:53,393 - Train: 4.08% [197800/4848000] [40.8/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 20:09:26,108 - Train: 4.08% [197900/4848000] [40.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:09:58,778 - Train: 4.08% [198000/4848000] [40.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:10:31,457 - Train: 4.09% [198100/4848000] [40.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:11:04,119 - Train: 4.09% [198200/4848000] [40.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:11:36,885 - Train: 4.09% [198300/4848000] [40.9/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 20:12:09,606 - Train: 4.09% [198400/4848000] [40.9/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 20:12:42,315 - Train: 4.09% [198500/4848000] [40.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:13:15,013 - Train: 4.10% [198600/4848000] [41.0/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 20:13:47,709 - Train: 4.10% [198700/4848000] [41.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:14:10,840 - ==> Total time: 20:48:09 Eta: 20 days, 6:34:39 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 20:14:22,630 - Train: 4.10% [198800/4848000] [41.0/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:14:55,297 - Train: 4.10% [198900/4848000] [41.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:15:28,117 - Train: 4.10% [199000/4848000] [41.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:16:00,829 - Train: 4.11% [199100/4848000] [41.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:16:33,510 - Train: 4.11% [199200/4848000] [41.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:17:06,206 - Train: 4.11% [199300/4848000] [41.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:17:38,918 - Train: 4.11% [199400/4848000] [41.1/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:18:11,634 - Train: 4.12% [199500/4848000] [41.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 20:18:44,295 - Train: 4.12% [199600/4848000] [41.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:19:16,984 - Train: 4.12% [199700/4848000] [41.2/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:19:49,631 - Train: 4.12% [199800/4848000] [41.2/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:20:22,328 - Train: 4.12% [199900/4848000] [41.2/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 20:20:55,008 - Train: 4.13% [200000/4848000] [41.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:21:27,674 - Train: 4.13% [200100/4848000] [41.3/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 20:22:00,376 - Train: 4.13% [200200/4848000] [41.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:22:33,093 - Train: 4.13% [200300/4848000] [41.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:23:05,832 - Train: 4.13% [200400/4848000] [41.3/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 20:23:38,579 - Train: 4.14% [200500/4848000] [41.4/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:24:11,286 - Train: 4.14% [200600/4848000] [41.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:24:44,038 - Train: 4.14% [200700/4848000] [41.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:25:16,786 - Train: 4.14% [200800/4848000] [41.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:25:49,489 - Train: 4.14% [200900/4848000] [41.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:26:22,176 - Train: 4.15% [201000/4848000] [41.5/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 20:26:54,874 - Train: 4.15% [201100/4848000] [41.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:27:27,572 - Train: 4.15% [201200/4848000] [41.5/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 20:28:00,264 - Train: 4.15% [201300/4848000] [41.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:28:34,474 - Train: 4.15% [201400/4848000] [41.5/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:29:07,179 - Train: 4.16% [201500/4848000] [41.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 20:29:39,888 - Train: 4.16% [201600/4848000] [41.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:30:12,589 - Train: 4.16% [201700/4848000] [41.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:30:45,422 - Train: 4.16% [201800/4848000] [41.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:31:19,525 - Train: 4.16% [201900/4848000] [41.6/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:31:52,210 - Train: 4.17% [202000/4848000] [41.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:32:25,534 - Train: 4.17% [202100/4848000] [41.7/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:32:58,278 - Train: 4.17% [202200/4848000] [41.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:33:30,975 - Train: 4.17% [202300/4848000] [41.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 20:34:03,725 - Train: 4.17% [202400/4848000] [41.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:34:36,455 - Train: 4.18% [202500/4848000] [41.8/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 20:35:09,127 - Train: 4.18% [202600/4848000] [41.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 20:35:41,813 - Train: 4.18% [202700/4848000] [41.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:36:14,527 - Train: 4.18% [202800/4848000] [41.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:36:47,219 - Train: 4.19% [202900/4848000] [41.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:37:19,923 - Train: 4.19% [203000/4848000] [41.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:37:52,649 - Train: 4.19% [203100/4848000] [41.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:38:26,938 - Train: 4.19% [203200/4848000] [41.9/1000.0] [batch_t 0.329 (0.343)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:38:59,758 - Train: 4.19% [203300/4848000] [41.9/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 20:39:34,090 - Train: 4.20% [203400/4848000] [42.0/1000.0] [batch_t 0.328 (0.343)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:40:07,951 - Train: 4.20% [203500/4848000] [42.0/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:40:40,673 - Train: 4.20% [203600/4848000] [42.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:40:45,905 - ==> Total time: 21:14:44 Eta: 20 days, 4:36:12 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 20:41:15,123 - Train: 4.20% [203700/4848000] [42.0/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:41:47,810 - Train: 4.20% [203800/4848000] [42.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:42:20,537 - Train: 4.21% [203900/4848000] [42.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:42:53,262 - Train: 4.21% [204000/4848000] [42.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:43:25,970 - Train: 4.21% [204100/4848000] [42.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:43:58,647 - Train: 4.21% [204200/4848000] [42.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:44:31,354 - Train: 4.21% [204300/4848000] [42.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:45:04,158 - Train: 4.22% [204400/4848000] [42.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:45:36,899 - Train: 4.22% [204500/4848000] [42.2/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:46:09,735 - Train: 4.22% [204600/4848000] [42.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:46:42,410 - Train: 4.22% [204700/4848000] [42.2/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:47:15,833 - Train: 4.22% [204800/4848000] [42.2/1000.0] [batch_t 0.325 (0.334)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 20:47:48,521 - Train: 4.23% [204900/4848000] [42.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:48:21,221 - Train: 4.23% [205000/4848000] [42.3/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 20:48:53,941 - Train: 4.23% [205100/4848000] [42.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:49:27,738 - Train: 4.23% [205200/4848000] [42.3/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:50:00,413 - Train: 4.23% [205300/4848000] [42.3/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:50:33,192 - Train: 4.24% [205400/4848000] [42.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 20:51:06,903 - Train: 4.24% [205500/4848000] [42.4/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:51:39,592 - Train: 4.24% [205600/4848000] [42.4/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 20:52:12,393 - Train: 4.24% [205700/4848000] [42.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:52:45,093 - Train: 4.25% [205800/4848000] [42.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:53:17,767 - Train: 4.25% [205900/4848000] [42.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:53:50,589 - Train: 4.25% [206000/4848000] [42.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:54:23,269 - Train: 4.25% [206100/4848000] [42.5/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 20:54:55,970 - Train: 4.25% [206200/4848000] [42.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:55:30,012 - Train: 4.26% [206300/4848000] [42.6/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:56:02,698 - Train: 4.26% [206400/4848000] [42.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:56:35,482 - Train: 4.26% [206500/4848000] [42.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:57:08,155 - Train: 4.26% [206600/4848000] [42.6/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 20:57:40,786 - Train: 4.26% [206700/4848000] [42.6/1000.0] [batch_t 0.325 (0.326)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 20:58:13,724 - Train: 4.27% [206800/4848000] [42.7/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 20:58:46,394 - Train: 4.27% [206900/4848000] [42.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 20:59:19,089 - Train: 4.27% [207000/4848000] [42.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 20:59:51,794 - Train: 4.27% [207100/4848000] [42.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:00:24,476 - Train: 4.27% [207200/4848000] [42.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:00:57,148 - Train: 4.28% [207300/4848000] [42.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 21:01:29,858 - Train: 4.28% [207400/4848000] [42.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 21:02:02,698 - Train: 4.28% [207500/4848000] [42.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:02:35,390 - Train: 4.28% [207600/4848000] [42.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 21:03:08,072 - Train: 4.28% [207700/4848000] [42.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 21:03:40,794 - Train: 4.29% [207800/4848000] [42.9/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 21:04:13,476 - Train: 4.29% [207900/4848000] [42.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 21:04:46,172 - Train: 4.29% [208000/4848000] [42.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 21:05:18,844 - Train: 4.29% [208100/4848000] [42.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:05:51,523 - Train: 4.29% [208200/4848000] [42.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:06:25,764 - Train: 4.30% [208300/4848000] [43.0/1000.0] [batch_t 0.326 (0.342)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 21:06:58,438 - Train: 4.30% [208400/4848000] [43.0/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 21:07:20,534 - ==> Total time: 21:41:19 Eta: 20 days, 2:41:52 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 21:07:33,638 - Train: 4.30% [208500/4848000] [43.0/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 21:08:06,312 - Train: 4.30% [208600/4848000] [43.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:08:39,008 - Train: 4.30% [208700/4848000] [43.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:09:11,793 - Train: 4.31% [208800/4848000] [43.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:09:44,483 - Train: 4.31% [208900/4848000] [43.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 21:10:17,153 - Train: 4.31% [209000/4848000] [43.1/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 21:10:49,841 - Train: 4.31% [209100/4848000] [43.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:11:23,770 - Train: 4.32% [209200/4848000] [43.2/1000.0] [batch_t 0.324 (0.339)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 21:11:56,439 - Train: 4.32% [209300/4848000] [43.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:12:29,151 - Train: 4.32% [209400/4848000] [43.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:13:01,865 - Train: 4.32% [209500/4848000] [43.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 21:13:35,412 - Train: 4.32% [209600/4848000] [43.2/1000.0] [batch_t 0.325 (0.335)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 21:14:08,265 - Train: 4.33% [209700/4848000] [43.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:14:41,015 - Train: 4.33% [209800/4848000] [43.3/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 21:15:14,775 - Train: 4.33% [209900/4848000] [43.3/1000.0] [batch_t 0.326 (0.337)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 21:15:47,489 - Train: 4.33% [210000/4848000] [43.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:16:20,829 - Train: 4.33% [210100/4848000] [43.3/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 21:16:53,648 - Train: 4.34% [210200/4848000] [43.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:17:26,501 - Train: 4.34% [210300/4848000] [43.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:17:59,217 - Train: 4.34% [210400/4848000] [43.4/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 21:18:32,656 - Train: 4.34% [210500/4848000] [43.4/1000.0] [batch_t 0.332 (0.334)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 21:19:06,156 - Train: 4.34% [210600/4848000] [43.4/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:19:38,906 - Train: 4.35% [210700/4848000] [43.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:20:11,579 - Train: 4.35% [210800/4848000] [43.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 21:20:44,269 - Train: 4.35% [210900/4848000] [43.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:21:16,985 - Train: 4.35% [211000/4848000] [43.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 21:21:49,713 - Train: 4.35% [211100/4848000] [43.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:22:22,399 - Train: 4.36% [211200/4848000] [43.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 21:22:55,104 - Train: 4.36% [211300/4848000] [43.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:23:27,904 - Train: 4.36% [211400/4848000] [43.6/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 21:24:00,574 - Train: 4.36% [211500/4848000] [43.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 21:24:33,543 - Train: 4.36% [211600/4848000] [43.6/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:25:07,738 - Train: 4.37% [211700/4848000] [43.7/1000.0] [batch_t 0.330 (0.342)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:25:40,438 - Train: 4.37% [211800/4848000] [43.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:26:13,935 - Train: 4.37% [211900/4848000] [43.7/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:26:46,668 - Train: 4.37% [212000/4848000] [43.7/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 21:27:19,413 - Train: 4.38% [212100/4848000] [43.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:27:52,086 - Train: 4.38% [212200/4848000] [43.8/1000.0] [batch_t 0.321 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-03 21:28:25,819 - Train: 4.38% [212300/4848000] [43.8/1000.0] [batch_t 0.326 (0.337)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 21:28:58,504 - Train: 4.38% [212400/4848000] [43.8/1000.0] [batch_t 0.335 (0.327)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-03 21:29:32,331 - Train: 4.38% [212500/4848000] [43.8/1000.0] [batch_t 0.332 (0.338)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 21:30:05,213 - Train: 4.39% [212600/4848000] [43.9/1000.0] [batch_t 0.448 (0.329)] [data_t 0.121] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:30:37,959 - Train: 4.39% [212700/4848000] [43.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:31:11,377 - Train: 4.39% [212800/4848000] [43.9/1000.0] [batch_t 0.325 (0.334)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 21:31:44,126 - Train: 4.39% [212900/4848000] [43.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 21:32:18,132 - Train: 4.39% [213000/4848000] [43.9/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:32:50,967 - Train: 4.40% [213100/4848000] [44.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:33:23,751 - Train: 4.40% [213200/4848000] [44.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:33:56,490 - Train: 4.40% [213300/4848000] [44.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:34:00,428 - ==> Total time: 22:07:58 Eta: 20 days, 0:53:25 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 21:34:33,236 - Train: 4.40% [213400/4848000] [44.0/1000.0] [batch_t 0.326 (0.356)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 21:35:06,109 - Train: 4.40% [213500/4848000] [44.0/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 21:35:38,820 - Train: 4.41% [213600/4848000] [44.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:36:12,663 - Train: 4.41% [213700/4848000] [44.1/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:36:45,319 - Train: 4.41% [213800/4848000] [44.1/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:37:18,058 - Train: 4.41% [213900/4848000] [44.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:37:50,752 - Train: 4.41% [214000/4848000] [44.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:38:24,976 - Train: 4.42% [214100/4848000] [44.2/1000.0] [batch_t 0.324 (0.342)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 21:38:57,637 - Train: 4.42% [214200/4848000] [44.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:39:30,517 - Train: 4.42% [214300/4848000] [44.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:40:03,307 - Train: 4.42% [214400/4848000] [44.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:40:37,382 - Train: 4.42% [214500/4848000] [44.2/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:41:11,024 - Train: 4.43% [214600/4848000] [44.3/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 21:41:43,712 - Train: 4.43% [214700/4848000] [44.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:42:17,019 - Train: 4.43% [214800/4848000] [44.3/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:42:49,689 - Train: 4.43% [214900/4848000] [44.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:43:22,346 - Train: 4.43% [215000/4848000] [44.3/1000.0] [batch_t 0.326 (0.326)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 21:43:55,062 - Train: 4.44% [215100/4848000] [44.4/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:44:28,049 - Train: 4.44% [215200/4848000] [44.4/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 21:45:00,780 - Train: 4.44% [215300/4848000] [44.4/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 21:45:35,061 - Train: 4.44% [215400/4848000] [44.4/1000.0] [batch_t 0.327 (0.343)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 21:46:12,013 - Train: 4.45% [215500/4848000] [44.5/1000.0] [batch_t 0.327 (0.369)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-03 21:46:44,672 - Train: 4.45% [215600/4848000] [44.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 21:47:21,787 - Train: 4.45% [215700/4848000] [44.5/1000.0] [batch_t 0.329 (0.371)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:47:54,573 - Train: 4.45% [215800/4848000] [44.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:48:33,291 - Train: 4.45% [215900/4848000] [44.5/1000.0] [batch_t 0.329 (0.387)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:49:07,007 - Train: 4.46% [216000/4848000] [44.6/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:49:39,713 - Train: 4.46% [216100/4848000] [44.6/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 21:50:13,993 - Train: 4.46% [216200/4848000] [44.6/1000.0] [batch_t 0.326 (0.343)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 21:50:46,687 - Train: 4.46% [216300/4848000] [44.6/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-03 21:51:19,367 - Train: 4.46% [216400/4848000] [44.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:51:52,034 - Train: 4.47% [216500/4848000] [44.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 21:52:24,810 - Train: 4.47% [216600/4848000] [44.7/1000.0] [batch_t 0.321 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-03 21:53:01,986 - Train: 4.47% [216700/4848000] [44.7/1000.0] [batch_t 0.328 (0.372)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:54:01,693 - Train: 4.47% [216800/4848000] [44.7/1000.0] [batch_t 0.329 (0.597)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:54:38,392 - Train: 4.47% [216900/4848000] [44.7/1000.0] [batch_t 0.330 (0.367)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 21:55:11,861 - Train: 4.48% [217000/4848000] [44.8/1000.0] [batch_t 0.325 (0.335)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 21:55:44,562 - Train: 4.48% [217100/4848000] [44.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:56:17,272 - Train: 4.48% [217200/4848000] [44.8/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 21:56:50,085 - Train: 4.48% [217300/4848000] [44.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:57:22,791 - Train: 4.48% [217400/4848000] [44.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 21:57:55,481 - Train: 4.49% [217500/4848000] [44.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 21:58:28,178 - Train: 4.49% [217600/4848000] [44.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 21:59:00,893 - Train: 4.49% [217700/4848000] [44.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 21:59:33,616 - Train: 4.49% [217800/4848000] [44.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:00:06,341 - Train: 4.49% [217900/4848000] [44.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:00:39,064 - Train: 4.50% [218000/4848000] [45.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:01:11,802 - Train: 4.50% [218100/4848000] [45.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:01:31,418 - ==> Total time: 22:35:29 Eta: 19 days, 23:26:41 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 22:01:45,840 - Train: 4.50% [218200/4848000] [45.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 22:02:18,553 - Train: 4.50% [218300/4848000] [45.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:02:51,239 - Train: 4.50% [218400/4848000] [45.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:03:24,153 - Train: 4.51% [218500/4848000] [45.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 22:03:56,958 - Train: 4.51% [218600/4848000] [45.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:04:29,642 - Train: 4.51% [218700/4848000] [45.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:05:02,348 - Train: 4.51% [218800/4848000] [45.1/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:05:35,554 - Train: 4.52% [218900/4848000] [45.2/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:06:08,957 - Train: 4.52% [219000/4848000] [45.2/1000.0] [batch_t 0.330 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:06:41,620 - Train: 4.52% [219100/4848000] [45.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:07:14,320 - Train: 4.52% [219200/4848000] [45.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:07:47,003 - Train: 4.52% [219300/4848000] [45.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:08:19,639 - Train: 4.53% [219400/4848000] [45.3/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:08:52,322 - Train: 4.53% [219500/4848000] [45.3/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 22:09:25,407 - Train: 4.53% [219600/4848000] [45.3/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:09:58,185 - Train: 4.53% [219700/4848000] [45.3/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 22:10:31,768 - Train: 4.53% [219800/4848000] [45.3/1000.0] [batch_t 0.331 (0.336)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 22:11:04,456 - Train: 4.54% [219900/4848000] [45.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:11:37,294 - Train: 4.54% [220000/4848000] [45.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 22:12:10,553 - Train: 4.54% [220100/4848000] [45.4/1000.0] [batch_t 0.334 (0.333)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-03 22:12:43,230 - Train: 4.54% [220200/4848000] [45.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:13:15,923 - Train: 4.54% [220300/4848000] [45.4/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 22:13:48,726 - Train: 4.55% [220400/4848000] [45.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:14:22,057 - Train: 4.55% [220500/4848000] [45.5/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:14:54,754 - Train: 4.55% [220600/4848000] [45.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:15:27,439 - Train: 4.55% [220700/4848000] [45.5/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:16:00,118 - Train: 4.55% [220800/4848000] [45.5/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 22:16:32,868 - Train: 4.56% [220900/4848000] [45.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:17:06,551 - Train: 4.56% [221000/4848000] [45.6/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:17:39,206 - Train: 4.56% [221100/4848000] [45.6/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:18:11,914 - Train: 4.56% [221200/4848000] [45.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:18:44,582 - Train: 4.56% [221300/4848000] [45.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:19:17,388 - Train: 4.57% [221400/4848000] [45.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:19:50,088 - Train: 4.57% [221500/4848000] [45.7/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 22:20:22,805 - Train: 4.57% [221600/4848000] [45.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:20:55,536 - Train: 4.57% [221700/4848000] [45.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:21:29,642 - Train: 4.58% [221800/4848000] [45.8/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:22:02,324 - Train: 4.58% [221900/4848000] [45.8/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 22:22:35,020 - Train: 4.58% [222000/4848000] [45.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:23:07,718 - Train: 4.58% [222100/4848000] [45.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:23:40,466 - Train: 4.58% [222200/4848000] [45.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:24:13,160 - Train: 4.59% [222300/4848000] [45.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:24:45,836 - Train: 4.59% [222400/4848000] [45.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:25:18,509 - Train: 4.59% [222500/4848000] [45.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:25:51,171 - Train: 4.59% [222600/4848000] [45.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:26:23,847 - Train: 4.59% [222700/4848000] [45.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:26:56,508 - Train: 4.60% [222800/4848000] [46.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:27:29,651 - Train: 4.60% [222900/4848000] [46.0/1000.0] [batch_t 0.325 (0.331)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 22:28:02,347 - Train: 4.60% [223000/4848000] [46.0/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 22:28:05,143 - ==> Total time: 23:02:03 Eta: 19 days, 21:42:44 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 22:28:37,124 - Train: 4.60% [223100/4848000] [46.0/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:29:09,827 - Train: 4.60% [223200/4848000] [46.0/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 22:29:42,487 - Train: 4.61% [223300/4848000] [46.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:30:15,156 - Train: 4.61% [223400/4848000] [46.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:30:47,848 - Train: 4.61% [223500/4848000] [46.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:31:20,555 - Train: 4.61% [223600/4848000] [46.1/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 22:31:53,266 - Train: 4.61% [223700/4848000] [46.1/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 22:32:25,937 - Train: 4.62% [223800/4848000] [46.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 22:32:58,672 - Train: 4.62% [223900/4848000] [46.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:33:31,349 - Train: 4.62% [224000/4848000] [46.2/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:34:03,992 - Train: 4.62% [224100/4848000] [46.2/1000.0] [batch_t 0.330 (0.326)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:34:36,801 - Train: 4.62% [224200/4848000] [46.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 22:35:09,517 - Train: 4.63% [224300/4848000] [46.3/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 22:35:42,236 - Train: 4.63% [224400/4848000] [46.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:36:14,948 - Train: 4.63% [224500/4848000] [46.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:36:47,637 - Train: 4.63% [224600/4848000] [46.3/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 22:37:20,342 - Train: 4.63% [224700/4848000] [46.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:37:53,069 - Train: 4.64% [224800/4848000] [46.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:38:25,828 - Train: 4.64% [224900/4848000] [46.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:38:58,575 - Train: 4.64% [225000/4848000] [46.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:39:31,343 - Train: 4.64% [225100/4848000] [46.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 22:40:04,078 - Train: 4.65% [225200/4848000] [46.5/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 22:40:36,914 - Train: 4.65% [225300/4848000] [46.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:41:09,762 - Train: 4.65% [225400/4848000] [46.5/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 22:41:42,504 - Train: 4.65% [225500/4848000] [46.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:42:15,410 - Train: 4.65% [225600/4848000] [46.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:42:48,117 - Train: 4.66% [225700/4848000] [46.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 22:43:20,860 - Train: 4.66% [225800/4848000] [46.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:43:53,609 - Train: 4.66% [225900/4848000] [46.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 22:44:26,334 - Train: 4.66% [226000/4848000] [46.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 22:44:59,072 - Train: 4.66% [226100/4848000] [46.6/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:45:31,791 - Train: 4.67% [226200/4848000] [46.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:46:04,526 - Train: 4.67% [226300/4848000] [46.7/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 22:46:38,213 - Train: 4.67% [226400/4848000] [46.7/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:47:10,933 - Train: 4.67% [226500/4848000] [46.7/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 22:47:43,633 - Train: 4.67% [226600/4848000] [46.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:48:17,284 - Train: 4.68% [226700/4848000] [46.8/1000.0] [batch_t 0.326 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:48:50,007 - Train: 4.68% [226800/4848000] [46.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:49:22,728 - Train: 4.68% [226900/4848000] [46.8/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 22:49:55,555 - Train: 4.68% [227000/4848000] [46.8/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 22:50:28,301 - Train: 4.68% [227100/4848000] [46.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:51:01,048 - Train: 4.69% [227200/4848000] [46.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:51:33,754 - Train: 4.69% [227300/4848000] [46.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:52:07,476 - Train: 4.69% [227400/4848000] [46.9/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:52:40,217 - Train: 4.69% [227500/4848000] [46.9/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 22:53:12,945 - Train: 4.69% [227600/4848000] [46.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:53:45,640 - Train: 4.70% [227700/4848000] [47.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:54:18,400 - Train: 4.70% [227800/4848000] [47.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:54:36,882 - ==> Total time: 23:28:35 Eta: 19 days, 20:01:24 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 22:54:52,644 - Train: 4.70% [227900/4848000] [47.0/1000.0] [batch_t 0.324 (0.331)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 22:55:26,693 - Train: 4.70% [228000/4848000] [47.0/1000.0] [batch_t 0.330 (0.340)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 22:55:59,386 - Train: 4.71% [228100/4848000] [47.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:56:33,319 - Train: 4.71% [228200/4848000] [47.1/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 22:57:06,968 - Train: 4.71% [228300/4848000] [47.1/1000.0] [batch_t 0.323 (0.336)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 22:57:39,820 - Train: 4.71% [228400/4848000] [47.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 22:58:12,524 - Train: 4.71% [228500/4848000] [47.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 22:58:45,236 - Train: 4.72% [228600/4848000] [47.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 22:59:17,964 - Train: 4.72% [228700/4848000] [47.2/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 22:59:50,677 - Train: 4.72% [228800/4848000] [47.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 23:00:23,409 - Train: 4.72% [228900/4848000] [47.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:00:56,118 - Train: 4.72% [229000/4848000] [47.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:01:28,810 - Train: 4.73% [229100/4848000] [47.3/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 23:02:01,535 - Train: 4.73% [229200/4848000] [47.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:02:34,224 - Train: 4.73% [229300/4848000] [47.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:03:06,926 - Train: 4.73% [229400/4848000] [47.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:03:39,691 - Train: 4.73% [229500/4848000] [47.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:04:12,436 - Train: 4.74% [229600/4848000] [47.4/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 23:04:45,117 - Train: 4.74% [229700/4848000] [47.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:05:17,931 - Train: 4.74% [229800/4848000] [47.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:05:50,636 - Train: 4.74% [229900/4848000] [47.4/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:06:23,351 - Train: 4.74% [230000/4848000] [47.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:06:56,025 - Train: 4.75% [230100/4848000] [47.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:07:28,687 - Train: 4.75% [230200/4848000] [47.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:08:01,388 - Train: 4.75% [230300/4848000] [47.5/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 23:08:34,065 - Train: 4.75% [230400/4848000] [47.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 23:09:06,750 - Train: 4.75% [230500/4848000] [47.5/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 23:09:39,428 - Train: 4.76% [230600/4848000] [47.6/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 23:10:12,132 - Train: 4.76% [230700/4848000] [47.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:10:44,806 - Train: 4.76% [230800/4848000] [47.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:11:17,513 - Train: 4.76% [230900/4848000] [47.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:11:50,234 - Train: 4.76% [231000/4848000] [47.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:12:22,909 - Train: 4.77% [231100/4848000] [47.7/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 23:12:55,665 - Train: 4.77% [231200/4848000] [47.7/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:13:28,329 - Train: 4.77% [231300/4848000] [47.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:14:01,017 - Train: 4.77% [231400/4848000] [47.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:14:33,638 - Train: 4.78% [231500/4848000] [47.8/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:15:06,350 - Train: 4.78% [231600/4848000] [47.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:15:39,067 - Train: 4.78% [231700/4848000] [47.8/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 23:16:11,757 - Train: 4.78% [231800/4848000] [47.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:16:44,433 - Train: 4.78% [231900/4848000] [47.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:17:17,110 - Train: 4.79% [232000/4848000] [47.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 23:17:49,760 - Train: 4.79% [232100/4848000] [47.9/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:18:22,433 - Train: 4.79% [232200/4848000] [47.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:18:55,136 - Train: 4.79% [232300/4848000] [47.9/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 23:19:28,740 - Train: 4.79% [232400/4848000] [47.9/1000.0] [batch_t 0.324 (0.336)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-03 23:20:01,417 - Train: 4.80% [232500/4848000] [48.0/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:20:34,127 - Train: 4.80% [232600/4848000] [48.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:21:06,873 - Train: 4.80% [232700/4848000] [48.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:21:08,191 - ==> Total time: 23:55:06 Eta: 19 days, 18:23:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 23:21:40,856 - Train: 4.80% [232800/4848000] [48.0/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 23:22:13,564 - Train: 4.80% [232900/4848000] [48.0/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:22:46,263 - Train: 4.81% [233000/4848000] [48.1/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 23:23:18,965 - Train: 4.81% [233100/4848000] [48.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:23:51,678 - Train: 4.81% [233200/4848000] [48.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:24:24,347 - Train: 4.81% [233300/4848000] [48.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:24:57,031 - Train: 4.81% [233400/4848000] [48.1/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 23:25:29,767 - Train: 4.82% [233500/4848000] [48.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:26:02,439 - Train: 4.82% [233600/4848000] [48.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:26:35,093 - Train: 4.82% [233700/4848000] [48.2/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:27:07,747 - Train: 4.82% [233800/4848000] [48.2/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:27:40,444 - Train: 4.82% [233900/4848000] [48.2/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 23:28:13,287 - Train: 4.83% [234000/4848000] [48.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:28:45,987 - Train: 4.83% [234100/4848000] [48.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:29:18,700 - Train: 4.83% [234200/4848000] [48.3/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 23:29:51,395 - Train: 4.83% [234300/4848000] [48.3/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 23:30:24,100 - Train: 4.83% [234400/4848000] [48.3/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 23:30:56,782 - Train: 4.84% [234500/4848000] [48.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:31:29,453 - Train: 4.84% [234600/4848000] [48.4/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 23:32:02,164 - Train: 4.84% [234700/4848000] [48.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:32:34,846 - Train: 4.84% [234800/4848000] [48.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:33:08,331 - Train: 4.85% [234900/4848000] [48.5/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 23:33:40,985 - Train: 4.85% [235000/4848000] [48.5/1000.0] [batch_t 0.327 (0.326)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:34:15,180 - Train: 4.85% [235100/4848000] [48.5/1000.0] [batch_t 0.325 (0.342)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 23:34:47,919 - Train: 4.85% [235200/4848000] [48.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:35:20,619 - Train: 4.85% [235300/4848000] [48.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:35:53,437 - Train: 4.86% [235400/4848000] [48.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 23:36:27,817 - Train: 4.86% [235500/4848000] [48.6/1000.0] [batch_t 0.328 (0.344)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:37:00,544 - Train: 4.86% [235600/4848000] [48.6/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 23:37:34,515 - Train: 4.86% [235700/4848000] [48.6/1000.0] [batch_t 0.326 (0.340)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 23:38:07,269 - Train: 4.86% [235800/4848000] [48.6/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-03 23:38:39,989 - Train: 4.87% [235900/4848000] [48.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:39:13,856 - Train: 4.87% [236000/4848000] [48.7/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:39:46,525 - Train: 4.87% [236100/4848000] [48.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:40:19,216 - Train: 4.87% [236200/4848000] [48.7/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 23:40:51,900 - Train: 4.87% [236300/4848000] [48.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:41:25,410 - Train: 4.88% [236400/4848000] [48.8/1000.0] [batch_t 0.325 (0.335)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 23:41:58,126 - Train: 4.88% [236500/4848000] [48.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:42:31,953 - Train: 4.88% [236600/4848000] [48.8/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:43:04,650 - Train: 4.88% [236700/4848000] [48.8/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 23:43:38,936 - Train: 4.88% [236800/4848000] [48.8/1000.0] [batch_t 0.329 (0.343)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:44:11,599 - Train: 4.89% [236900/4848000] [48.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 23:44:44,293 - Train: 4.89% [237000/4848000] [48.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 23:45:17,251 - Train: 4.89% [237100/4848000] [48.9/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:45:49,971 - Train: 4.89% [237200/4848000] [48.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:46:22,687 - Train: 4.89% [237300/4848000] [48.9/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-03 23:46:55,392 - Train: 4.90% [237400/4848000] [49.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:47:28,116 - Train: 4.90% [237500/4848000] [49.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:47:45,079 - ==> Total time: 1 day, 0:21:43 Eta: 19 days, 16:49:25 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-03 23:48:02,221 - Train: 4.90% [237600/4848000] [49.0/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:48:35,165 - Train: 4.90% [237700/4848000] [49.0/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 23:49:08,002 - Train: 4.91% [237800/4848000] [49.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:49:40,694 - Train: 4.91% [237900/4848000] [49.1/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-03 23:50:14,861 - Train: 4.91% [238000/4848000] [49.1/1000.0] [batch_t 0.331 (0.342)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-03 23:50:47,582 - Train: 4.91% [238100/4848000] [49.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:51:21,466 - Train: 4.91% [238200/4848000] [49.1/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:51:54,140 - Train: 4.92% [238300/4848000] [49.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 23:52:26,925 - Train: 4.92% [238400/4848000] [49.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:52:59,607 - Train: 4.92% [238500/4848000] [49.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 23:53:33,450 - Train: 4.92% [238600/4848000] [49.2/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 23:54:06,312 - Train: 4.92% [238700/4848000] [49.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:54:38,993 - Train: 4.93% [238800/4848000] [49.3/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-03 23:55:11,692 - Train: 4.93% [238900/4848000] [49.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-03 23:55:44,439 - Train: 4.93% [239000/4848000] [49.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:56:17,158 - Train: 4.93% [239100/4848000] [49.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:56:49,879 - Train: 4.93% [239200/4848000] [49.3/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 23:57:22,572 - Train: 4.94% [239300/4848000] [49.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-03 23:57:55,243 - Train: 4.94% [239400/4848000] [49.4/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-03 23:58:27,963 - Train: 4.94% [239500/4848000] [49.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:59:00,749 - Train: 4.94% [239600/4848000] [49.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-03 23:59:34,270 - Train: 4.94% [239700/4848000] [49.4/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 00:00:07,419 - Train: 4.95% [239800/4848000] [49.5/1000.0] [batch_t 0.325 (0.331)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 00:00:40,131 - Train: 4.95% [239900/4848000] [49.5/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 00:01:12,810 - Train: 4.95% [240000/4848000] [49.5/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 00:01:45,542 - Train: 4.95% [240100/4848000] [49.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:02:18,202 - Train: 4.95% [240200/4848000] [49.5/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 00:02:50,947 - Train: 4.96% [240300/4848000] [49.6/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 00:03:23,636 - Train: 4.96% [240400/4848000] [49.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:03:56,311 - Train: 4.96% [240500/4848000] [49.6/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 00:04:29,032 - Train: 4.96% [240600/4848000] [49.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 00:05:01,714 - Train: 4.96% [240700/4848000] [49.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:05:34,397 - Train: 4.97% [240800/4848000] [49.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:06:07,053 - Train: 4.97% [240900/4848000] [49.7/1000.0] [batch_t 0.328 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:06:39,825 - Train: 4.97% [241000/4848000] [49.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 00:07:12,497 - Train: 4.97% [241100/4848000] [49.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 00:07:45,154 - Train: 4.98% [241200/4848000] [49.8/1000.0] [batch_t 0.329 (0.326)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:08:17,847 - Train: 4.98% [241300/4848000] [49.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 00:08:50,507 - Train: 4.98% [241400/4848000] [49.8/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 00:09:23,209 - Train: 4.98% [241500/4848000] [49.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 00:09:55,918 - Train: 4.98% [241600/4848000] [49.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 00:10:28,661 - Train: 4.99% [241700/4848000] [49.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:11:01,394 - Train: 4.99% [241800/4848000] [49.9/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 00:11:34,093 - Train: 4.99% [241900/4848000] [49.9/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 00:12:06,771 - Train: 4.99% [242000/4848000] [49.9/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 00:12:39,465 - Train: 4.99% [242100/4848000] [49.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:13:12,137 - Train: 5.00% [242200/4848000] [50.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 00:13:44,807 - Train: 5.00% [242300/4848000] [50.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 00:14:17,556 - Train: 5.00% [242400/4848000] [50.0/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 00:14:25,834 - Test: 16.13% [50/310] [batch_t 0.153 (0.160)] 2024-04-04 00:14:33,416 - Test: 32.26% [100/310] [batch_t 0.155 (0.156)] 2024-04-04 00:14:40,990 - Test: 48.39% [150/310] [batch_t 0.156 (0.154)] 2024-04-04 00:14:48,772 - Test: 64.52% [200/310] [batch_t 0.163 (0.155)] 2024-04-04 00:14:56,429 - Test: 80.65% [250/310] [batch_t 0.155 (0.154)] 2024-04-04 00:15:06,835 - Test: 96.77% [300/310] [batch_t 0.155 (0.163)] 2024-04-04 00:15:08,927 - Test: 100.00% [310/310] [batch_t 0.720 (0.165)] 2024-04-04 00:39:27,447 - ==> Metric Time for coco : 0.004 (mAUROC_sp_max) 0.001 (mAP_sp_max) 0.001 (mF1_max_sp_max) 341.800 (mAUROC_px) 263.352 (mAP_px) 31.484 (mF1_max_px) 750.325 (mAUPRO_px) 10.732 (mF1_px_0.2_0.8_0.1) 10.649 (mAcc_px_0.2_0.8_0.1) 10.615 (mIoU_px_0.2_0.8_0.1) 30.416 (mIoU_max_px) 2024-04-04 00:39:27,834 - | Name | mAUROC_sp_max | mAUROC_sp_max (Max) | mAP_sp_max | mAP_sp_max (Max) | mF1_max_sp_max | mF1_max_sp_max (Max) | mAUROC_px | mAUROC_px (Max) | mAP_px | mAP_px (Max) | mF1_max_px | mF1_max_px (Max) | mAUPRO_px | mAUPRO_px (Max) | mF1_px_0.2_0.8_0.1 | mF1_px_0.2_0.8_0.1 (Max) | mAcc_px_0.2_0.8_0.1 | mAcc_px_0.2_0.8_0.1 (Max) | mIoU_px_0.2_0.8_0.1 | mIoU_px_0.2_0.8_0.1 (Max) | mIoU_max_px | mIoU_max_px (Max) | |:------:|:---------------:|:---------------------:|:------------:|:------------------:|:----------------:|:----------------------:|:-----------:|:------------------:|:--------:|:-----------------:|:------------:|:------------------:|:-----------:|:------------------:|:--------------------:|:--------------------------:|:---------------------:|:---------------------------:|:---------------------:|:---------------------------:|:-------------:|:-------------------:| | coco | 58.404 | 58.404 (50 epoch) | 40.479 | 40.479 (50 epoch) | 51.858 | 51.858 (50 epoch) | 58.979 | 58.979 (50 epoch) | 8.601 | 8.601 (50 epoch) | 14.397 | 14.397 (50 epoch) | 34.736 | 34.736 (50 epoch) | 9.008 | 9.008 (50 epoch) | 38.999 | 38.999 (50 epoch) | 4.810 | 4.810 (50 epoch) | 7.757 | 7.757 (50 epoch) | 2024-04-04 00:39:28,234 - ==> Total time: 1 day, 1:13:26 Eta: 19 days, 23:15:27 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 00:40:05,560 - Train: 5.00% [242500/4848000] [50.0/1000.0] [batch_t 0.329 (0.352)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:40:38,484 - Train: 5.00% [242600/4848000] [50.0/1000.0] [batch_t 0.336 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-04 00:41:11,409 - Train: 5.01% [242700/4848000] [50.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 00:41:44,348 - Train: 5.01% [242800/4848000] [50.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 00:42:17,278 - Train: 5.01% [242900/4848000] [50.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 00:42:50,203 - Train: 5.01% [243000/4848000] [50.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 00:43:23,124 - Train: 5.01% [243100/4848000] [50.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 00:43:56,079 - Train: 5.02% [243200/4848000] [50.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 00:44:29,031 - Train: 5.02% [243300/4848000] [50.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:45:01,939 - Train: 5.02% [243400/4848000] [50.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 00:45:34,887 - Train: 5.02% [243500/4848000] [50.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:46:07,797 - Train: 5.02% [243600/4848000] [50.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 00:46:40,754 - Train: 5.03% [243700/4848000] [50.3/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 00:47:13,798 - Train: 5.03% [243800/4848000] [50.3/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 00:47:46,715 - Train: 5.03% [243900/4848000] [50.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 00:48:19,652 - Train: 5.03% [244000/4848000] [50.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 00:48:52,558 - Train: 5.04% [244100/4848000] [50.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 00:49:25,465 - Train: 5.04% [244200/4848000] [50.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 00:49:58,391 - Train: 5.04% [244300/4848000] [50.4/1000.0] [batch_t 0.336 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-04 00:50:31,285 - Train: 5.04% [244400/4848000] [50.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 00:51:04,194 - Train: 5.04% [244500/4848000] [50.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 00:51:38,467 - Train: 5.05% [244600/4848000] [50.5/1000.0] [batch_t 0.332 (0.343)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 00:52:11,367 - Train: 5.05% [244700/4848000] [50.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:52:44,309 - Train: 5.05% [244800/4848000] [50.5/1000.0] [batch_t 0.338 (0.329)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-04 00:53:17,236 - Train: 5.05% [244900/4848000] [50.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 00:53:50,176 - Train: 5.05% [245000/4848000] [50.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 00:54:23,466 - Train: 5.06% [245100/4848000] [50.6/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 00:54:56,451 - Train: 5.06% [245200/4848000] [50.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:55:29,380 - Train: 5.06% [245300/4848000] [50.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:56:02,287 - Train: 5.06% [245400/4848000] [50.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:56:35,222 - Train: 5.06% [245500/4848000] [50.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 00:57:08,152 - Train: 5.07% [245600/4848000] [50.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 00:57:41,070 - Train: 5.07% [245700/4848000] [50.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 00:58:14,037 - Train: 5.07% [245800/4848000] [50.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 00:58:46,950 - Train: 5.07% [245900/4848000] [50.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 00:59:20,080 - Train: 5.07% [246000/4848000] [50.7/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 00:59:53,019 - Train: 5.08% [246100/4848000] [50.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:00:25,916 - Train: 5.08% [246200/4848000] [50.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:00:58,836 - Train: 5.08% [246300/4848000] [50.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:01:31,745 - Train: 5.08% [246400/4848000] [50.8/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 01:02:04,655 - Train: 5.08% [246500/4848000] [50.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:02:37,660 - Train: 5.09% [246600/4848000] [50.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 01:03:10,561 - Train: 5.09% [246700/4848000] [50.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:03:43,491 - Train: 5.09% [246800/4848000] [50.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:04:16,412 - Train: 5.09% [246900/4848000] [50.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:04:49,334 - Train: 5.09% [247000/4848000] [50.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:05:22,239 - Train: 5.10% [247100/4848000] [51.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:05:55,158 - Train: 5.10% [247200/4848000] [51.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:06:10,958 - ==> Total time: 1 day, 1:40:09 Eta: 19 days, 21:39:00 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 01:06:29,903 - Train: 5.10% [247300/4848000] [51.0/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 01:07:02,827 - Train: 5.10% [247400/4848000] [51.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 01:07:36,411 - Train: 5.11% [247500/4848000] [51.1/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:08:09,344 - Train: 5.11% [247600/4848000] [51.1/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 01:08:42,285 - Train: 5.11% [247700/4848000] [51.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:09:15,232 - Train: 5.11% [247800/4848000] [51.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:09:48,318 - Train: 5.11% [247900/4848000] [51.1/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:10:21,296 - Train: 5.12% [248000/4848000] [51.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 01:10:54,203 - Train: 5.12% [248100/4848000] [51.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:11:27,107 - Train: 5.12% [248200/4848000] [51.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:11:59,988 - Train: 5.12% [248300/4848000] [51.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:12:33,978 - Train: 5.12% [248400/4848000] [51.2/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 01:13:06,922 - Train: 5.13% [248500/4848000] [51.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:13:39,837 - Train: 5.13% [248600/4848000] [51.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:14:12,723 - Train: 5.13% [248700/4848000] [51.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:14:45,615 - Train: 5.13% [248800/4848000] [51.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:15:18,910 - Train: 5.13% [248900/4848000] [51.3/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:15:51,804 - Train: 5.14% [249000/4848000] [51.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:16:25,656 - Train: 5.14% [249100/4848000] [51.4/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:16:58,612 - Train: 5.14% [249200/4848000] [51.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:17:31,531 - Train: 5.14% [249300/4848000] [51.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:18:04,602 - Train: 5.14% [249400/4848000] [51.4/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 01:18:37,526 - Train: 5.15% [249500/4848000] [51.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:19:10,432 - Train: 5.15% [249600/4848000] [51.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:19:43,365 - Train: 5.15% [249700/4848000] [51.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:20:16,337 - Train: 5.15% [249800/4848000] [51.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:20:49,268 - Train: 5.15% [249900/4848000] [51.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:21:22,194 - Train: 5.16% [250000/4848000] [51.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 01:21:55,179 - Train: 5.16% [250100/4848000] [51.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:22:28,175 - Train: 5.16% [250200/4848000] [51.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:23:01,099 - Train: 5.16% [250300/4848000] [51.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:23:34,054 - Train: 5.17% [250400/4848000] [51.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:24:06,994 - Train: 5.17% [250500/4848000] [51.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 01:24:39,935 - Train: 5.17% [250600/4848000] [51.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:25:12,882 - Train: 5.17% [250700/4848000] [51.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:25:45,989 - Train: 5.17% [250800/4848000] [51.7/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 01:26:18,943 - Train: 5.18% [250900/4848000] [51.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:26:51,885 - Train: 5.18% [251000/4848000] [51.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:27:24,848 - Train: 5.18% [251100/4848000] [51.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:27:57,793 - Train: 5.18% [251200/4848000] [51.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 01:28:30,787 - Train: 5.18% [251300/4848000] [51.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:29:03,738 - Train: 5.19% [251400/4848000] [51.9/1000.0] [batch_t 0.337 (0.329)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-04 01:29:36,655 - Train: 5.19% [251500/4848000] [51.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:30:09,595 - Train: 5.19% [251600/4848000] [51.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 01:30:42,479 - Train: 5.19% [251700/4848000] [51.9/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 01:31:15,379 - Train: 5.19% [251800/4848000] [51.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:31:48,282 - Train: 5.20% [251900/4848000] [52.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:32:21,563 - Train: 5.20% [252000/4848000] [52.0/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 01:32:53,164 - ==> Total time: 1 day, 2:06:51 Eta: 19 days, 20:05:04 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 01:32:56,551 - Train: 5.20% [252100/4848000] [52.0/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:33:29,458 - Train: 5.20% [252200/4848000] [52.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:34:02,385 - Train: 5.20% [252300/4848000] [52.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:34:35,281 - Train: 5.21% [252400/4848000] [52.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:35:08,181 - Train: 5.21% [252500/4848000] [52.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:35:41,095 - Train: 5.21% [252600/4848000] [52.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 01:36:14,017 - Train: 5.21% [252700/4848000] [52.1/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 01:36:46,907 - Train: 5.21% [252800/4848000] [52.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:37:19,890 - Train: 5.22% [252900/4848000] [52.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:37:52,844 - Train: 5.22% [253000/4848000] [52.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:38:25,800 - Train: 5.22% [253100/4848000] [52.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:38:58,714 - Train: 5.22% [253200/4848000] [52.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 01:39:31,647 - Train: 5.22% [253300/4848000] [52.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:40:05,117 - Train: 5.23% [253400/4848000] [52.3/1000.0] [batch_t 0.815 (0.335)] [data_t 0.487] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:40:38,040 - Train: 5.23% [253500/4848000] [52.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:41:11,112 - Train: 5.23% [253600/4848000] [52.3/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:41:44,075 - Train: 5.23% [253700/4848000] [52.3/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 01:42:17,052 - Train: 5.24% [253800/4848000] [52.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:42:49,950 - Train: 5.24% [253900/4848000] [52.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:43:23,686 - Train: 5.24% [254000/4848000] [52.4/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:43:56,647 - Train: 5.24% [254100/4848000] [52.4/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 01:44:29,538 - Train: 5.24% [254200/4848000] [52.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:45:02,498 - Train: 5.25% [254300/4848000] [52.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:45:35,404 - Train: 5.25% [254400/4848000] [52.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:46:08,373 - Train: 5.25% [254500/4848000] [52.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:46:41,320 - Train: 5.25% [254600/4848000] [52.5/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 01:47:14,285 - Train: 5.25% [254700/4848000] [52.5/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 01:47:47,240 - Train: 5.26% [254800/4848000] [52.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:48:20,151 - Train: 5.26% [254900/4848000] [52.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:48:53,195 - Train: 5.26% [255000/4848000] [52.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:49:26,094 - Train: 5.26% [255100/4848000] [52.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:49:59,024 - Train: 5.26% [255200/4848000] [52.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:50:31,945 - Train: 5.27% [255300/4848000] [52.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:51:04,855 - Train: 5.27% [255400/4848000] [52.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:51:37,770 - Train: 5.27% [255500/4848000] [52.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:52:10,718 - Train: 5.27% [255600/4848000] [52.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:52:43,653 - Train: 5.27% [255700/4848000] [52.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:53:16,610 - Train: 5.28% [255800/4848000] [52.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:53:49,554 - Train: 5.28% [255900/4848000] [52.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:54:22,488 - Train: 5.28% [256000/4848000] [52.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:54:55,450 - Train: 5.28% [256100/4848000] [52.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:55:28,380 - Train: 5.28% [256200/4848000] [52.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:56:01,316 - Train: 5.29% [256300/4848000] [52.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:56:34,424 - Train: 5.29% [256400/4848000] [52.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:57:07,354 - Train: 5.29% [256500/4848000] [52.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:57:40,292 - Train: 5.29% [256600/4848000] [52.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 01:58:13,194 - Train: 5.29% [256700/4848000] [52.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:58:46,113 - Train: 5.30% [256800/4848000] [53.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 01:59:19,768 - Train: 5.30% [256900/4848000] [53.0/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 01:59:34,249 - ==> Total time: 1 day, 2:33:32 Eta: 19 days, 18:33:21 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 01:59:54,543 - Train: 5.30% [257000/4848000] [53.0/1000.0] [batch_t 0.336 (0.330)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-04 02:00:27,521 - Train: 5.30% [257100/4848000] [53.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:01:00,488 - Train: 5.31% [257200/4848000] [53.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:01:33,423 - Train: 5.31% [257300/4848000] [53.1/1000.0] [batch_t 0.336 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-04 02:02:06,380 - Train: 5.31% [257400/4848000] [53.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:02:39,364 - Train: 5.31% [257500/4848000] [53.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:03:12,342 - Train: 5.31% [257600/4848000] [53.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:03:45,363 - Train: 5.32% [257700/4848000] [53.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:04:18,266 - Train: 5.32% [257800/4848000] [53.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:04:51,213 - Train: 5.32% [257900/4848000] [53.2/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 02:05:24,120 - Train: 5.32% [258000/4848000] [53.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:05:57,045 - Train: 5.32% [258100/4848000] [53.2/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 02:06:29,964 - Train: 5.33% [258200/4848000] [53.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:07:02,922 - Train: 5.33% [258300/4848000] [53.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:07:35,845 - Train: 5.33% [258400/4848000] [53.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:08:08,781 - Train: 5.33% [258500/4848000] [53.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 02:08:41,763 - Train: 5.33% [258600/4848000] [53.3/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 02:09:14,635 - Train: 5.34% [258700/4848000] [53.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:09:47,567 - Train: 5.34% [258800/4848000] [53.4/1000.0] [batch_t 0.336 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-04 02:10:20,503 - Train: 5.34% [258900/4848000] [53.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 02:10:53,431 - Train: 5.34% [259000/4848000] [53.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:11:26,372 - Train: 5.34% [259100/4848000] [53.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:11:59,388 - Train: 5.35% [259200/4848000] [53.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:12:32,315 - Train: 5.35% [259300/4848000] [53.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:13:05,262 - Train: 5.35% [259400/4848000] [53.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 02:13:38,194 - Train: 5.35% [259500/4848000] [53.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:14:11,133 - Train: 5.35% [259600/4848000] [53.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:14:44,072 - Train: 5.36% [259700/4848000] [53.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:15:16,991 - Train: 5.36% [259800/4848000] [53.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:15:49,933 - Train: 5.36% [259900/4848000] [53.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:16:22,865 - Train: 5.36% [260000/4848000] [53.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:16:55,828 - Train: 5.37% [260100/4848000] [53.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:17:28,777 - Train: 5.37% [260200/4848000] [53.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:18:01,754 - Train: 5.37% [260300/4848000] [53.7/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 02:18:34,702 - Train: 5.37% [260400/4848000] [53.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:19:07,618 - Train: 5.37% [260500/4848000] [53.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:19:40,689 - Train: 5.38% [260600/4848000] [53.8/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:20:13,609 - Train: 5.38% [260700/4848000] [53.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:20:46,519 - Train: 5.38% [260800/4848000] [53.8/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 02:21:19,437 - Train: 5.38% [260900/4848000] [53.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 02:21:52,349 - Train: 5.38% [261000/4848000] [53.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:22:25,275 - Train: 5.39% [261100/4848000] [53.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:22:58,175 - Train: 5.39% [261200/4848000] [53.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 02:23:31,090 - Train: 5.39% [261300/4848000] [53.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:24:04,022 - Train: 5.39% [261400/4848000] [53.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:24:36,922 - Train: 5.39% [261500/4848000] [53.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:25:09,871 - Train: 5.40% [261600/4848000] [54.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:25:42,791 - Train: 5.40% [261700/4848000] [54.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 02:26:14,225 - ==> Total time: 1 day, 3:00:12 Eta: 19 days, 17:03:42 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 02:26:18,864 - Train: 5.40% [261800/4848000] [54.0/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:26:51,958 - Train: 5.40% [261900/4848000] [54.0/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:27:24,902 - Train: 5.40% [262000/4848000] [54.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 02:27:57,857 - Train: 5.41% [262100/4848000] [54.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:28:30,800 - Train: 5.41% [262200/4848000] [54.1/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 02:29:03,710 - Train: 5.41% [262300/4848000] [54.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:29:37,471 - Train: 5.41% [262400/4848000] [54.1/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:30:10,407 - Train: 5.41% [262500/4848000] [54.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 02:30:43,300 - Train: 5.42% [262600/4848000] [54.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:31:16,718 - Train: 5.42% [262700/4848000] [54.2/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:31:49,671 - Train: 5.42% [262800/4848000] [54.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:32:22,575 - Train: 5.42% [262900/4848000] [54.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:32:55,479 - Train: 5.42% [263000/4848000] [54.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:33:28,411 - Train: 5.43% [263100/4848000] [54.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:34:01,387 - Train: 5.43% [263200/4848000] [54.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:34:34,538 - Train: 5.43% [263300/4848000] [54.3/1000.0] [batch_t 0.330 (0.331)] [data_t 0.004] [optim_t 0.326] [lr 0.005000] 2024-04-04 02:35:07,496 - Train: 5.43% [263400/4848000] [54.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:35:40,427 - Train: 5.44% [263500/4848000] [54.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:36:13,346 - Train: 5.44% [263600/4848000] [54.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:36:46,231 - Train: 5.44% [263700/4848000] [54.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:37:19,222 - Train: 5.44% [263800/4848000] [54.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:37:52,160 - Train: 5.44% [263900/4848000] [54.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:38:25,088 - Train: 5.45% [264000/4848000] [54.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 02:38:57,969 - Train: 5.45% [264100/4848000] [54.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 02:39:30,912 - Train: 5.45% [264200/4848000] [54.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:40:03,833 - Train: 5.45% [264300/4848000] [54.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:40:36,765 - Train: 5.45% [264400/4848000] [54.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:41:09,711 - Train: 5.46% [264500/4848000] [54.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:41:42,644 - Train: 5.46% [264600/4848000] [54.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 02:42:15,558 - Train: 5.46% [264700/4848000] [54.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 02:42:48,568 - Train: 5.46% [264800/4848000] [54.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 02:43:21,692 - Train: 5.46% [264900/4848000] [54.6/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 02:43:54,609 - Train: 5.47% [265000/4848000] [54.7/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 02:44:27,537 - Train: 5.47% [265100/4848000] [54.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:45:00,445 - Train: 5.47% [265200/4848000] [54.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:45:33,394 - Train: 5.47% [265300/4848000] [54.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:46:06,300 - Train: 5.47% [265400/4848000] [54.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:46:39,207 - Train: 5.48% [265500/4848000] [54.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:47:12,116 - Train: 5.48% [265600/4848000] [54.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:47:45,067 - Train: 5.48% [265700/4848000] [54.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:48:17,996 - Train: 5.48% [265800/4848000] [54.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:48:50,918 - Train: 5.48% [265900/4848000] [54.8/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 02:49:23,868 - Train: 5.49% [266000/4848000] [54.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:49:56,798 - Train: 5.49% [266100/4848000] [54.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:50:29,817 - Train: 5.49% [266200/4848000] [54.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:51:02,760 - Train: 5.49% [266300/4848000] [54.9/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 02:51:35,671 - Train: 5.50% [266400/4848000] [55.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:52:08,584 - Train: 5.50% [266500/4848000] [55.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:52:41,535 - Train: 5.50% [266600/4848000] [55.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:52:54,708 - ==> Total time: 1 day, 3:26:53 Eta: 19 days, 15:36:30 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 02:53:16,545 - Train: 5.50% [266700/4848000] [55.0/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:53:49,470 - Train: 5.50% [266800/4848000] [55.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:54:22,630 - Train: 5.51% [266900/4848000] [55.1/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 02:54:55,538 - Train: 5.51% [267000/4848000] [55.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:55:28,506 - Train: 5.51% [267100/4848000] [55.1/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:56:01,456 - Train: 5.51% [267200/4848000] [55.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 02:56:34,337 - Train: 5.51% [267300/4848000] [55.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:57:07,724 - Train: 5.52% [267400/4848000] [55.2/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:57:40,779 - Train: 5.52% [267500/4848000] [55.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 02:58:13,740 - Train: 5.52% [267600/4848000] [55.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 02:58:46,680 - Train: 5.52% [267700/4848000] [55.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:59:20,096 - Train: 5.52% [267800/4848000] [55.2/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 02:59:53,029 - Train: 5.53% [267900/4848000] [55.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 03:00:25,976 - Train: 5.53% [268000/4848000] [55.3/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 03:00:58,912 - Train: 5.53% [268100/4848000] [55.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:01:31,862 - Train: 5.53% [268200/4848000] [55.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:02:04,854 - Train: 5.53% [268300/4848000] [55.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:02:37,773 - Train: 5.54% [268400/4848000] [55.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:03:10,693 - Train: 5.54% [268500/4848000] [55.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:03:43,671 - Train: 5.54% [268600/4848000] [55.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:04:16,570 - Train: 5.54% [268700/4848000] [55.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:04:49,484 - Train: 5.54% [268800/4848000] [55.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:05:22,518 - Train: 5.55% [268900/4848000] [55.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:05:55,435 - Train: 5.55% [269000/4848000] [55.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:06:28,364 - Train: 5.55% [269100/4848000] [55.5/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 03:07:01,277 - Train: 5.55% [269200/4848000] [55.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 03:07:34,252 - Train: 5.55% [269300/4848000] [55.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:08:07,210 - Train: 5.56% [269400/4848000] [55.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:08:40,158 - Train: 5.56% [269500/4848000] [55.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:09:13,091 - Train: 5.56% [269600/4848000] [55.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 03:09:46,028 - Train: 5.56% [269700/4848000] [55.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:10:18,954 - Train: 5.57% [269800/4848000] [55.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:10:51,884 - Train: 5.57% [269900/4848000] [55.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:11:24,811 - Train: 5.57% [270000/4848000] [55.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 03:11:57,751 - Train: 5.57% [270100/4848000] [55.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:12:30,696 - Train: 5.57% [270200/4848000] [55.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:13:03,747 - Train: 5.58% [270300/4848000] [55.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:13:36,672 - Train: 5.58% [270400/4848000] [55.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:14:09,595 - Train: 5.58% [270500/4848000] [55.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 03:14:42,539 - Train: 5.58% [270600/4848000] [55.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 03:15:15,460 - Train: 5.58% [270700/4848000] [55.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 03:15:48,434 - Train: 5.59% [270800/4848000] [55.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:16:21,397 - Train: 5.59% [270900/4848000] [55.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:16:54,351 - Train: 5.59% [271000/4848000] [55.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:17:27,259 - Train: 5.59% [271100/4848000] [55.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:18:00,163 - Train: 5.59% [271200/4848000] [55.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:18:33,074 - Train: 5.60% [271300/4848000] [56.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:19:06,275 - Train: 5.60% [271400/4848000] [56.0/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:19:35,286 - ==> Total time: 1 day, 3:53:33 Eta: 19 days, 14:11:29 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 03:19:41,243 - Train: 5.60% [271500/4848000] [56.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:20:14,341 - Train: 5.60% [271600/4848000] [56.0/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 03:20:47,275 - Train: 5.60% [271700/4848000] [56.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:21:20,219 - Train: 5.61% [271800/4848000] [56.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:21:53,182 - Train: 5.61% [271900/4848000] [56.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:22:26,094 - Train: 5.61% [272000/4848000] [56.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:22:59,037 - Train: 5.61% [272100/4848000] [56.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:23:31,991 - Train: 5.61% [272200/4848000] [56.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:24:04,904 - Train: 5.62% [272300/4848000] [56.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:24:37,869 - Train: 5.62% [272400/4848000] [56.2/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 03:25:10,826 - Train: 5.62% [272500/4848000] [56.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:25:43,754 - Train: 5.62% [272600/4848000] [56.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 03:26:16,676 - Train: 5.62% [272700/4848000] [56.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:26:49,598 - Train: 5.63% [272800/4848000] [56.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:27:22,546 - Train: 5.63% [272900/4848000] [56.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:27:55,460 - Train: 5.63% [273000/4848000] [56.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 03:28:28,513 - Train: 5.63% [273100/4848000] [56.3/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:29:01,454 - Train: 5.64% [273200/4848000] [56.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:29:34,425 - Train: 5.64% [273300/4848000] [56.4/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 03:30:07,445 - Train: 5.64% [273400/4848000] [56.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:30:40,398 - Train: 5.64% [273500/4848000] [56.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:31:13,316 - Train: 5.64% [273600/4848000] [56.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:31:46,273 - Train: 5.65% [273700/4848000] [56.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:32:19,902 - Train: 5.65% [273800/4848000] [56.5/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:32:52,822 - Train: 5.65% [273900/4848000] [56.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:33:25,685 - Train: 5.65% [274000/4848000] [56.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 03:33:58,647 - Train: 5.65% [274100/4848000] [56.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:34:31,598 - Train: 5.66% [274200/4848000] [56.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 03:35:04,535 - Train: 5.66% [274300/4848000] [56.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 03:35:37,451 - Train: 5.66% [274400/4848000] [56.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 03:36:10,537 - Train: 5.66% [274500/4848000] [56.6/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:36:43,453 - Train: 5.66% [274600/4848000] [56.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 03:37:16,359 - Train: 5.67% [274700/4848000] [56.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:37:49,291 - Train: 5.67% [274800/4848000] [56.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:38:22,235 - Train: 5.67% [274900/4848000] [56.7/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 03:38:55,171 - Train: 5.67% [275000/4848000] [56.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:39:28,721 - Train: 5.67% [275100/4848000] [56.7/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:40:01,678 - Train: 5.68% [275200/4848000] [56.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:40:34,637 - Train: 5.68% [275300/4848000] [56.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 03:41:07,581 - Train: 5.68% [275400/4848000] [56.8/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 03:41:40,540 - Train: 5.68% [275500/4848000] [56.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:42:13,477 - Train: 5.68% [275600/4848000] [56.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:42:46,386 - Train: 5.69% [275700/4848000] [56.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:43:19,328 - Train: 5.69% [275800/4848000] [56.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:43:52,426 - Train: 5.69% [275900/4848000] [56.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:44:25,364 - Train: 5.69% [276000/4848000] [56.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 03:44:58,326 - Train: 5.70% [276100/4848000] [57.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:45:31,248 - Train: 5.70% [276200/4848000] [57.0/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 03:46:04,209 - Train: 5.70% [276300/4848000] [57.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:46:16,063 - ==> Total time: 1 day, 4:20:14 Eta: 19 days, 12:48:34 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 03:46:39,362 - Train: 5.70% [276400/4848000] [57.0/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 03:47:12,329 - Train: 5.70% [276500/4848000] [57.0/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 03:47:45,273 - Train: 5.71% [276600/4848000] [57.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:48:18,201 - Train: 5.71% [276700/4848000] [57.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:48:51,126 - Train: 5.71% [276800/4848000] [57.1/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 03:49:24,069 - Train: 5.71% [276900/4848000] [57.1/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 03:49:57,069 - Train: 5.71% [277000/4848000] [57.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:50:30,026 - Train: 5.72% [277100/4848000] [57.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 03:51:03,090 - Train: 5.72% [277200/4848000] [57.2/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 03:51:36,013 - Train: 5.72% [277300/4848000] [57.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:52:08,930 - Train: 5.72% [277400/4848000] [57.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:52:41,896 - Train: 5.72% [277500/4848000] [57.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:53:14,843 - Train: 5.73% [277600/4848000] [57.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:53:47,751 - Train: 5.73% [277700/4848000] [57.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:54:20,683 - Train: 5.73% [277800/4848000] [57.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 03:54:53,620 - Train: 5.73% [277900/4848000] [57.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:55:26,577 - Train: 5.73% [278000/4848000] [57.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 03:55:59,489 - Train: 5.74% [278100/4848000] [57.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 03:56:32,422 - Train: 5.74% [278200/4848000] [57.4/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 03:57:05,338 - Train: 5.74% [278300/4848000] [57.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:57:38,243 - Train: 5.74% [278400/4848000] [57.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:58:11,185 - Train: 5.74% [278500/4848000] [57.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:58:44,101 - Train: 5.75% [278600/4848000] [57.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:59:17,110 - Train: 5.75% [278700/4848000] [57.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 03:59:50,032 - Train: 5.75% [278800/4848000] [57.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 04:00:22,966 - Train: 5.75% [278900/4848000] [57.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:00:55,929 - Train: 5.75% [279000/4848000] [57.5/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 04:01:28,885 - Train: 5.76% [279100/4848000] [57.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 04:02:01,884 - Train: 5.76% [279200/4848000] [57.6/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 04:02:34,801 - Train: 5.76% [279300/4848000] [57.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 04:03:07,735 - Train: 5.76% [279400/4848000] [57.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:03:40,615 - Train: 5.77% [279500/4848000] [57.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:04:13,559 - Train: 5.77% [279600/4848000] [57.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:04:46,513 - Train: 5.77% [279700/4848000] [57.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:05:19,443 - Train: 5.77% [279800/4848000] [57.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:05:52,329 - Train: 5.77% [279900/4848000] [57.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 04:06:25,297 - Train: 5.78% [280000/4848000] [57.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:06:58,393 - Train: 5.78% [280100/4848000] [57.8/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 04:07:31,343 - Train: 5.78% [280200/4848000] [57.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 04:08:04,307 - Train: 5.78% [280300/4848000] [57.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:08:37,301 - Train: 5.78% [280400/4848000] [57.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:09:10,930 - Train: 5.79% [280500/4848000] [57.9/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:09:43,849 - Train: 5.79% [280600/4848000] [57.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:10:16,788 - Train: 5.79% [280700/4848000] [57.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:10:49,729 - Train: 5.79% [280800/4848000] [57.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:11:22,648 - Train: 5.79% [280900/4848000] [57.9/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 04:11:55,579 - Train: 5.80% [281000/4848000] [58.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 04:12:28,489 - Train: 5.80% [281100/4848000] [58.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 04:12:56,126 - ==> Total time: 1 day, 4:46:54 Eta: 19 days, 11:27:24 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 04:13:03,360 - Train: 5.80% [281200/4848000] [58.0/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:13:36,264 - Train: 5.80% [281300/4848000] [58.0/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 04:14:09,336 - Train: 5.80% [281400/4848000] [58.0/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:14:42,299 - Train: 5.81% [281500/4848000] [58.1/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 04:15:15,242 - Train: 5.81% [281600/4848000] [58.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:15:48,195 - Train: 5.81% [281700/4848000] [58.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 04:16:21,118 - Train: 5.81% [281800/4848000] [58.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 04:16:54,031 - Train: 5.81% [281900/4848000] [58.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:17:26,957 - Train: 5.82% [282000/4848000] [58.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 04:17:59,916 - Train: 5.82% [282100/4848000] [58.2/1000.0] [batch_t 0.336 (0.330)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-04 04:18:32,793 - Train: 5.82% [282200/4848000] [58.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 04:19:05,726 - Train: 5.82% [282300/4848000] [58.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:19:38,672 - Train: 5.83% [282400/4848000] [58.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:20:11,607 - Train: 5.83% [282500/4848000] [58.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:20:44,519 - Train: 5.83% [282600/4848000] [58.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:21:17,487 - Train: 5.83% [282700/4848000] [58.3/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 04:21:50,487 - Train: 5.83% [282800/4848000] [58.3/1000.0] [batch_t 0.323 (0.330)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-04 04:22:23,464 - Train: 5.84% [282900/4848000] [58.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:22:56,452 - Train: 5.84% [283000/4848000] [58.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 04:23:29,458 - Train: 5.84% [283100/4848000] [58.4/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 04:24:02,402 - Train: 5.84% [283200/4848000] [58.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 04:24:35,376 - Train: 5.84% [283300/4848000] [58.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:25:08,289 - Train: 5.85% [283400/4848000] [58.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 04:25:41,206 - Train: 5.85% [283500/4848000] [58.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 04:26:14,209 - Train: 5.85% [283600/4848000] [58.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:26:47,126 - Train: 5.85% [283700/4848000] [58.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 04:27:20,068 - Train: 5.85% [283800/4848000] [58.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:27:52,963 - Train: 5.86% [283900/4848000] [58.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 04:28:25,879 - Train: 5.86% [284000/4848000] [58.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:28:58,841 - Train: 5.86% [284100/4848000] [58.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 04:29:31,774 - Train: 5.86% [284200/4848000] [58.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 04:30:04,777 - Train: 5.86% [284300/4848000] [58.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 04:30:37,815 - Train: 5.87% [284400/4848000] [58.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:31:10,683 - Train: 5.87% [284500/4848000] [58.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 04:31:43,593 - Train: 5.87% [284600/4848000] [58.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:32:16,530 - Train: 5.87% [284700/4848000] [58.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 04:32:49,492 - Train: 5.87% [284800/4848000] [58.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 04:33:22,476 - Train: 5.88% [284900/4848000] [58.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:33:55,405 - Train: 5.88% [285000/4848000] [58.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:34:28,335 - Train: 5.88% [285100/4848000] [58.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:35:01,222 - Train: 5.88% [285200/4848000] [58.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:35:34,141 - Train: 5.88% [285300/4848000] [58.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 04:36:07,045 - Train: 5.89% [285400/4848000] [58.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:36:39,970 - Train: 5.89% [285500/4848000] [58.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:37:12,897 - Train: 5.89% [285600/4848000] [58.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:37:45,865 - Train: 5.89% [285700/4848000] [58.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 04:38:18,752 - Train: 5.90% [285800/4848000] [59.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 04:38:51,657 - Train: 5.90% [285900/4848000] [59.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:39:24,592 - Train: 5.90% [286000/4848000] [59.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 04:39:35,140 - ==> Total time: 1 day, 5:13:33 Eta: 19 days, 10:07:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 04:39:59,405 - Train: 5.90% [286100/4848000] [59.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:40:32,336 - Train: 5.90% [286200/4848000] [59.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:41:05,288 - Train: 5.91% [286300/4848000] [59.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 04:41:38,263 - Train: 5.91% [286400/4848000] [59.1/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 04:42:11,189 - Train: 5.91% [286500/4848000] [59.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 04:42:44,104 - Train: 5.91% [286600/4848000] [59.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:43:17,030 - Train: 5.91% [286700/4848000] [59.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 04:43:49,963 - Train: 5.92% [286800/4848000] [59.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:44:22,930 - Train: 5.92% [286900/4848000] [59.2/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 04:44:55,996 - Train: 5.92% [287000/4848000] [59.2/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:45:29,000 - Train: 5.92% [287100/4848000] [59.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:46:01,958 - Train: 5.92% [287200/4848000] [59.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:46:34,878 - Train: 5.93% [287300/4848000] [59.3/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 04:47:07,796 - Train: 5.93% [287400/4848000] [59.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:47:40,760 - Train: 5.93% [287500/4848000] [59.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 04:48:13,715 - Train: 5.93% [287600/4848000] [59.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 04:48:46,697 - Train: 5.93% [287700/4848000] [59.3/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 04:49:19,738 - Train: 5.94% [287800/4848000] [59.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:49:52,713 - Train: 5.94% [287900/4848000] [59.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:50:25,643 - Train: 5.94% [288000/4848000] [59.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:50:58,626 - Train: 5.94% [288100/4848000] [59.4/1000.0] [batch_t 0.340 (0.330)] [data_t 0.002] [optim_t 0.338] [lr 0.005000] 2024-04-04 04:51:31,616 - Train: 5.94% [288200/4848000] [59.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 04:52:04,588 - Train: 5.95% [288300/4848000] [59.5/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 04:52:37,664 - Train: 5.95% [288400/4848000] [59.5/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 04:53:13,631 - Train: 5.95% [288500/4848000] [59.5/1000.0] [batch_t 0.329 (0.360)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:53:46,575 - Train: 5.95% [288600/4848000] [59.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 04:54:19,485 - Train: 5.96% [288700/4848000] [59.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 04:54:52,510 - Train: 5.96% [288800/4848000] [59.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 04:55:26,908 - Train: 5.96% [288900/4848000] [59.6/1000.0] [batch_t 0.331 (0.344)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:55:59,834 - Train: 5.96% [289000/4848000] [59.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 04:56:32,777 - Train: 5.96% [289100/4848000] [59.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:57:06,699 - Train: 5.97% [289200/4848000] [59.7/1000.0] [batch_t 0.331 (0.339)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:57:39,640 - Train: 5.97% [289300/4848000] [59.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 04:58:13,879 - Train: 5.97% [289400/4848000] [59.7/1000.0] [batch_t 0.331 (0.342)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:58:46,820 - Train: 5.97% [289500/4848000] [59.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 04:59:19,739 - Train: 5.97% [289600/4848000] [59.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 04:59:52,655 - Train: 5.98% [289700/4848000] [59.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:00:25,602 - Train: 5.98% [289800/4848000] [59.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 05:00:58,628 - Train: 5.98% [289900/4848000] [59.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:01:31,528 - Train: 5.98% [290000/4848000] [59.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 05:02:04,479 - Train: 5.98% [290100/4848000] [59.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 05:02:37,388 - Train: 5.99% [290200/4848000] [59.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 05:03:10,336 - Train: 5.99% [290300/4848000] [59.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:03:43,234 - Train: 5.99% [290400/4848000] [59.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:04:16,171 - Train: 5.99% [290500/4848000] [59.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 05:04:49,119 - Train: 5.99% [290600/4848000] [59.9/1000.0] [batch_t 0.340 (0.329)] [data_t 0.002] [optim_t 0.338] [lr 0.005000] 2024-04-04 05:05:22,044 - Train: 6.00% [290700/4848000] [60.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 05:05:55,027 - Train: 6.00% [290800/4848000] [60.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:06:21,341 - ==> Total time: 1 day, 5:40:19 Eta: 19 days, 8:51:50 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 05:06:29,785 - Train: 6.00% [290900/4848000] [60.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 05:07:02,717 - Train: 6.00% [291000/4848000] [60.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 05:07:35,626 - Train: 6.00% [291100/4848000] [60.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:08:08,705 - Train: 6.01% [291200/4848000] [60.1/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:08:41,646 - Train: 6.01% [291300/4848000] [60.1/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 05:09:14,617 - Train: 6.01% [291400/4848000] [60.1/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 05:09:47,577 - Train: 6.01% [291500/4848000] [60.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:10:20,540 - Train: 6.01% [291600/4848000] [60.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:10:53,452 - Train: 6.02% [291700/4848000] [60.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:11:26,404 - Train: 6.02% [291800/4848000] [60.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 05:11:59,342 - Train: 6.02% [291900/4848000] [60.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:12:32,263 - Train: 6.02% [292000/4848000] [60.2/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 05:13:05,186 - Train: 6.03% [292100/4848000] [60.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:13:38,100 - Train: 6.03% [292200/4848000] [60.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 05:14:10,984 - Train: 6.03% [292300/4848000] [60.3/1000.0] [batch_t 0.336 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-04 05:14:43,888 - Train: 6.03% [292400/4848000] [60.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:15:16,824 - Train: 6.03% [292500/4848000] [60.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:15:49,858 - Train: 6.04% [292600/4848000] [60.4/1000.0] [batch_t 0.416 (0.330)] [data_t 0.002] [optim_t 0.414] [lr 0.005000] 2024-04-04 05:16:22,802 - Train: 6.04% [292700/4848000] [60.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 05:16:55,764 - Train: 6.04% [292800/4848000] [60.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:17:28,683 - Train: 6.04% [292900/4848000] [60.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 05:18:01,704 - Train: 6.04% [293000/4848000] [60.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:18:34,692 - Train: 6.05% [293100/4848000] [60.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:19:07,605 - Train: 6.05% [293200/4848000] [60.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 05:19:40,554 - Train: 6.05% [293300/4848000] [60.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:20:13,498 - Train: 6.05% [293400/4848000] [60.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:20:46,449 - Train: 6.05% [293500/4848000] [60.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 05:21:19,847 - Train: 6.06% [293600/4848000] [60.6/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:21:52,826 - Train: 6.06% [293700/4848000] [60.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 05:22:25,724 - Train: 6.06% [293800/4848000] [60.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 05:22:58,690 - Train: 6.06% [293900/4848000] [60.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:23:31,578 - Train: 6.06% [294000/4848000] [60.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:24:04,623 - Train: 6.07% [294100/4848000] [60.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:24:37,548 - Train: 6.07% [294200/4848000] [60.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:25:10,477 - Train: 6.07% [294300/4848000] [60.7/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 05:25:43,429 - Train: 6.07% [294400/4848000] [60.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:26:16,349 - Train: 6.07% [294500/4848000] [60.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:26:49,267 - Train: 6.08% [294600/4848000] [60.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 05:27:22,188 - Train: 6.08% [294700/4848000] [60.8/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 05:27:55,043 - Train: 6.08% [294800/4848000] [60.8/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 05:28:27,986 - Train: 6.08% [294900/4848000] [60.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:29:00,940 - Train: 6.08% [295000/4848000] [60.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:29:33,858 - Train: 6.09% [295100/4848000] [60.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:30:06,767 - Train: 6.09% [295200/4848000] [60.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:30:39,677 - Train: 6.09% [295300/4848000] [60.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:31:12,677 - Train: 6.09% [295400/4848000] [60.9/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 05:31:45,829 - Train: 6.10% [295500/4848000] [61.0/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:32:18,809 - Train: 6.10% [295600/4848000] [61.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 05:32:51,761 - Train: 6.10% [295700/4848000] [61.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 05:33:00,974 - ==> Total time: 1 day, 6:06:59 Eta: 19 days, 7:35:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 05:33:29,408 - Train: 6.10% [295800/4848000] [61.0/1000.0] [batch_t 0.330 (0.368)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:34:02,330 - Train: 6.10% [295900/4848000] [61.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:34:35,506 - Train: 6.11% [296000/4848000] [61.1/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 05:35:09,915 - Train: 6.11% [296100/4848000] [61.1/1000.0] [batch_t 0.330 (0.344)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:35:42,837 - Train: 6.11% [296200/4848000] [61.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:36:16,151 - Train: 6.11% [296300/4848000] [61.1/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:36:49,027 - Train: 6.11% [296400/4848000] [61.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 05:37:22,786 - Train: 6.12% [296500/4848000] [61.2/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 05:37:55,691 - Train: 6.12% [296600/4848000] [61.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 05:38:28,642 - Train: 6.12% [296700/4848000] [61.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 05:39:01,761 - Train: 6.12% [296800/4848000] [61.2/1000.0] [batch_t 0.484 (0.331)] [data_t 0.002] [optim_t 0.482] [lr 0.005000] 2024-04-04 05:39:34,646 - Train: 6.12% [296900/4848000] [61.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:40:07,584 - Train: 6.13% [297000/4848000] [61.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:40:40,521 - Train: 6.13% [297100/4848000] [61.3/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 05:41:13,453 - Train: 6.13% [297200/4848000] [61.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:41:46,397 - Train: 6.13% [297300/4848000] [61.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 05:42:19,274 - Train: 6.13% [297400/4848000] [61.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:42:52,205 - Train: 6.14% [297500/4848000] [61.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:43:25,110 - Train: 6.14% [297600/4848000] [61.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:43:58,056 - Train: 6.14% [297700/4848000] [61.4/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 05:44:30,976 - Train: 6.14% [297800/4848000] [61.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 05:45:03,897 - Train: 6.14% [297900/4848000] [61.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:45:36,825 - Train: 6.15% [298000/4848000] [61.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:46:09,779 - Train: 6.15% [298100/4848000] [61.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 05:46:42,733 - Train: 6.15% [298200/4848000] [61.5/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 05:47:15,750 - Train: 6.15% [298300/4848000] [61.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:47:48,653 - Train: 6.16% [298400/4848000] [61.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:48:21,621 - Train: 6.16% [298500/4848000] [61.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:48:54,561 - Train: 6.16% [298600/4848000] [61.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:49:27,532 - Train: 6.16% [298700/4848000] [61.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:50:00,482 - Train: 6.16% [298800/4848000] [61.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:50:33,435 - Train: 6.17% [298900/4848000] [61.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:51:06,386 - Train: 6.17% [299000/4848000] [61.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 05:51:39,298 - Train: 6.17% [299100/4848000] [61.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:52:12,200 - Train: 6.17% [299200/4848000] [61.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 05:52:45,188 - Train: 6.17% [299300/4848000] [61.7/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 05:53:18,164 - Train: 6.18% [299400/4848000] [61.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:53:51,085 - Train: 6.18% [299500/4848000] [61.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:54:24,031 - Train: 6.18% [299600/4848000] [61.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:54:57,045 - Train: 6.18% [299700/4848000] [61.8/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 05:55:29,974 - Train: 6.18% [299800/4848000] [61.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:56:02,909 - Train: 6.19% [299900/4848000] [61.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 05:56:35,844 - Train: 6.19% [300000/4848000] [61.9/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 05:57:08,800 - Train: 6.19% [300100/4848000] [61.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 05:57:41,782 - Train: 6.19% [300200/4848000] [61.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 05:58:14,739 - Train: 6.19% [300300/4848000] [61.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 05:58:47,654 - Train: 6.20% [300400/4848000] [62.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 05:59:20,585 - Train: 6.20% [300500/4848000] [62.0/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 05:59:45,620 - ==> Total time: 1 day, 6:33:44 Eta: 19 days, 6:22:38 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 05:59:55,497 - Train: 6.20% [300600/4848000] [62.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:00:28,432 - Train: 6.20% [300700/4848000] [62.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 06:01:01,369 - Train: 6.20% [300800/4848000] [62.0/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 06:01:34,334 - Train: 6.21% [300900/4848000] [62.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:02:07,420 - Train: 6.21% [301000/4848000] [62.1/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:02:40,336 - Train: 6.21% [301100/4848000] [62.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:03:13,264 - Train: 6.21% [301200/4848000] [62.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:03:46,197 - Train: 6.21% [301300/4848000] [62.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:04:19,141 - Train: 6.22% [301400/4848000] [62.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 06:04:52,102 - Train: 6.22% [301500/4848000] [62.2/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:05:25,012 - Train: 6.22% [301600/4848000] [62.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 06:05:57,931 - Train: 6.22% [301700/4848000] [62.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:06:30,882 - Train: 6.23% [301800/4848000] [62.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:07:03,797 - Train: 6.23% [301900/4848000] [62.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:07:36,748 - Train: 6.23% [302000/4848000] [62.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:08:09,684 - Train: 6.23% [302100/4848000] [62.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:08:42,623 - Train: 6.23% [302200/4848000] [62.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 06:09:15,549 - Train: 6.24% [302300/4848000] [62.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 06:09:48,461 - Train: 6.24% [302400/4848000] [62.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:10:21,445 - Train: 6.24% [302500/4848000] [62.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:10:54,361 - Train: 6.24% [302600/4848000] [62.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:11:27,285 - Train: 6.24% [302700/4848000] [62.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 06:12:00,233 - Train: 6.25% [302800/4848000] [62.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 06:12:33,184 - Train: 6.25% [302900/4848000] [62.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:13:06,095 - Train: 6.25% [303000/4848000] [62.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:13:39,022 - Train: 6.25% [303100/4848000] [62.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:14:11,932 - Train: 6.25% [303200/4848000] [62.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:14:44,849 - Train: 6.26% [303300/4848000] [62.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:15:17,793 - Train: 6.26% [303400/4848000] [62.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:15:50,689 - Train: 6.26% [303500/4848000] [62.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:16:23,594 - Train: 6.26% [303600/4848000] [62.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:16:56,488 - Train: 6.26% [303700/4848000] [62.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:17:29,386 - Train: 6.27% [303800/4848000] [62.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:18:02,393 - Train: 6.27% [303900/4848000] [62.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 06:18:35,307 - Train: 6.27% [304000/4848000] [62.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 06:19:08,230 - Train: 6.27% [304100/4848000] [62.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 06:19:41,155 - Train: 6.27% [304200/4848000] [62.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 06:20:14,042 - Train: 6.28% [304300/4848000] [62.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:20:47,007 - Train: 6.28% [304400/4848000] [62.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 06:21:19,925 - Train: 6.28% [304500/4848000] [62.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:21:52,865 - Train: 6.28% [304600/4848000] [62.8/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 06:22:25,835 - Train: 6.29% [304700/4848000] [62.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 06:22:58,749 - Train: 6.29% [304800/4848000] [62.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 06:23:31,644 - Train: 6.29% [304900/4848000] [62.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:24:04,585 - Train: 6.29% [305000/4848000] [62.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:24:37,505 - Train: 6.29% [305100/4848000] [62.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:25:10,447 - Train: 6.30% [305200/4848000] [63.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:25:43,392 - Train: 6.30% [305300/4848000] [63.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 06:26:16,481 - Train: 6.30% [305400/4848000] [63.0/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 06:26:24,387 - ==> Total time: 1 day, 7:00:22 Eta: 19 days, 5:09:28 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 06:26:51,460 - Train: 6.30% [305500/4848000] [63.0/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 06:27:24,379 - Train: 6.30% [305600/4848000] [63.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 06:27:57,278 - Train: 6.31% [305700/4848000] [63.1/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 06:28:30,166 - Train: 6.31% [305800/4848000] [63.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 06:29:03,075 - Train: 6.31% [305900/4848000] [63.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:29:35,998 - Train: 6.31% [306000/4848000] [63.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 06:30:08,894 - Train: 6.31% [306100/4848000] [63.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:30:41,987 - Train: 6.32% [306200/4848000] [63.2/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 06:31:15,029 - Train: 6.32% [306300/4848000] [63.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:31:47,903 - Train: 6.32% [306400/4848000] [63.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:32:20,816 - Train: 6.32% [306500/4848000] [63.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 06:32:53,709 - Train: 6.32% [306600/4848000] [63.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 06:33:26,763 - Train: 6.33% [306700/4848000] [63.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:33:59,702 - Train: 6.33% [306800/4848000] [63.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 06:34:32,599 - Train: 6.33% [306900/4848000] [63.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 06:35:05,531 - Train: 6.33% [307000/4848000] [63.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 06:35:38,469 - Train: 6.33% [307100/4848000] [63.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:36:11,380 - Train: 6.34% [307200/4848000] [63.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:36:44,306 - Train: 6.34% [307300/4848000] [63.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:37:17,261 - Train: 6.34% [307400/4848000] [63.4/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 06:37:50,184 - Train: 6.34% [307500/4848000] [63.4/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 06:38:23,111 - Train: 6.34% [307600/4848000] [63.4/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 06:38:56,064 - Train: 6.35% [307700/4848000] [63.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 06:39:29,026 - Train: 6.35% [307800/4848000] [63.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 06:40:01,948 - Train: 6.35% [307900/4848000] [63.5/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 06:40:34,912 - Train: 6.35% [308000/4848000] [63.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:41:08,021 - Train: 6.36% [308100/4848000] [63.6/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:41:40,990 - Train: 6.36% [308200/4848000] [63.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 06:42:13,879 - Train: 6.36% [308300/4848000] [63.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 06:42:46,821 - Train: 6.36% [308400/4848000] [63.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:43:19,725 - Train: 6.36% [308500/4848000] [63.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:43:52,642 - Train: 6.37% [308600/4848000] [63.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:44:25,569 - Train: 6.37% [308700/4848000] [63.7/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 06:44:58,485 - Train: 6.37% [308800/4848000] [63.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 06:45:31,434 - Train: 6.37% [308900/4848000] [63.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:46:04,389 - Train: 6.37% [309000/4848000] [63.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 06:46:37,601 - Train: 6.38% [309100/4848000] [63.8/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 06:47:10,527 - Train: 6.38% [309200/4848000] [63.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 06:47:43,521 - Train: 6.38% [309300/4848000] [63.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:48:16,427 - Train: 6.38% [309400/4848000] [63.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:48:49,338 - Train: 6.38% [309500/4848000] [63.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:49:22,401 - Train: 6.39% [309600/4848000] [63.9/1000.0] [batch_t 0.324 (0.331)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 06:49:55,310 - Train: 6.39% [309700/4848000] [63.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 06:50:28,231 - Train: 6.39% [309800/4848000] [63.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:51:01,160 - Train: 6.39% [309900/4848000] [63.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 06:51:34,062 - Train: 6.39% [310000/4848000] [63.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:52:07,516 - Train: 6.40% [310100/4848000] [64.0/1000.0] [batch_t 0.331 (0.334)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 06:52:40,431 - Train: 6.40% [310200/4848000] [64.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:53:04,197 - ==> Total time: 1 day, 7:27:02 Eta: 19 days, 3:58:01 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 06:53:15,807 - Train: 6.40% [310300/4848000] [64.0/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 06:53:48,758 - Train: 6.40% [310400/4848000] [64.0/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 06:54:21,702 - Train: 6.40% [310500/4848000] [64.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:54:54,630 - Train: 6.41% [310600/4848000] [64.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 06:55:27,571 - Train: 6.41% [310700/4848000] [64.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 06:56:00,547 - Train: 6.41% [310800/4848000] [64.1/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 06:56:33,563 - Train: 6.41% [310900/4848000] [64.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:57:06,458 - Train: 6.42% [311000/4848000] [64.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:57:39,422 - Train: 6.42% [311100/4848000] [64.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-04 06:58:12,379 - Train: 6.42% [311200/4848000] [64.2/1000.0] [batch_t 0.336 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-04 06:58:45,335 - Train: 6.42% [311300/4848000] [64.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 06:59:18,289 - Train: 6.42% [311400/4848000] [64.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 06:59:51,237 - Train: 6.43% [311500/4848000] [64.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:00:24,240 - Train: 6.43% [311600/4848000] [64.3/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 07:00:57,233 - Train: 6.43% [311700/4848000] [64.3/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 07:01:30,199 - Train: 6.43% [311800/4848000] [64.3/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:02:03,128 - Train: 6.43% [311900/4848000] [64.3/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 07:02:36,036 - Train: 6.44% [312000/4848000] [64.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 07:03:08,996 - Train: 6.44% [312100/4848000] [64.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:03:41,957 - Train: 6.44% [312200/4848000] [64.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 07:04:15,034 - Train: 6.44% [312300/4848000] [64.4/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:04:47,959 - Train: 6.44% [312400/4848000] [64.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:05:20,889 - Train: 6.45% [312500/4848000] [64.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:05:53,838 - Train: 6.45% [312600/4848000] [64.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:06:26,794 - Train: 6.45% [312700/4848000] [64.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:06:59,754 - Train: 6.45% [312800/4848000] [64.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:07:32,722 - Train: 6.45% [312900/4848000] [64.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:08:05,680 - Train: 6.46% [313000/4848000] [64.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:08:38,625 - Train: 6.46% [313100/4848000] [64.6/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 07:09:11,570 - Train: 6.46% [313200/4848000] [64.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 07:09:44,475 - Train: 6.46% [313300/4848000] [64.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:10:17,421 - Train: 6.46% [313400/4848000] [64.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:10:50,336 - Train: 6.47% [313500/4848000] [64.7/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 07:11:23,288 - Train: 6.47% [313600/4848000] [64.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:11:56,230 - Train: 6.47% [313700/4848000] [64.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:12:29,294 - Train: 6.47% [313800/4848000] [64.7/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:13:02,259 - Train: 6.47% [313900/4848000] [64.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:13:35,213 - Train: 6.48% [314000/4848000] [64.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:14:08,161 - Train: 6.48% [314100/4848000] [64.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:14:41,052 - Train: 6.48% [314200/4848000] [64.8/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 07:15:13,942 - Train: 6.48% [314300/4848000] [64.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:15:46,872 - Train: 6.49% [314400/4848000] [64.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:16:19,831 - Train: 6.49% [314500/4848000] [64.9/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 07:16:52,765 - Train: 6.49% [314600/4848000] [64.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:17:25,729 - Train: 6.49% [314700/4848000] [64.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:17:58,664 - Train: 6.49% [314800/4848000] [64.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 07:18:31,641 - Train: 6.50% [314900/4848000] [65.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 07:19:04,581 - Train: 6.50% [315000/4848000] [65.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:19:37,534 - Train: 6.50% [315100/4848000] [65.0/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 07:19:44,131 - ==> Total time: 1 day, 7:53:42 Eta: 19 days, 2:47:59 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 07:20:12,598 - Train: 6.50% [315200/4848000] [65.0/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 07:20:45,654 - Train: 6.50% [315300/4848000] [65.0/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 07:21:18,671 - Train: 6.51% [315400/4848000] [65.1/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 07:21:51,575 - Train: 6.51% [315500/4848000] [65.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:22:24,506 - Train: 6.51% [315600/4848000] [65.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 07:22:57,443 - Train: 6.51% [315700/4848000] [65.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:23:30,387 - Train: 6.51% [315800/4848000] [65.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:24:03,310 - Train: 6.52% [315900/4848000] [65.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 07:24:36,240 - Train: 6.52% [316000/4848000] [65.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 07:25:09,165 - Train: 6.52% [316100/4848000] [65.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 07:25:42,087 - Train: 6.52% [316200/4848000] [65.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 07:26:15,002 - Train: 6.52% [316300/4848000] [65.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:26:47,914 - Train: 6.53% [316400/4848000] [65.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:27:20,815 - Train: 6.53% [316500/4848000] [65.3/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 07:27:53,873 - Train: 6.53% [316600/4848000] [65.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:28:26,812 - Train: 6.53% [316700/4848000] [65.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:28:59,720 - Train: 6.53% [316800/4848000] [65.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 07:29:32,646 - Train: 6.54% [316900/4848000] [65.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:30:05,570 - Train: 6.54% [317000/4848000] [65.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:30:38,499 - Train: 6.54% [317100/4848000] [65.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:31:11,386 - Train: 6.54% [317200/4848000] [65.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:31:44,302 - Train: 6.54% [317300/4848000] [65.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 07:32:17,212 - Train: 6.55% [317400/4848000] [65.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:32:50,142 - Train: 6.55% [317500/4848000] [65.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 07:33:23,087 - Train: 6.55% [317600/4848000] [65.5/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 07:33:56,031 - Train: 6.55% [317700/4848000] [65.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:34:28,933 - Train: 6.56% [317800/4848000] [65.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 07:35:01,888 - Train: 6.56% [317900/4848000] [65.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:35:34,936 - Train: 6.56% [318000/4848000] [65.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:36:07,879 - Train: 6.56% [318100/4848000] [65.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 07:36:40,776 - Train: 6.56% [318200/4848000] [65.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:37:13,703 - Train: 6.57% [318300/4848000] [65.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:37:46,650 - Train: 6.57% [318400/4848000] [65.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:38:19,544 - Train: 6.57% [318500/4848000] [65.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 07:38:52,465 - Train: 6.57% [318600/4848000] [65.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 07:39:25,399 - Train: 6.57% [318700/4848000] [65.7/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 07:39:58,327 - Train: 6.58% [318800/4848000] [65.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:40:31,245 - Train: 6.58% [318900/4848000] [65.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:41:04,201 - Train: 6.58% [319000/4848000] [65.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:41:37,980 - Train: 6.58% [319100/4848000] [65.8/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:42:10,963 - Train: 6.58% [319200/4848000] [65.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 07:42:43,874 - Train: 6.59% [319300/4848000] [65.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:43:16,857 - Train: 6.59% [319400/4848000] [65.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:43:49,880 - Train: 6.59% [319500/4848000] [65.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 07:44:22,794 - Train: 6.59% [319600/4848000] [65.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:44:55,820 - Train: 6.59% [319700/4848000] [65.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 07:45:28,752 - Train: 6.60% [319800/4848000] [66.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 07:46:01,747 - Train: 6.60% [319900/4848000] [66.0/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 07:46:24,131 - ==> Total time: 1 day, 8:20:22 Eta: 19 days, 1:39:16 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 07:46:36,547 - Train: 6.60% [320000/4848000] [66.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 07:47:09,464 - Train: 6.60% [320100/4848000] [66.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 07:47:42,361 - Train: 6.60% [320200/4848000] [66.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 07:48:15,298 - Train: 6.61% [320300/4848000] [66.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:48:48,215 - Train: 6.61% [320400/4848000] [66.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 07:49:21,126 - Train: 6.61% [320500/4848000] [66.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:49:54,007 - Train: 6.61% [320600/4848000] [66.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 07:50:26,924 - Train: 6.62% [320700/4848000] [66.2/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 07:50:59,960 - Train: 6.62% [320800/4848000] [66.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 07:51:32,845 - Train: 6.62% [320900/4848000] [66.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:52:05,742 - Train: 6.62% [321000/4848000] [66.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 07:52:38,675 - Train: 6.62% [321100/4848000] [66.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:53:11,600 - Train: 6.63% [321200/4848000] [66.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 07:53:44,535 - Train: 6.63% [321300/4848000] [66.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:54:17,484 - Train: 6.63% [321400/4848000] [66.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:54:50,470 - Train: 6.63% [321500/4848000] [66.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:55:23,416 - Train: 6.63% [321600/4848000] [66.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:55:56,364 - Train: 6.64% [321700/4848000] [66.4/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 07:56:29,332 - Train: 6.64% [321800/4848000] [66.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:57:02,283 - Train: 6.64% [321900/4848000] [66.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 07:57:35,253 - Train: 6.64% [322000/4848000] [66.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:58:08,228 - Train: 6.64% [322100/4848000] [66.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 07:58:41,327 - Train: 6.65% [322200/4848000] [66.5/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 07:59:14,308 - Train: 6.65% [322300/4848000] [66.5/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 07:59:47,238 - Train: 6.65% [322400/4848000] [66.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 08:00:20,167 - Train: 6.65% [322500/4848000] [66.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 08:00:53,119 - Train: 6.65% [322600/4848000] [66.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 08:01:26,077 - Train: 6.66% [322700/4848000] [66.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 08:01:58,987 - Train: 6.66% [322800/4848000] [66.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 08:02:31,902 - Train: 6.66% [322900/4848000] [66.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 08:03:04,784 - Train: 6.66% [323000/4848000] [66.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:03:37,758 - Train: 6.66% [323100/4848000] [66.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 08:04:10,691 - Train: 6.67% [323200/4848000] [66.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 08:04:43,577 - Train: 6.67% [323300/4848000] [66.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 08:05:16,461 - Train: 6.67% [323400/4848000] [66.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:05:49,469 - Train: 6.67% [323500/4848000] [66.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:06:22,530 - Train: 6.67% [323600/4848000] [66.7/1000.0] [batch_t 0.334 (0.331)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 08:06:55,497 - Train: 6.68% [323700/4848000] [66.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:07:28,434 - Train: 6.68% [323800/4848000] [66.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:08:01,339 - Train: 6.68% [323900/4848000] [66.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 08:08:34,285 - Train: 6.68% [324000/4848000] [66.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:09:07,167 - Train: 6.69% [324100/4848000] [66.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 08:09:40,088 - Train: 6.69% [324200/4848000] [66.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 08:10:13,026 - Train: 6.69% [324300/4848000] [66.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:10:45,923 - Train: 6.69% [324400/4848000] [66.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:11:18,809 - Train: 6.69% [324500/4848000] [66.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 08:11:51,740 - Train: 6.70% [324600/4848000] [67.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 08:12:24,684 - Train: 6.70% [324700/4848000] [67.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 08:12:57,633 - Train: 6.70% [324800/4848000] [67.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:13:02,906 - ==> Total time: 1 day, 8:47:01 Eta: 19 days, 0:31:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 08:13:32,597 - Train: 6.70% [324900/4848000] [67.0/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:14:05,500 - Train: 6.70% [325000/4848000] [67.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 08:14:38,423 - Train: 6.71% [325100/4848000] [67.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:15:11,325 - Train: 6.71% [325200/4848000] [67.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:15:44,235 - Train: 6.71% [325300/4848000] [67.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 08:16:17,149 - Train: 6.71% [325400/4848000] [67.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 08:16:50,168 - Train: 6.71% [325500/4848000] [67.1/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 08:17:23,051 - Train: 6.72% [325600/4848000] [67.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:17:55,934 - Train: 6.72% [325700/4848000] [67.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:18:28,845 - Train: 6.72% [325800/4848000] [67.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:19:01,752 - Train: 6.72% [325900/4848000] [67.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 08:19:34,666 - Train: 6.72% [326000/4848000] [67.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:20:07,562 - Train: 6.73% [326100/4848000] [67.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 08:20:40,458 - Train: 6.73% [326200/4848000] [67.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:21:13,384 - Train: 6.73% [326300/4848000] [67.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 08:21:46,364 - Train: 6.73% [326400/4848000] [67.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:22:20,356 - Train: 6.73% [326500/4848000] [67.3/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:22:53,242 - Train: 6.74% [326600/4848000] [67.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:23:26,162 - Train: 6.74% [326700/4848000] [67.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 08:23:59,052 - Train: 6.74% [326800/4848000] [67.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:24:31,932 - Train: 6.74% [326900/4848000] [67.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 08:25:04,866 - Train: 6.75% [327000/4848000] [67.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:25:37,759 - Train: 6.75% [327100/4848000] [67.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:26:10,637 - Train: 6.75% [327200/4848000] [67.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 08:26:43,546 - Train: 6.75% [327300/4848000] [67.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 08:27:16,468 - Train: 6.75% [327400/4848000] [67.5/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 08:27:49,420 - Train: 6.76% [327500/4848000] [67.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 08:28:22,373 - Train: 6.76% [327600/4848000] [67.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 08:28:55,329 - Train: 6.76% [327700/4848000] [67.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:29:28,499 - Train: 6.76% [327800/4848000] [67.6/1000.0] [batch_t 0.332 (0.332)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 08:30:01,473 - Train: 6.76% [327900/4848000] [67.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:30:34,940 - Train: 6.77% [328000/4848000] [67.7/1000.0] [batch_t 0.332 (0.335)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 08:31:07,929 - Train: 6.77% [328100/4848000] [67.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:31:40,861 - Train: 6.77% [328200/4848000] [67.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:32:13,790 - Train: 6.77% [328300/4848000] [67.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:32:46,741 - Train: 6.77% [328400/4848000] [67.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 08:33:19,701 - Train: 6.78% [328500/4848000] [67.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:33:52,645 - Train: 6.78% [328600/4848000] [67.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 08:34:25,560 - Train: 6.78% [328700/4848000] [67.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:34:58,493 - Train: 6.78% [328800/4848000] [67.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:35:31,427 - Train: 6.78% [328900/4848000] [67.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:36:04,328 - Train: 6.79% [329000/4848000] [67.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 08:36:37,225 - Train: 6.79% [329100/4848000] [67.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:37:10,267 - Train: 6.79% [329200/4848000] [67.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:37:43,204 - Train: 6.79% [329300/4848000] [67.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 08:38:16,141 - Train: 6.79% [329400/4848000] [67.9/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 08:38:49,076 - Train: 6.80% [329500/4848000] [68.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:39:22,044 - Train: 6.80% [329600/4848000] [68.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:39:43,105 - ==> Total time: 1 day, 9:13:41 Eta: 18 days, 23:25:19 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 08:39:56,975 - Train: 6.80% [329700/4848000] [68.0/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:40:29,969 - Train: 6.80% [329800/4848000] [68.0/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 08:41:02,941 - Train: 6.80% [329900/4848000] [68.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 08:41:35,900 - Train: 6.81% [330000/4848000] [68.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 08:42:08,823 - Train: 6.81% [330100/4848000] [68.1/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 08:42:41,718 - Train: 6.81% [330200/4848000] [68.1/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 08:43:14,631 - Train: 6.81% [330300/4848000] [68.1/1000.0] [batch_t 0.336 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-04 08:43:47,532 - Train: 6.82% [330400/4848000] [68.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:44:20,621 - Train: 6.82% [330500/4848000] [68.2/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 08:44:53,524 - Train: 6.82% [330600/4848000] [68.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 08:45:26,476 - Train: 6.82% [330700/4848000] [68.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:45:59,402 - Train: 6.82% [330800/4848000] [68.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 08:46:32,362 - Train: 6.83% [330900/4848000] [68.3/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 08:47:05,297 - Train: 6.83% [331000/4848000] [68.3/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 08:47:38,212 - Train: 6.83% [331100/4848000] [68.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 08:48:11,172 - Train: 6.83% [331200/4848000] [68.3/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 08:48:44,130 - Train: 6.83% [331300/4848000] [68.3/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 08:49:17,047 - Train: 6.84% [331400/4848000] [68.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:49:49,993 - Train: 6.84% [331500/4848000] [68.4/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 08:50:22,971 - Train: 6.84% [331600/4848000] [68.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:50:55,893 - Train: 6.84% [331700/4848000] [68.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:51:28,792 - Train: 6.84% [331800/4848000] [68.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 08:52:01,761 - Train: 6.85% [331900/4848000] [68.5/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 08:52:34,805 - Train: 6.85% [332000/4848000] [68.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 08:53:07,793 - Train: 6.85% [332100/4848000] [68.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:53:40,723 - Train: 6.85% [332200/4848000] [68.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:54:13,651 - Train: 6.85% [332300/4848000] [68.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 08:54:46,585 - Train: 6.86% [332400/4848000] [68.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 08:55:19,504 - Train: 6.86% [332500/4848000] [68.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 08:55:52,451 - Train: 6.86% [332600/4848000] [68.6/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 08:56:25,372 - Train: 6.86% [332700/4848000] [68.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 08:56:58,290 - Train: 6.86% [332800/4848000] [68.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 08:57:31,177 - Train: 6.87% [332900/4848000] [68.7/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 08:58:04,091 - Train: 6.87% [333000/4848000] [68.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 08:58:37,501 - Train: 6.87% [333100/4848000] [68.7/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 08:59:10,403 - Train: 6.87% [333200/4848000] [68.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 08:59:43,396 - Train: 6.88% [333300/4848000] [68.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 09:00:16,440 - Train: 6.88% [333400/4848000] [68.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:00:49,381 - Train: 6.88% [333500/4848000] [68.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:01:22,279 - Train: 6.88% [333600/4848000] [68.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:01:55,181 - Train: 6.88% [333700/4848000] [68.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:02:28,146 - Train: 6.89% [333800/4848000] [68.9/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 09:03:01,120 - Train: 6.89% [333900/4848000] [68.9/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 09:03:34,055 - Train: 6.89% [334000/4848000] [68.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 09:04:07,004 - Train: 6.89% [334100/4848000] [68.9/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 09:04:39,917 - Train: 6.89% [334200/4848000] [68.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:05:12,846 - Train: 6.90% [334300/4848000] [69.0/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 09:05:45,818 - Train: 6.90% [334400/4848000] [69.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:06:19,082 - Train: 6.90% [334500/4848000] [69.0/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:06:23,037 - ==> Total time: 1 day, 9:40:21 Eta: 18 days, 22:20:12 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 09:06:54,126 - Train: 6.90% [334600/4848000] [69.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:07:27,190 - Train: 6.90% [334700/4848000] [69.0/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 09:08:00,128 - Train: 6.91% [334800/4848000] [69.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 09:08:33,777 - Train: 6.91% [334900/4848000] [69.1/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:09:06,762 - Train: 6.91% [335000/4848000] [69.1/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 09:09:39,702 - Train: 6.91% [335100/4848000] [69.1/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 09:10:12,616 - Train: 6.91% [335200/4848000] [69.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:10:45,535 - Train: 6.92% [335300/4848000] [69.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:11:19,491 - Train: 6.92% [335400/4848000] [69.2/1000.0] [batch_t 0.331 (0.339)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:11:52,448 - Train: 6.92% [335500/4848000] [69.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:12:26,022 - Train: 6.92% [335600/4848000] [69.2/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 09:12:58,989 - Train: 6.92% [335700/4848000] [69.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 09:13:31,910 - Train: 6.93% [335800/4848000] [69.3/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 09:14:04,840 - Train: 6.93% [335900/4848000] [69.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:14:37,756 - Train: 6.93% [336000/4848000] [69.3/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 09:15:10,653 - Train: 6.93% [336100/4848000] [69.3/1000.0] [batch_t 0.336 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-04 09:15:43,742 - Train: 6.93% [336200/4848000] [69.3/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:16:16,684 - Train: 6.94% [336300/4848000] [69.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:16:49,626 - Train: 6.94% [336400/4848000] [69.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 09:17:22,484 - Train: 6.94% [336500/4848000] [69.4/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 09:17:55,457 - Train: 6.94% [336600/4848000] [69.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:18:28,396 - Train: 6.95% [336700/4848000] [69.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:19:01,353 - Train: 6.95% [336800/4848000] [69.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:19:34,273 - Train: 6.95% [336900/4848000] [69.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:20:07,218 - Train: 6.95% [337000/4848000] [69.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 09:20:40,145 - Train: 6.95% [337100/4848000] [69.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:21:13,117 - Train: 6.96% [337200/4848000] [69.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 09:21:46,033 - Train: 6.96% [337300/4848000] [69.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:22:18,966 - Train: 6.96% [337400/4848000] [69.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:22:51,904 - Train: 6.96% [337500/4848000] [69.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:23:24,953 - Train: 6.96% [337600/4848000] [69.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:23:57,880 - Train: 6.97% [337700/4848000] [69.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:24:30,813 - Train: 6.97% [337800/4848000] [69.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:25:03,747 - Train: 6.97% [337900/4848000] [69.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:25:36,678 - Train: 6.97% [338000/4848000] [69.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:26:09,565 - Train: 6.97% [338100/4848000] [69.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:26:42,500 - Train: 6.98% [338200/4848000] [69.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:27:15,464 - Train: 6.98% [338300/4848000] [69.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:27:48,357 - Train: 6.98% [338400/4848000] [69.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:28:21,286 - Train: 6.98% [338500/4848000] [69.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 09:28:54,174 - Train: 6.98% [338600/4848000] [69.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:29:27,123 - Train: 6.99% [338700/4848000] [69.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 09:30:00,048 - Train: 6.99% [338800/4848000] [69.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:30:33,011 - Train: 6.99% [338900/4848000] [69.9/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 09:31:06,125 - Train: 6.99% [339000/4848000] [69.9/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 09:31:39,053 - Train: 6.99% [339100/4848000] [69.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:32:11,960 - Train: 7.00% [339200/4848000] [70.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:32:44,938 - Train: 7.00% [339300/4848000] [70.0/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 09:33:04,739 - ==> Total time: 1 day, 10:07:03 Eta: 18 days, 21:16:34 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 09:33:19,874 - Train: 7.00% [339400/4848000] [70.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:33:52,779 - Train: 7.00% [339500/4848000] [70.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 09:34:25,685 - Train: 7.00% [339600/4848000] [70.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 09:34:58,621 - Train: 7.01% [339700/4848000] [70.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:35:31,544 - Train: 7.01% [339800/4848000] [70.1/1000.0] [batch_t 0.336 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-04 09:36:04,527 - Train: 7.01% [339900/4848000] [70.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:36:37,509 - Train: 7.01% [340000/4848000] [70.1/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 09:37:10,406 - Train: 7.02% [340100/4848000] [70.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:37:43,330 - Train: 7.02% [340200/4848000] [70.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:38:16,993 - Train: 7.02% [340300/4848000] [70.2/1000.0] [batch_t 0.331 (0.337)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:38:49,932 - Train: 7.02% [340400/4848000] [70.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 09:39:22,838 - Train: 7.02% [340500/4848000] [70.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:39:55,723 - Train: 7.03% [340600/4848000] [70.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:40:28,605 - Train: 7.03% [340700/4848000] [70.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:41:01,536 - Train: 7.03% [340800/4848000] [70.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:41:34,472 - Train: 7.03% [340900/4848000] [70.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:42:07,431 - Train: 7.03% [341000/4848000] [70.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:42:40,372 - Train: 7.04% [341100/4848000] [70.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:43:13,274 - Train: 7.04% [341200/4848000] [70.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:43:46,216 - Train: 7.04% [341300/4848000] [70.4/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 09:44:19,160 - Train: 7.04% [341400/4848000] [70.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:44:52,082 - Train: 7.04% [341500/4848000] [70.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:45:25,044 - Train: 7.05% [341600/4848000] [70.5/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 09:45:57,977 - Train: 7.05% [341700/4848000] [70.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 09:46:31,107 - Train: 7.05% [341800/4848000] [70.5/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:47:04,094 - Train: 7.05% [341900/4848000] [70.5/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 09:47:37,033 - Train: 7.05% [342000/4848000] [70.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 09:48:09,949 - Train: 7.06% [342100/4848000] [70.6/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 09:48:42,899 - Train: 7.06% [342200/4848000] [70.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:49:15,793 - Train: 7.06% [342300/4848000] [70.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 09:49:48,764 - Train: 7.06% [342400/4848000] [70.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 09:50:21,695 - Train: 7.06% [342500/4848000] [70.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 09:50:54,617 - Train: 7.07% [342600/4848000] [70.7/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 09:51:27,546 - Train: 7.07% [342700/4848000] [70.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:52:00,509 - Train: 7.07% [342800/4848000] [70.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:52:33,435 - Train: 7.07% [342900/4848000] [70.7/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 09:53:06,397 - Train: 7.08% [343000/4848000] [70.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:53:39,343 - Train: 7.08% [343100/4848000] [70.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:54:12,438 - Train: 7.08% [343200/4848000] [70.8/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 09:54:45,338 - Train: 7.08% [343300/4848000] [70.8/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 09:55:18,290 - Train: 7.08% [343400/4848000] [70.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 09:55:51,255 - Train: 7.09% [343500/4848000] [70.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 09:56:24,222 - Train: 7.09% [343600/4848000] [70.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:56:57,180 - Train: 7.09% [343700/4848000] [70.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:57:30,120 - Train: 7.09% [343800/4848000] [70.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:58:03,037 - Train: 7.09% [343900/4848000] [70.9/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 09:58:35,970 - Train: 7.10% [344000/4848000] [71.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 09:59:08,913 - Train: 7.10% [344100/4848000] [71.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 09:59:41,830 - Train: 7.10% [344200/4848000] [71.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 09:59:44,470 - ==> Total time: 1 day, 10:33:42 Eta: 18 days, 20:13:33 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 10:00:16,642 - Train: 7.10% [344300/4848000] [71.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:00:49,598 - Train: 7.10% [344400/4848000] [71.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:01:22,556 - Train: 7.11% [344500/4848000] [71.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:01:55,653 - Train: 7.11% [344600/4848000] [71.1/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:02:28,572 - Train: 7.11% [344700/4848000] [71.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:03:01,492 - Train: 7.11% [344800/4848000] [71.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:03:34,453 - Train: 7.11% [344900/4848000] [71.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:04:07,390 - Train: 7.12% [345000/4848000] [71.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:04:40,338 - Train: 7.12% [345100/4848000] [71.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:05:13,301 - Train: 7.12% [345200/4848000] [71.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:05:46,243 - Train: 7.12% [345300/4848000] [71.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:06:19,190 - Train: 7.12% [345400/4848000] [71.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:06:52,071 - Train: 7.13% [345500/4848000] [71.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:07:24,967 - Train: 7.13% [345600/4848000] [71.3/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 10:07:57,884 - Train: 7.13% [345700/4848000] [71.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:08:30,850 - Train: 7.13% [345800/4848000] [71.3/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:09:03,769 - Train: 7.13% [345900/4848000] [71.3/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 10:09:36,737 - Train: 7.14% [346000/4848000] [71.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:10:09,669 - Train: 7.14% [346100/4848000] [71.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:10:42,596 - Train: 7.14% [346200/4848000] [71.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:11:16,135 - Train: 7.14% [346300/4848000] [71.4/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:11:49,108 - Train: 7.15% [346400/4848000] [71.5/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 10:12:23,342 - Train: 7.15% [346500/4848000] [71.5/1000.0] [batch_t 0.330 (0.342)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:12:56,271 - Train: 7.15% [346600/4848000] [71.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:13:30,430 - Train: 7.15% [346700/4848000] [71.5/1000.0] [batch_t 0.331 (0.341)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:14:03,361 - Train: 7.15% [346800/4848000] [71.5/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 10:14:36,309 - Train: 7.16% [346900/4848000] [71.6/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 10:15:10,588 - Train: 7.16% [347000/4848000] [71.6/1000.0] [batch_t 0.327 (0.343)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 10:15:43,467 - Train: 7.16% [347100/4848000] [71.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 10:16:17,989 - Train: 7.16% [347200/4848000] [71.6/1000.0] [batch_t 0.330 (0.345)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:16:50,921 - Train: 7.16% [347300/4848000] [71.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:17:24,004 - Train: 7.17% [347400/4848000] [71.7/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 10:17:56,909 - Train: 7.17% [347500/4848000] [71.7/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 10:18:30,379 - Train: 7.17% [347600/4848000] [71.7/1000.0] [batch_t 0.332 (0.335)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 10:19:03,307 - Train: 7.17% [347700/4848000] [71.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:19:36,242 - Train: 7.17% [347800/4848000] [71.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 10:20:09,497 - Train: 7.18% [347900/4848000] [71.8/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:20:42,434 - Train: 7.18% [348000/4848000] [71.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:21:15,387 - Train: 7.18% [348100/4848000] [71.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:21:48,328 - Train: 7.18% [348200/4848000] [71.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:22:21,269 - Train: 7.18% [348300/4848000] [71.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 10:22:54,205 - Train: 7.19% [348400/4848000] [71.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 10:23:27,154 - Train: 7.19% [348500/4848000] [71.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 10:24:00,116 - Train: 7.19% [348600/4848000] [71.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 10:24:33,102 - Train: 7.19% [348700/4848000] [71.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:25:06,068 - Train: 7.19% [348800/4848000] [71.9/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 10:25:39,075 - Train: 7.20% [348900/4848000] [72.0/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 10:26:11,970 - Train: 7.20% [349000/4848000] [72.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:26:30,417 - ==> Total time: 1 day, 11:00:28 Eta: 18 days, 19:12:52 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 10:26:46,765 - Train: 7.20% [349100/4848000] [72.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:27:19,660 - Train: 7.20% [349200/4848000] [72.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:27:52,611 - Train: 7.21% [349300/4848000] [72.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:28:25,507 - Train: 7.21% [349400/4848000] [72.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 10:28:58,372 - Train: 7.21% [349500/4848000] [72.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:29:31,323 - Train: 7.21% [349600/4848000] [72.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:30:04,198 - Train: 7.21% [349700/4848000] [72.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 10:30:37,099 - Train: 7.22% [349800/4848000] [72.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:31:10,025 - Train: 7.22% [349900/4848000] [72.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:31:42,929 - Train: 7.22% [350000/4848000] [72.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:32:15,880 - Train: 7.22% [350100/4848000] [72.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:32:48,995 - Train: 7.22% [350200/4848000] [72.2/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:33:21,940 - Train: 7.23% [350300/4848000] [72.3/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 10:33:54,959 - Train: 7.23% [350400/4848000] [72.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:34:27,939 - Train: 7.23% [350500/4848000] [72.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:35:00,942 - Train: 7.23% [350600/4848000] [72.3/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 10:35:33,918 - Train: 7.23% [350700/4848000] [72.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:36:06,875 - Train: 7.24% [350800/4848000] [72.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:36:39,840 - Train: 7.24% [350900/4848000] [72.4/1000.0] [batch_t 0.336 (0.330)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-04 10:37:12,806 - Train: 7.24% [351000/4848000] [72.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:37:45,794 - Train: 7.24% [351100/4848000] [72.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:38:18,726 - Train: 7.24% [351200/4848000] [72.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:38:51,661 - Train: 7.25% [351300/4848000] [72.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:39:24,616 - Train: 7.25% [351400/4848000] [72.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:39:57,527 - Train: 7.25% [351500/4848000] [72.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 10:40:30,485 - Train: 7.25% [351600/4848000] [72.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 10:41:03,598 - Train: 7.25% [351700/4848000] [72.5/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 10:41:36,541 - Train: 7.26% [351800/4848000] [72.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 10:42:09,476 - Train: 7.26% [351900/4848000] [72.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:42:42,409 - Train: 7.26% [352000/4848000] [72.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:43:15,339 - Train: 7.26% [352100/4848000] [72.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:43:48,310 - Train: 7.26% [352200/4848000] [72.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 10:44:21,245 - Train: 7.27% [352300/4848000] [72.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 10:44:54,172 - Train: 7.27% [352400/4848000] [72.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:45:27,143 - Train: 7.27% [352500/4848000] [72.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:46:00,080 - Train: 7.27% [352600/4848000] [72.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:46:33,040 - Train: 7.28% [352700/4848000] [72.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 10:47:06,004 - Train: 7.28% [352800/4848000] [72.8/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 10:47:38,948 - Train: 7.28% [352900/4848000] [72.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:48:11,886 - Train: 7.28% [353000/4848000] [72.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:48:44,952 - Train: 7.28% [353100/4848000] [72.8/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 10:49:17,916 - Train: 7.29% [353200/4848000] [72.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 10:49:50,858 - Train: 7.29% [353300/4848000] [72.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 10:50:23,840 - Train: 7.29% [353400/4848000] [72.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:50:56,820 - Train: 7.29% [353500/4848000] [72.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:51:29,790 - Train: 7.29% [353600/4848000] [72.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:52:02,748 - Train: 7.30% [353700/4848000] [73.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:52:35,721 - Train: 7.30% [353800/4848000] [73.0/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 10:53:08,663 - Train: 7.30% [353900/4848000] [73.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:53:09,984 - ==> Total time: 1 day, 11:27:08 Eta: 18 days, 18:11:46 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 10:53:43,563 - Train: 7.30% [354000/4848000] [73.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 10:54:16,459 - Train: 7.30% [354100/4848000] [73.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:54:49,391 - Train: 7.31% [354200/4848000] [73.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:55:22,357 - Train: 7.31% [354300/4848000] [73.1/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 10:55:55,381 - Train: 7.31% [354400/4848000] [73.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:56:28,325 - Train: 7.31% [354500/4848000] [73.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 10:57:01,255 - Train: 7.31% [354600/4848000] [73.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:57:34,211 - Train: 7.32% [354700/4848000] [73.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 10:58:07,144 - Train: 7.32% [354800/4848000] [73.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:58:40,073 - Train: 7.32% [354900/4848000] [73.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 10:59:14,060 - Train: 7.32% [355000/4848000] [73.2/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 10:59:46,985 - Train: 7.32% [355100/4848000] [73.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:00:19,922 - Train: 7.33% [355200/4848000] [73.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:00:52,867 - Train: 7.33% [355300/4848000] [73.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:01:25,791 - Train: 7.33% [355400/4848000] [73.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:01:58,704 - Train: 7.33% [355500/4848000] [73.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:02:31,605 - Train: 7.33% [355600/4848000] [73.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:03:04,496 - Train: 7.34% [355700/4848000] [73.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:03:37,483 - Train: 7.34% [355800/4848000] [73.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 11:04:10,395 - Train: 7.34% [355900/4848000] [73.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:04:43,315 - Train: 7.34% [356000/4848000] [73.4/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 11:05:16,205 - Train: 7.35% [356100/4848000] [73.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:05:49,124 - Train: 7.35% [356200/4848000] [73.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:06:22,027 - Train: 7.35% [356300/4848000] [73.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:06:54,927 - Train: 7.35% [356400/4848000] [73.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:07:27,798 - Train: 7.35% [356500/4848000] [73.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:08:00,686 - Train: 7.36% [356600/4848000] [73.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:08:33,580 - Train: 7.36% [356700/4848000] [73.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:09:06,530 - Train: 7.36% [356800/4848000] [73.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:09:39,436 - Train: 7.36% [356900/4848000] [73.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:10:12,345 - Train: 7.36% [357000/4848000] [73.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:10:45,245 - Train: 7.37% [357100/4848000] [73.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:11:18,156 - Train: 7.37% [357200/4848000] [73.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 11:11:51,226 - Train: 7.37% [357300/4848000] [73.7/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 11:12:24,131 - Train: 7.37% [357400/4848000] [73.7/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 11:12:57,047 - Train: 7.37% [357500/4848000] [73.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:13:29,971 - Train: 7.38% [357600/4848000] [73.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:14:02,875 - Train: 7.38% [357700/4848000] [73.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:14:35,830 - Train: 7.38% [357800/4848000] [73.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:15:08,733 - Train: 7.38% [357900/4848000] [73.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:15:41,711 - Train: 7.38% [358000/4848000] [73.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:16:14,618 - Train: 7.39% [358100/4848000] [73.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:16:47,590 - Train: 7.39% [358200/4848000] [73.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:17:20,533 - Train: 7.39% [358300/4848000] [73.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:17:53,529 - Train: 7.39% [358400/4848000] [73.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:18:26,489 - Train: 7.39% [358500/4848000] [73.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:18:59,443 - Train: 7.40% [358600/4848000] [74.0/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 11:19:32,629 - Train: 7.40% [358700/4848000] [74.0/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:19:49,779 - ==> Total time: 1 day, 11:53:48 Eta: 18 days, 17:11:39 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 11:20:07,423 - Train: 7.40% [358800/4848000] [74.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:20:40,295 - Train: 7.40% [358900/4848000] [74.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:21:13,207 - Train: 7.41% [359000/4848000] [74.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:21:46,111 - Train: 7.41% [359100/4848000] [74.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:22:19,107 - Train: 7.41% [359200/4848000] [74.1/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 11:22:52,021 - Train: 7.41% [359300/4848000] [74.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 11:23:25,000 - Train: 7.41% [359400/4848000] [74.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:23:57,977 - Train: 7.42% [359500/4848000] [74.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 11:24:30,906 - Train: 7.42% [359600/4848000] [74.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 11:25:03,833 - Train: 7.42% [359700/4848000] [74.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:25:36,761 - Train: 7.42% [359800/4848000] [74.2/1000.0] [batch_t 0.337 (0.329)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-04 11:26:09,918 - Train: 7.42% [359900/4848000] [74.2/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:26:42,996 - Train: 7.43% [360000/4848000] [74.3/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:27:15,910 - Train: 7.43% [360100/4848000] [74.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:27:48,854 - Train: 7.43% [360200/4848000] [74.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 11:28:21,771 - Train: 7.43% [360300/4848000] [74.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:28:54,738 - Train: 7.43% [360400/4848000] [74.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:29:27,673 - Train: 7.44% [360500/4848000] [74.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:30:00,609 - Train: 7.44% [360600/4848000] [74.4/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 11:30:33,574 - Train: 7.44% [360700/4848000] [74.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:31:06,529 - Train: 7.44% [360800/4848000] [74.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:31:39,478 - Train: 7.44% [360900/4848000] [74.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:32:12,496 - Train: 7.45% [361000/4848000] [74.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:32:45,501 - Train: 7.45% [361100/4848000] [74.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:33:18,511 - Train: 7.45% [361200/4848000] [74.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 11:33:51,491 - Train: 7.45% [361300/4848000] [74.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:34:25,552 - Train: 7.45% [361400/4848000] [74.5/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:34:58,561 - Train: 7.46% [361500/4848000] [74.6/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 11:35:31,506 - Train: 7.46% [361600/4848000] [74.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 11:36:04,490 - Train: 7.46% [361700/4848000] [74.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:36:37,458 - Train: 7.46% [361800/4848000] [74.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:37:10,422 - Train: 7.46% [361900/4848000] [74.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:37:43,387 - Train: 7.47% [362000/4848000] [74.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:38:16,382 - Train: 7.47% [362100/4848000] [74.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:38:49,299 - Train: 7.47% [362200/4848000] [74.7/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 11:39:22,252 - Train: 7.47% [362300/4848000] [74.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:39:55,167 - Train: 7.48% [362400/4848000] [74.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:40:28,079 - Train: 7.48% [362500/4848000] [74.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:41:01,040 - Train: 7.48% [362600/4848000] [74.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:41:34,012 - Train: 7.48% [362700/4848000] [74.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 11:42:06,972 - Train: 7.48% [362800/4848000] [74.8/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 11:42:40,061 - Train: 7.49% [362900/4848000] [74.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:43:13,021 - Train: 7.49% [363000/4848000] [74.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:43:46,004 - Train: 7.49% [363100/4848000] [74.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:44:18,973 - Train: 7.49% [363200/4848000] [74.9/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 11:44:51,940 - Train: 7.49% [363300/4848000] [74.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:45:24,883 - Train: 7.50% [363400/4848000] [75.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 11:45:57,808 - Train: 7.50% [363500/4848000] [75.0/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 11:46:30,792 - Train: 7.50% [363600/4848000] [75.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:46:30,796 - ==> Total time: 1 day, 12:20:29 Eta: 18 days, 16:12:41 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 11:47:05,711 - Train: 7.50% [363700/4848000] [75.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:47:38,671 - Train: 7.50% [363800/4848000] [75.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:48:11,642 - Train: 7.51% [363900/4848000] [75.1/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 11:48:44,605 - Train: 7.51% [364000/4848000] [75.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:49:17,554 - Train: 7.51% [364100/4848000] [75.1/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 11:49:50,660 - Train: 7.51% [364200/4848000] [75.1/1000.0] [batch_t 0.335 (0.331)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 11:50:23,655 - Train: 7.51% [364300/4848000] [75.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:50:56,564 - Train: 7.52% [364400/4848000] [75.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:51:29,496 - Train: 7.52% [364500/4848000] [75.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:52:02,430 - Train: 7.52% [364600/4848000] [75.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:52:35,370 - Train: 7.52% [364700/4848000] [75.2/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 11:53:08,333 - Train: 7.52% [364800/4848000] [75.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:53:41,299 - Train: 7.53% [364900/4848000] [75.3/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:54:14,266 - Train: 7.53% [365000/4848000] [75.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 11:54:47,204 - Train: 7.53% [365100/4848000] [75.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:55:20,143 - Train: 7.53% [365200/4848000] [75.3/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 11:55:53,092 - Train: 7.54% [365300/4848000] [75.4/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 11:56:26,048 - Train: 7.54% [365400/4848000] [75.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:56:58,975 - Train: 7.54% [365500/4848000] [75.4/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 11:57:32,034 - Train: 7.54% [365600/4848000] [75.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:58:04,968 - Train: 7.54% [365700/4848000] [75.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 11:58:37,925 - Train: 7.55% [365800/4848000] [75.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 11:59:10,890 - Train: 7.55% [365900/4848000] [75.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 11:59:43,859 - Train: 7.55% [366000/4848000] [75.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:00:16,787 - Train: 7.55% [366100/4848000] [75.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:00:49,744 - Train: 7.55% [366200/4848000] [75.5/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 12:01:22,682 - Train: 7.56% [366300/4848000] [75.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 12:01:55,591 - Train: 7.56% [366400/4848000] [75.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 12:02:28,522 - Train: 7.56% [366500/4848000] [75.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 12:03:01,433 - Train: 7.56% [366600/4848000] [75.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 12:03:34,336 - Train: 7.56% [366700/4848000] [75.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 12:04:07,286 - Train: 7.57% [366800/4848000] [75.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 12:04:40,203 - Train: 7.57% [366900/4848000] [75.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:05:13,141 - Train: 7.57% [367000/4848000] [75.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 12:05:46,209 - Train: 7.57% [367100/4848000] [75.7/1000.0] [batch_t 0.333 (0.331)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 12:06:19,161 - Train: 7.57% [367200/4848000] [75.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 12:06:52,092 - Train: 7.58% [367300/4848000] [75.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 12:07:25,044 - Train: 7.58% [367400/4848000] [75.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:07:57,960 - Train: 7.58% [367500/4848000] [75.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 12:08:30,886 - Train: 7.58% [367600/4848000] [75.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 12:09:03,948 - Train: 7.58% [367700/4848000] [75.8/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 12:09:36,893 - Train: 7.59% [367800/4848000] [75.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 12:10:09,807 - Train: 7.59% [367900/4848000] [75.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:10:42,765 - Train: 7.59% [368000/4848000] [75.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 12:11:15,709 - Train: 7.59% [368100/4848000] [75.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 12:11:48,668 - Train: 7.59% [368200/4848000] [75.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 12:12:21,620 - Train: 7.60% [368300/4848000] [76.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:12:54,570 - Train: 7.60% [368400/4848000] [76.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 12:13:11,314 - ==> Total time: 1 day, 12:47:09 Eta: 18 days, 15:14:27 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 12:13:30,471 - Train: 7.60% [368500/4848000] [76.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:14:03,417 - Train: 7.60% [368600/4848000] [76.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:14:36,367 - Train: 7.61% [368700/4848000] [76.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:15:09,342 - Train: 7.61% [368800/4848000] [76.1/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 12:15:42,278 - Train: 7.61% [368900/4848000] [76.1/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 12:16:15,183 - Train: 7.61% [369000/4848000] [76.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 12:16:48,152 - Train: 7.61% [369100/4848000] [76.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 12:17:21,095 - Train: 7.62% [369200/4848000] [76.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:17:54,025 - Train: 7.62% [369300/4848000] [76.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 12:18:26,984 - Train: 7.62% [369400/4848000] [76.2/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 12:18:59,921 - Train: 7.62% [369500/4848000] [76.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:19:32,882 - Train: 7.62% [369600/4848000] [76.2/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 12:20:05,807 - Train: 7.63% [369700/4848000] [76.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 12:20:38,857 - Train: 7.63% [369800/4848000] [76.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 12:21:11,784 - Train: 7.63% [369900/4848000] [76.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 12:21:44,694 - Train: 7.63% [370000/4848000] [76.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:22:17,658 - Train: 7.63% [370100/4848000] [76.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 12:22:50,636 - Train: 7.64% [370200/4848000] [76.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:23:23,571 - Train: 7.64% [370300/4848000] [76.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 12:23:56,496 - Train: 7.64% [370400/4848000] [76.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 12:24:29,414 - Train: 7.64% [370500/4848000] [76.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 12:25:02,332 - Train: 7.64% [370600/4848000] [76.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 12:25:35,269 - Train: 7.65% [370700/4848000] [76.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:26:08,208 - Train: 7.65% [370800/4848000] [76.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:26:41,073 - Train: 7.65% [370900/4848000] [76.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 12:27:14,076 - Train: 7.65% [371000/4848000] [76.5/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 12:27:46,993 - Train: 7.65% [371100/4848000] [76.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:28:20,087 - Train: 7.66% [371200/4848000] [76.6/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 12:28:53,001 - Train: 7.66% [371300/4848000] [76.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:29:25,949 - Train: 7.66% [371400/4848000] [76.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 12:29:58,880 - Train: 7.66% [371500/4848000] [76.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 12:30:31,824 - Train: 7.67% [371600/4848000] [76.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 12:31:04,711 - Train: 7.67% [371700/4848000] [76.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:31:37,655 - Train: 7.67% [371800/4848000] [76.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:32:10,578 - Train: 7.67% [371900/4848000] [76.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 12:32:43,475 - Train: 7.67% [372000/4848000] [76.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 12:33:16,407 - Train: 7.68% [372100/4848000] [76.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 12:33:49,313 - Train: 7.68% [372200/4848000] [76.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:34:22,267 - Train: 7.68% [372300/4848000] [76.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 12:34:55,190 - Train: 7.68% [372400/4848000] [76.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 12:35:28,138 - Train: 7.68% [372500/4848000] [76.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 12:36:01,105 - Train: 7.69% [372600/4848000] [76.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:36:34,176 - Train: 7.69% [372700/4848000] [76.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 12:37:07,123 - Train: 7.69% [372800/4848000] [76.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:37:40,078 - Train: 7.69% [372900/4848000] [76.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:38:13,066 - Train: 7.69% [373000/4848000] [76.9/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 12:38:46,157 - Train: 7.70% [373100/4848000] [77.0/1000.0] [batch_t 0.333 (0.331)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 12:39:19,079 - Train: 7.70% [373200/4848000] [77.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 12:39:50,675 - ==> Total time: 1 day, 13:13:49 Eta: 18 days, 14:16:49 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 12:39:53,931 - Train: 7.70% [373300/4848000] [77.0/1000.0] [batch_t 0.333 (0.346)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 12:40:26,893 - Train: 7.70% [373400/4848000] [77.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 12:40:59,890 - Train: 7.70% [373500/4848000] [77.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 12:41:32,855 - Train: 7.71% [373600/4848000] [77.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:42:05,832 - Train: 7.71% [373700/4848000] [77.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 12:42:38,745 - Train: 7.71% [373800/4848000] [77.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 12:43:11,738 - Train: 7.71% [373900/4848000] [77.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:43:44,866 - Train: 7.71% [374000/4848000] [77.1/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 12:44:17,829 - Train: 7.72% [374100/4848000] [77.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 12:44:50,773 - Train: 7.72% [374200/4848000] [77.2/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 12:45:23,737 - Train: 7.72% [374300/4848000] [77.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 12:45:56,666 - Train: 7.72% [374400/4848000] [77.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:46:29,628 - Train: 7.72% [374500/4848000] [77.2/1000.0] [batch_t 0.341 (0.330)] [data_t 0.002] [optim_t 0.340] [lr 0.005000] 2024-04-04 12:47:02,616 - Train: 7.73% [374600/4848000] [77.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 12:47:35,587 - Train: 7.73% [374700/4848000] [77.3/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 12:48:08,502 - Train: 7.73% [374800/4848000] [77.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:48:41,464 - Train: 7.73% [374900/4848000] [77.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 12:49:14,405 - Train: 7.74% [375000/4848000] [77.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 12:49:47,320 - Train: 7.74% [375100/4848000] [77.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:50:20,254 - Train: 7.74% [375200/4848000] [77.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 12:50:53,198 - Train: 7.74% [375300/4848000] [77.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 12:51:26,299 - Train: 7.74% [375400/4848000] [77.4/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:51:59,232 - Train: 7.75% [375500/4848000] [77.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:52:32,135 - Train: 7.75% [375600/4848000] [77.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 12:53:05,093 - Train: 7.75% [375700/4848000] [77.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:53:38,033 - Train: 7.75% [375800/4848000] [77.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 12:54:10,970 - Train: 7.75% [375900/4848000] [77.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 12:54:43,922 - Train: 7.76% [376000/4848000] [77.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:55:16,842 - Train: 7.76% [376100/4848000] [77.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:55:49,817 - Train: 7.76% [376200/4848000] [77.6/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 12:56:22,760 - Train: 7.76% [376300/4848000] [77.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:56:55,720 - Train: 7.76% [376400/4848000] [77.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:57:28,612 - Train: 7.77% [376500/4848000] [77.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:58:01,591 - Train: 7.77% [376600/4848000] [77.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 12:58:35,028 - Train: 7.77% [376700/4848000] [77.7/1000.0] [batch_t 0.334 (0.334)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 12:59:08,068 - Train: 7.77% [376800/4848000] [77.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 12:59:41,027 - Train: 7.77% [376900/4848000] [77.7/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 13:00:13,968 - Train: 7.78% [377000/4848000] [77.8/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 13:00:46,911 - Train: 7.78% [377100/4848000] [77.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 13:01:19,882 - Train: 7.78% [377200/4848000] [77.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:01:52,831 - Train: 7.78% [377300/4848000] [77.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:02:25,775 - Train: 7.78% [377400/4848000] [77.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:02:58,696 - Train: 7.79% [377500/4848000] [77.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:03:31,631 - Train: 7.79% [377600/4848000] [77.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 13:04:04,630 - Train: 7.79% [377700/4848000] [77.9/1000.0] [batch_t 0.336 (0.330)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-04 13:04:37,542 - Train: 7.79% [377800/4848000] [77.9/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 13:05:10,465 - Train: 7.79% [377900/4848000] [77.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 13:05:43,413 - Train: 7.80% [378000/4848000] [78.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:06:16,370 - Train: 7.80% [378100/4848000] [78.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 13:06:30,887 - ==> Total time: 1 day, 13:40:29 Eta: 18 days, 13:20:08 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 13:06:51,606 - Train: 7.80% [378200/4848000] [78.0/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 13:07:24,573 - Train: 7.80% [378300/4848000] [78.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:07:57,526 - Train: 7.81% [378400/4848000] [78.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:08:30,452 - Train: 7.81% [378500/4848000] [78.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 13:09:03,364 - Train: 7.81% [378600/4848000] [78.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:09:36,272 - Train: 7.81% [378700/4848000] [78.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 13:10:09,218 - Train: 7.81% [378800/4848000] [78.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 13:10:42,153 - Train: 7.82% [378900/4848000] [78.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:11:15,081 - Train: 7.82% [379000/4848000] [78.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:11:47,986 - Train: 7.82% [379100/4848000] [78.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:12:21,690 - Train: 7.82% [379200/4848000] [78.2/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:12:54,583 - Train: 7.82% [379300/4848000] [78.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:13:27,509 - Train: 7.83% [379400/4848000] [78.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:14:00,392 - Train: 7.83% [379500/4848000] [78.3/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 13:14:34,376 - Train: 7.83% [379600/4848000] [78.3/1000.0] [batch_t 0.326 (0.340)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 13:15:08,116 - Train: 7.83% [379700/4848000] [78.3/1000.0] [batch_t 0.331 (0.337)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 13:15:41,064 - Train: 7.83% [379800/4848000] [78.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 13:16:13,970 - Train: 7.84% [379900/4848000] [78.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 13:16:46,900 - Train: 7.84% [380000/4848000] [78.4/1000.0] [batch_t 0.333 (0.329)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-04 13:17:19,805 - Train: 7.84% [380100/4848000] [78.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 13:17:52,745 - Train: 7.84% [380200/4848000] [78.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 13:18:25,670 - Train: 7.84% [380300/4848000] [78.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:18:58,583 - Train: 7.85% [380400/4848000] [78.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:19:31,459 - Train: 7.85% [380500/4848000] [78.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:20:04,352 - Train: 7.85% [380600/4848000] [78.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:20:37,278 - Train: 7.85% [380700/4848000] [78.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:21:10,195 - Train: 7.85% [380800/4848000] [78.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:21:43,124 - Train: 7.86% [380900/4848000] [78.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:22:16,054 - Train: 7.86% [381000/4848000] [78.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 13:22:49,024 - Train: 7.86% [381100/4848000] [78.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:23:21,953 - Train: 7.86% [381200/4848000] [78.6/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 13:23:54,877 - Train: 7.87% [381300/4848000] [78.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:24:27,772 - Train: 7.87% [381400/4848000] [78.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 13:25:00,682 - Train: 7.87% [381500/4848000] [78.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:25:33,608 - Train: 7.87% [381600/4848000] [78.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:26:06,519 - Train: 7.87% [381700/4848000] [78.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:26:39,439 - Train: 7.88% [381800/4848000] [78.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:27:13,616 - Train: 7.88% [381900/4848000] [78.8/1000.0] [batch_t 0.328 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:27:46,515 - Train: 7.88% [382000/4848000] [78.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:28:19,456 - Train: 7.88% [382100/4848000] [78.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:28:52,377 - Train: 7.88% [382200/4848000] [78.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:29:25,272 - Train: 7.89% [382300/4848000] [78.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:29:58,192 - Train: 7.89% [382400/4848000] [78.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:30:31,157 - Train: 7.89% [382500/4848000] [78.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 13:31:04,093 - Train: 7.89% [382600/4848000] [78.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:31:36,990 - Train: 7.89% [382700/4848000] [78.9/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 13:32:10,642 - Train: 7.90% [382800/4848000] [79.0/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 13:32:43,533 - Train: 7.90% [382900/4848000] [79.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:33:13,839 - ==> Total time: 1 day, 14:07:12 Eta: 18 days, 12:24:45 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 13:33:19,023 - Train: 7.90% [383000/4848000] [79.0/1000.0] [batch_t 0.331 (0.368)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 13:33:51,941 - Train: 7.90% [383100/4848000] [79.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:34:24,867 - Train: 7.90% [383200/4848000] [79.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:34:57,781 - Train: 7.91% [383300/4848000] [79.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:35:30,689 - Train: 7.91% [383400/4848000] [79.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:36:03,645 - Train: 7.91% [383500/4848000] [79.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:36:36,544 - Train: 7.91% [383600/4848000] [79.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 13:37:09,435 - Train: 7.91% [383700/4848000] [79.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:37:42,491 - Train: 7.92% [383800/4848000] [79.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:38:15,399 - Train: 7.92% [383900/4848000] [79.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:38:48,357 - Train: 7.92% [384000/4848000] [79.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:39:21,265 - Train: 7.92% [384100/4848000] [79.2/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 13:39:54,183 - Train: 7.92% [384200/4848000] [79.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:40:27,148 - Train: 7.93% [384300/4848000] [79.3/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 13:41:00,108 - Train: 7.93% [384400/4848000] [79.3/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 13:41:32,980 - Train: 7.93% [384500/4848000] [79.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 13:42:05,889 - Train: 7.93% [384600/4848000] [79.3/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 13:42:38,783 - Train: 7.94% [384700/4848000] [79.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 13:43:11,679 - Train: 7.94% [384800/4848000] [79.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:43:44,624 - Train: 7.94% [384900/4848000] [79.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 13:44:17,532 - Train: 7.94% [385000/4848000] [79.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:44:50,453 - Train: 7.94% [385100/4848000] [79.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:45:23,457 - Train: 7.95% [385200/4848000] [79.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:45:56,353 - Train: 7.95% [385300/4848000] [79.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 13:46:29,325 - Train: 7.95% [385400/4848000] [79.5/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 13:47:02,239 - Train: 7.95% [385500/4848000] [79.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:47:35,131 - Train: 7.95% [385600/4848000] [79.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 13:48:08,084 - Train: 7.96% [385700/4848000] [79.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 13:48:41,037 - Train: 7.96% [385800/4848000] [79.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:49:13,989 - Train: 7.96% [385900/4848000] [79.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:49:46,907 - Train: 7.96% [386000/4848000] [79.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 13:50:21,343 - Train: 7.96% [386100/4848000] [79.6/1000.0] [batch_t 0.327 (0.344)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 13:50:56,224 - Train: 7.97% [386200/4848000] [79.7/1000.0] [batch_t 0.331 (0.349)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 13:51:29,150 - Train: 7.97% [386300/4848000] [79.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:52:02,120 - Train: 7.97% [386400/4848000] [79.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:52:35,093 - Train: 7.97% [386500/4848000] [79.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:53:08,244 - Train: 7.97% [386600/4848000] [79.7/1000.0] [batch_t 0.335 (0.331)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 13:53:41,197 - Train: 7.98% [386700/4848000] [79.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:54:14,144 - Train: 7.98% [386800/4848000] [79.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 13:54:47,090 - Train: 7.98% [386900/4848000] [79.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:55:20,029 - Train: 7.98% [387000/4848000] [79.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:55:52,991 - Train: 7.98% [387100/4848000] [79.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 13:56:25,898 - Train: 7.99% [387200/4848000] [79.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:56:58,806 - Train: 7.99% [387300/4848000] [79.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 13:57:31,719 - Train: 7.99% [387400/4848000] [79.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:58:04,588 - Train: 7.99% [387500/4848000] [79.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 13:58:37,489 - Train: 8.00% [387600/4848000] [80.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 13:59:12,626 - Train: 8.00% [387700/4848000] [80.0/1000.0] [batch_t 0.329 (0.351)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 13:59:45,541 - Train: 8.00% [387800/4848000] [80.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 13:59:58,709 - ==> Total time: 1 day, 14:33:57 Eta: 18 days, 11:30:27 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 14:00:23,253 - Train: 8.00% [387900/4848000] [80.0/1000.0] [batch_t 0.328 (0.377)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:00:56,151 - Train: 8.00% [388000/4848000] [80.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 14:01:29,082 - Train: 8.01% [388100/4848000] [80.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 14:02:02,001 - Train: 8.01% [388200/4848000] [80.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:02:34,946 - Train: 8.01% [388300/4848000] [80.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 14:03:07,848 - Train: 8.01% [388400/4848000] [80.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 14:03:40,794 - Train: 8.01% [388500/4848000] [80.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:04:13,699 - Train: 8.02% [388600/4848000] [80.2/1000.0] [batch_t 0.336 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-04 14:04:46,600 - Train: 8.02% [388700/4848000] [80.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 14:05:19,517 - Train: 8.02% [388800/4848000] [80.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 14:05:52,434 - Train: 8.02% [388900/4848000] [80.2/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 14:06:25,835 - Train: 8.02% [389000/4848000] [80.2/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:06:58,853 - Train: 8.03% [389100/4848000] [80.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 14:07:33,117 - Train: 8.03% [389200/4848000] [80.3/1000.0] [batch_t 0.330 (0.343)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 14:08:06,487 - Train: 8.03% [389300/4848000] [80.3/1000.0] [batch_t 0.330 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 14:08:39,614 - Train: 8.03% [389400/4848000] [80.3/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 14:09:12,654 - Train: 8.03% [389500/4848000] [80.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 14:09:45,626 - Train: 8.04% [389600/4848000] [80.4/1000.0] [batch_t 0.336 (0.330)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-04 14:10:19,434 - Train: 8.04% [389700/4848000] [80.4/1000.0] [batch_t 0.334 (0.338)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 14:10:52,334 - Train: 8.04% [389800/4848000] [80.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 14:11:26,952 - Train: 8.04% [389900/4848000] [80.4/1000.0] [batch_t 0.330 (0.346)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 14:11:59,886 - Train: 8.04% [390000/4848000] [80.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:12:33,860 - Train: 8.05% [390100/4848000] [80.5/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:13:07,460 - Train: 8.05% [390200/4848000] [80.5/1000.0] [batch_t 0.332 (0.336)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 14:13:40,413 - Train: 8.05% [390300/4848000] [80.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 14:14:14,212 - Train: 8.05% [390400/4848000] [80.5/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 14:14:47,135 - Train: 8.05% [390500/4848000] [80.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 14:15:21,729 - Train: 8.06% [390600/4848000] [80.6/1000.0] [batch_t 0.328 (0.346)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:15:54,658 - Train: 8.06% [390700/4848000] [80.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 14:16:28,680 - Train: 8.06% [390800/4848000] [80.6/1000.0] [batch_t 0.331 (0.340)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 14:17:01,600 - Train: 8.06% [390900/4848000] [80.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 14:17:35,271 - Train: 8.07% [391000/4848000] [80.7/1000.0] [batch_t 0.331 (0.337)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 14:18:08,680 - Train: 8.07% [391100/4848000] [80.7/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:18:41,580 - Train: 8.07% [391200/4848000] [80.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 14:19:14,497 - Train: 8.07% [391300/4848000] [80.7/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 14:19:47,423 - Train: 8.07% [391400/4848000] [80.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 14:20:21,454 - Train: 8.08% [391500/4848000] [80.8/1000.0] [batch_t 0.331 (0.340)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 14:20:54,371 - Train: 8.08% [391600/4848000] [80.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 14:21:27,280 - Train: 8.08% [391700/4848000] [80.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 14:22:00,162 - Train: 8.08% [391800/4848000] [80.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 14:22:33,060 - Train: 8.08% [391900/4848000] [80.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 14:23:06,045 - Train: 8.09% [392000/4848000] [80.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 14:23:38,933 - Train: 8.09% [392100/4848000] [80.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 14:24:12,605 - Train: 8.09% [392200/4848000] [80.9/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 14:24:45,483 - Train: 8.09% [392300/4848000] [80.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 14:25:18,449 - Train: 8.09% [392400/4848000] [80.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 14:25:51,384 - Train: 8.10% [392500/4848000] [81.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 14:26:24,453 - Train: 8.10% [392600/4848000] [81.0/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 14:26:53,410 - ==> Total time: 1 day, 15:00:51 Eta: 18 days, 10:38:42 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 14:26:59,247 - Train: 8.10% [392700/4848000] [81.0/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 14:27:32,136 - Train: 8.10% [392800/4848000] [81.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 14:28:05,058 - Train: 8.10% [392900/4848000] [81.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 14:28:37,942 - Train: 8.11% [393000/4848000] [81.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 14:29:10,848 - Train: 8.11% [393100/4848000] [81.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 14:29:43,753 - Train: 8.11% [393200/4848000] [81.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 14:30:16,700 - Train: 8.11% [393300/4848000] [81.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 14:30:49,664 - Train: 8.11% [393400/4848000] [81.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 14:31:22,712 - Train: 8.12% [393500/4848000] [81.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:31:55,601 - Train: 8.12% [393600/4848000] [81.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 14:32:28,513 - Train: 8.12% [393700/4848000] [81.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 14:33:01,426 - Train: 8.12% [393800/4848000] [81.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 14:33:34,344 - Train: 8.12% [393900/4848000] [81.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 14:34:07,307 - Train: 8.13% [394000/4848000] [81.3/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 14:34:40,191 - Train: 8.13% [394100/4848000] [81.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 14:35:13,130 - Train: 8.13% [394200/4848000] [81.3/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 14:35:46,089 - Train: 8.13% [394300/4848000] [81.3/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 14:36:18,988 - Train: 8.14% [394400/4848000] [81.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 14:36:51,953 - Train: 8.14% [394500/4848000] [81.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 14:37:24,885 - Train: 8.14% [394600/4848000] [81.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 14:37:57,817 - Train: 8.14% [394700/4848000] [81.4/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 14:38:30,744 - Train: 8.14% [394800/4848000] [81.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 14:39:03,699 - Train: 8.15% [394900/4848000] [81.5/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 14:39:36,705 - Train: 8.15% [395000/4848000] [81.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:40:09,634 - Train: 8.15% [395100/4848000] [81.5/1000.0] [batch_t 0.344 (0.329)] [data_t 0.002] [optim_t 0.342] [lr 0.005000] 2024-04-04 14:40:42,600 - Train: 8.15% [395200/4848000] [81.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 14:41:15,569 - Train: 8.15% [395300/4848000] [81.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 14:41:48,465 - Train: 8.16% [395400/4848000] [81.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 14:42:21,396 - Train: 8.16% [395500/4848000] [81.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 14:42:54,420 - Train: 8.16% [395600/4848000] [81.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 14:43:27,336 - Train: 8.16% [395700/4848000] [81.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 14:44:00,269 - Train: 8.16% [395800/4848000] [81.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 14:44:33,252 - Train: 8.17% [395900/4848000] [81.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 14:45:06,131 - Train: 8.17% [396000/4848000] [81.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 14:45:39,076 - Train: 8.17% [396100/4848000] [81.7/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 14:46:12,051 - Train: 8.17% [396200/4848000] [81.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 14:46:44,984 - Train: 8.17% [396300/4848000] [81.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:47:17,987 - Train: 8.18% [396400/4848000] [81.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 14:47:50,920 - Train: 8.18% [396500/4848000] [81.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:48:23,828 - Train: 8.18% [396600/4848000] [81.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 14:48:56,716 - Train: 8.18% [396700/4848000] [81.8/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 14:49:29,644 - Train: 8.18% [396800/4848000] [81.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 14:50:02,550 - Train: 8.19% [396900/4848000] [81.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 14:50:36,281 - Train: 8.19% [397000/4848000] [81.9/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 14:51:09,140 - Train: 8.19% [397100/4848000] [81.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 14:51:42,005 - Train: 8.19% [397200/4848000] [81.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 14:52:14,901 - Train: 8.20% [397300/4848000] [82.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 14:52:47,781 - Train: 8.20% [397400/4848000] [82.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 14:53:20,724 - Train: 8.20% [397500/4848000] [82.0/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-04 14:53:32,582 - ==> Total time: 1 day, 15:27:31 Eta: 18 days, 9:44:39 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 14:53:55,482 - Train: 8.20% [397600/4848000] [82.0/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 14:54:28,529 - Train: 8.20% [397700/4848000] [82.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 14:55:01,445 - Train: 8.21% [397800/4848000] [82.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 14:55:34,410 - Train: 8.21% [397900/4848000] [82.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 14:56:07,408 - Train: 8.21% [398000/4848000] [82.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 14:56:42,183 - Train: 8.21% [398100/4848000] [82.1/1000.0] [batch_t 0.330 (0.348)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 14:57:15,121 - Train: 8.21% [398200/4848000] [82.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 14:57:48,026 - Train: 8.22% [398300/4848000] [82.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 14:58:20,962 - Train: 8.22% [398400/4848000] [82.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 14:58:53,870 - Train: 8.22% [398500/4848000] [82.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 14:59:26,780 - Train: 8.22% [398600/4848000] [82.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 14:59:59,692 - Train: 8.22% [398700/4848000] [82.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:00:32,562 - Train: 8.23% [398800/4848000] [82.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 15:01:05,490 - Train: 8.23% [398900/4848000] [82.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:01:38,401 - Train: 8.23% [399000/4848000] [82.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 15:02:11,395 - Train: 8.23% [399100/4848000] [82.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 15:02:44,313 - Train: 8.23% [399200/4848000] [82.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:03:17,225 - Train: 8.24% [399300/4848000] [82.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:03:50,140 - Train: 8.24% [399400/4848000] [82.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 15:04:23,070 - Train: 8.24% [399500/4848000] [82.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 15:04:55,989 - Train: 8.24% [399600/4848000] [82.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:05:28,876 - Train: 8.24% [399700/4848000] [82.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 15:06:01,799 - Train: 8.25% [399800/4848000] [82.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:06:34,732 - Train: 8.25% [399900/4848000] [82.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:07:07,687 - Train: 8.25% [400000/4848000] [82.5/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 15:07:40,617 - Train: 8.25% [400100/4848000] [82.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:08:13,569 - Train: 8.25% [400200/4848000] [82.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:08:46,527 - Train: 8.26% [400300/4848000] [82.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 15:09:19,506 - Train: 8.26% [400400/4848000] [82.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 15:09:52,499 - Train: 8.26% [400500/4848000] [82.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:10:26,038 - Train: 8.26% [400600/4848000] [82.6/1000.0] [batch_t 0.334 (0.335)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 15:10:59,008 - Train: 8.27% [400700/4848000] [82.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:11:32,045 - Train: 8.27% [400800/4848000] [82.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:12:05,010 - Train: 8.27% [400900/4848000] [82.7/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 15:12:38,004 - Train: 8.27% [401000/4848000] [82.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 15:13:10,970 - Train: 8.27% [401100/4848000] [82.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:13:43,885 - Train: 8.28% [401200/4848000] [82.8/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 15:14:16,788 - Train: 8.28% [401300/4848000] [82.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:14:49,705 - Train: 8.28% [401400/4848000] [82.8/1000.0] [batch_t 0.337 (0.329)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-04 15:15:22,656 - Train: 8.28% [401500/4848000] [82.8/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 15:15:55,554 - Train: 8.28% [401600/4848000] [82.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 15:16:28,477 - Train: 8.29% [401700/4848000] [82.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 15:17:01,415 - Train: 8.29% [401800/4848000] [82.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 15:17:34,317 - Train: 8.29% [401900/4848000] [82.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:18:07,351 - Train: 8.29% [402000/4848000] [82.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:18:40,287 - Train: 8.29% [402100/4848000] [82.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:19:13,183 - Train: 8.30% [402200/4848000] [83.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:19:46,106 - Train: 8.30% [402300/4848000] [83.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 15:20:13,785 - ==> Total time: 1 day, 15:54:12 Eta: 18 days, 8:51:38 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 15:20:21,001 - Train: 8.30% [402400/4848000] [83.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:20:53,917 - Train: 8.30% [402500/4848000] [83.0/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 15:21:26,853 - Train: 8.30% [402600/4848000] [83.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 15:21:59,794 - Train: 8.31% [402700/4848000] [83.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 15:22:32,755 - Train: 8.31% [402800/4848000] [83.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:23:05,714 - Train: 8.31% [402900/4848000] [83.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:23:38,654 - Train: 8.31% [403000/4848000] [83.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:24:11,591 - Train: 8.31% [403100/4848000] [83.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:24:44,532 - Train: 8.32% [403200/4848000] [83.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:25:17,629 - Train: 8.32% [403300/4848000] [83.2/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 15:25:50,581 - Train: 8.32% [403400/4848000] [83.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 15:26:23,544 - Train: 8.32% [403500/4848000] [83.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 15:26:56,495 - Train: 8.33% [403600/4848000] [83.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:27:29,459 - Train: 8.33% [403700/4848000] [83.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 15:28:02,391 - Train: 8.33% [403800/4848000] [83.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:28:35,356 - Train: 8.33% [403900/4848000] [83.3/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:29:08,290 - Train: 8.33% [404000/4848000] [83.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:29:41,243 - Train: 8.34% [404100/4848000] [83.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:30:14,225 - Train: 8.34% [404200/4848000] [83.4/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 15:30:47,167 - Train: 8.34% [404300/4848000] [83.4/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 15:31:20,067 - Train: 8.34% [404400/4848000] [83.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 15:31:53,017 - Train: 8.34% [404500/4848000] [83.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 15:32:25,972 - Train: 8.35% [404600/4848000] [83.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 15:32:58,985 - Train: 8.35% [404700/4848000] [83.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 15:33:31,923 - Train: 8.35% [404800/4848000] [83.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:34:04,878 - Train: 8.35% [404900/4848000] [83.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:34:37,803 - Train: 8.35% [405000/4848000] [83.5/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 15:35:10,756 - Train: 8.36% [405100/4848000] [83.6/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 15:35:43,683 - Train: 8.36% [405200/4848000] [83.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:36:16,621 - Train: 8.36% [405300/4848000] [83.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 15:36:49,572 - Train: 8.36% [405400/4848000] [83.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 15:37:22,552 - Train: 8.36% [405500/4848000] [83.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:37:55,499 - Train: 8.37% [405600/4848000] [83.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 15:38:28,427 - Train: 8.37% [405700/4848000] [83.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 15:39:01,330 - Train: 8.37% [405800/4848000] [83.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 15:39:34,240 - Train: 8.37% [405900/4848000] [83.7/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 15:40:07,164 - Train: 8.37% [406000/4848000] [83.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:40:40,241 - Train: 8.38% [406100/4848000] [83.8/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:41:13,172 - Train: 8.38% [406200/4848000] [83.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:41:46,121 - Train: 8.38% [406300/4848000] [83.8/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 15:42:19,097 - Train: 8.38% [406400/4848000] [83.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:42:52,073 - Train: 8.38% [406500/4848000] [83.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:43:25,023 - Train: 8.39% [406600/4848000] [83.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:43:57,941 - Train: 8.39% [406700/4848000] [83.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:44:30,901 - Train: 8.39% [406800/4848000] [83.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:45:03,850 - Train: 8.39% [406900/4848000] [83.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:45:36,758 - Train: 8.40% [407000/4848000] [84.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 15:46:09,697 - Train: 8.40% [407100/4848000] [84.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:46:42,617 - Train: 8.40% [407200/4848000] [84.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:46:53,156 - ==> Total time: 1 day, 16:20:51 Eta: 18 days, 7:58:54 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 15:47:17,374 - Train: 8.40% [407300/4848000] [84.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 15:47:50,347 - Train: 8.40% [407400/4848000] [84.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 15:48:23,430 - Train: 8.41% [407500/4848000] [84.1/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 15:48:56,353 - Train: 8.41% [407600/4848000] [84.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:49:29,391 - Train: 8.41% [407700/4848000] [84.1/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 15:50:02,301 - Train: 8.41% [407800/4848000] [84.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:50:35,267 - Train: 8.41% [407900/4848000] [84.1/1000.0] [batch_t 0.343 (0.330)] [data_t 0.002] [optim_t 0.341] [lr 0.005000] 2024-04-04 15:51:08,228 - Train: 8.42% [408000/4848000] [84.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 15:51:41,139 - Train: 8.42% [408100/4848000] [84.2/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 15:52:14,078 - Train: 8.42% [408200/4848000] [84.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:52:47,099 - Train: 8.42% [408300/4848000] [84.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 15:53:20,017 - Train: 8.42% [408400/4848000] [84.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:53:53,018 - Train: 8.43% [408500/4848000] [84.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 15:54:25,934 - Train: 8.43% [408600/4848000] [84.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 15:54:58,843 - Train: 8.43% [408700/4848000] [84.3/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 15:55:32,113 - Train: 8.43% [408800/4848000] [84.3/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 15:56:05,106 - Train: 8.43% [408900/4848000] [84.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 15:56:37,993 - Train: 8.44% [409000/4848000] [84.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 15:57:10,893 - Train: 8.44% [409100/4848000] [84.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 15:57:43,770 - Train: 8.44% [409200/4848000] [84.4/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 15:58:16,876 - Train: 8.44% [409300/4848000] [84.4/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 15:58:49,806 - Train: 8.44% [409400/4848000] [84.4/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 15:59:22,762 - Train: 8.45% [409500/4848000] [84.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 15:59:55,687 - Train: 8.45% [409600/4848000] [84.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 16:00:28,636 - Train: 8.45% [409700/4848000] [84.5/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 16:01:01,642 - Train: 8.45% [409800/4848000] [84.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:01:34,594 - Train: 8.46% [409900/4848000] [84.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 16:02:07,532 - Train: 8.46% [410000/4848000] [84.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 16:02:40,488 - Train: 8.46% [410100/4848000] [84.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:03:13,431 - Train: 8.46% [410200/4848000] [84.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:03:46,457 - Train: 8.46% [410300/4848000] [84.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:04:19,361 - Train: 8.47% [410400/4848000] [84.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 16:04:52,251 - Train: 8.47% [410500/4848000] [84.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:05:25,186 - Train: 8.47% [410600/4848000] [84.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:05:58,181 - Train: 8.47% [410700/4848000] [84.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:06:31,092 - Train: 8.47% [410800/4848000] [84.7/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 16:07:03,967 - Train: 8.48% [410900/4848000] [84.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 16:07:36,898 - Train: 8.48% [411000/4848000] [84.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 16:08:11,355 - Train: 8.48% [411100/4848000] [84.8/1000.0] [batch_t 0.326 (0.344)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 16:08:44,291 - Train: 8.48% [411200/4848000] [84.8/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 16:09:17,255 - Train: 8.48% [411300/4848000] [84.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 16:09:50,207 - Train: 8.49% [411400/4848000] [84.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:10:23,117 - Train: 8.49% [411500/4848000] [84.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 16:10:56,088 - Train: 8.49% [411600/4848000] [84.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 16:11:29,034 - Train: 8.49% [411700/4848000] [84.9/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 16:12:02,055 - Train: 8.49% [411800/4848000] [84.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:12:35,226 - Train: 8.50% [411900/4848000] [85.0/1000.0] [batch_t 0.326 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 16:13:08,102 - Train: 8.50% [412000/4848000] [85.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 16:13:34,445 - ==> Total time: 1 day, 16:47:32 Eta: 18 days, 7:07:08 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 16:13:43,212 - Train: 8.50% [412100/4848000] [85.0/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 16:14:16,823 - Train: 8.50% [412200/4848000] [85.0/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 16:14:49,738 - Train: 8.50% [412300/4848000] [85.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 16:15:23,416 - Train: 8.51% [412400/4848000] [85.1/1000.0] [batch_t 0.331 (0.337)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 16:15:56,345 - Train: 8.51% [412500/4848000] [85.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 16:16:29,913 - Train: 8.51% [412600/4848000] [85.1/1000.0] [batch_t 0.331 (0.336)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 16:17:02,850 - Train: 8.51% [412700/4848000] [85.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 16:17:47,496 - Train: 8.51% [412800/4848000] [85.1/1000.0] [batch_t 0.330 (0.446)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 16:18:20,841 - Train: 8.52% [412900/4848000] [85.2/1000.0] [batch_t 0.331 (0.333)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 16:18:53,725 - Train: 8.52% [413000/4848000] [85.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 16:19:26,771 - Train: 8.52% [413100/4848000] [85.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 16:19:59,674 - Train: 8.52% [413200/4848000] [85.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:20:32,616 - Train: 8.53% [413300/4848000] [85.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 16:21:05,550 - Train: 8.53% [413400/4848000] [85.3/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 16:21:38,482 - Train: 8.53% [413500/4848000] [85.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 16:22:11,398 - Train: 8.53% [413600/4848000] [85.3/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 16:22:44,367 - Train: 8.53% [413700/4848000] [85.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:23:18,189 - Train: 8.54% [413800/4848000] [85.4/1000.0] [batch_t 0.333 (0.338)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 16:23:51,118 - Train: 8.54% [413900/4848000] [85.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 16:24:24,927 - Train: 8.54% [414000/4848000] [85.4/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 16:24:57,845 - Train: 8.54% [414100/4848000] [85.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 16:25:31,700 - Train: 8.54% [414200/4848000] [85.4/1000.0] [batch_t 0.332 (0.338)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 16:26:04,653 - Train: 8.55% [414300/4848000] [85.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 16:26:37,653 - Train: 8.55% [414400/4848000] [85.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 16:27:12,757 - Train: 8.55% [414500/4848000] [85.5/1000.0] [batch_t 0.331 (0.351)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 16:27:45,730 - Train: 8.55% [414600/4848000] [85.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 16:28:19,496 - Train: 8.55% [414700/4848000] [85.5/1000.0] [batch_t 0.331 (0.338)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 16:28:52,430 - Train: 8.56% [414800/4848000] [85.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:29:26,266 - Train: 8.56% [414900/4848000] [85.6/1000.0] [batch_t 0.335 (0.338)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 16:30:03,498 - Train: 8.56% [415000/4848000] [85.6/1000.0] [batch_t 0.329 (0.372)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 16:30:36,442 - Train: 8.56% [415100/4848000] [85.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:31:09,400 - Train: 8.56% [415200/4848000] [85.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 16:31:42,308 - Train: 8.57% [415300/4848000] [85.7/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 16:32:15,859 - Train: 8.57% [415400/4848000] [85.7/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 16:32:48,788 - Train: 8.57% [415500/4848000] [85.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:33:21,771 - Train: 8.57% [415600/4848000] [85.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 16:33:54,735 - Train: 8.57% [415700/4848000] [85.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 16:34:27,686 - Train: 8.58% [415800/4848000] [85.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 16:35:00,644 - Train: 8.58% [415900/4848000] [85.8/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 16:35:33,691 - Train: 8.58% [416000/4848000] [85.8/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 16:36:06,615 - Train: 8.58% [416100/4848000] [85.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 16:36:39,540 - Train: 8.58% [416200/4848000] [85.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 16:37:12,491 - Train: 8.59% [416300/4848000] [85.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 16:37:45,435 - Train: 8.59% [416400/4848000] [85.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 16:38:18,357 - Train: 8.59% [416500/4848000] [85.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 16:38:51,253 - Train: 8.59% [416600/4848000] [85.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 16:39:24,185 - Train: 8.60% [416700/4848000] [86.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 16:39:57,099 - Train: 8.60% [416800/4848000] [86.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:40:30,002 - Train: 8.60% [416900/4848000] [86.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 16:40:39,231 - ==> Total time: 1 day, 17:14:37 Eta: 18 days, 6:20:07 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 16:41:04,813 - Train: 8.60% [417000/4848000] [86.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:41:37,716 - Train: 8.60% [417100/4848000] [86.0/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 16:42:10,689 - Train: 8.61% [417200/4848000] [86.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 16:42:43,788 - Train: 8.61% [417300/4848000] [86.1/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 16:43:16,792 - Train: 8.61% [417400/4848000] [86.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:43:49,760 - Train: 8.61% [417500/4848000] [86.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:44:22,785 - Train: 8.61% [417600/4848000] [86.1/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 16:44:55,750 - Train: 8.62% [417700/4848000] [86.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:45:28,651 - Train: 8.62% [417800/4848000] [86.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 16:46:01,562 - Train: 8.62% [417900/4848000] [86.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 16:46:34,924 - Train: 8.62% [418000/4848000] [86.2/1000.0] [batch_t 0.331 (0.334)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 16:47:07,823 - Train: 8.62% [418100/4848000] [86.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 16:47:40,714 - Train: 8.63% [418200/4848000] [86.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:48:13,597 - Train: 8.63% [418300/4848000] [86.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:48:46,502 - Train: 8.63% [418400/4848000] [86.3/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 16:49:19,467 - Train: 8.63% [418500/4848000] [86.3/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 16:49:52,381 - Train: 8.63% [418600/4848000] [86.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:50:25,417 - Train: 8.64% [418700/4848000] [86.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:50:58,331 - Train: 8.64% [418800/4848000] [86.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 16:51:31,229 - Train: 8.64% [418900/4848000] [86.4/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 16:52:04,131 - Train: 8.64% [419000/4848000] [86.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:52:37,037 - Train: 8.64% [419100/4848000] [86.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 16:53:10,829 - Train: 8.65% [419200/4848000] [86.5/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:53:43,719 - Train: 8.65% [419300/4848000] [86.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 16:54:16,621 - Train: 8.65% [419400/4848000] [86.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 16:54:49,504 - Train: 8.65% [419500/4848000] [86.5/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 16:55:22,397 - Train: 8.66% [419600/4848000] [86.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 16:55:55,311 - Train: 8.66% [419700/4848000] [86.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 16:56:29,269 - Train: 8.66% [419800/4848000] [86.6/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:57:02,191 - Train: 8.66% [419900/4848000] [86.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 16:57:35,271 - Train: 8.66% [420000/4848000] [86.6/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 16:58:08,192 - Train: 8.67% [420100/4848000] [86.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 16:58:41,240 - Train: 8.67% [420200/4848000] [86.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 16:59:14,993 - Train: 8.67% [420300/4848000] [86.7/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 16:59:47,975 - Train: 8.67% [420400/4848000] [86.7/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 17:00:20,960 - Train: 8.67% [420500/4848000] [86.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:00:53,960 - Train: 8.68% [420600/4848000] [86.8/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 17:01:27,601 - Train: 8.68% [420700/4848000] [86.8/1000.0] [batch_t 0.331 (0.336)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:02:00,577 - Train: 8.68% [420800/4848000] [86.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:02:33,669 - Train: 8.68% [420900/4848000] [86.8/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:03:07,264 - Train: 8.68% [421000/4848000] [86.8/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 17:03:40,200 - Train: 8.69% [421100/4848000] [86.9/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 17:04:14,284 - Train: 8.69% [421200/4848000] [86.9/1000.0] [batch_t 0.324 (0.341)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 17:04:47,223 - Train: 8.69% [421300/4848000] [86.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 17:05:22,723 - Train: 8.69% [421400/4848000] [86.9/1000.0] [batch_t 0.331 (0.355)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:05:55,731 - Train: 8.69% [421500/4848000] [86.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:06:30,022 - Train: 8.70% [421600/4848000] [87.0/1000.0] [batch_t 0.330 (0.343)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:07:02,959 - Train: 8.70% [421700/4848000] [87.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:07:29,022 - ==> Total time: 1 day, 17:41:27 Eta: 18 days, 5:30:56 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 17:07:39,076 - Train: 8.70% [421800/4848000] [87.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 17:08:13,238 - Train: 8.70% [421900/4848000] [87.0/1000.0] [batch_t 0.332 (0.342)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 17:08:46,205 - Train: 8.70% [422000/4848000] [87.0/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 17:09:21,135 - Train: 8.71% [422100/4848000] [87.1/1000.0] [batch_t 0.330 (0.349)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:09:54,046 - Train: 8.71% [422200/4848000] [87.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:10:27,880 - Train: 8.71% [422300/4848000] [87.1/1000.0] [batch_t 0.333 (0.338)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 17:11:00,821 - Train: 8.71% [422400/4848000] [87.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:11:34,264 - Train: 8.71% [422500/4848000] [87.1/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 17:12:07,234 - Train: 8.72% [422600/4848000] [87.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:12:40,167 - Train: 8.72% [422700/4848000] [87.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:13:13,502 - Train: 8.72% [422800/4848000] [87.2/1000.0] [batch_t 0.331 (0.333)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 17:13:46,387 - Train: 8.72% [422900/4848000] [87.2/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 17:14:19,457 - Train: 8.73% [423000/4848000] [87.3/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:14:52,387 - Train: 8.73% [423100/4848000] [87.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:15:26,701 - Train: 8.73% [423200/4848000] [87.3/1000.0] [batch_t 0.328 (0.343)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 17:15:59,613 - Train: 8.73% [423300/4848000] [87.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 17:16:35,017 - Train: 8.73% [423400/4848000] [87.3/1000.0] [batch_t 0.329 (0.354)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 17:17:08,023 - Train: 8.74% [423500/4848000] [87.4/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 17:17:40,940 - Train: 8.74% [423600/4848000] [87.4/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 17:18:14,877 - Train: 8.74% [423700/4848000] [87.4/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:18:47,805 - Train: 8.74% [423800/4848000] [87.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:19:20,754 - Train: 8.74% [423900/4848000] [87.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 17:19:53,683 - Train: 8.75% [424000/4848000] [87.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:20:27,344 - Train: 8.75% [424100/4848000] [87.5/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:21:00,264 - Train: 8.75% [424200/4848000] [87.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:21:33,199 - Train: 8.75% [424300/4848000] [87.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 17:22:06,221 - Train: 8.75% [424400/4848000] [87.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:22:39,149 - Train: 8.76% [424500/4848000] [87.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:23:12,088 - Train: 8.76% [424600/4848000] [87.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:23:45,074 - Train: 8.76% [424700/4848000] [87.6/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 17:24:18,004 - Train: 8.76% [424800/4848000] [87.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:24:50,945 - Train: 8.76% [424900/4848000] [87.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:25:24,729 - Train: 8.77% [425000/4848000] [87.7/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:25:57,684 - Train: 8.77% [425100/4848000] [87.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:26:32,701 - Train: 8.77% [425200/4848000] [87.7/1000.0] [batch_t 0.330 (0.350)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:27:07,264 - Train: 8.77% [425300/4848000] [87.7/1000.0] [batch_t 0.471 (0.346)] [data_t 0.136] [optim_t 0.335] [lr 0.005000] 2024-04-04 17:27:40,217 - Train: 8.77% [425400/4848000] [87.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 17:28:13,134 - Train: 8.78% [425500/4848000] [87.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:28:46,061 - Train: 8.78% [425600/4848000] [87.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:29:19,556 - Train: 8.78% [425700/4848000] [87.8/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 17:29:52,607 - Train: 8.78% [425800/4848000] [87.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:30:25,713 - Train: 8.79% [425900/4848000] [87.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:30:58,664 - Train: 8.79% [426000/4848000] [87.9/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 17:31:32,833 - Train: 8.79% [426100/4848000] [87.9/1000.0] [batch_t 0.334 (0.342)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 17:32:05,795 - Train: 8.79% [426200/4848000] [87.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:32:38,724 - Train: 8.79% [426300/4848000] [87.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:33:12,396 - Train: 8.80% [426400/4848000] [88.0/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 17:33:45,340 - Train: 8.80% [426500/4848000] [88.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:34:19,275 - Train: 8.80% [426600/4848000] [88.0/1000.0] [batch_t 0.336 (0.339)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-04 17:34:27,187 - ==> Total time: 1 day, 18:08:25 Eta: 18 days, 4:43:42 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 17:34:54,293 - Train: 8.80% [426700/4848000] [88.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:35:27,975 - Train: 8.80% [426800/4848000] [88.0/1000.0] [batch_t 0.331 (0.337)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 17:36:00,904 - Train: 8.81% [426900/4848000] [88.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:36:33,882 - Train: 8.81% [427000/4848000] [88.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:37:07,635 - Train: 8.81% [427100/4848000] [88.1/1000.0] [batch_t 0.334 (0.337)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 17:37:40,699 - Train: 8.81% [427200/4848000] [88.1/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:38:13,859 - Train: 8.81% [427300/4848000] [88.1/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:38:46,806 - Train: 8.82% [427400/4848000] [88.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:39:20,683 - Train: 8.82% [427500/4848000] [88.2/1000.0] [batch_t 0.336 (0.339)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-04 17:39:53,592 - Train: 8.82% [427600/4848000] [88.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:40:27,435 - Train: 8.82% [427700/4848000] [88.2/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:41:00,379 - Train: 8.82% [427800/4848000] [88.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:41:33,951 - Train: 8.83% [427900/4848000] [88.3/1000.0] [batch_t 0.331 (0.336)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:42:11,462 - Train: 8.83% [428000/4848000] [88.3/1000.0] [batch_t 0.327 (0.375)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 17:42:44,881 - Train: 8.83% [428100/4848000] [88.3/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 17:43:18,553 - Train: 8.83% [428200/4848000] [88.3/1000.0] [batch_t 0.334 (0.337)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 17:43:51,497 - Train: 8.83% [428300/4848000] [88.3/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 17:44:24,669 - Train: 8.84% [428400/4848000] [88.4/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:44:57,605 - Train: 8.84% [428500/4848000] [88.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:45:32,844 - Train: 8.84% [428600/4848000] [88.4/1000.0] [batch_t 0.328 (0.352)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 17:46:09,116 - Train: 8.84% [428700/4848000] [88.4/1000.0] [batch_t 0.331 (0.363)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:46:42,017 - Train: 8.84% [428800/4848000] [88.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:47:14,916 - Train: 8.85% [428900/4848000] [88.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 17:47:47,803 - Train: 8.85% [429000/4848000] [88.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:48:24,431 - Train: 8.85% [429100/4848000] [88.5/1000.0] [batch_t 0.330 (0.366)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:48:57,356 - Train: 8.85% [429200/4848000] [88.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:49:32,486 - Train: 8.86% [429300/4848000] [88.6/1000.0] [batch_t 0.329 (0.351)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 17:50:08,814 - Train: 8.86% [429400/4848000] [88.6/1000.0] [batch_t 0.330 (0.363)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:50:41,755 - Train: 8.86% [429500/4848000] [88.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:51:17,009 - Train: 8.86% [429600/4848000] [88.6/1000.0] [batch_t 0.330 (0.352)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:51:49,952 - Train: 8.86% [429700/4848000] [88.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:52:24,964 - Train: 8.87% [429800/4848000] [88.7/1000.0] [batch_t 0.328 (0.350)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 17:52:57,982 - Train: 8.87% [429900/4848000] [88.7/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 17:53:31,573 - Train: 8.87% [430000/4848000] [88.7/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 17:54:04,720 - Train: 8.87% [430100/4848000] [88.7/1000.0] [batch_t 0.554 (0.331)] [data_t 0.225] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:54:41,106 - Train: 8.87% [430200/4848000] [88.7/1000.0] [batch_t 0.328 (0.364)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 17:55:14,077 - Train: 8.88% [430300/4848000] [88.8/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 17:55:47,002 - Train: 8.88% [430400/4848000] [88.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:56:22,451 - Train: 8.88% [430500/4848000] [88.8/1000.0] [batch_t 0.336 (0.354)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-04 17:56:55,374 - Train: 8.88% [430600/4848000] [88.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 17:57:29,153 - Train: 8.88% [430700/4848000] [88.8/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 17:58:02,103 - Train: 8.89% [430800/4848000] [88.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 17:58:35,046 - Train: 8.89% [430900/4848000] [88.9/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 17:59:08,014 - Train: 8.89% [431000/4848000] [88.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 17:59:40,977 - Train: 8.89% [431100/4848000] [88.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 18:00:16,312 - Train: 8.89% [431200/4848000] [88.9/1000.0] [batch_t 0.331 (0.353)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:00:49,271 - Train: 8.90% [431300/4848000] [89.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:01:22,680 - Train: 8.90% [431400/4848000] [89.0/1000.0] [batch_t 0.330 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 18:01:46,587 - ==> Total time: 1 day, 18:35:45 Eta: 18 days, 4:00:33 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 18:01:57,688 - Train: 8.90% [431500/4848000] [89.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 18:02:31,299 - Train: 8.90% [431600/4848000] [89.0/1000.0] [batch_t 0.332 (0.336)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 18:03:04,218 - Train: 8.90% [431700/4848000] [89.0/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 18:03:38,018 - Train: 8.91% [431800/4848000] [89.1/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 18:04:11,526 - Train: 8.91% [431900/4848000] [89.1/1000.0] [batch_t 0.331 (0.335)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:04:44,451 - Train: 8.91% [432000/4848000] [89.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 18:05:17,385 - Train: 8.91% [432100/4848000] [89.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 18:05:50,294 - Train: 8.92% [432200/4848000] [89.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 18:06:24,161 - Train: 8.92% [432300/4848000] [89.2/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 18:06:57,086 - Train: 8.92% [432400/4848000] [89.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 18:07:30,026 - Train: 8.92% [432500/4848000] [89.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:08:02,967 - Train: 8.92% [432600/4848000] [89.2/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 18:08:36,390 - Train: 8.93% [432700/4848000] [89.3/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 18:09:09,465 - Train: 8.93% [432800/4848000] [89.3/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 18:09:42,446 - Train: 8.93% [432900/4848000] [89.3/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 18:10:16,014 - Train: 8.93% [433000/4848000] [89.3/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 18:10:48,980 - Train: 8.93% [433100/4848000] [89.3/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 18:11:23,029 - Train: 8.94% [433200/4848000] [89.4/1000.0] [batch_t 0.336 (0.340)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-04 18:11:56,005 - Train: 8.94% [433300/4848000] [89.4/1000.0] [batch_t 0.336 (0.330)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-04 18:12:29,949 - Train: 8.94% [433400/4848000] [89.4/1000.0] [batch_t 0.332 (0.339)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 18:13:02,888 - Train: 8.94% [433500/4848000] [89.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 18:13:37,316 - Train: 8.94% [433600/4848000] [89.4/1000.0] [batch_t 0.330 (0.344)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:14:11,792 - Train: 8.95% [433700/4848000] [89.5/1000.0] [batch_t 0.330 (0.345)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 18:14:44,747 - Train: 8.95% [433800/4848000] [89.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:15:18,362 - Train: 8.95% [433900/4848000] [89.5/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:15:51,370 - Train: 8.95% [434000/4848000] [89.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 18:16:24,944 - Train: 8.95% [434100/4848000] [89.5/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 18:16:57,920 - Train: 8.96% [434200/4848000] [89.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 18:17:32,092 - Train: 8.96% [434300/4848000] [89.6/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 18:18:05,783 - Train: 8.96% [434400/4848000] [89.6/1000.0] [batch_t 1.112 (0.337)] [data_t 0.781] [optim_t 0.330] [lr 0.005000] 2024-04-04 18:18:38,690 - Train: 8.96% [434500/4848000] [89.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:19:12,814 - Train: 8.96% [434600/4848000] [89.6/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 18:19:45,732 - Train: 8.97% [434700/4848000] [89.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 18:20:18,971 - Train: 8.97% [434800/4848000] [89.7/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:20:51,910 - Train: 8.97% [434900/4848000] [89.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:21:25,379 - Train: 8.97% [435000/4848000] [89.7/1000.0] [batch_t 0.331 (0.335)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:21:58,326 - Train: 8.97% [435100/4848000] [89.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:22:31,269 - Train: 8.98% [435200/4848000] [89.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 18:23:04,249 - Train: 8.98% [435300/4848000] [89.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 18:23:37,642 - Train: 8.98% [435400/4848000] [89.8/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 18:24:11,746 - Train: 8.98% [435500/4848000] [89.8/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 18:24:44,701 - Train: 8.99% [435600/4848000] [89.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 18:25:18,476 - Train: 8.99% [435700/4848000] [89.9/1000.0] [batch_t 0.334 (0.338)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 18:25:51,451 - Train: 8.99% [435800/4848000] [89.9/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 18:26:25,288 - Train: 8.99% [435900/4848000] [89.9/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 18:26:58,237 - Train: 8.99% [436000/4848000] [89.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 18:27:31,191 - Train: 9.00% [436100/4848000] [90.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:28:04,135 - Train: 9.00% [436200/4848000] [90.0/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 18:28:37,798 - Train: 9.00% [436300/4848000] [90.0/1000.0] [batch_t 0.331 (0.337)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:28:44,409 - ==> Total time: 1 day, 19:02:42 Eta: 18 days, 3:14:07 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 18:29:14,195 - Train: 9.00% [436400/4848000] [90.0/1000.0] [batch_t 0.331 (0.342)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:29:47,140 - Train: 9.00% [436500/4848000] [90.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 18:30:20,059 - Train: 9.01% [436600/4848000] [90.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 18:30:52,991 - Train: 9.01% [436700/4848000] [90.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:31:25,887 - Train: 9.01% [436800/4848000] [90.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 18:31:58,807 - Train: 9.01% [436900/4848000] [90.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 18:32:31,889 - Train: 9.01% [437000/4848000] [90.1/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 18:33:05,415 - Train: 9.02% [437100/4848000] [90.2/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:33:38,380 - Train: 9.02% [437200/4848000] [90.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 18:34:12,235 - Train: 9.02% [437300/4848000] [90.2/1000.0] [batch_t 0.333 (0.338)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 18:34:45,225 - Train: 9.02% [437400/4848000] [90.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 18:35:18,933 - Train: 9.02% [437500/4848000] [90.2/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:35:51,878 - Train: 9.03% [437600/4848000] [90.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:36:24,782 - Train: 9.03% [437700/4848000] [90.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:36:57,775 - Train: 9.03% [437800/4848000] [90.3/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 18:37:31,417 - Train: 9.03% [437900/4848000] [90.3/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 18:38:04,368 - Train: 9.03% [438000/4848000] [90.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 18:38:38,910 - Train: 9.04% [438100/4848000] [90.4/1000.0] [batch_t 0.332 (0.345)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 18:39:12,727 - Train: 9.04% [438200/4848000] [90.4/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 18:39:45,642 - Train: 9.04% [438300/4848000] [90.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 18:40:18,705 - Train: 9.04% [438400/4848000] [90.4/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 18:40:51,604 - Train: 9.04% [438500/4848000] [90.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 18:41:24,533 - Train: 9.05% [438600/4848000] [90.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 18:41:57,472 - Train: 9.05% [438700/4848000] [90.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 18:42:30,408 - Train: 9.05% [438800/4848000] [90.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:43:03,326 - Train: 9.05% [438900/4848000] [90.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:43:36,766 - Train: 9.06% [439000/4848000] [90.6/1000.0] [batch_t 0.332 (0.334)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 18:44:09,781 - Train: 9.06% [439100/4848000] [90.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 18:44:42,677 - Train: 9.06% [439200/4848000] [90.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 18:45:15,628 - Train: 9.06% [439300/4848000] [90.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 18:45:48,608 - Train: 9.06% [439400/4848000] [90.6/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 18:46:21,578 - Train: 9.07% [439500/4848000] [90.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 18:46:54,579 - Train: 9.07% [439600/4848000] [90.7/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 18:47:27,566 - Train: 9.07% [439700/4848000] [90.7/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 18:48:00,475 - Train: 9.07% [439800/4848000] [90.7/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 18:48:34,396 - Train: 9.07% [439900/4848000] [90.7/1000.0] [batch_t 0.331 (0.339)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:49:09,313 - Train: 9.08% [440000/4848000] [90.8/1000.0] [batch_t 0.330 (0.349)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 18:49:42,226 - Train: 9.08% [440100/4848000] [90.8/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 18:50:15,176 - Train: 9.08% [440200/4848000] [90.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 18:50:48,062 - Train: 9.08% [440300/4848000] [90.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:51:21,022 - Train: 9.08% [440400/4848000] [90.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:51:53,937 - Train: 9.09% [440500/4848000] [90.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 18:52:26,841 - Train: 9.09% [440600/4848000] [90.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:52:59,769 - Train: 9.09% [440700/4848000] [90.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 18:53:32,683 - Train: 9.09% [440800/4848000] [90.9/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 18:54:06,822 - Train: 9.09% [440900/4848000] [90.9/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:54:39,740 - Train: 9.10% [441000/4848000] [91.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:55:12,699 - Train: 9.10% [441100/4848000] [91.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:55:35,121 - ==> Total time: 1 day, 19:29:33 Eta: 18 days, 2:26:55 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 18:55:47,673 - Train: 9.10% [441200/4848000] [91.0/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 18:56:20,598 - Train: 9.10% [441300/4848000] [91.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:56:53,514 - Train: 9.10% [441400/4848000] [91.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:57:27,078 - Train: 9.11% [441500/4848000] [91.1/1000.0] [batch_t 0.331 (0.336)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:57:59,999 - Train: 9.11% [441600/4848000] [91.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:58:32,975 - Train: 9.11% [441700/4848000] [91.1/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 18:59:06,975 - Train: 9.11% [441800/4848000] [91.1/1000.0] [batch_t 0.330 (0.340)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 18:59:39,871 - Train: 9.12% [441900/4848000] [91.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:00:13,681 - Train: 9.12% [442000/4848000] [91.2/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:00:46,649 - Train: 9.12% [442100/4848000] [91.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:01:20,700 - Train: 9.12% [442200/4848000] [91.2/1000.0] [batch_t 0.332 (0.340)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 19:01:53,625 - Train: 9.12% [442300/4848000] [91.2/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 19:02:26,968 - Train: 9.13% [442400/4848000] [91.3/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:02:59,902 - Train: 9.13% [442500/4848000] [91.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:03:34,877 - Train: 9.13% [442600/4848000] [91.3/1000.0] [batch_t 0.330 (0.350)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 19:04:09,043 - Train: 9.13% [442700/4848000] [91.3/1000.0] [batch_t 0.328 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:04:42,005 - Train: 9.13% [442800/4848000] [91.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:05:15,701 - Train: 9.14% [442900/4848000] [91.4/1000.0] [batch_t 0.333 (0.337)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 19:05:48,634 - Train: 9.14% [443000/4848000] [91.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 19:06:22,556 - Train: 9.14% [443100/4848000] [91.4/1000.0] [batch_t 0.331 (0.339)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 19:06:55,446 - Train: 9.14% [443200/4848000] [91.4/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 19:07:29,128 - Train: 9.14% [443300/4848000] [91.4/1000.0] [batch_t 0.333 (0.337)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 19:08:02,061 - Train: 9.15% [443400/4848000] [91.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 19:08:35,147 - Train: 9.15% [443500/4848000] [91.5/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:09:08,214 - Train: 9.15% [443600/4848000] [91.5/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 19:09:41,103 - Train: 9.15% [443700/4848000] [91.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 19:10:16,118 - Train: 9.15% [443800/4848000] [91.5/1000.0] [batch_t 0.330 (0.350)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 19:10:49,054 - Train: 9.16% [443900/4848000] [91.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 19:11:22,414 - Train: 9.16% [444000/4848000] [91.6/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 19:11:55,480 - Train: 9.16% [444100/4848000] [91.6/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:12:28,401 - Train: 9.16% [444200/4848000] [91.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:13:01,332 - Train: 9.16% [444300/4848000] [91.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:13:34,305 - Train: 9.17% [444400/4848000] [91.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 19:14:07,238 - Train: 9.17% [444500/4848000] [91.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:14:40,161 - Train: 9.17% [444600/4848000] [91.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:15:13,142 - Train: 9.17% [444700/4848000] [91.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:15:46,060 - Train: 9.17% [444800/4848000] [91.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 19:16:18,975 - Train: 9.18% [444900/4848000] [91.8/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 19:16:51,980 - Train: 9.18% [445000/4848000] [91.8/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 19:17:26,145 - Train: 9.18% [445100/4848000] [91.8/1000.0] [batch_t 0.330 (0.342)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 19:17:59,050 - Train: 9.18% [445200/4848000] [91.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:18:34,711 - Train: 9.19% [445300/4848000] [91.9/1000.0] [batch_t 0.332 (0.357)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 19:19:07,644 - Train: 9.19% [445400/4848000] [91.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:19:40,730 - Train: 9.19% [445500/4848000] [91.9/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 19:20:13,693 - Train: 9.19% [445600/4848000] [91.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 19:20:46,626 - Train: 9.19% [445700/4848000] [91.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:21:19,587 - Train: 9.20% [445800/4848000] [92.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:21:52,537 - Train: 9.20% [445900/4848000] [92.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:22:25,486 - Train: 9.20% [446000/4848000] [92.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 19:22:30,755 - ==> Total time: 1 day, 19:56:29 Eta: 18 days, 1:40:59 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 19:23:00,317 - Train: 9.20% [446100/4848000] [92.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:23:33,233 - Train: 9.20% [446200/4848000] [92.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 19:24:06,164 - Train: 9.21% [446300/4848000] [92.1/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 19:24:39,102 - Train: 9.21% [446400/4848000] [92.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 19:25:12,062 - Train: 9.21% [446500/4848000] [92.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:25:44,997 - Train: 9.21% [446600/4848000] [92.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 19:26:17,919 - Train: 9.21% [446700/4848000] [92.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:26:51,012 - Train: 9.22% [446800/4848000] [92.2/1000.0] [batch_t 0.333 (0.331)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 19:27:23,915 - Train: 9.22% [446900/4848000] [92.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 19:27:56,803 - Train: 9.22% [447000/4848000] [92.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 19:28:29,715 - Train: 9.22% [447100/4848000] [92.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 19:29:02,666 - Train: 9.22% [447200/4848000] [92.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:29:36,537 - Train: 9.23% [447300/4848000] [92.3/1000.0] [batch_t 0.325 (0.339)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 19:30:10,327 - Train: 9.23% [447400/4848000] [92.3/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 19:30:43,278 - Train: 9.23% [447500/4848000] [92.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 19:31:16,228 - Train: 9.23% [447600/4848000] [92.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 19:31:49,183 - Train: 9.23% [447700/4848000] [92.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:32:22,106 - Train: 9.24% [447800/4848000] [92.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:32:55,057 - Train: 9.24% [447900/4848000] [92.4/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 19:33:28,635 - Train: 9.24% [448000/4848000] [92.4/1000.0] [batch_t 0.331 (0.336)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:34:01,588 - Train: 9.24% [448100/4848000] [92.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:34:34,623 - Train: 9.25% [448200/4848000] [92.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:35:08,331 - Train: 9.25% [448300/4848000] [92.5/1000.0] [batch_t 0.331 (0.337)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 19:35:41,252 - Train: 9.25% [448400/4848000] [92.5/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 19:36:14,177 - Train: 9.25% [448500/4848000] [92.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:36:47,176 - Train: 9.25% [448600/4848000] [92.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:37:20,323 - Train: 9.26% [448700/4848000] [92.6/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:37:53,261 - Train: 9.26% [448800/4848000] [92.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:38:27,393 - Train: 9.26% [448900/4848000] [92.6/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 19:39:00,320 - Train: 9.26% [449000/4848000] [92.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 19:39:34,641 - Train: 9.26% [449100/4848000] [92.6/1000.0] [batch_t 0.332 (0.343)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 19:40:07,724 - Train: 9.27% [449200/4848000] [92.7/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:40:40,656 - Train: 9.27% [449300/4848000] [92.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:41:14,541 - Train: 9.27% [449400/4848000] [92.7/1000.0] [batch_t 0.331 (0.339)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:41:47,503 - Train: 9.27% [449500/4848000] [92.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 19:42:21,111 - Train: 9.27% [449600/4848000] [92.7/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:42:54,163 - Train: 9.28% [449700/4848000] [92.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 19:43:27,820 - Train: 9.28% [449800/4848000] [92.8/1000.0] [batch_t 0.334 (0.336)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 19:44:00,780 - Train: 9.28% [449900/4848000] [92.8/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 19:44:34,344 - Train: 9.28% [450000/4848000] [92.8/1000.0] [batch_t 0.331 (0.336)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:45:07,478 - Train: 9.28% [450100/4848000] [92.8/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:45:40,408 - Train: 9.29% [450200/4848000] [92.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:46:15,577 - Train: 9.29% [450300/4848000] [92.9/1000.0] [batch_t 0.328 (0.352)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 19:46:48,512 - Train: 9.29% [450400/4848000] [92.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:47:21,495 - Train: 9.29% [450500/4848000] [92.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:47:54,433 - Train: 9.29% [450600/4848000] [92.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:48:27,988 - Train: 9.30% [450700/4848000] [93.0/1000.0] [batch_t 0.331 (0.335)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:49:01,025 - Train: 9.30% [450800/4848000] [93.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 19:49:22,132 - ==> Total time: 1 day, 20:23:20 Eta: 18 days, 0:54:45 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 19:49:36,109 - Train: 9.30% [450900/4848000] [93.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:50:09,545 - Train: 9.30% [451000/4848000] [93.0/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:50:42,469 - Train: 9.30% [451100/4848000] [93.0/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 19:51:15,660 - Train: 9.31% [451200/4848000] [93.1/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:51:48,604 - Train: 9.31% [451300/4848000] [93.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 19:52:21,579 - Train: 9.31% [451400/4848000] [93.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:52:54,512 - Train: 9.31% [451500/4848000] [93.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 19:53:29,566 - Train: 9.32% [451600/4848000] [93.2/1000.0] [batch_t 0.330 (0.350)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 19:54:02,524 - Train: 9.32% [451700/4848000] [93.2/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 19:54:35,958 - Train: 9.32% [451800/4848000] [93.2/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 19:55:09,545 - Train: 9.32% [451900/4848000] [93.2/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 19:55:42,488 - Train: 9.32% [452000/4848000] [93.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 19:56:16,940 - Train: 9.33% [452100/4848000] [93.3/1000.0] [batch_t 0.331 (0.344)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:56:49,825 - Train: 9.33% [452200/4848000] [93.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 19:57:24,082 - Train: 9.33% [452300/4848000] [93.3/1000.0] [batch_t 0.328 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 19:57:57,139 - Train: 9.33% [452400/4848000] [93.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 19:58:30,068 - Train: 9.33% [452500/4848000] [93.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 19:59:02,997 - Train: 9.34% [452600/4848000] [93.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 19:59:35,933 - Train: 9.34% [452700/4848000] [93.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:00:10,181 - Train: 9.34% [452800/4848000] [93.4/1000.0] [batch_t 0.324 (0.342)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-04 20:00:43,150 - Train: 9.34% [452900/4848000] [93.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:01:16,110 - Train: 9.34% [453000/4848000] [93.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:01:49,090 - Train: 9.35% [453100/4848000] [93.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 20:02:24,088 - Train: 9.35% [453200/4848000] [93.5/1000.0] [batch_t 0.328 (0.350)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 20:02:57,066 - Train: 9.35% [453300/4848000] [93.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 20:03:30,495 - Train: 9.35% [453400/4848000] [93.5/1000.0] [batch_t 0.331 (0.334)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:04:03,418 - Train: 9.35% [453500/4848000] [93.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 20:04:36,346 - Train: 9.36% [453600/4848000] [93.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 20:05:09,245 - Train: 9.36% [453700/4848000] [93.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:05:42,315 - Train: 9.36% [453800/4848000] [93.6/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:06:15,212 - Train: 9.36% [453900/4848000] [93.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:06:48,104 - Train: 9.36% [454000/4848000] [93.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:07:26,223 - Train: 9.37% [454100/4848000] [93.7/1000.0] [batch_t 0.331 (0.381)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 20:07:59,116 - Train: 9.37% [454200/4848000] [93.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:08:32,077 - Train: 9.37% [454300/4848000] [93.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 20:09:05,014 - Train: 9.37% [454400/4848000] [93.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 20:09:37,977 - Train: 9.38% [454500/4848000] [93.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:10:11,760 - Train: 9.38% [454600/4848000] [93.8/1000.0] [batch_t 0.331 (0.338)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:10:44,706 - Train: 9.38% [454700/4848000] [93.8/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 20:11:20,485 - Train: 9.38% [454800/4848000] [93.8/1000.0] [batch_t 0.329 (0.358)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 20:11:53,378 - Train: 9.38% [454900/4848000] [93.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:12:26,303 - Train: 9.39% [455000/4848000] [93.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:12:59,329 - Train: 9.39% [455100/4848000] [93.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 20:13:32,251 - Train: 9.39% [455200/4848000] [93.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 20:14:05,305 - Train: 9.39% [455300/4848000] [93.9/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 20:14:38,223 - Train: 9.39% [455400/4848000] [93.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:15:11,118 - Train: 9.40% [455500/4848000] [94.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:15:44,035 - Train: 9.40% [455600/4848000] [94.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 20:16:17,174 - Train: 9.40% [455700/4848000] [94.0/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 20:16:21,130 - ==> Total time: 1 day, 20:50:19 Eta: 18 days, 0:10:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 20:16:52,164 - Train: 9.40% [455800/4848000] [94.0/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 20:17:26,005 - Train: 9.40% [455900/4848000] [94.0/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:17:58,910 - Train: 9.41% [456000/4848000] [94.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 20:18:31,833 - Train: 9.41% [456100/4848000] [94.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:19:04,778 - Train: 9.41% [456200/4848000] [94.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:19:37,699 - Train: 9.41% [456300/4848000] [94.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:20:11,970 - Train: 9.41% [456400/4848000] [94.1/1000.0] [batch_t 0.330 (0.343)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 20:20:44,860 - Train: 9.42% [456500/4848000] [94.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:21:18,517 - Train: 9.42% [456600/4848000] [94.2/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:21:51,465 - Train: 9.42% [456700/4848000] [94.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:22:25,819 - Train: 9.42% [456800/4848000] [94.2/1000.0] [batch_t 0.329 (0.343)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:22:58,752 - Train: 9.42% [456900/4848000] [94.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:23:32,336 - Train: 9.43% [457000/4848000] [94.3/1000.0] [batch_t 0.331 (0.336)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 20:24:06,294 - Train: 9.43% [457100/4848000] [94.3/1000.0] [batch_t 0.333 (0.339)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 20:24:39,219 - Train: 9.43% [457200/4848000] [94.3/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 20:25:13,235 - Train: 9.43% [457300/4848000] [94.3/1000.0] [batch_t 0.331 (0.340)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:25:46,192 - Train: 9.43% [457400/4848000] [94.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:26:21,061 - Train: 9.44% [457500/4848000] [94.4/1000.0] [batch_t 0.331 (0.349)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:26:53,963 - Train: 9.44% [457600/4848000] [94.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 20:27:27,799 - Train: 9.44% [457700/4848000] [94.4/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 20:28:00,708 - Train: 9.44% [457800/4848000] [94.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:28:34,293 - Train: 9.45% [457900/4848000] [94.5/1000.0] [batch_t 0.332 (0.336)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 20:29:07,977 - Train: 9.45% [458000/4848000] [94.5/1000.0] [batch_t 0.332 (0.337)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 20:29:40,931 - Train: 9.45% [458100/4848000] [94.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 20:30:14,905 - Train: 9.45% [458200/4848000] [94.5/1000.0] [batch_t 0.332 (0.340)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 20:30:47,833 - Train: 9.45% [458300/4848000] [94.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 20:31:20,779 - Train: 9.46% [458400/4848000] [94.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:31:53,742 - Train: 9.46% [458500/4848000] [94.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 20:32:26,713 - Train: 9.46% [458600/4848000] [94.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:32:59,717 - Train: 9.46% [458700/4848000] [94.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 20:33:32,668 - Train: 9.46% [458800/4848000] [94.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 20:34:06,064 - Train: 9.47% [458900/4848000] [94.7/1000.0] [batch_t 0.712 (0.334)] [data_t 0.386] [optim_t 0.326] [lr 0.005000] 2024-04-04 20:34:39,033 - Train: 9.47% [459000/4848000] [94.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 20:35:11,991 - Train: 9.47% [459100/4848000] [94.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:35:44,911 - Train: 9.47% [459200/4848000] [94.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 20:36:17,871 - Train: 9.47% [459300/4848000] [94.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:36:50,812 - Train: 9.48% [459400/4848000] [94.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:37:23,830 - Train: 9.48% [459500/4848000] [94.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:37:56,766 - Train: 9.48% [459600/4848000] [94.8/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 20:38:29,720 - Train: 9.48% [459700/4848000] [94.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:39:02,653 - Train: 9.48% [459800/4848000] [94.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:39:36,687 - Train: 9.49% [459900/4848000] [94.9/1000.0] [batch_t 0.331 (0.340)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:40:09,659 - Train: 9.49% [460000/4848000] [94.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 20:40:42,625 - Train: 9.49% [460100/4848000] [94.9/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 20:41:17,046 - Train: 9.49% [460200/4848000] [94.9/1000.0] [batch_t 0.329 (0.344)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:41:50,000 - Train: 9.49% [460300/4848000] [94.9/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 20:42:23,804 - Train: 9.50% [460400/4848000] [95.0/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:42:56,697 - Train: 9.50% [460500/4848000] [95.0/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 20:43:17,933 - ==> Total time: 1 day, 21:17:16 Eta: 17 days, 23:25:36 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 20:43:32,981 - Train: 9.50% [460600/4848000] [95.0/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 20:44:05,898 - Train: 9.50% [460700/4848000] [95.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:44:38,998 - Train: 9.50% [460800/4848000] [95.0/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 20:45:13,683 - Train: 9.51% [460900/4848000] [95.1/1000.0] [batch_t 0.331 (0.347)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:45:46,779 - Train: 9.51% [461000/4848000] [95.1/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 20:46:20,846 - Train: 9.51% [461100/4848000] [95.1/1000.0] [batch_t 0.334 (0.341)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 20:46:53,826 - Train: 9.51% [461200/4848000] [95.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 20:47:26,772 - Train: 9.52% [461300/4848000] [95.2/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 20:47:59,764 - Train: 9.52% [461400/4848000] [95.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:48:33,255 - Train: 9.52% [461500/4848000] [95.2/1000.0] [batch_t 0.331 (0.335)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:49:06,842 - Train: 9.52% [461600/4848000] [95.2/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 20:49:39,758 - Train: 9.52% [461700/4848000] [95.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:50:13,973 - Train: 9.53% [461800/4848000] [95.3/1000.0] [batch_t 0.332 (0.342)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 20:50:46,867 - Train: 9.53% [461900/4848000] [95.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 20:51:20,599 - Train: 9.53% [462000/4848000] [95.3/1000.0] [batch_t 0.326 (0.337)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 20:51:53,527 - Train: 9.53% [462100/4848000] [95.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:52:27,504 - Train: 9.53% [462200/4848000] [95.3/1000.0] [batch_t 0.330 (0.340)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 20:53:00,652 - Train: 9.54% [462300/4848000] [95.4/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:53:34,667 - Train: 9.54% [462400/4848000] [95.4/1000.0] [batch_t 0.332 (0.340)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 20:54:08,611 - Train: 9.54% [462500/4848000] [95.4/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 20:54:41,530 - Train: 9.54% [462600/4848000] [95.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 20:55:15,430 - Train: 9.54% [462700/4848000] [95.4/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:55:48,337 - Train: 9.55% [462800/4848000] [95.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:56:22,620 - Train: 9.55% [462900/4848000] [95.5/1000.0] [batch_t 0.331 (0.343)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 20:56:55,587 - Train: 9.55% [463000/4848000] [95.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 20:57:28,577 - Train: 9.55% [463100/4848000] [95.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 20:58:01,558 - Train: 9.55% [463200/4848000] [95.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 20:58:35,205 - Train: 9.56% [463300/4848000] [95.6/1000.0] [batch_t 0.326 (0.336)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 20:59:08,164 - Train: 9.56% [463400/4848000] [95.6/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 20:59:41,127 - Train: 9.56% [463500/4848000] [95.6/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 21:00:14,796 - Train: 9.56% [463600/4848000] [95.6/1000.0] [batch_t 0.325 (0.337)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-04 21:00:47,988 - Train: 9.56% [463700/4848000] [95.6/1000.0] [batch_t 0.338 (0.332)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-04 21:01:21,024 - Train: 9.57% [463800/4848000] [95.7/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 21:01:54,063 - Train: 9.57% [463900/4848000] [95.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 21:02:27,055 - Train: 9.57% [464000/4848000] [95.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:03:00,054 - Train: 9.57% [464100/4848000] [95.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:03:32,979 - Train: 9.58% [464200/4848000] [95.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 21:04:05,954 - Train: 9.58% [464300/4848000] [95.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 21:04:38,896 - Train: 9.58% [464400/4848000] [95.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 21:05:11,849 - Train: 9.58% [464500/4848000] [95.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 21:05:44,814 - Train: 9.58% [464600/4848000] [95.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 21:06:18,108 - Train: 9.59% [464700/4848000] [95.9/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 21:06:51,062 - Train: 9.59% [464800/4848000] [95.9/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 21:07:24,039 - Train: 9.59% [464900/4848000] [95.9/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 21:07:57,014 - Train: 9.59% [465000/4848000] [95.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 21:08:30,100 - Train: 9.59% [465100/4848000] [95.9/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 21:09:03,085 - Train: 9.60% [465200/4848000] [96.0/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 21:09:37,147 - Train: 9.60% [465300/4848000] [96.0/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:10:10,077 - Train: 9.60% [465400/4848000] [96.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:10:12,726 - ==> Total time: 1 day, 21:44:11 Eta: 17 days, 22:41:05 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 21:10:45,036 - Train: 9.60% [465500/4848000] [96.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:11:17,986 - Train: 9.60% [465600/4848000] [96.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:11:50,903 - Train: 9.61% [465700/4848000] [96.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 21:12:23,836 - Train: 9.61% [465800/4848000] [96.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 21:12:56,756 - Train: 9.61% [465900/4848000] [96.1/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 21:13:29,656 - Train: 9.61% [466000/4848000] [96.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 21:14:02,613 - Train: 9.61% [466100/4848000] [96.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:14:36,770 - Train: 9.62% [466200/4848000] [96.2/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:15:09,698 - Train: 9.62% [466300/4848000] [96.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:15:42,598 - Train: 9.62% [466400/4848000] [96.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 21:16:15,675 - Train: 9.62% [466500/4848000] [96.2/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 21:16:48,664 - Train: 9.62% [466600/4848000] [96.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:17:23,383 - Train: 9.63% [466700/4848000] [96.3/1000.0] [batch_t 0.330 (0.347)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:17:56,303 - Train: 9.63% [466800/4848000] [96.3/1000.0] [batch_t 0.336 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-04 21:18:30,553 - Train: 9.63% [466900/4848000] [96.3/1000.0] [batch_t 0.330 (0.342)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:19:19,509 - Train: 9.63% [467000/4848000] [96.3/1000.0] [batch_t 0.333 (0.489)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 21:19:52,491 - Train: 9.63% [467100/4848000] [96.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 21:20:26,794 - Train: 9.64% [467200/4848000] [96.4/1000.0] [batch_t 0.328 (0.343)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 21:20:59,771 - Train: 9.64% [467300/4848000] [96.4/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 21:21:33,365 - Train: 9.64% [467400/4848000] [96.4/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:22:06,411 - Train: 9.64% [467500/4848000] [96.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:22:39,361 - Train: 9.65% [467600/4848000] [96.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 21:23:12,407 - Train: 9.65% [467700/4848000] [96.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:23:45,422 - Train: 9.65% [467800/4848000] [96.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 21:24:31,408 - Train: 9.65% [467900/4848000] [96.5/1000.0] [batch_t 0.335 (0.460)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 21:25:04,467 - Train: 9.65% [468000/4848000] [96.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 21:25:37,457 - Train: 9.66% [468100/4848000] [96.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 21:26:10,998 - Train: 9.66% [468200/4848000] [96.6/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 21:26:43,991 - Train: 9.66% [468300/4848000] [96.6/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 21:27:17,821 - Train: 9.66% [468400/4848000] [96.6/1000.0] [batch_t 0.339 (0.338)] [data_t 0.002] [optim_t 0.337] [lr 0.005000] 2024-04-04 21:27:50,863 - Train: 9.66% [468500/4848000] [96.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 21:28:23,891 - Train: 9.67% [468600/4848000] [96.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 21:28:57,005 - Train: 9.67% [468700/4848000] [96.7/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:29:30,845 - Train: 9.67% [468800/4848000] [96.7/1000.0] [batch_t 0.337 (0.338)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-04 21:30:03,896 - Train: 9.67% [468900/4848000] [96.7/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 21:30:37,508 - Train: 9.67% [469000/4848000] [96.7/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:31:12,459 - Train: 9.68% [469100/4848000] [96.8/1000.0] [batch_t 0.331 (0.349)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 21:31:45,477 - Train: 9.68% [469200/4848000] [96.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 21:32:19,998 - Train: 9.68% [469300/4848000] [96.8/1000.0] [batch_t 0.332 (0.345)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 21:32:53,189 - Train: 9.68% [469400/4848000] [96.8/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:33:27,570 - Train: 9.68% [469500/4848000] [96.8/1000.0] [batch_t 0.335 (0.344)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 21:34:00,645 - Train: 9.69% [469600/4848000] [96.9/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 21:34:37,003 - Train: 9.69% [469700/4848000] [96.9/1000.0] [batch_t 0.330 (0.363)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:35:10,574 - Train: 9.69% [469800/4848000] [96.9/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 21:35:43,588 - Train: 9.69% [469900/4848000] [96.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:36:17,680 - Train: 9.69% [470000/4848000] [96.9/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:36:50,702 - Train: 9.70% [470100/4848000] [97.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 21:37:24,478 - Train: 9.70% [470200/4848000] [97.0/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:37:44,867 - ==> Total time: 1 day, 22:11:43 Eta: 17 days, 22:02:44 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 21:38:01,496 - Train: 9.70% [470300/4848000] [97.0/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 21:38:34,906 - Train: 9.70% [470400/4848000] [97.0/1000.0] [batch_t 0.331 (0.334)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 21:39:08,838 - Train: 9.71% [470500/4848000] [97.1/1000.0] [batch_t 0.331 (0.339)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 21:39:41,862 - Train: 9.71% [470600/4848000] [97.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:40:16,332 - Train: 9.71% [470700/4848000] [97.1/1000.0] [batch_t 0.329 (0.345)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 21:40:52,368 - Train: 9.71% [470800/4848000] [97.1/1000.0] [batch_t 0.326 (0.360)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-04 21:41:26,400 - Train: 9.71% [470900/4848000] [97.1/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 21:41:59,394 - Train: 9.72% [471000/4848000] [97.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 21:42:34,168 - Train: 9.72% [471100/4848000] [97.2/1000.0] [batch_t 0.330 (0.348)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:43:08,084 - Train: 9.72% [471200/4848000] [97.2/1000.0] [batch_t 0.331 (0.339)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 21:43:41,054 - Train: 9.72% [471300/4848000] [97.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 21:44:22,767 - Train: 9.72% [471400/4848000] [97.2/1000.0] [batch_t 0.336 (0.417)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-04 21:44:55,767 - Train: 9.73% [471500/4848000] [97.3/1000.0] [batch_t 0.336 (0.330)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-04 21:45:30,060 - Train: 9.73% [471600/4848000] [97.3/1000.0] [batch_t 0.330 (0.343)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:46:03,012 - Train: 9.73% [471700/4848000] [97.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 21:46:37,164 - Train: 9.73% [471800/4848000] [97.3/1000.0] [batch_t 0.331 (0.341)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 21:47:10,798 - Train: 9.73% [471900/4848000] [97.3/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 21:47:43,772 - Train: 9.74% [472000/4848000] [97.4/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 21:48:17,395 - Train: 9.74% [472100/4848000] [97.4/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:48:50,532 - Train: 9.74% [472200/4848000] [97.4/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 21:49:24,293 - Train: 9.74% [472300/4848000] [97.4/1000.0] [batch_t 0.331 (0.337)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 21:49:57,254 - Train: 9.74% [472400/4848000] [97.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:50:33,043 - Train: 9.75% [472500/4848000] [97.5/1000.0] [batch_t 0.336 (0.358)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-04 21:51:07,078 - Train: 9.75% [472600/4848000] [97.5/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 21:51:40,103 - Train: 9.75% [472700/4848000] [97.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:52:13,535 - Train: 9.75% [472800/4848000] [97.5/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 21:52:46,481 - Train: 9.75% [472900/4848000] [97.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:53:20,135 - Train: 9.76% [473000/4848000] [97.6/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 21:53:53,116 - Train: 9.76% [473100/4848000] [97.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:54:28,063 - Train: 9.76% [473200/4848000] [97.6/1000.0] [batch_t 0.329 (0.349)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 21:55:01,019 - Train: 9.76% [473300/4848000] [97.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:55:33,973 - Train: 9.76% [473400/4848000] [97.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 21:56:07,495 - Train: 9.77% [473500/4848000] [97.7/1000.0] [batch_t 0.332 (0.335)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 21:56:40,587 - Train: 9.77% [473600/4848000] [97.7/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 21:57:13,755 - Train: 9.77% [473700/4848000] [97.7/1000.0] [batch_t 0.332 (0.332)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 21:57:46,717 - Train: 9.77% [473800/4848000] [97.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 21:58:21,720 - Train: 9.78% [473900/4848000] [97.8/1000.0] [batch_t 0.330 (0.350)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 21:58:54,714 - Train: 9.78% [474000/4848000] [97.8/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 21:59:28,310 - Train: 9.78% [474100/4848000] [97.8/1000.0] [batch_t 0.336 (0.336)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-04 22:00:01,283 - Train: 9.78% [474200/4848000] [97.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:00:34,220 - Train: 9.78% [474300/4848000] [97.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 22:01:07,768 - Train: 9.79% [474400/4848000] [97.9/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:01:40,724 - Train: 9.79% [474500/4848000] [97.9/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 22:02:14,188 - Train: 9.79% [474600/4848000] [97.9/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:02:47,113 - Train: 9.79% [474700/4848000] [97.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 22:03:20,554 - Train: 9.79% [474800/4848000] [97.9/1000.0] [batch_t 0.334 (0.334)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 22:03:53,548 - Train: 9.80% [474900/4848000] [98.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:04:27,351 - Train: 9.80% [475000/4848000] [98.0/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 22:05:02,952 - Train: 9.80% [475100/4848000] [98.0/1000.0] [batch_t 0.335 (0.356)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 22:05:04,286 - ==> Total time: 1 day, 22:39:02 Eta: 17 days, 21:22:39 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 22:05:39,284 - Train: 9.80% [475200/4848000] [98.0/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 22:06:13,529 - Train: 9.80% [475300/4848000] [98.0/1000.0] [batch_t 0.331 (0.342)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 22:06:46,493 - Train: 9.81% [475400/4848000] [98.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:07:19,998 - Train: 9.81% [475500/4848000] [98.1/1000.0] [batch_t 0.332 (0.335)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 22:07:52,961 - Train: 9.81% [475600/4848000] [98.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 22:08:27,080 - Train: 9.81% [475700/4848000] [98.1/1000.0] [batch_t 0.332 (0.341)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 22:09:00,073 - Train: 9.81% [475800/4848000] [98.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 22:09:34,212 - Train: 9.82% [475900/4848000] [98.2/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:10:08,209 - Train: 9.82% [476000/4848000] [98.2/1000.0] [batch_t 0.339 (0.340)] [data_t 0.002] [optim_t 0.337] [lr 0.005000] 2024-04-04 22:10:41,117 - Train: 9.82% [476100/4848000] [98.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 22:11:16,107 - Train: 9.82% [476200/4848000] [98.2/1000.0] [batch_t 0.344 (0.350)] [data_t 0.002] [optim_t 0.342] [lr 0.005000] 2024-04-04 22:11:49,106 - Train: 9.82% [476300/4848000] [98.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 22:12:23,839 - Train: 9.83% [476400/4848000] [98.3/1000.0] [batch_t 0.329 (0.347)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:12:56,838 - Train: 9.83% [476500/4848000] [98.3/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 22:13:30,980 - Train: 9.83% [476600/4848000] [98.3/1000.0] [batch_t 0.332 (0.341)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 22:14:03,963 - Train: 9.83% [476700/4848000] [98.3/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 22:14:37,752 - Train: 9.83% [476800/4848000] [98.3/1000.0] [batch_t 0.332 (0.338)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 22:15:12,144 - Train: 9.84% [476900/4848000] [98.4/1000.0] [batch_t 0.327 (0.344)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 22:15:45,098 - Train: 9.84% [477000/4848000] [98.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 22:16:19,111 - Train: 9.84% [477100/4848000] [98.4/1000.0] [batch_t 0.330 (0.340)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:16:52,061 - Train: 9.84% [477200/4848000] [98.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 22:17:27,367 - Train: 9.85% [477300/4848000] [98.5/1000.0] [batch_t 0.331 (0.353)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 22:18:00,335 - Train: 9.85% [477400/4848000] [98.5/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 22:18:34,084 - Train: 9.85% [477500/4848000] [98.5/1000.0] [batch_t 0.336 (0.337)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-04 22:19:08,139 - Train: 9.85% [477600/4848000] [98.5/1000.0] [batch_t 0.332 (0.340)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 22:19:41,127 - Train: 9.85% [477700/4848000] [98.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 22:20:14,985 - Train: 9.86% [477800/4848000] [98.6/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:20:48,118 - Train: 9.86% [477900/4848000] [98.6/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:21:21,115 - Train: 9.86% [478000/4848000] [98.6/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 22:21:54,108 - Train: 9.86% [478100/4848000] [98.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:22:27,676 - Train: 9.86% [478200/4848000] [98.6/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:23:00,643 - Train: 9.87% [478300/4848000] [98.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:23:34,097 - Train: 9.87% [478400/4848000] [98.7/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-04 22:24:07,582 - Train: 9.87% [478500/4848000] [98.7/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:24:40,572 - Train: 9.87% [478600/4848000] [98.7/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 22:25:14,558 - Train: 9.87% [478700/4848000] [98.7/1000.0] [batch_t 0.330 (0.340)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 22:25:47,562 - Train: 9.88% [478800/4848000] [98.8/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 22:26:20,551 - Train: 9.88% [478900/4848000] [98.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:26:53,564 - Train: 9.88% [479000/4848000] [98.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:27:26,555 - Train: 9.88% [479100/4848000] [98.8/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 22:27:59,541 - Train: 9.88% [479200/4848000] [98.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 22:28:33,718 - Train: 9.89% [479300/4848000] [98.9/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 22:29:07,641 - Train: 9.89% [479400/4848000] [98.9/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 22:29:40,633 - Train: 9.89% [479500/4848000] [98.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 22:30:13,624 - Train: 9.89% [479600/4848000] [98.9/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 22:30:46,574 - Train: 9.89% [479700/4848000] [98.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 22:31:20,364 - Train: 9.90% [479800/4848000] [99.0/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 22:31:53,353 - Train: 9.90% [479900/4848000] [99.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 22:32:12,993 - ==> Total time: 1 day, 23:06:11 Eta: 17 days, 20:41:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 22:32:30,738 - Train: 9.90% [480000/4848000] [99.0/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 22:33:03,681 - Train: 9.90% [480100/4848000] [99.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 22:33:37,912 - Train: 9.91% [480200/4848000] [99.1/1000.0] [batch_t 0.330 (0.342)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:34:10,920 - Train: 9.91% [480300/4848000] [99.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:34:43,840 - Train: 9.91% [480400/4848000] [99.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 22:35:17,858 - Train: 9.91% [480500/4848000] [99.1/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 22:35:50,949 - Train: 9.91% [480600/4848000] [99.1/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 22:36:25,283 - Train: 9.92% [480700/4848000] [99.2/1000.0] [batch_t 0.331 (0.343)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 22:36:58,216 - Train: 9.92% [480800/4848000] [99.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:37:31,080 - Train: 9.92% [480900/4848000] [99.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 22:38:04,035 - Train: 9.92% [481000/4848000] [99.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 22:38:38,777 - Train: 9.92% [481100/4848000] [99.2/1000.0] [batch_t 0.331 (0.347)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 22:39:12,228 - Train: 9.93% [481200/4848000] [99.3/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 22:39:45,159 - Train: 9.93% [481300/4848000] [99.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 22:40:18,422 - Train: 9.93% [481400/4848000] [99.3/1000.0] [batch_t 0.340 (0.333)] [data_t 0.002] [optim_t 0.338] [lr 0.005000] 2024-04-04 22:40:51,441 - Train: 9.93% [481500/4848000] [99.3/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 22:41:24,718 - Train: 9.93% [481600/4848000] [99.3/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:41:57,742 - Train: 9.94% [481700/4848000] [99.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 22:42:31,344 - Train: 9.94% [481800/4848000] [99.4/1000.0] [batch_t 0.333 (0.336)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 22:43:04,370 - Train: 9.94% [481900/4848000] [99.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:43:38,290 - Train: 9.94% [482000/4848000] [99.4/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:44:12,547 - Train: 9.94% [482100/4848000] [99.4/1000.0] [batch_t 0.331 (0.342)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 22:44:45,551 - Train: 9.95% [482200/4848000] [99.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 22:45:20,216 - Train: 9.95% [482300/4848000] [99.5/1000.0] [batch_t 0.330 (0.347)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:45:53,145 - Train: 9.95% [482400/4848000] [99.5/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 22:46:27,209 - Train: 9.95% [482500/4848000] [99.5/1000.0] [batch_t 0.331 (0.341)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 22:47:00,246 - Train: 9.95% [482600/4848000] [99.5/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-04 22:47:33,759 - Train: 9.96% [482700/4848000] [99.6/1000.0] [batch_t 0.332 (0.335)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 22:48:07,093 - Train: 9.96% [482800/4848000] [99.6/1000.0] [batch_t 0.331 (0.333)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-04 22:48:40,031 - Train: 9.96% [482900/4848000] [99.6/1000.0] [batch_t 0.336 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-04 22:49:13,448 - Train: 9.96% [483000/4848000] [99.6/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 22:49:46,376 - Train: 9.96% [483100/4848000] [99.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:50:20,432 - Train: 9.97% [483200/4848000] [99.7/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 22:50:53,530 - Train: 9.97% [483300/4848000] [99.7/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 22:51:27,660 - Train: 9.97% [483400/4848000] [99.7/1000.0] [batch_t 0.331 (0.341)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 22:52:00,808 - Train: 9.97% [483500/4848000] [99.7/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 22:52:35,801 - Train: 9.98% [483600/4848000] [99.8/1000.0] [batch_t 0.334 (0.350)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 22:53:08,727 - Train: 9.98% [483700/4848000] [99.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-04 22:53:41,682 - Train: 9.98% [483800/4848000] [99.8/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 22:54:16,733 - Train: 9.98% [483900/4848000] [99.8/1000.0] [batch_t 0.333 (0.350)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-04 22:54:49,683 - Train: 9.98% [484000/4848000] [99.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 22:55:22,606 - Train: 9.99% [484100/4848000] [99.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 22:55:55,550 - Train: 9.99% [484200/4848000] [99.9/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-04 22:56:29,584 - Train: 9.99% [484300/4848000] [99.9/1000.0] [batch_t 0.332 (0.340)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 22:57:02,579 - Train: 9.99% [484400/4848000] [99.9/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 22:57:37,118 - Train: 9.99% [484500/4848000] [99.9/1000.0] [batch_t 0.329 (0.345)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-04 22:58:11,559 - Train: 10.00% [484600/4848000] [100.0/1000.0] [batch_t 0.330 (0.344)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:58:44,609 - Train: 10.00% [484700/4848000] [100.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-04 22:59:19,404 - Train: 10.00% [484800/4848000] [100.0/1000.0] [batch_t 0.329 (0.348)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-04 22:59:28,681 - Test: 16.13% [50/310] [batch_t 0.158 (0.167)] 2024-04-04 22:59:36,394 - Test: 32.26% [100/310] [batch_t 0.159 (0.161)] 2024-04-04 22:59:45,425 - Test: 48.39% [150/310] [batch_t 0.278 (0.167)] 2024-04-04 22:59:57,557 - Test: 64.52% [200/310] [batch_t 0.178 (0.186)] 2024-04-04 23:00:11,455 - Test: 80.65% [250/310] [batch_t 0.159 (0.205)] 2024-04-04 23:00:21,901 - Test: 96.77% [300/310] [batch_t 0.594 (0.205)] 2024-04-04 23:00:24,357 - Test: 100.00% [310/310] [batch_t 0.081 (0.207)] 2024-04-04 23:26:12,071 - ==> Metric Time for coco : 0.005 (mAUROC_sp_max) 0.002 (mAP_sp_max) 0.002 (mF1_max_sp_max) 357.699 (mAUROC_px) 270.512 (mAP_px) 33.127 (mF1_max_px) 806.477 (mAUPRO_px) 12.266 (mF1_px_0.2_0.8_0.1) 12.293 (mAcc_px_0.2_0.8_0.1) 11.876 (mIoU_px_0.2_0.8_0.1) 34.376 (mIoU_max_px) 2024-04-04 23:26:12,464 - | Name | mAUROC_sp_max | mAUROC_sp_max (Max) | mAP_sp_max | mAP_sp_max (Max) | mF1_max_sp_max | mF1_max_sp_max (Max) | mAUROC_px | mAUROC_px (Max) | mAP_px | mAP_px (Max) | mF1_max_px | mF1_max_px (Max) | mAUPRO_px | mAUPRO_px (Max) | mF1_px_0.2_0.8_0.1 | mF1_px_0.2_0.8_0.1 (Max) | mAcc_px_0.2_0.8_0.1 | mAcc_px_0.2_0.8_0.1 (Max) | mIoU_px_0.2_0.8_0.1 | mIoU_px_0.2_0.8_0.1 (Max) | mIoU_max_px | mIoU_max_px (Max) | |:------:|:---------------:|:---------------------:|:------------:|:------------------:|:----------------:|:----------------------:|:-----------:|:------------------:|:--------:|:-----------------:|:------------:|:------------------:|:-----------:|:------------------:|:--------------------:|:--------------------------:|:---------------------:|:---------------------------:|:---------------------:|:---------------------------:|:-------------:|:-------------------:| | coco | 56.836 | 58.404 (50 epoch) | 38.398 | 40.479 (50 epoch) | 51.509 | 51.858 (50 epoch) | 59.488 | 59.488 (100 epoch) | 8.714 | 8.714 (100 epoch) | 14.504 | 14.504 (100 epoch) | 31.503 | 34.736 (50 epoch) | 8.354 | 9.008 (50 epoch) | 36.138 | 38.999 (50 epoch) | 4.462 | 4.810 (50 epoch) | 7.819 | 7.819 (100 epoch) | 2024-04-04 23:26:13,012 - ==> Total time: 2 days, 0:00:11 Eta: 18 days, 0:01:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-04 23:27:32,030 - Train: 10.00% [484900/4848000] [100.0/1000.0] [batch_t 0.760 (0.761)] [data_t 0.002] [optim_t 0.758] [lr 0.005000] 2024-04-04 23:28:47,834 - Train: 10.00% [485000/4848000] [100.0/1000.0] [batch_t 0.759 (0.758)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-04 23:30:03,643 - Train: 10.01% [485100/4848000] [100.1/1000.0] [batch_t 0.762 (0.758)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-04 23:31:19,571 - Train: 10.01% [485200/4848000] [100.1/1000.0] [batch_t 0.753 (0.758)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-04 23:32:35,440 - Train: 10.01% [485300/4848000] [100.1/1000.0] [batch_t 0.759 (0.759)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-04 23:33:51,241 - Train: 10.01% [485400/4848000] [100.1/1000.0] [batch_t 0.759 (0.758)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-04 23:35:06,994 - Train: 10.01% [485500/4848000] [100.1/1000.0] [batch_t 0.753 (0.757)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-04 23:36:22,822 - Train: 10.02% [485600/4848000] [100.2/1000.0] [batch_t 0.758 (0.758)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-04 23:37:38,749 - Train: 10.02% [485700/4848000] [100.2/1000.0] [batch_t 0.759 (0.759)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-04 23:38:54,621 - Train: 10.02% [485800/4848000] [100.2/1000.0] [batch_t 0.762 (0.759)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-04 23:40:10,416 - Train: 10.02% [485900/4848000] [100.2/1000.0] [batch_t 0.763 (0.758)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-04 23:41:26,290 - Train: 10.02% [486000/4848000] [100.2/1000.0] [batch_t 0.763 (0.759)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-04 23:42:42,112 - Train: 10.03% [486100/4848000] [100.3/1000.0] [batch_t 0.754 (0.758)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-04 23:43:57,996 - Train: 10.03% [486200/4848000] [100.3/1000.0] [batch_t 0.749 (0.759)] [data_t 0.002] [optim_t 0.746] [lr 0.005000] 2024-04-04 23:45:13,827 - Train: 10.03% [486300/4848000] [100.3/1000.0] [batch_t 0.750 (0.758)] [data_t 0.002] [optim_t 0.748] [lr 0.005000] 2024-04-04 23:46:29,724 - Train: 10.03% [486400/4848000] [100.3/1000.0] [batch_t 0.760 (0.759)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-04 23:47:45,528 - Train: 10.04% [486500/4848000] [100.4/1000.0] [batch_t 0.764 (0.758)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-04 23:49:01,362 - Train: 10.04% [486600/4848000] [100.4/1000.0] [batch_t 0.756 (0.758)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-04 23:50:17,063 - Train: 10.04% [486700/4848000] [100.4/1000.0] [batch_t 0.746 (0.757)] [data_t 0.002] [optim_t 0.744] [lr 0.005000] 2024-04-04 23:51:32,876 - Train: 10.04% [486800/4848000] [100.4/1000.0] [batch_t 0.767 (0.758)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-04 23:52:48,599 - Train: 10.04% [486900/4848000] [100.4/1000.0] [batch_t 0.750 (0.757)] [data_t 0.002] [optim_t 0.748] [lr 0.005000] 2024-04-04 23:54:04,460 - Train: 10.05% [487000/4848000] [100.5/1000.0] [batch_t 0.764 (0.759)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-04 23:55:20,226 - Train: 10.05% [487100/4848000] [100.5/1000.0] [batch_t 0.753 (0.758)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-04 23:56:35,934 - Train: 10.05% [487200/4848000] [100.5/1000.0] [batch_t 0.759 (0.757)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-04 23:57:51,760 - Train: 10.05% [487300/4848000] [100.5/1000.0] [batch_t 0.754 (0.758)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-04 23:59:07,555 - Train: 10.05% [487400/4848000] [100.5/1000.0] [batch_t 0.759 (0.758)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-05 00:00:23,418 - Train: 10.06% [487500/4848000] [100.6/1000.0] [batch_t 0.750 (0.759)] [data_t 0.002] [optim_t 0.748] [lr 0.005000] 2024-04-05 00:01:39,144 - Train: 10.06% [487600/4848000] [100.6/1000.0] [batch_t 0.760 (0.757)] [data_t 0.002] [optim_t 0.758] [lr 0.005000] 2024-04-05 00:02:55,023 - Train: 10.06% [487700/4848000] [100.6/1000.0] [batch_t 0.755 (0.759)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-05 00:04:10,805 - Train: 10.06% [487800/4848000] [100.6/1000.0] [batch_t 0.761 (0.758)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-05 00:05:26,539 - Train: 10.06% [487900/4848000] [100.6/1000.0] [batch_t 0.755 (0.757)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-05 00:06:42,358 - Train: 10.07% [488000/4848000] [100.7/1000.0] [batch_t 0.759 (0.758)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-05 00:07:58,059 - Train: 10.07% [488100/4848000] [100.7/1000.0] [batch_t 0.748 (0.757)] [data_t 0.002] [optim_t 0.746] [lr 0.005000] 2024-04-05 00:09:13,896 - Train: 10.07% [488200/4848000] [100.7/1000.0] [batch_t 0.757 (0.758)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-05 00:10:29,655 - Train: 10.07% [488300/4848000] [100.7/1000.0] [batch_t 0.756 (0.757)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-05 00:11:45,354 - Train: 10.07% [488400/4848000] [100.7/1000.0] [batch_t 0.749 (0.757)] [data_t 0.002] [optim_t 0.747] [lr 0.005000] 2024-04-05 00:13:01,201 - Train: 10.08% [488500/4848000] [100.8/1000.0] [batch_t 0.760 (0.758)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-05 00:14:16,995 - Train: 10.08% [488600/4848000] [100.8/1000.0] [batch_t 0.758 (0.758)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-05 00:15:32,729 - Train: 10.08% [488700/4848000] [100.8/1000.0] [batch_t 0.749 (0.757)] [data_t 0.002] [optim_t 0.747] [lr 0.005000] 2024-04-05 00:16:48,513 - Train: 10.08% [488800/4848000] [100.8/1000.0] [batch_t 0.745 (0.758)] [data_t 0.002] [optim_t 0.743] [lr 0.005000] 2024-04-05 00:18:04,376 - Train: 10.08% [488900/4848000] [100.8/1000.0] [batch_t 0.760 (0.759)] [data_t 0.002] [optim_t 0.758] [lr 0.005000] 2024-04-05 00:19:20,196 - Train: 10.09% [489000/4848000] [100.9/1000.0] [batch_t 0.755 (0.758)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-05 00:20:36,094 - Train: 10.09% [489100/4848000] [100.9/1000.0] [batch_t 0.759 (0.759)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-05 00:21:51,829 - Train: 10.09% [489200/4848000] [100.9/1000.0] [batch_t 0.749 (0.757)] [data_t 0.002] [optim_t 0.747] [lr 0.005000] 2024-04-05 00:23:07,589 - Train: 10.09% [489300/4848000] [100.9/1000.0] [batch_t 0.768 (0.758)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 00:24:23,471 - Train: 10.09% [489400/4848000] [100.9/1000.0] [batch_t 0.758 (0.759)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-05 00:25:39,246 - Train: 10.10% [489500/4848000] [101.0/1000.0] [batch_t 0.757 (0.758)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-05 00:26:55,038 - Train: 10.10% [489600/4848000] [101.0/1000.0] [batch_t 0.759 (0.758)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-05 00:27:31,418 - ==> Total time: 2 days, 1:01:29 Eta: 18 days, 4:22:15 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 00:28:12,996 - Train: 10.10% [489700/4848000] [101.0/1000.0] [batch_t 0.761 (0.762)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-05 00:29:28,857 - Train: 10.10% [489800/4848000] [101.0/1000.0] [batch_t 0.762 (0.759)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-05 00:30:18,535 - Train: 10.11% [489900/4848000] [101.1/1000.0] [batch_t 0.327 (0.497)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:30:51,363 - Train: 10.11% [490000/4848000] [101.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 00:31:25,289 - Train: 10.11% [490100/4848000] [101.1/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 00:31:58,093 - Train: 10.11% [490200/4848000] [101.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 00:32:32,785 - Train: 10.11% [490300/4848000] [101.1/1000.0] [batch_t 0.322 (0.347)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-05 00:33:06,686 - Train: 10.12% [490400/4848000] [101.2/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 00:33:39,614 - Train: 10.12% [490500/4848000] [101.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 00:34:12,417 - Train: 10.12% [490600/4848000] [101.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 00:34:45,261 - Train: 10.12% [490700/4848000] [101.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 00:35:19,812 - Train: 10.12% [490800/4848000] [101.2/1000.0] [batch_t 0.328 (0.345)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 00:35:52,599 - Train: 10.13% [490900/4848000] [101.3/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 00:36:26,944 - Train: 10.13% [491000/4848000] [101.3/1000.0] [batch_t 0.327 (0.343)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 00:36:59,873 - Train: 10.13% [491100/4848000] [101.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 00:37:35,316 - Train: 10.13% [491200/4848000] [101.3/1000.0] [batch_t 0.325 (0.354)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 00:38:08,261 - Train: 10.13% [491300/4848000] [101.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:38:41,072 - Train: 10.14% [491400/4848000] [101.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 00:39:13,957 - Train: 10.14% [491500/4848000] [101.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 00:39:46,794 - Train: 10.14% [491600/4848000] [101.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:40:21,757 - Train: 10.14% [491700/4848000] [101.4/1000.0] [batch_t 0.327 (0.350)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:40:54,580 - Train: 10.14% [491800/4848000] [101.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 00:41:27,358 - Train: 10.15% [491900/4848000] [101.5/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 00:42:00,270 - Train: 10.15% [492000/4848000] [101.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 00:42:35,189 - Train: 10.15% [492100/4848000] [101.5/1000.0] [batch_t 0.327 (0.349)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:43:08,016 - Train: 10.15% [492200/4848000] [101.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:43:40,874 - Train: 10.15% [492300/4848000] [101.5/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 00:44:15,306 - Train: 10.16% [492400/4848000] [101.6/1000.0] [batch_t 0.329 (0.344)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 00:44:48,134 - Train: 10.16% [492500/4848000] [101.6/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 00:45:22,022 - Train: 10.16% [492600/4848000] [101.6/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:45:54,920 - Train: 10.16% [492700/4848000] [101.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 00:46:27,763 - Train: 10.17% [492800/4848000] [101.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 00:47:00,662 - Train: 10.17% [492900/4848000] [101.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 00:47:34,087 - Train: 10.17% [493000/4848000] [101.7/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 00:48:08,396 - Train: 10.17% [493100/4848000] [101.7/1000.0] [batch_t 0.324 (0.343)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 00:48:41,258 - Train: 10.17% [493200/4848000] [101.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 00:49:15,206 - Train: 10.18% [493300/4848000] [101.8/1000.0] [batch_t 0.324 (0.339)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 00:49:48,183 - Train: 10.18% [493400/4848000] [101.8/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 00:50:21,893 - Train: 10.18% [493500/4848000] [101.8/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:50:54,783 - Train: 10.18% [493600/4848000] [101.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 00:51:28,904 - Train: 10.18% [493700/4848000] [101.8/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:52:01,695 - Train: 10.19% [493800/4848000] [101.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 00:52:35,297 - Train: 10.19% [493900/4848000] [101.9/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:53:08,487 - Train: 10.19% [494000/4848000] [101.9/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 00:53:41,268 - Train: 10.19% [494100/4848000] [101.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 00:54:15,002 - Train: 10.19% [494200/4848000] [101.9/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:54:47,845 - Train: 10.20% [494300/4848000] [102.0/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-05 00:55:21,698 - Train: 10.20% [494400/4848000] [102.0/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 00:55:53,203 - ==> Total time: 2 days, 1:29:51 Eta: 18 days, 3:46:25 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 00:55:56,668 - Train: 10.20% [494500/4848000] [102.0/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:56:29,470 - Train: 10.20% [494600/4848000] [102.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 00:57:02,420 - Train: 10.20% [494700/4848000] [102.0/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 00:57:35,503 - Train: 10.21% [494800/4848000] [102.1/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 00:58:09,356 - Train: 10.21% [494900/4848000] [102.1/1000.0] [batch_t 0.333 (0.338)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 00:58:42,190 - Train: 10.21% [495000/4848000] [102.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:59:15,979 - Train: 10.21% [495100/4848000] [102.1/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 00:59:48,848 - Train: 10.21% [495200/4848000] [102.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 01:00:21,677 - Train: 10.22% [495300/4848000] [102.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:00:54,560 - Train: 10.22% [495400/4848000] [102.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:01:28,933 - Train: 10.22% [495500/4848000] [102.2/1000.0] [batch_t 0.333 (0.344)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 01:02:01,852 - Train: 10.22% [495600/4848000] [102.2/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 01:02:35,048 - Train: 10.22% [495700/4848000] [102.2/1000.0] [batch_t 0.334 (0.332)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 01:03:08,529 - Train: 10.23% [495800/4848000] [102.3/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 01:03:41,491 - Train: 10.23% [495900/4848000] [102.3/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:04:16,659 - Train: 10.23% [496000/4848000] [102.3/1000.0] [batch_t 0.329 (0.352)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 01:04:49,527 - Train: 10.23% [496100/4848000] [102.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 01:05:22,620 - Train: 10.24% [496200/4848000] [102.4/1000.0] [batch_t 0.322 (0.331)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-05 01:05:55,484 - Train: 10.24% [496300/4848000] [102.4/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 01:06:30,454 - Train: 10.24% [496400/4848000] [102.4/1000.0] [batch_t 0.327 (0.350)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:07:03,217 - Train: 10.24% [496500/4848000] [102.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:07:37,110 - Train: 10.24% [496600/4848000] [102.4/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:08:10,653 - Train: 10.25% [496700/4848000] [102.5/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:08:43,386 - Train: 10.25% [496800/4848000] [102.5/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 01:09:17,483 - Train: 10.25% [496900/4848000] [102.5/1000.0] [batch_t 0.325 (0.341)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 01:09:50,249 - Train: 10.25% [497000/4848000] [102.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 01:10:24,173 - Train: 10.25% [497100/4848000] [102.5/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 01:10:56,970 - Train: 10.26% [497200/4848000] [102.6/1000.0] [batch_t 0.321 (0.328)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-05 01:11:29,781 - Train: 10.26% [497300/4848000] [102.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:12:02,593 - Train: 10.26% [497400/4848000] [102.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:12:36,135 - Train: 10.26% [497500/4848000] [102.6/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 01:13:09,010 - Train: 10.26% [497600/4848000] [102.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 01:13:41,803 - Train: 10.27% [497700/4848000] [102.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 01:14:14,693 - Train: 10.27% [497800/4848000] [102.7/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:14:47,523 - Train: 10.27% [497900/4848000] [102.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 01:15:21,321 - Train: 10.27% [498000/4848000] [102.7/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:15:54,123 - Train: 10.27% [498100/4848000] [102.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:16:27,873 - Train: 10.28% [498200/4848000] [102.8/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:17:00,730 - Train: 10.28% [498300/4848000] [102.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 01:17:34,772 - Train: 10.28% [498400/4848000] [102.8/1000.0] [batch_t 0.325 (0.340)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 01:18:08,158 - Train: 10.28% [498500/4848000] [102.8/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:18:40,958 - Train: 10.28% [498600/4848000] [102.8/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 01:19:16,130 - Train: 10.29% [498700/4848000] [102.9/1000.0] [batch_t 0.328 (0.352)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:19:48,902 - Train: 10.29% [498800/4848000] [102.9/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 01:20:22,310 - Train: 10.29% [498900/4848000] [102.9/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 01:20:55,161 - Train: 10.29% [499000/4848000] [102.9/1000.0] [batch_t 0.343 (0.328)] [data_t 0.002] [optim_t 0.341] [lr 0.005000] 2024-04-05 01:21:28,487 - Train: 10.29% [499100/4848000] [102.9/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:22:01,261 - Train: 10.30% [499200/4848000] [103.0/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 01:22:35,884 - Train: 10.30% [499300/4848000] [103.0/1000.0] [batch_t 0.329 (0.346)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 01:22:50,308 - ==> Total time: 2 days, 1:56:48 Eta: 18 days, 2:58:27 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 01:23:11,177 - Train: 10.30% [499400/4848000] [103.0/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:23:44,088 - Train: 10.30% [499500/4848000] [103.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:24:18,344 - Train: 10.31% [499600/4848000] [103.1/1000.0] [batch_t 0.327 (0.342)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:24:51,179 - Train: 10.31% [499700/4848000] [103.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:25:24,754 - Train: 10.31% [499800/4848000] [103.1/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:25:57,615 - Train: 10.31% [499900/4848000] [103.1/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 01:26:31,139 - Train: 10.31% [500000/4848000] [103.1/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:27:03,899 - Train: 10.32% [500100/4848000] [103.2/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 01:27:36,739 - Train: 10.32% [500200/4848000] [103.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 01:28:09,691 - Train: 10.32% [500300/4848000] [103.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:28:42,580 - Train: 10.32% [500400/4848000] [103.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:29:16,242 - Train: 10.32% [500500/4848000] [103.2/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:29:49,093 - Train: 10.33% [500600/4848000] [103.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 01:30:21,886 - Train: 10.33% [500700/4848000] [103.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 01:30:54,717 - Train: 10.33% [500800/4848000] [103.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 01:31:28,699 - Train: 10.33% [500900/4848000] [103.3/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:32:01,592 - Train: 10.33% [501000/4848000] [103.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 01:32:35,664 - Train: 10.34% [501100/4848000] [103.4/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 01:33:08,929 - Train: 10.34% [501200/4848000] [103.4/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 01:33:41,756 - Train: 10.34% [501300/4848000] [103.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:34:14,674 - Train: 10.34% [501400/4848000] [103.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:34:47,630 - Train: 10.34% [501500/4848000] [103.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 01:35:20,917 - Train: 10.35% [501600/4848000] [103.5/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 01:35:53,760 - Train: 10.35% [501700/4848000] [103.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 01:36:26,809 - Train: 10.35% [501800/4848000] [103.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:36:59,655 - Train: 10.35% [501900/4848000] [103.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 01:37:33,701 - Train: 10.35% [502000/4848000] [103.5/1000.0] [batch_t 0.331 (0.340)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 01:38:06,523 - Train: 10.36% [502100/4848000] [103.6/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-05 01:38:39,338 - Train: 10.36% [502200/4848000] [103.6/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 01:39:12,469 - Train: 10.36% [502300/4848000] [103.6/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 01:39:45,258 - Train: 10.36% [502400/4848000] [103.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 01:40:18,080 - Train: 10.37% [502500/4848000] [103.7/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 01:40:50,904 - Train: 10.37% [502600/4848000] [103.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:41:23,771 - Train: 10.37% [502700/4848000] [103.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:41:56,611 - Train: 10.37% [502800/4848000] [103.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:42:29,430 - Train: 10.37% [502900/4848000] [103.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 01:43:02,231 - Train: 10.38% [503000/4848000] [103.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:43:35,080 - Train: 10.38% [503100/4848000] [103.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:44:08,047 - Train: 10.38% [503200/4848000] [103.8/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 01:44:40,945 - Train: 10.38% [503300/4848000] [103.8/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 01:45:13,779 - Train: 10.38% [503400/4848000] [103.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:45:46,585 - Train: 10.39% [503500/4848000] [103.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 01:46:19,406 - Train: 10.39% [503600/4848000] [103.9/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 01:46:52,267 - Train: 10.39% [503700/4848000] [103.9/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 01:47:25,862 - Train: 10.39% [503800/4848000] [103.9/1000.0] [batch_t 0.331 (0.336)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 01:47:58,680 - Train: 10.39% [503900/4848000] [103.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:48:31,457 - Train: 10.40% [504000/4848000] [104.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:49:04,327 - Train: 10.40% [504100/4848000] [104.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:49:34,762 - ==> Total time: 2 days, 2:23:33 Eta: 18 days, 2:09:05 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 01:49:39,434 - Train: 10.40% [504200/4848000] [104.0/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:50:12,285 - Train: 10.40% [504300/4848000] [104.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 01:50:45,062 - Train: 10.40% [504400/4848000] [104.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 01:51:18,011 - Train: 10.41% [504500/4848000] [104.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 01:51:50,852 - Train: 10.41% [504600/4848000] [104.1/1000.0] [batch_t 0.336 (0.328)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-05 01:52:23,773 - Train: 10.41% [504700/4848000] [104.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 01:52:56,636 - Train: 10.41% [504800/4848000] [104.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 01:53:30,229 - Train: 10.41% [504900/4848000] [104.1/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:54:03,083 - Train: 10.42% [505000/4848000] [104.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 01:54:35,942 - Train: 10.42% [505100/4848000] [104.2/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 01:55:09,132 - Train: 10.42% [505200/4848000] [104.2/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 01:55:41,975 - Train: 10.42% [505300/4848000] [104.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:56:15,305 - Train: 10.42% [505400/4848000] [104.2/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 01:56:48,082 - Train: 10.43% [505500/4848000] [104.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 01:57:22,267 - Train: 10.43% [505600/4848000] [104.3/1000.0] [batch_t 0.326 (0.342)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 01:57:55,118 - Train: 10.43% [505700/4848000] [104.3/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 01:58:28,775 - Train: 10.43% [505800/4848000] [104.3/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 01:59:01,584 - Train: 10.44% [505900/4848000] [104.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 01:59:35,362 - Train: 10.44% [506000/4848000] [104.4/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:00:09,218 - Train: 10.44% [506100/4848000] [104.4/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 02:00:42,030 - Train: 10.44% [506200/4848000] [104.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 02:01:16,035 - Train: 10.44% [506300/4848000] [104.4/1000.0] [batch_t 0.326 (0.340)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 02:01:48,842 - Train: 10.45% [506400/4848000] [104.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 02:02:22,744 - Train: 10.45% [506500/4848000] [104.5/1000.0] [batch_t 0.323 (0.339)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 02:02:55,592 - Train: 10.45% [506600/4848000] [104.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:03:31,167 - Train: 10.45% [506700/4848000] [104.5/1000.0] [batch_t 0.331 (0.356)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 02:04:03,987 - Train: 10.45% [506800/4848000] [104.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 02:04:36,827 - Train: 10.46% [506900/4848000] [104.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 02:05:09,653 - Train: 10.46% [507000/4848000] [104.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 02:05:42,483 - Train: 10.46% [507100/4848000] [104.6/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-05 02:06:16,747 - Train: 10.46% [507200/4848000] [104.6/1000.0] [batch_t 0.329 (0.343)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 02:06:49,527 - Train: 10.46% [507300/4848000] [104.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 02:07:23,905 - Train: 10.47% [507400/4848000] [104.7/1000.0] [batch_t 0.329 (0.344)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 02:07:56,774 - Train: 10.47% [507500/4848000] [104.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 02:08:29,980 - Train: 10.47% [507600/4848000] [104.7/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:09:02,857 - Train: 10.47% [507700/4848000] [104.7/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 02:09:38,350 - Train: 10.47% [507800/4848000] [104.7/1000.0] [batch_t 0.325 (0.355)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 02:10:12,093 - Train: 10.48% [507900/4848000] [104.8/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 02:10:44,980 - Train: 10.48% [508000/4848000] [104.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 02:11:17,801 - Train: 10.48% [508100/4848000] [104.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 02:11:50,660 - Train: 10.48% [508200/4848000] [104.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 02:12:23,484 - Train: 10.48% [508300/4848000] [104.8/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 02:12:56,292 - Train: 10.49% [508400/4848000] [104.9/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 02:13:31,244 - Train: 10.49% [508500/4848000] [104.9/1000.0] [batch_t 0.333 (0.349)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 02:14:04,110 - Train: 10.49% [508600/4848000] [104.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 02:14:36,952 - Train: 10.49% [508700/4848000] [104.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 02:15:09,768 - Train: 10.50% [508800/4848000] [105.0/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 02:15:42,716 - Train: 10.50% [508900/4848000] [105.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:16:15,606 - Train: 10.50% [509000/4848000] [105.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 02:16:28,739 - ==> Total time: 2 days, 2:50:27 Eta: 18 days, 1:21:29 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 02:16:50,739 - Train: 10.50% [509100/4848000] [105.0/1000.0] [batch_t 0.324 (0.332)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 02:17:23,615 - Train: 10.50% [509200/4848000] [105.0/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 02:17:56,446 - Train: 10.51% [509300/4848000] [105.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 02:18:30,612 - Train: 10.51% [509400/4848000] [105.1/1000.0] [batch_t 0.333 (0.342)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 02:19:03,391 - Train: 10.51% [509500/4848000] [105.1/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 02:19:36,304 - Train: 10.51% [509600/4848000] [105.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 02:20:10,186 - Train: 10.51% [509700/4848000] [105.1/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 02:20:43,034 - Train: 10.52% [509800/4848000] [105.2/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 02:21:16,748 - Train: 10.52% [509900/4848000] [105.2/1000.0] [batch_t 0.325 (0.337)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 02:21:49,627 - Train: 10.52% [510000/4848000] [105.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 02:22:22,459 - Train: 10.52% [510100/4848000] [105.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 02:22:55,454 - Train: 10.52% [510200/4848000] [105.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:23:29,111 - Train: 10.53% [510300/4848000] [105.3/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:24:01,919 - Train: 10.53% [510400/4848000] [105.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 02:24:35,786 - Train: 10.53% [510500/4848000] [105.3/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 02:25:09,725 - Train: 10.53% [510600/4848000] [105.3/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 02:25:42,558 - Train: 10.53% [510700/4848000] [105.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 02:26:15,385 - Train: 10.54% [510800/4848000] [105.4/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 02:26:48,159 - Train: 10.54% [510900/4848000] [105.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 02:27:22,202 - Train: 10.54% [511000/4848000] [105.4/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 02:27:55,083 - Train: 10.54% [511100/4848000] [105.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 02:28:29,074 - Train: 10.54% [511200/4848000] [105.4/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:29:01,858 - Train: 10.55% [511300/4848000] [105.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 02:29:35,138 - Train: 10.55% [511400/4848000] [105.5/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 02:30:09,212 - Train: 10.55% [511500/4848000] [105.5/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:30:42,118 - Train: 10.55% [511600/4848000] [105.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 02:31:15,372 - Train: 10.55% [511700/4848000] [105.5/1000.0] [batch_t 0.326 (0.332)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 02:31:48,174 - Train: 10.56% [511800/4848000] [105.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 02:32:20,959 - Train: 10.56% [511900/4848000] [105.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:32:53,764 - Train: 10.56% [512000/4848000] [105.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:33:27,625 - Train: 10.56% [512100/4848000] [105.6/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 02:34:00,446 - Train: 10.57% [512200/4848000] [105.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 02:34:33,330 - Train: 10.57% [512300/4848000] [105.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 02:35:06,968 - Train: 10.57% [512400/4848000] [105.7/1000.0] [batch_t 0.325 (0.336)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 02:35:39,793 - Train: 10.57% [512500/4848000] [105.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 02:36:12,641 - Train: 10.57% [512600/4848000] [105.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 02:36:45,497 - Train: 10.58% [512700/4848000] [105.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:37:18,623 - Train: 10.58% [512800/4848000] [105.8/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:37:51,420 - Train: 10.58% [512900/4848000] [105.8/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 02:38:25,183 - Train: 10.58% [513000/4848000] [105.8/1000.0] [batch_t 0.496 (0.338)] [data_t 0.002] [optim_t 0.494] [lr 0.005000] 2024-04-05 02:38:57,983 - Train: 10.58% [513100/4848000] [105.8/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 02:39:30,778 - Train: 10.59% [513200/4848000] [105.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 02:40:03,534 - Train: 10.59% [513300/4848000] [105.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 02:40:36,317 - Train: 10.59% [513400/4848000] [105.9/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 02:41:09,140 - Train: 10.59% [513500/4848000] [105.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 02:41:42,004 - Train: 10.59% [513600/4848000] [105.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 02:42:15,770 - Train: 10.60% [513700/4848000] [106.0/1000.0] [batch_t 0.332 (0.338)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 02:42:48,572 - Train: 10.60% [513800/4848000] [106.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 02:43:17,544 - ==> Total time: 2 days, 3:17:16 Eta: 18 days, 0:33:33 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 02:43:23,537 - Train: 10.60% [513900/4848000] [106.0/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 02:43:56,355 - Train: 10.60% [514000/4848000] [106.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 02:44:29,428 - Train: 10.60% [514100/4848000] [106.0/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 02:45:02,207 - Train: 10.61% [514200/4848000] [106.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 02:45:35,092 - Train: 10.61% [514300/4848000] [106.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 02:46:08,052 - Train: 10.61% [514400/4848000] [106.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 02:46:40,784 - Train: 10.61% [514500/4848000] [106.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:47:13,592 - Train: 10.61% [514600/4848000] [106.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 02:47:46,463 - Train: 10.62% [514700/4848000] [106.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:48:19,320 - Train: 10.62% [514800/4848000] [106.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 02:48:52,268 - Train: 10.62% [514900/4848000] [106.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 02:49:25,195 - Train: 10.62% [515000/4848000] [106.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 02:49:58,032 - Train: 10.62% [515100/4848000] [106.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 02:50:30,911 - Train: 10.63% [515200/4848000] [106.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 02:51:03,795 - Train: 10.63% [515300/4848000] [106.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 02:51:37,671 - Train: 10.63% [515400/4848000] [106.3/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 02:52:10,561 - Train: 10.63% [515500/4848000] [106.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 02:52:43,535 - Train: 10.64% [515600/4848000] [106.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 02:53:17,180 - Train: 10.64% [515700/4848000] [106.4/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 02:53:50,270 - Train: 10.64% [515800/4848000] [106.4/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:54:23,415 - Train: 10.64% [515900/4848000] [106.4/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 02:54:56,344 - Train: 10.64% [516000/4848000] [106.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:55:29,237 - Train: 10.65% [516100/4848000] [106.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:56:02,174 - Train: 10.65% [516200/4848000] [106.5/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 02:56:35,124 - Train: 10.65% [516300/4848000] [106.5/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 02:57:07,990 - Train: 10.65% [516400/4848000] [106.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 02:57:40,800 - Train: 10.65% [516500/4848000] [106.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 02:58:13,635 - Train: 10.66% [516600/4848000] [106.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 02:58:46,432 - Train: 10.66% [516700/4848000] [106.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 02:59:19,194 - Train: 10.66% [516800/4848000] [106.6/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 02:59:52,007 - Train: 10.66% [516900/4848000] [106.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 03:00:24,792 - Train: 10.66% [517000/4848000] [106.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 03:00:57,576 - Train: 10.67% [517100/4848000] [106.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 03:01:31,884 - Train: 10.67% [517200/4848000] [106.7/1000.0] [batch_t 0.459 (0.343)] [data_t 0.002] [optim_t 0.458] [lr 0.005000] 2024-04-05 03:02:04,725 - Train: 10.67% [517300/4848000] [106.7/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 03:02:37,508 - Train: 10.67% [517400/4848000] [106.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 03:03:10,982 - Train: 10.67% [517500/4848000] [106.7/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 03:03:43,804 - Train: 10.68% [517600/4848000] [106.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 03:04:16,671 - Train: 10.68% [517700/4848000] [106.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 03:04:49,479 - Train: 10.68% [517800/4848000] [106.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 03:05:22,360 - Train: 10.68% [517900/4848000] [106.8/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 03:05:55,254 - Train: 10.68% [518000/4848000] [106.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 03:06:28,779 - Train: 10.69% [518100/4848000] [106.9/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 03:07:01,645 - Train: 10.69% [518200/4848000] [106.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 03:07:34,988 - Train: 10.69% [518300/4848000] [106.9/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 03:08:07,772 - Train: 10.69% [518400/4848000] [106.9/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 03:08:40,523 - Train: 10.70% [518500/4848000] [107.0/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-05 03:09:14,561 - Train: 10.70% [518600/4848000] [107.0/1000.0] [batch_t 0.330 (0.340)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 03:09:47,482 - Train: 10.70% [518700/4848000] [107.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 03:09:59,312 - ==> Total time: 2 days, 3:43:57 Eta: 17 days, 23:45:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 03:10:23,436 - Train: 10.70% [518800/4848000] [107.0/1000.0] [batch_t 0.330 (0.345)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 03:10:56,258 - Train: 10.70% [518900/4848000] [107.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 03:11:29,133 - Train: 10.71% [519000/4848000] [107.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 03:12:01,955 - Train: 10.71% [519100/4848000] [107.1/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 03:12:34,789 - Train: 10.71% [519200/4848000] [107.1/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 03:13:07,949 - Train: 10.71% [519300/4848000] [107.1/1000.0] [batch_t 0.323 (0.332)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 03:13:40,747 - Train: 10.71% [519400/4848000] [107.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 03:14:13,606 - Train: 10.72% [519500/4848000] [107.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 03:14:46,435 - Train: 10.72% [519600/4848000] [107.2/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 03:15:19,855 - Train: 10.72% [519700/4848000] [107.2/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 03:15:52,706 - Train: 10.72% [519800/4848000] [107.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 03:16:26,818 - Train: 10.72% [519900/4848000] [107.2/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 03:16:59,785 - Train: 10.73% [520000/4848000] [107.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 03:17:32,592 - Train: 10.73% [520100/4848000] [107.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 03:18:05,489 - Train: 10.73% [520200/4848000] [107.3/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 03:18:38,334 - Train: 10.73% [520300/4848000] [107.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 03:19:11,183 - Train: 10.73% [520400/4848000] [107.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 03:19:43,950 - Train: 10.74% [520500/4848000] [107.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 03:20:17,553 - Train: 10.74% [520600/4848000] [107.4/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 03:20:50,366 - Train: 10.74% [520700/4848000] [107.4/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 03:21:23,258 - Train: 10.74% [520800/4848000] [107.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 03:21:56,059 - Train: 10.74% [520900/4848000] [107.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 03:22:28,942 - Train: 10.75% [521000/4848000] [107.5/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 03:23:01,799 - Train: 10.75% [521100/4848000] [107.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 03:23:34,618 - Train: 10.75% [521200/4848000] [107.5/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 03:24:07,541 - Train: 10.75% [521300/4848000] [107.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 03:24:40,358 - Train: 10.75% [521400/4848000] [107.5/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 03:25:13,334 - Train: 10.76% [521500/4848000] [107.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 03:25:46,212 - Train: 10.76% [521600/4848000] [107.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 03:26:19,385 - Train: 10.76% [521700/4848000] [107.6/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 03:26:52,199 - Train: 10.76% [521800/4848000] [107.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 03:27:25,024 - Train: 10.77% [521900/4848000] [107.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 03:27:57,829 - Train: 10.77% [522000/4848000] [107.7/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-05 03:28:30,686 - Train: 10.77% [522100/4848000] [107.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 03:29:03,527 - Train: 10.77% [522200/4848000] [107.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 03:29:37,236 - Train: 10.77% [522300/4848000] [107.7/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 03:30:10,506 - Train: 10.78% [522400/4848000] [107.8/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 03:30:43,363 - Train: 10.78% [522500/4848000] [107.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 03:31:16,590 - Train: 10.78% [522600/4848000] [107.8/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 03:31:49,404 - Train: 10.78% [522700/4848000] [107.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 03:32:22,264 - Train: 10.78% [522800/4848000] [107.8/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 03:32:55,213 - Train: 10.79% [522900/4848000] [107.9/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 03:33:28,076 - Train: 10.79% [523000/4848000] [107.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 03:34:00,862 - Train: 10.79% [523100/4848000] [107.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 03:34:34,546 - Train: 10.79% [523200/4848000] [107.9/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 03:35:07,401 - Train: 10.79% [523300/4848000] [107.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 03:35:40,224 - Train: 10.80% [523400/4848000] [108.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 03:36:13,086 - Train: 10.80% [523500/4848000] [108.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 03:36:40,667 - ==> Total time: 2 days, 4:10:39 Eta: 17 days, 22:56:52 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 03:36:48,176 - Train: 10.80% [523600/4848000] [108.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 03:37:21,017 - Train: 10.80% [523700/4848000] [108.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 03:37:53,810 - Train: 10.80% [523800/4848000] [108.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 03:38:26,650 - Train: 10.81% [523900/4848000] [108.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 03:38:59,439 - Train: 10.81% [524000/4848000] [108.1/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 03:39:32,314 - Train: 10.81% [524100/4848000] [108.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 03:40:05,281 - Train: 10.81% [524200/4848000] [108.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 03:40:38,160 - Train: 10.81% [524300/4848000] [108.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 03:41:11,034 - Train: 10.82% [524400/4848000] [108.2/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 03:41:43,821 - Train: 10.82% [524500/4848000] [108.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 03:42:16,725 - Train: 10.82% [524600/4848000] [108.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 03:42:49,580 - Train: 10.82% [524700/4848000] [108.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 03:43:22,485 - Train: 10.83% [524800/4848000] [108.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 03:43:55,324 - Train: 10.83% [524900/4848000] [108.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 03:44:28,230 - Train: 10.83% [525000/4848000] [108.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 03:45:01,024 - Train: 10.83% [525100/4848000] [108.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 03:45:33,838 - Train: 10.83% [525200/4848000] [108.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 03:46:07,559 - Train: 10.84% [525300/4848000] [108.4/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 03:46:40,348 - Train: 10.84% [525400/4848000] [108.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 03:47:13,862 - Train: 10.84% [525500/4848000] [108.4/1000.0] [batch_t 0.325 (0.335)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 03:47:46,721 - Train: 10.84% [525600/4848000] [108.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 03:48:20,237 - Train: 10.84% [525700/4848000] [108.4/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 03:48:53,097 - Train: 10.85% [525800/4848000] [108.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 03:49:26,552 - Train: 10.85% [525900/4848000] [108.5/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 03:49:59,347 - Train: 10.85% [526000/4848000] [108.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 03:50:32,256 - Train: 10.85% [526100/4848000] [108.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 03:51:05,048 - Train: 10.85% [526200/4848000] [108.5/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 03:51:38,003 - Train: 10.86% [526300/4848000] [108.6/1000.0] [batch_t 0.340 (0.329)] [data_t 0.002] [optim_t 0.338] [lr 0.005000] 2024-04-05 03:52:11,781 - Train: 10.86% [526400/4848000] [108.6/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 03:52:44,574 - Train: 10.86% [526500/4848000] [108.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 03:53:17,901 - Train: 10.86% [526600/4848000] [108.6/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 03:53:50,675 - Train: 10.86% [526700/4848000] [108.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 03:54:24,101 - Train: 10.87% [526800/4848000] [108.7/1000.0] [batch_t 0.340 (0.334)] [data_t 0.002] [optim_t 0.338] [lr 0.005000] 2024-04-05 03:54:56,908 - Train: 10.87% [526900/4848000] [108.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 03:55:29,715 - Train: 10.87% [527000/4848000] [108.7/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 03:56:02,761 - Train: 10.87% [527100/4848000] [108.7/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 03:56:35,606 - Train: 10.87% [527200/4848000] [108.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 03:57:08,908 - Train: 10.88% [527300/4848000] [108.8/1000.0] [batch_t 0.326 (0.333)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 03:57:41,734 - Train: 10.88% [527400/4848000] [108.8/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 03:58:14,565 - Train: 10.88% [527500/4848000] [108.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 03:58:47,327 - Train: 10.88% [527600/4848000] [108.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 03:59:20,106 - Train: 10.88% [527700/4848000] [108.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 03:59:52,926 - Train: 10.89% [527800/4848000] [108.9/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 04:00:25,765 - Train: 10.89% [527900/4848000] [108.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 04:00:58,610 - Train: 10.89% [528000/4848000] [108.9/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 04:01:31,468 - Train: 10.89% [528100/4848000] [108.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 04:02:04,297 - Train: 10.90% [528200/4848000] [109.0/1000.0] [batch_t 0.340 (0.328)] [data_t 0.002] [optim_t 0.339] [lr 0.005000] 2024-04-05 04:02:37,066 - Train: 10.90% [528300/4848000] [109.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 04:03:09,860 - Train: 10.90% [528400/4848000] [109.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 04:03:20,382 - ==> Total time: 2 days, 4:37:18 Eta: 17 days, 22:08:52 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 04:03:45,011 - Train: 10.90% [528500/4848000] [109.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 04:04:17,747 - Train: 10.90% [528600/4848000] [109.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 04:04:50,504 - Train: 10.91% [528700/4848000] [109.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 04:05:23,341 - Train: 10.91% [528800/4848000] [109.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 04:05:56,162 - Train: 10.91% [528900/4848000] [109.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 04:06:29,366 - Train: 10.91% [529000/4848000] [109.1/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 04:07:02,255 - Train: 10.91% [529100/4848000] [109.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 04:07:35,100 - Train: 10.92% [529200/4848000] [109.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 04:08:08,086 - Train: 10.92% [529300/4848000] [109.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 04:08:40,917 - Train: 10.92% [529400/4848000] [109.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 04:09:13,819 - Train: 10.92% [529500/4848000] [109.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 04:09:46,672 - Train: 10.92% [529600/4848000] [109.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 04:10:19,555 - Train: 10.93% [529700/4848000] [109.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 04:10:52,415 - Train: 10.93% [529800/4848000] [109.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 04:11:28,157 - Train: 10.93% [529900/4848000] [109.3/1000.0] [batch_t 0.326 (0.357)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 04:12:01,023 - Train: 10.93% [530000/4848000] [109.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 04:12:33,924 - Train: 10.93% [530100/4848000] [109.3/1000.0] [batch_t 0.322 (0.329)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-05 04:13:06,784 - Train: 10.94% [530200/4848000] [109.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 04:13:39,622 - Train: 10.94% [530300/4848000] [109.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 04:14:12,491 - Train: 10.94% [530400/4848000] [109.4/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 04:14:45,369 - Train: 10.94% [530500/4848000] [109.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 04:15:19,182 - Train: 10.94% [530600/4848000] [109.4/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 04:15:52,078 - Train: 10.95% [530700/4848000] [109.5/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 04:16:26,457 - Train: 10.95% [530800/4848000] [109.5/1000.0] [batch_t 0.330 (0.344)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 04:16:59,298 - Train: 10.95% [530900/4848000] [109.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 04:17:32,129 - Train: 10.95% [531000/4848000] [109.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 04:18:04,938 - Train: 10.96% [531100/4848000] [109.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 04:18:37,753 - Train: 10.96% [531200/4848000] [109.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 04:19:10,697 - Train: 10.96% [531300/4848000] [109.6/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 04:19:43,495 - Train: 10.96% [531400/4848000] [109.6/1000.0] [batch_t 0.338 (0.328)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-05 04:20:16,949 - Train: 10.96% [531500/4848000] [109.6/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 04:20:49,736 - Train: 10.97% [531600/4848000] [109.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 04:21:23,693 - Train: 10.97% [531700/4848000] [109.7/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 04:21:56,520 - Train: 10.97% [531800/4848000] [109.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 04:22:31,928 - Train: 10.97% [531900/4848000] [109.7/1000.0] [batch_t 0.328 (0.354)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 04:23:04,828 - Train: 10.97% [532000/4848000] [109.7/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 04:23:37,814 - Train: 10.98% [532100/4848000] [109.8/1000.0] [batch_t 0.320 (0.330)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-05 04:24:12,415 - Train: 10.98% [532200/4848000] [109.8/1000.0] [batch_t 0.328 (0.346)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 04:24:45,303 - Train: 10.98% [532300/4848000] [109.8/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 04:25:18,594 - Train: 10.98% [532400/4848000] [109.8/1000.0] [batch_t 0.331 (0.333)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 04:25:51,477 - Train: 10.98% [532500/4848000] [109.8/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 04:26:25,123 - Train: 10.99% [532600/4848000] [109.9/1000.0] [batch_t 0.332 (0.336)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 04:26:57,985 - Train: 10.99% [532700/4848000] [109.9/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 04:27:32,345 - Train: 10.99% [532800/4848000] [109.9/1000.0] [batch_t 0.330 (0.344)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 04:28:05,169 - Train: 10.99% [532900/4848000] [109.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 04:28:38,094 - Train: 10.99% [533000/4848000] [109.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 04:29:10,941 - Train: 11.00% [533100/4848000] [110.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 04:29:43,850 - Train: 11.00% [533200/4848000] [110.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 04:30:10,147 - ==> Total time: 2 days, 5:04:08 Eta: 17 days, 21:22:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 04:30:18,817 - Train: 11.00% [533300/4848000] [110.0/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-05 04:30:51,665 - Train: 11.00% [533400/4848000] [110.0/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 04:31:24,465 - Train: 11.00% [533500/4848000] [110.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 04:31:57,325 - Train: 11.01% [533600/4848000] [110.1/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-05 04:32:30,189 - Train: 11.01% [533700/4848000] [110.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 04:33:03,031 - Train: 11.01% [533800/4848000] [110.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 04:33:35,831 - Train: 11.01% [533900/4848000] [110.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 04:34:08,668 - Train: 11.01% [534000/4848000] [110.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 04:34:41,583 - Train: 11.02% [534100/4848000] [110.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 04:35:14,370 - Train: 11.02% [534200/4848000] [110.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 04:35:47,140 - Train: 11.02% [534300/4848000] [110.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 04:36:20,782 - Train: 11.02% [534400/4848000] [110.2/1000.0] [batch_t 0.326 (0.336)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 04:36:53,632 - Train: 11.03% [534500/4848000] [110.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 04:37:26,532 - Train: 11.03% [534600/4848000] [110.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 04:37:59,399 - Train: 11.03% [534700/4848000] [110.3/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 04:38:32,267 - Train: 11.03% [534800/4848000] [110.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 04:39:05,124 - Train: 11.03% [534900/4848000] [110.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 04:39:37,939 - Train: 11.04% [535000/4848000] [110.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 04:40:11,855 - Train: 11.04% [535100/4848000] [110.4/1000.0] [batch_t 0.334 (0.339)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 04:40:44,675 - Train: 11.04% [535200/4848000] [110.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 04:41:17,468 - Train: 11.04% [535300/4848000] [110.4/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 04:41:50,310 - Train: 11.04% [535400/4848000] [110.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 04:42:23,231 - Train: 11.05% [535500/4848000] [110.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 04:42:56,081 - Train: 11.05% [535600/4848000] [110.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 04:43:29,042 - Train: 11.05% [535700/4848000] [110.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 04:44:01,869 - Train: 11.05% [535800/4848000] [110.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 04:44:35,705 - Train: 11.05% [535900/4848000] [110.5/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 04:45:08,543 - Train: 11.06% [536000/4848000] [110.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 04:45:41,385 - Train: 11.06% [536100/4848000] [110.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 04:46:14,367 - Train: 11.06% [536200/4848000] [110.6/1000.0] [batch_t 0.342 (0.330)] [data_t 0.002] [optim_t 0.340] [lr 0.005000] 2024-04-05 04:46:47,269 - Train: 11.06% [536300/4848000] [110.6/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 04:47:20,072 - Train: 11.06% [536400/4848000] [110.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 04:47:52,854 - Train: 11.07% [536500/4848000] [110.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 04:48:25,671 - Train: 11.07% [536600/4848000] [110.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 04:48:58,450 - Train: 11.07% [536700/4848000] [110.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 04:49:32,007 - Train: 11.07% [536800/4848000] [110.7/1000.0] [batch_t 0.334 (0.335)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 04:50:05,300 - Train: 11.07% [536900/4848000] [110.7/1000.0] [batch_t 0.324 (0.333)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 04:50:38,124 - Train: 11.08% [537000/4848000] [110.8/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 04:51:11,397 - Train: 11.08% [537100/4848000] [110.8/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 04:51:44,226 - Train: 11.08% [537200/4848000] [110.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 04:52:18,644 - Train: 11.08% [537300/4848000] [110.8/1000.0] [batch_t 0.328 (0.344)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 04:52:51,450 - Train: 11.08% [537400/4848000] [110.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 04:53:25,087 - Train: 11.09% [537500/4848000] [110.9/1000.0] [batch_t 0.326 (0.336)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 04:53:57,947 - Train: 11.09% [537600/4848000] [110.9/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 04:54:31,298 - Train: 11.09% [537700/4848000] [110.9/1000.0] [batch_t 0.325 (0.333)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 04:55:04,104 - Train: 11.09% [537800/4848000] [110.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 04:55:37,446 - Train: 11.10% [537900/4848000] [111.0/1000.0] [batch_t 0.326 (0.333)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 04:56:11,231 - Train: 11.10% [538000/4848000] [111.0/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 04:56:44,051 - Train: 11.10% [538100/4848000] [111.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 04:56:53,263 - ==> Total time: 2 days, 5:30:51 Eta: 17 days, 20:35:49 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 04:57:20,154 - Train: 11.10% [538200/4848000] [111.0/1000.0] [batch_t 0.328 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 04:57:53,050 - Train: 11.10% [538300/4848000] [111.0/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 04:58:27,086 - Train: 11.11% [538400/4848000] [111.1/1000.0] [batch_t 0.331 (0.340)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 04:58:59,996 - Train: 11.11% [538500/4848000] [111.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 04:59:34,206 - Train: 11.11% [538600/4848000] [111.1/1000.0] [batch_t 0.326 (0.342)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 05:00:08,377 - Train: 11.11% [538700/4848000] [111.1/1000.0] [batch_t 0.325 (0.342)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 05:00:41,187 - Train: 11.11% [538800/4848000] [111.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 05:01:14,657 - Train: 11.12% [538900/4848000] [111.2/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 05:01:47,517 - Train: 11.12% [539000/4848000] [111.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 05:02:22,338 - Train: 11.12% [539100/4848000] [111.2/1000.0] [batch_t 0.324 (0.348)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 05:02:55,122 - Train: 11.12% [539200/4848000] [111.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 05:03:29,142 - Train: 11.12% [539300/4848000] [111.2/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 05:04:02,002 - Train: 11.13% [539400/4848000] [111.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 05:04:37,352 - Train: 11.13% [539500/4848000] [111.3/1000.0] [batch_t 0.328 (0.353)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 05:05:10,140 - Train: 11.13% [539600/4848000] [111.3/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 05:05:43,081 - Train: 11.13% [539700/4848000] [111.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 05:06:17,125 - Train: 11.13% [539800/4848000] [111.3/1000.0] [batch_t 0.332 (0.340)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 05:06:49,873 - Train: 11.14% [539900/4848000] [111.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 05:07:22,753 - Train: 11.14% [540000/4848000] [111.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 05:07:55,608 - Train: 11.14% [540100/4848000] [111.4/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 05:08:28,548 - Train: 11.14% [540200/4848000] [111.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 05:09:01,444 - Train: 11.14% [540300/4848000] [111.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 05:09:34,295 - Train: 11.15% [540400/4848000] [111.5/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 05:10:07,124 - Train: 11.15% [540500/4848000] [111.5/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 05:10:39,959 - Train: 11.15% [540600/4848000] [111.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 05:11:12,934 - Train: 11.15% [540700/4848000] [111.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 05:11:45,853 - Train: 11.16% [540800/4848000] [111.6/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 05:12:18,651 - Train: 11.16% [540900/4848000] [111.6/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 05:12:51,565 - Train: 11.16% [541000/4848000] [111.6/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 05:13:24,592 - Train: 11.16% [541100/4848000] [111.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 05:13:57,387 - Train: 11.16% [541200/4848000] [111.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 05:14:30,204 - Train: 11.17% [541300/4848000] [111.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 05:15:03,082 - Train: 11.17% [541400/4848000] [111.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 05:15:35,913 - Train: 11.17% [541500/4848000] [111.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 05:16:08,796 - Train: 11.17% [541600/4848000] [111.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 05:16:41,635 - Train: 11.17% [541700/4848000] [111.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 05:17:14,466 - Train: 11.18% [541800/4848000] [111.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 05:17:47,275 - Train: 11.18% [541900/4848000] [111.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 05:18:20,163 - Train: 11.18% [542000/4848000] [111.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 05:18:53,026 - Train: 11.18% [542100/4848000] [111.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 05:19:25,827 - Train: 11.18% [542200/4848000] [111.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 05:19:58,573 - Train: 11.19% [542300/4848000] [111.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 05:20:31,416 - Train: 11.19% [542400/4848000] [111.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 05:21:04,176 - Train: 11.19% [542500/4848000] [111.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 05:21:38,731 - Train: 11.19% [542600/4848000] [111.9/1000.0] [batch_t 0.328 (0.345)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 05:22:11,517 - Train: 11.19% [542700/4848000] [111.9/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 05:22:44,343 - Train: 11.20% [542800/4848000] [112.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 05:23:17,834 - Train: 11.20% [542900/4848000] [112.0/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 05:23:42,750 - ==> Total time: 2 days, 5:57:41 Eta: 17 days, 19:50:14 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 05:23:52,477 - Train: 11.20% [543000/4848000] [112.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 05:24:25,295 - Train: 11.20% [543100/4848000] [112.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 05:24:58,276 - Train: 11.20% [543200/4848000] [112.0/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 05:25:31,541 - Train: 11.21% [543300/4848000] [112.1/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 05:26:04,366 - Train: 11.21% [543400/4848000] [112.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 05:26:38,186 - Train: 11.21% [543500/4848000] [112.1/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 05:27:10,978 - Train: 11.21% [543600/4848000] [112.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 05:27:43,797 - Train: 11.21% [543700/4848000] [112.1/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 05:28:16,673 - Train: 11.22% [543800/4848000] [112.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 05:28:49,624 - Train: 11.22% [543900/4848000] [112.2/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 05:29:22,445 - Train: 11.22% [544000/4848000] [112.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 05:29:55,276 - Train: 11.22% [544100/4848000] [112.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 05:30:28,061 - Train: 11.23% [544200/4848000] [112.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 05:31:00,910 - Train: 11.23% [544300/4848000] [112.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 05:31:33,726 - Train: 11.23% [544400/4848000] [112.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 05:32:06,495 - Train: 11.23% [544500/4848000] [112.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 05:32:39,301 - Train: 11.23% [544600/4848000] [112.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 05:33:12,123 - Train: 11.24% [544700/4848000] [112.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 05:33:44,958 - Train: 11.24% [544800/4848000] [112.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 05:34:17,695 - Train: 11.24% [544900/4848000] [112.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 05:34:50,583 - Train: 11.24% [545000/4848000] [112.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 05:35:23,321 - Train: 11.24% [545100/4848000] [112.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 05:35:56,094 - Train: 11.25% [545200/4848000] [112.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 05:36:28,963 - Train: 11.25% [545300/4848000] [112.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 05:37:01,702 - Train: 11.25% [545400/4848000] [112.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 05:37:34,564 - Train: 11.25% [545500/4848000] [112.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 05:38:07,342 - Train: 11.25% [545600/4848000] [112.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 05:38:40,150 - Train: 11.26% [545700/4848000] [112.6/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 05:39:13,023 - Train: 11.26% [545800/4848000] [112.6/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 05:39:45,899 - Train: 11.26% [545900/4848000] [112.6/1000.0] [batch_t 0.322 (0.329)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-05 05:40:18,659 - Train: 11.26% [546000/4848000] [112.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 05:40:51,436 - Train: 11.26% [546100/4848000] [112.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 05:41:24,241 - Train: 11.27% [546200/4848000] [112.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 05:41:57,023 - Train: 11.27% [546300/4848000] [112.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 05:42:30,345 - Train: 11.27% [546400/4848000] [112.7/1000.0] [batch_t 0.326 (0.333)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 05:43:03,261 - Train: 11.27% [546500/4848000] [112.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 05:43:36,044 - Train: 11.27% [546600/4848000] [112.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 05:44:08,891 - Train: 11.28% [546700/4848000] [112.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 05:44:41,752 - Train: 11.28% [546800/4848000] [112.8/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 05:45:14,608 - Train: 11.28% [546900/4848000] [112.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 05:45:47,394 - Train: 11.28% [547000/4848000] [112.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 05:46:20,198 - Train: 11.29% [547100/4848000] [112.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 05:46:52,994 - Train: 11.29% [547200/4848000] [112.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 05:47:25,821 - Train: 11.29% [547300/4848000] [112.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 05:47:58,627 - Train: 11.29% [547400/4848000] [112.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 05:48:31,431 - Train: 11.29% [547500/4848000] [112.9/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 05:49:04,195 - Train: 11.30% [547600/4848000] [113.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 05:49:38,392 - Train: 11.30% [547700/4848000] [113.0/1000.0] [batch_t 0.323 (0.342)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 05:50:11,201 - Train: 11.30% [547800/4848000] [113.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 05:50:19,061 - ==> Total time: 2 days, 6:24:17 Eta: 17 days, 19:03:15 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 05:50:46,089 - Train: 11.30% [547900/4848000] [113.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 05:51:18,954 - Train: 11.30% [548000/4848000] [113.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 05:51:51,898 - Train: 11.31% [548100/4848000] [113.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 05:52:24,696 - Train: 11.31% [548200/4848000] [113.1/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 05:52:57,593 - Train: 11.31% [548300/4848000] [113.1/1000.0] [batch_t 0.322 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 05:53:30,402 - Train: 11.31% [548400/4848000] [113.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 05:54:03,249 - Train: 11.31% [548500/4848000] [113.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 05:54:36,178 - Train: 11.32% [548600/4848000] [113.2/1000.0] [batch_t 0.336 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-05 05:55:09,089 - Train: 11.32% [548700/4848000] [113.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 05:55:41,995 - Train: 11.32% [548800/4848000] [113.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 05:56:15,189 - Train: 11.32% [548900/4848000] [113.2/1000.0] [batch_t 0.333 (0.332)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 05:56:48,096 - Train: 11.32% [549000/4848000] [113.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 05:57:21,660 - Train: 11.33% [549100/4848000] [113.3/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 05:57:54,522 - Train: 11.33% [549200/4848000] [113.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 05:58:27,354 - Train: 11.33% [549300/4848000] [113.3/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 05:59:00,211 - Train: 11.33% [549400/4848000] [113.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 05:59:33,771 - Train: 11.33% [549500/4848000] [113.3/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 06:00:06,672 - Train: 11.34% [549600/4848000] [113.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 06:00:39,597 - Train: 11.34% [549700/4848000] [113.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 06:01:13,594 - Train: 11.34% [549800/4848000] [113.4/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 06:01:46,454 - Train: 11.34% [549900/4848000] [113.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 06:02:19,590 - Train: 11.34% [550000/4848000] [113.4/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 06:02:52,517 - Train: 11.35% [550100/4848000] [113.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 06:03:25,439 - Train: 11.35% [550200/4848000] [113.5/1000.0] [batch_t 0.338 (0.329)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-05 06:03:58,313 - Train: 11.35% [550300/4848000] [113.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 06:04:31,250 - Train: 11.35% [550400/4848000] [113.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 06:05:04,147 - Train: 11.36% [550500/4848000] [113.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 06:05:37,026 - Train: 11.36% [550600/4848000] [113.6/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 06:06:09,888 - Train: 11.36% [550700/4848000] [113.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 06:06:42,676 - Train: 11.36% [550800/4848000] [113.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 06:07:15,452 - Train: 11.36% [550900/4848000] [113.6/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 06:07:48,393 - Train: 11.37% [551000/4848000] [113.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 06:08:22,312 - Train: 11.37% [551100/4848000] [113.7/1000.0] [batch_t 0.323 (0.339)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 06:08:55,104 - Train: 11.37% [551200/4848000] [113.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 06:09:27,879 - Train: 11.37% [551300/4848000] [113.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 06:10:00,727 - Train: 11.37% [551400/4848000] [113.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 06:10:33,536 - Train: 11.38% [551500/4848000] [113.8/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 06:11:06,357 - Train: 11.38% [551600/4848000] [113.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 06:11:39,155 - Train: 11.38% [551700/4848000] [113.8/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 06:12:11,920 - Train: 11.38% [551800/4848000] [113.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 06:12:44,748 - Train: 11.38% [551900/4848000] [113.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 06:13:17,676 - Train: 11.39% [552000/4848000] [113.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 06:13:50,500 - Train: 11.39% [552100/4848000] [113.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 06:14:23,712 - Train: 11.39% [552200/4848000] [113.9/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 06:14:56,481 - Train: 11.39% [552300/4848000] [113.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 06:15:29,451 - Train: 11.39% [552400/4848000] [113.9/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 06:16:02,230 - Train: 11.40% [552500/4848000] [114.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 06:16:35,998 - Train: 11.40% [552600/4848000] [114.0/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 06:16:59,597 - ==> Total time: 2 days, 6:50:58 Eta: 17 days, 18:17:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 06:17:10,890 - Train: 11.40% [552700/4848000] [114.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 06:17:43,696 - Train: 11.40% [552800/4848000] [114.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 06:18:16,549 - Train: 11.40% [552900/4848000] [114.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 06:18:49,338 - Train: 11.41% [553000/4848000] [114.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 06:19:22,121 - Train: 11.41% [553100/4848000] [114.1/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 06:19:54,891 - Train: 11.41% [553200/4848000] [114.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 06:20:27,817 - Train: 11.41% [553300/4848000] [114.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 06:21:00,779 - Train: 11.42% [553400/4848000] [114.2/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 06:21:33,678 - Train: 11.42% [553500/4848000] [114.2/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 06:22:06,647 - Train: 11.42% [553600/4848000] [114.2/1000.0] [batch_t 0.336 (0.330)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-05 06:22:39,722 - Train: 11.42% [553700/4848000] [114.2/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 06:23:12,535 - Train: 11.42% [553800/4848000] [114.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 06:23:45,415 - Train: 11.43% [553900/4848000] [114.3/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 06:24:18,322 - Train: 11.43% [554000/4848000] [114.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 06:24:51,224 - Train: 11.43% [554100/4848000] [114.3/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 06:25:24,138 - Train: 11.43% [554200/4848000] [114.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 06:25:57,007 - Train: 11.43% [554300/4848000] [114.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 06:26:29,958 - Train: 11.44% [554400/4848000] [114.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 06:27:02,882 - Train: 11.44% [554500/4848000] [114.4/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 06:27:35,771 - Train: 11.44% [554600/4848000] [114.4/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 06:28:08,665 - Train: 11.44% [554700/4848000] [114.4/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 06:28:41,540 - Train: 11.44% [554800/4848000] [114.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 06:29:14,492 - Train: 11.45% [554900/4848000] [114.5/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 06:29:47,386 - Train: 11.45% [555000/4848000] [114.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 06:30:20,274 - Train: 11.45% [555100/4848000] [114.5/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 06:30:53,313 - Train: 11.45% [555200/4848000] [114.5/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 06:31:26,180 - Train: 11.45% [555300/4848000] [114.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 06:31:58,998 - Train: 11.46% [555400/4848000] [114.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 06:32:31,828 - Train: 11.46% [555500/4848000] [114.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 06:33:04,653 - Train: 11.46% [555600/4848000] [114.6/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 06:33:37,552 - Train: 11.46% [555700/4848000] [114.6/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 06:34:10,479 - Train: 11.46% [555800/4848000] [114.6/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 06:34:43,305 - Train: 11.47% [555900/4848000] [114.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 06:35:16,147 - Train: 11.47% [556000/4848000] [114.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 06:35:48,965 - Train: 11.47% [556100/4848000] [114.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 06:36:21,799 - Train: 11.47% [556200/4848000] [114.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 06:36:54,676 - Train: 11.47% [556300/4848000] [114.7/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 06:37:27,497 - Train: 11.48% [556400/4848000] [114.8/1000.0] [batch_t 0.336 (0.328)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-05 06:38:00,315 - Train: 11.48% [556500/4848000] [114.8/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 06:38:33,299 - Train: 11.48% [556600/4848000] [114.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 06:39:06,097 - Train: 11.48% [556700/4848000] [114.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 06:39:38,972 - Train: 11.49% [556800/4848000] [114.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 06:40:11,825 - Train: 11.49% [556900/4848000] [114.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 06:40:44,683 - Train: 11.49% [557000/4848000] [114.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 06:41:17,599 - Train: 11.49% [557100/4848000] [114.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 06:41:50,499 - Train: 11.49% [557200/4848000] [114.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 06:42:23,319 - Train: 11.50% [557300/4848000] [115.0/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 06:42:56,098 - Train: 11.50% [557400/4848000] [115.0/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 06:43:28,897 - Train: 11.50% [557500/4848000] [115.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 06:43:35,463 - ==> Total time: 2 days, 7:17:33 Eta: 17 days, 17:30:50 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 06:44:03,704 - Train: 11.50% [557600/4848000] [115.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 06:44:36,564 - Train: 11.50% [557700/4848000] [115.0/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-05 06:45:09,497 - Train: 11.51% [557800/4848000] [115.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 06:45:42,484 - Train: 11.51% [557900/4848000] [115.1/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 06:46:15,278 - Train: 11.51% [558000/4848000] [115.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 06:46:48,085 - Train: 11.51% [558100/4848000] [115.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 06:47:20,857 - Train: 11.51% [558200/4848000] [115.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 06:47:53,637 - Train: 11.52% [558300/4848000] [115.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 06:48:26,441 - Train: 11.52% [558400/4848000] [115.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 06:48:59,216 - Train: 11.52% [558500/4848000] [115.2/1000.0] [batch_t 0.320 (0.328)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-05 06:49:32,696 - Train: 11.52% [558600/4848000] [115.2/1000.0] [batch_t 0.325 (0.335)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 06:50:05,466 - Train: 11.52% [558700/4848000] [115.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 06:50:38,275 - Train: 11.53% [558800/4848000] [115.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 06:51:11,899 - Train: 11.53% [558900/4848000] [115.3/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 06:51:44,709 - Train: 11.53% [559000/4848000] [115.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 06:52:18,241 - Train: 11.53% [559100/4848000] [115.3/1000.0] [batch_t 0.331 (0.335)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 06:52:51,075 - Train: 11.53% [559200/4848000] [115.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 06:53:24,012 - Train: 11.54% [559300/4848000] [115.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 06:53:57,002 - Train: 11.54% [559400/4848000] [115.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 06:54:29,858 - Train: 11.54% [559500/4848000] [115.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 06:55:02,724 - Train: 11.54% [559600/4848000] [115.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 06:55:35,612 - Train: 11.54% [559700/4848000] [115.4/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 06:56:09,714 - Train: 11.55% [559800/4848000] [115.5/1000.0] [batch_t 0.332 (0.341)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 06:56:42,639 - Train: 11.55% [559900/4848000] [115.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 06:57:15,557 - Train: 11.55% [560000/4848000] [115.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 06:57:48,401 - Train: 11.55% [560100/4848000] [115.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 06:58:21,298 - Train: 11.56% [560200/4848000] [115.6/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 06:58:54,155 - Train: 11.56% [560300/4848000] [115.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 06:59:27,062 - Train: 11.56% [560400/4848000] [115.6/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 06:59:59,890 - Train: 11.56% [560500/4848000] [115.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 07:00:32,725 - Train: 11.56% [560600/4848000] [115.6/1000.0] [batch_t 0.339 (0.328)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-05 07:01:05,544 - Train: 11.57% [560700/4848000] [115.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 07:01:38,553 - Train: 11.57% [560800/4848000] [115.7/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 07:02:12,248 - Train: 11.57% [560900/4848000] [115.7/1000.0] [batch_t 0.335 (0.337)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-05 07:02:45,133 - Train: 11.57% [561000/4848000] [115.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 07:03:18,070 - Train: 11.57% [561100/4848000] [115.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 07:03:50,890 - Train: 11.58% [561200/4848000] [115.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 07:04:23,813 - Train: 11.58% [561300/4848000] [115.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 07:04:56,656 - Train: 11.58% [561400/4848000] [115.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 07:05:30,798 - Train: 11.58% [561500/4848000] [115.8/1000.0] [batch_t 0.326 (0.341)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 07:06:03,556 - Train: 11.58% [561600/4848000] [115.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 07:06:36,336 - Train: 11.59% [561700/4848000] [115.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 07:07:09,136 - Train: 11.59% [561800/4848000] [115.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 07:07:41,989 - Train: 11.59% [561900/4848000] [115.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 07:08:14,792 - Train: 11.59% [562000/4848000] [115.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 07:08:47,624 - Train: 11.59% [562100/4848000] [115.9/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 07:09:20,544 - Train: 11.60% [562200/4848000] [116.0/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 07:09:53,411 - Train: 11.60% [562300/4848000] [116.0/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 07:10:15,726 - ==> Total time: 2 days, 7:44:14 Eta: 17 days, 16:45:23 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 07:10:28,305 - Train: 11.60% [562400/4848000] [116.0/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 07:11:01,140 - Train: 11.60% [562500/4848000] [116.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 07:11:33,929 - Train: 11.60% [562600/4848000] [116.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 07:12:06,797 - Train: 11.61% [562700/4848000] [116.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 07:12:39,558 - Train: 11.61% [562800/4848000] [116.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 07:13:12,315 - Train: 11.61% [562900/4848000] [116.1/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 07:13:45,145 - Train: 11.61% [563000/4848000] [116.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 07:14:17,930 - Train: 11.62% [563100/4848000] [116.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 07:14:50,730 - Train: 11.62% [563200/4848000] [116.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 07:15:23,627 - Train: 11.62% [563300/4848000] [116.2/1000.0] [batch_t 0.321 (0.329)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-05 07:15:56,542 - Train: 11.62% [563400/4848000] [116.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 07:16:29,587 - Train: 11.62% [563500/4848000] [116.2/1000.0] [batch_t 0.484 (0.330)] [data_t 0.002] [optim_t 0.482] [lr 0.005000] 2024-04-05 07:17:02,446 - Train: 11.63% [563600/4848000] [116.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 07:17:35,267 - Train: 11.63% [563700/4848000] [116.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 07:18:08,236 - Train: 11.63% [563800/4848000] [116.3/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 07:18:41,141 - Train: 11.63% [563900/4848000] [116.3/1000.0] [batch_t 0.337 (0.329)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-05 07:19:14,009 - Train: 11.63% [564000/4848000] [116.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 07:19:46,891 - Train: 11.64% [564100/4848000] [116.4/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 07:20:19,709 - Train: 11.64% [564200/4848000] [116.4/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 07:20:52,612 - Train: 11.64% [564300/4848000] [116.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 07:21:25,498 - Train: 11.64% [564400/4848000] [116.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 07:21:58,370 - Train: 11.64% [564500/4848000] [116.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 07:22:31,227 - Train: 11.65% [564600/4848000] [116.5/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 07:23:04,185 - Train: 11.65% [564700/4848000] [116.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 07:23:37,138 - Train: 11.65% [564800/4848000] [116.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 07:24:10,083 - Train: 11.65% [564900/4848000] [116.5/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 07:24:43,074 - Train: 11.65% [565000/4848000] [116.5/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 07:25:15,870 - Train: 11.66% [565100/4848000] [116.6/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 07:25:48,656 - Train: 11.66% [565200/4848000] [116.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 07:26:21,491 - Train: 11.66% [565300/4848000] [116.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 07:26:54,264 - Train: 11.66% [565400/4848000] [116.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 07:27:27,146 - Train: 11.66% [565500/4848000] [116.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 07:28:00,036 - Train: 11.67% [565600/4848000] [116.7/1000.0] [batch_t 0.336 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-05 07:28:34,433 - Train: 11.67% [565700/4848000] [116.7/1000.0] [batch_t 0.332 (0.344)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 07:29:07,346 - Train: 11.67% [565800/4848000] [116.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 07:29:40,201 - Train: 11.67% [565900/4848000] [116.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 07:30:13,088 - Train: 11.67% [566000/4848000] [116.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 07:30:45,974 - Train: 11.68% [566100/4848000] [116.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 07:31:18,760 - Train: 11.68% [566200/4848000] [116.8/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 07:31:51,557 - Train: 11.68% [566300/4848000] [116.8/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 07:32:25,043 - Train: 11.68% [566400/4848000] [116.8/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 07:32:57,927 - Train: 11.69% [566500/4848000] [116.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 07:33:30,725 - Train: 11.69% [566600/4848000] [116.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 07:34:03,596 - Train: 11.69% [566700/4848000] [116.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 07:34:37,157 - Train: 11.69% [566800/4848000] [116.9/1000.0] [batch_t 0.331 (0.336)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 07:35:10,533 - Train: 11.69% [566900/4848000] [116.9/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 07:35:43,470 - Train: 11.70% [567000/4848000] [117.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 07:36:17,548 - Train: 11.70% [567100/4848000] [117.0/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 07:36:50,355 - Train: 11.70% [567200/4848000] [117.0/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 07:36:55,636 - ==> Total time: 2 days, 8:10:54 Eta: 17 days, 16:00:13 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 07:37:26,051 - Train: 11.70% [567300/4848000] [117.0/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 07:37:58,818 - Train: 11.70% [567400/4848000] [117.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 07:38:32,005 - Train: 11.71% [567500/4848000] [117.1/1000.0] [batch_t 0.323 (0.332)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 07:39:04,849 - Train: 11.71% [567600/4848000] [117.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 07:39:37,794 - Train: 11.71% [567700/4848000] [117.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 07:40:11,528 - Train: 11.71% [567800/4848000] [117.1/1000.0] [batch_t 0.326 (0.337)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 07:40:44,399 - Train: 11.71% [567900/4848000] [117.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 07:41:17,477 - Train: 11.72% [568000/4848000] [117.2/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 07:41:50,426 - Train: 11.72% [568100/4848000] [117.2/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 07:42:24,488 - Train: 11.72% [568200/4848000] [117.2/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 07:42:57,362 - Train: 11.72% [568300/4848000] [117.2/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 07:43:31,054 - Train: 11.72% [568400/4848000] [117.2/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 07:44:03,842 - Train: 11.73% [568500/4848000] [117.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 07:44:36,911 - Train: 11.73% [568600/4848000] [117.3/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 07:45:09,703 - Train: 11.73% [568700/4848000] [117.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 07:45:42,599 - Train: 11.73% [568800/4848000] [117.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 07:46:16,419 - Train: 11.73% [568900/4848000] [117.3/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 07:46:49,240 - Train: 11.74% [569000/4848000] [117.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 07:47:22,098 - Train: 11.74% [569100/4848000] [117.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 07:47:55,016 - Train: 11.74% [569200/4848000] [117.4/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 07:48:27,807 - Train: 11.74% [569300/4848000] [117.4/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 07:49:00,556 - Train: 11.75% [569400/4848000] [117.5/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 07:49:33,500 - Train: 11.75% [569500/4848000] [117.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 07:50:06,289 - Train: 11.75% [569600/4848000] [117.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 07:50:39,118 - Train: 11.75% [569700/4848000] [117.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 07:51:11,917 - Train: 11.75% [569800/4848000] [117.5/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 07:51:44,694 - Train: 11.76% [569900/4848000] [117.6/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 07:52:17,469 - Train: 11.76% [570000/4848000] [117.6/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 07:52:50,243 - Train: 11.76% [570100/4848000] [117.6/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 07:53:23,090 - Train: 11.76% [570200/4848000] [117.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 07:53:55,973 - Train: 11.76% [570300/4848000] [117.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 07:54:29,846 - Train: 11.77% [570400/4848000] [117.7/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 07:55:02,594 - Train: 11.77% [570500/4848000] [117.7/1000.0] [batch_t 0.320 (0.327)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-05 07:55:35,589 - Train: 11.77% [570600/4848000] [117.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 07:56:08,508 - Train: 11.77% [570700/4848000] [117.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 07:56:41,496 - Train: 11.77% [570800/4848000] [117.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 07:57:14,815 - Train: 11.78% [570900/4848000] [117.8/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 07:57:47,743 - Train: 11.78% [571000/4848000] [117.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 07:58:22,143 - Train: 11.78% [571100/4848000] [117.8/1000.0] [batch_t 0.324 (0.344)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 07:58:55,072 - Train: 11.78% [571200/4848000] [117.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 07:59:27,951 - Train: 11.78% [571300/4848000] [117.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 08:00:00,847 - Train: 11.79% [571400/4848000] [117.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 08:00:33,764 - Train: 11.79% [571500/4848000] [117.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 08:01:06,648 - Train: 11.79% [571600/4848000] [117.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 08:01:39,440 - Train: 11.79% [571700/4848000] [117.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 08:02:12,265 - Train: 11.79% [571800/4848000] [117.9/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 08:02:45,143 - Train: 11.80% [571900/4848000] [118.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 08:03:18,099 - Train: 11.80% [572000/4848000] [118.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 08:03:39,066 - ==> Total time: 2 days, 8:37:37 Eta: 17 days, 15:15:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 08:03:52,991 - Train: 11.80% [572100/4848000] [118.0/1000.0] [batch_t 0.330 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 08:04:25,776 - Train: 11.80% [572200/4848000] [118.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 08:04:58,599 - Train: 11.80% [572300/4848000] [118.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 08:05:31,407 - Train: 11.81% [572400/4848000] [118.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 08:06:04,184 - Train: 11.81% [572500/4848000] [118.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 08:06:36,985 - Train: 11.81% [572600/4848000] [118.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 08:07:09,754 - Train: 11.81% [572700/4848000] [118.1/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 08:07:42,585 - Train: 11.82% [572800/4848000] [118.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 08:08:15,467 - Train: 11.82% [572900/4848000] [118.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 08:08:48,297 - Train: 11.82% [573000/4848000] [118.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 08:09:21,200 - Train: 11.82% [573100/4848000] [118.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 08:09:54,075 - Train: 11.82% [573200/4848000] [118.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 08:10:26,959 - Train: 11.83% [573300/4848000] [118.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 08:10:59,877 - Train: 11.83% [573400/4848000] [118.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 08:11:33,422 - Train: 11.83% [573500/4848000] [118.3/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 08:12:06,338 - Train: 11.83% [573600/4848000] [118.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 08:12:39,177 - Train: 11.83% [573700/4848000] [118.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 08:13:12,078 - Train: 11.84% [573800/4848000] [118.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 08:13:44,948 - Train: 11.84% [573900/4848000] [118.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 08:14:17,888 - Train: 11.84% [574000/4848000] [118.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 08:14:50,736 - Train: 11.84% [574100/4848000] [118.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 08:15:23,711 - Train: 11.84% [574200/4848000] [118.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 08:15:56,603 - Train: 11.85% [574300/4848000] [118.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 08:16:29,481 - Train: 11.85% [574400/4848000] [118.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 08:17:02,346 - Train: 11.85% [574500/4848000] [118.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 08:17:35,788 - Train: 11.85% [574600/4848000] [118.5/1000.0] [batch_t 0.326 (0.334)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 08:18:09,633 - Train: 11.85% [574700/4848000] [118.5/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 08:18:42,640 - Train: 11.86% [574800/4848000] [118.6/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 08:19:16,454 - Train: 11.86% [574900/4848000] [118.6/1000.0] [batch_t 0.325 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 08:19:49,335 - Train: 11.86% [575000/4848000] [118.6/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 08:20:23,621 - Train: 11.86% [575100/4848000] [118.6/1000.0] [batch_t 0.327 (0.343)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 08:20:56,461 - Train: 11.86% [575200/4848000] [118.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 08:21:29,795 - Train: 11.87% [575300/4848000] [118.7/1000.0] [batch_t 0.336 (0.333)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-05 08:22:02,639 - Train: 11.87% [575400/4848000] [118.7/1000.0] [batch_t 0.338 (0.328)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-05 08:22:36,479 - Train: 11.87% [575500/4848000] [118.7/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 08:23:10,540 - Train: 11.87% [575600/4848000] [118.7/1000.0] [batch_t 0.336 (0.340)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-05 08:23:43,432 - Train: 11.88% [575700/4848000] [118.8/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 08:24:16,305 - Train: 11.88% [575800/4848000] [118.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 08:24:49,055 - Train: 11.88% [575900/4848000] [118.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 08:25:21,819 - Train: 11.88% [576000/4848000] [118.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 08:25:54,613 - Train: 11.88% [576100/4848000] [118.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 08:26:27,617 - Train: 11.89% [576200/4848000] [118.9/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 08:27:00,513 - Train: 11.89% [576300/4848000] [118.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 08:27:35,185 - Train: 11.89% [576400/4848000] [118.9/1000.0] [batch_t 0.329 (0.347)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 08:28:08,132 - Train: 11.89% [576500/4848000] [118.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 08:28:41,004 - Train: 11.89% [576600/4848000] [118.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 08:29:13,857 - Train: 11.90% [576700/4848000] [119.0/1000.0] [batch_t 0.339 (0.328)] [data_t 0.002] [optim_t 0.337] [lr 0.005000] 2024-04-05 08:29:46,751 - Train: 11.90% [576800/4848000] [119.0/1000.0] [batch_t 0.338 (0.329)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-05 08:30:21,054 - Train: 11.90% [576900/4848000] [119.0/1000.0] [batch_t 0.331 (0.343)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 08:30:25,032 - ==> Total time: 2 days, 9:04:23 Eta: 17 days, 14:32:00 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 08:30:56,177 - Train: 11.90% [577000/4848000] [119.0/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 08:31:29,056 - Train: 11.90% [577100/4848000] [119.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 08:32:01,899 - Train: 11.91% [577200/4848000] [119.1/1000.0] [batch_t 0.337 (0.328)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-05 08:32:34,681 - Train: 11.91% [577300/4848000] [119.1/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-05 08:33:07,518 - Train: 11.91% [577400/4848000] [119.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 08:33:40,327 - Train: 11.91% [577500/4848000] [119.1/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 08:34:13,339 - Train: 11.91% [577600/4848000] [119.1/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 08:34:46,140 - Train: 11.92% [577700/4848000] [119.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 08:35:18,973 - Train: 11.92% [577800/4848000] [119.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 08:35:51,794 - Train: 11.92% [577900/4848000] [119.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 08:36:24,653 - Train: 11.92% [578000/4848000] [119.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 08:36:57,505 - Train: 11.92% [578100/4848000] [119.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 08:37:31,186 - Train: 11.93% [578200/4848000] [119.3/1000.0] [batch_t 0.323 (0.337)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 08:38:04,034 - Train: 11.93% [578300/4848000] [119.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 08:38:36,898 - Train: 11.93% [578400/4848000] [119.3/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-05 08:39:10,433 - Train: 11.93% [578500/4848000] [119.3/1000.0] [batch_t 0.325 (0.335)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 08:39:43,284 - Train: 11.93% [578600/4848000] [119.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 08:40:16,089 - Train: 11.94% [578700/4848000] [119.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 08:40:48,977 - Train: 11.94% [578800/4848000] [119.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 08:41:21,809 - Train: 11.94% [578900/4848000] [119.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-05 08:41:54,739 - Train: 11.94% [579000/4848000] [119.4/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 08:42:27,541 - Train: 11.95% [579100/4848000] [119.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 08:43:00,359 - Train: 11.95% [579200/4848000] [119.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 08:43:33,188 - Train: 11.95% [579300/4848000] [119.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 08:44:06,000 - Train: 11.95% [579400/4848000] [119.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 08:44:38,831 - Train: 11.95% [579500/4848000] [119.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 08:45:11,652 - Train: 11.96% [579600/4848000] [119.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 08:45:44,518 - Train: 11.96% [579700/4848000] [119.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 08:46:17,891 - Train: 11.96% [579800/4848000] [119.6/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 08:46:50,706 - Train: 11.96% [579900/4848000] [119.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 08:47:23,499 - Train: 11.96% [580000/4848000] [119.6/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 08:47:56,319 - Train: 11.97% [580100/4848000] [119.7/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 08:48:30,302 - Train: 11.97% [580200/4848000] [119.7/1000.0] [batch_t 0.324 (0.340)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 08:49:03,115 - Train: 11.97% [580300/4848000] [119.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 08:49:36,570 - Train: 11.97% [580400/4848000] [119.7/1000.0] [batch_t 0.326 (0.334)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 08:50:09,348 - Train: 11.97% [580500/4848000] [119.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 08:50:42,150 - Train: 11.98% [580600/4848000] [119.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 08:51:15,191 - Train: 11.98% [580700/4848000] [119.8/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 08:51:47,980 - Train: 11.98% [580800/4848000] [119.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 08:52:21,056 - Train: 11.98% [580900/4848000] [119.8/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 08:52:53,887 - Train: 11.98% [581000/4848000] [119.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 08:53:26,733 - Train: 11.99% [581100/4848000] [119.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 08:53:59,581 - Train: 11.99% [581200/4848000] [119.9/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 08:54:32,960 - Train: 11.99% [581300/4848000] [119.9/1000.0] [batch_t 0.326 (0.334)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 08:55:06,859 - Train: 11.99% [581400/4848000] [119.9/1000.0] [batch_t 0.332 (0.339)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 08:55:39,697 - Train: 11.99% [581500/4848000] [119.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 08:56:12,549 - Train: 12.00% [581600/4848000] [120.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 08:56:45,418 - Train: 12.00% [581700/4848000] [120.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 08:57:05,141 - ==> Total time: 2 days, 9:31:03 Eta: 17 days, 13:47:46 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 08:57:20,422 - Train: 12.00% [581800/4848000] [120.0/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 08:57:53,234 - Train: 12.00% [581900/4848000] [120.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 08:58:26,819 - Train: 12.00% [582000/4848000] [120.0/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 08:58:59,674 - Train: 12.01% [582100/4848000] [120.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 08:59:33,707 - Train: 12.01% [582200/4848000] [120.1/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 09:00:06,596 - Train: 12.01% [582300/4848000] [120.1/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 09:00:39,439 - Train: 12.01% [582400/4848000] [120.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 09:01:12,616 - Train: 12.02% [582500/4848000] [120.2/1000.0] [batch_t 0.326 (0.332)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 09:01:45,452 - Train: 12.02% [582600/4848000] [120.2/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 09:02:18,289 - Train: 12.02% [582700/4848000] [120.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 09:02:51,144 - Train: 12.02% [582800/4848000] [120.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 09:03:23,962 - Train: 12.02% [582900/4848000] [120.2/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 09:03:56,842 - Train: 12.03% [583000/4848000] [120.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 09:04:30,733 - Train: 12.03% [583100/4848000] [120.3/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 09:05:03,532 - Train: 12.03% [583200/4848000] [120.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 09:05:36,472 - Train: 12.03% [583300/4848000] [120.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 09:06:09,295 - Train: 12.03% [583400/4848000] [120.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 09:06:42,148 - Train: 12.04% [583500/4848000] [120.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 09:07:15,008 - Train: 12.04% [583600/4848000] [120.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 09:07:47,857 - Train: 12.04% [583700/4848000] [120.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 09:08:20,761 - Train: 12.04% [583800/4848000] [120.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 09:08:53,594 - Train: 12.04% [583900/4848000] [120.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 09:09:26,559 - Train: 12.05% [584000/4848000] [120.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 09:09:59,425 - Train: 12.05% [584100/4848000] [120.5/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 09:10:32,308 - Train: 12.05% [584200/4848000] [120.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 09:11:05,179 - Train: 12.05% [584300/4848000] [120.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 09:11:37,988 - Train: 12.05% [584400/4848000] [120.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 09:12:13,165 - Train: 12.06% [584500/4848000] [120.6/1000.0] [batch_t 0.326 (0.352)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 09:12:45,973 - Train: 12.06% [584600/4848000] [120.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 09:13:19,785 - Train: 12.06% [584700/4848000] [120.6/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 09:13:52,621 - Train: 12.06% [584800/4848000] [120.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 09:14:26,173 - Train: 12.06% [584900/4848000] [120.6/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 09:14:58,934 - Train: 12.07% [585000/4848000] [120.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 09:15:31,793 - Train: 12.07% [585100/4848000] [120.7/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 09:16:04,620 - Train: 12.07% [585200/4848000] [120.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 09:16:37,437 - Train: 12.07% [585300/4848000] [120.7/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 09:17:10,241 - Train: 12.08% [585400/4848000] [120.8/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-05 09:17:43,096 - Train: 12.08% [585500/4848000] [120.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 09:18:15,888 - Train: 12.08% [585600/4848000] [120.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 09:18:48,735 - Train: 12.08% [585700/4848000] [120.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 09:19:21,616 - Train: 12.08% [585800/4848000] [120.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 09:19:54,457 - Train: 12.09% [585900/4848000] [120.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 09:20:27,311 - Train: 12.09% [586000/4848000] [120.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 09:21:00,298 - Train: 12.09% [586100/4848000] [120.9/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 09:21:33,139 - Train: 12.09% [586200/4848000] [120.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 09:22:05,998 - Train: 12.09% [586300/4848000] [120.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 09:22:38,799 - Train: 12.10% [586400/4848000] [121.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 09:23:12,340 - Train: 12.10% [586500/4848000] [121.0/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 09:23:45,280 - Train: 12.10% [586600/4848000] [121.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 09:23:47,908 - ==> Total time: 2 days, 9:57:46 Eta: 17 days, 13:04:09 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 09:24:20,849 - Train: 12.10% [586700/4848000] [121.0/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 09:24:53,642 - Train: 12.10% [586800/4848000] [121.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 09:25:26,457 - Train: 12.11% [586900/4848000] [121.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 09:25:59,311 - Train: 12.11% [587000/4848000] [121.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 09:26:32,185 - Train: 12.11% [587100/4848000] [121.1/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 09:27:05,189 - Train: 12.11% [587200/4848000] [121.1/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 09:27:38,063 - Train: 12.11% [587300/4848000] [121.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 09:28:11,129 - Train: 12.12% [587400/4848000] [121.2/1000.0] [batch_t 0.513 (0.331)] [data_t 0.002] [optim_t 0.511] [lr 0.005000] 2024-04-05 09:28:44,014 - Train: 12.12% [587500/4848000] [121.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 09:29:16,917 - Train: 12.12% [587600/4848000] [121.2/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 09:29:49,852 - Train: 12.12% [587700/4848000] [121.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 09:30:23,878 - Train: 12.12% [587800/4848000] [121.2/1000.0] [batch_t 0.333 (0.340)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 09:30:56,753 - Train: 12.13% [587900/4848000] [121.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 09:31:30,807 - Train: 12.13% [588000/4848000] [121.3/1000.0] [batch_t 0.336 (0.340)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-05 09:32:03,712 - Train: 12.13% [588100/4848000] [121.3/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 09:32:36,549 - Train: 12.13% [588200/4848000] [121.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 09:33:09,422 - Train: 12.13% [588300/4848000] [121.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 09:33:42,265 - Train: 12.14% [588400/4848000] [121.4/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 09:34:15,055 - Train: 12.14% [588500/4848000] [121.4/1000.0] [batch_t 0.337 (0.328)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-05 09:34:47,835 - Train: 12.14% [588600/4848000] [121.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 09:35:20,711 - Train: 12.14% [588700/4848000] [121.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 09:35:53,521 - Train: 12.15% [588800/4848000] [121.5/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 09:36:26,520 - Train: 12.15% [588900/4848000] [121.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 09:36:59,478 - Train: 12.15% [589000/4848000] [121.5/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 09:37:32,385 - Train: 12.15% [589100/4848000] [121.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 09:38:05,248 - Train: 12.15% [589200/4848000] [121.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 09:38:38,141 - Train: 12.16% [589300/4848000] [121.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 09:39:10,944 - Train: 12.16% [589400/4848000] [121.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 09:39:43,840 - Train: 12.16% [589500/4848000] [121.6/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-05 09:40:16,729 - Train: 12.16% [589600/4848000] [121.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 09:40:49,606 - Train: 12.16% [589700/4848000] [121.6/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 09:41:23,130 - Train: 12.17% [589800/4848000] [121.7/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 09:41:56,014 - Train: 12.17% [589900/4848000] [121.7/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 09:42:28,919 - Train: 12.17% [590000/4848000] [121.7/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 09:43:01,800 - Train: 12.17% [590100/4848000] [121.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 09:43:35,580 - Train: 12.17% [590200/4848000] [121.7/1000.0] [batch_t 0.332 (0.338)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 09:44:08,543 - Train: 12.18% [590300/4848000] [121.8/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 09:44:41,400 - Train: 12.18% [590400/4848000] [121.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 09:45:15,307 - Train: 12.18% [590500/4848000] [121.8/1000.0] [batch_t 0.323 (0.339)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 09:45:48,096 - Train: 12.18% [590600/4848000] [121.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 09:46:20,879 - Train: 12.18% [590700/4848000] [121.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 09:46:53,683 - Train: 12.19% [590800/4848000] [121.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 09:47:26,486 - Train: 12.19% [590900/4848000] [121.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 09:47:59,312 - Train: 12.19% [591000/4848000] [121.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 09:48:32,103 - Train: 12.19% [591100/4848000] [121.9/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 09:49:05,021 - Train: 12.19% [591200/4848000] [121.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 09:49:37,859 - Train: 12.20% [591300/4848000] [122.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 09:50:13,105 - Train: 12.20% [591400/4848000] [122.0/1000.0] [batch_t 0.327 (0.352)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 09:50:31,485 - ==> Total time: 2 days, 10:24:29 Eta: 17 days, 12:20:54 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 09:50:48,239 - Train: 12.20% [591500/4848000] [122.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 09:51:21,189 - Train: 12.20% [591600/4848000] [122.0/1000.0] [batch_t 0.471 (0.329)] [data_t 0.002] [optim_t 0.469] [lr 0.005000] 2024-04-05 09:51:53,994 - Train: 12.21% [591700/4848000] [122.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 09:52:27,401 - Train: 12.21% [591800/4848000] [122.1/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 09:53:00,221 - Train: 12.21% [591900/4848000] [122.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 09:53:33,172 - Train: 12.21% [592000/4848000] [122.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 09:54:08,473 - Train: 12.21% [592100/4848000] [122.1/1000.0] [batch_t 0.325 (0.353)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 09:54:41,338 - Train: 12.22% [592200/4848000] [122.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 09:55:18,761 - Train: 12.22% [592300/4848000] [122.2/1000.0] [batch_t 0.328 (0.374)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 09:55:51,737 - Train: 12.22% [592400/4848000] [122.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 09:56:29,240 - Train: 12.22% [592500/4848000] [122.2/1000.0] [batch_t 0.323 (0.375)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 09:57:02,137 - Train: 12.22% [592600/4848000] [122.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 09:57:36,314 - Train: 12.23% [592700/4848000] [122.3/1000.0] [batch_t 0.323 (0.342)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 09:58:10,312 - Train: 12.23% [592800/4848000] [122.3/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 09:58:43,218 - Train: 12.23% [592900/4848000] [122.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 10:00:15,422 - Train: 12.23% [593000/4848000] [122.3/1000.0] [batch_t 0.327 (0.922)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 10:00:48,448 - Train: 12.23% [593100/4848000] [122.3/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 10:01:21,356 - Train: 12.24% [593200/4848000] [122.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 10:01:54,239 - Train: 12.24% [593300/4848000] [122.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 10:02:29,603 - Train: 12.24% [593400/4848000] [122.4/1000.0] [batch_t 0.328 (0.354)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 10:03:02,499 - Train: 12.24% [593500/4848000] [122.4/1000.0] [batch_t 0.337 (0.329)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-05 10:03:37,386 - Train: 12.24% [593600/4848000] [122.4/1000.0] [batch_t 0.324 (0.349)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 10:04:13,112 - Train: 12.25% [593700/4848000] [122.5/1000.0] [batch_t 0.328 (0.357)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 10:04:46,071 - Train: 12.25% [593800/4848000] [122.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 10:05:20,218 - Train: 12.25% [593900/4848000] [122.5/1000.0] [batch_t 0.324 (0.341)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 10:05:52,999 - Train: 12.25% [594000/4848000] [122.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 10:06:27,242 - Train: 12.25% [594100/4848000] [122.5/1000.0] [batch_t 0.326 (0.342)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 10:07:00,096 - Train: 12.26% [594200/4848000] [122.6/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 10:07:44,586 - Train: 12.26% [594300/4848000] [122.6/1000.0] [batch_t 0.326 (0.445)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 10:08:18,003 - Train: 12.26% [594400/4848000] [122.6/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 10:08:51,046 - Train: 12.26% [594500/4848000] [122.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 10:09:23,903 - Train: 12.26% [594600/4848000] [122.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 10:09:56,818 - Train: 12.27% [594700/4848000] [122.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 10:10:35,357 - Train: 12.27% [594800/4848000] [122.7/1000.0] [batch_t 0.330 (0.385)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 10:11:09,140 - Train: 12.27% [594900/4848000] [122.7/1000.0] [batch_t 0.334 (0.338)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 10:11:42,076 - Train: 12.27% [595000/4848000] [122.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 10:12:15,032 - Train: 12.28% [595100/4848000] [122.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 10:12:47,930 - Train: 12.28% [595200/4848000] [122.8/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 10:13:20,909 - Train: 12.28% [595300/4848000] [122.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 10:13:53,831 - Train: 12.28% [595400/4848000] [122.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 10:14:26,825 - Train: 12.28% [595500/4848000] [122.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 10:14:59,788 - Train: 12.29% [595600/4848000] [122.9/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 10:15:32,807 - Train: 12.29% [595700/4848000] [122.9/1000.0] [batch_t 0.323 (0.330)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 10:16:05,844 - Train: 12.29% [595800/4848000] [122.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 10:16:38,765 - Train: 12.29% [595900/4848000] [122.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 10:17:11,866 - Train: 12.29% [596000/4848000] [122.9/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 10:17:44,818 - Train: 12.30% [596100/4848000] [123.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 10:18:17,748 - Train: 12.30% [596200/4848000] [123.0/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 10:18:50,802 - Train: 12.30% [596300/4848000] [123.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 10:18:52,133 - ==> Total time: 2 days, 10:52:50 Eta: 17 days, 11:49:27 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 10:19:25,940 - Train: 12.30% [596400/4848000] [123.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 10:19:58,837 - Train: 12.30% [596500/4848000] [123.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 10:20:31,718 - Train: 12.31% [596600/4848000] [123.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 10:21:04,567 - Train: 12.31% [596700/4848000] [123.1/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 10:21:37,391 - Train: 12.31% [596800/4848000] [123.1/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 10:22:10,236 - Train: 12.31% [596900/4848000] [123.1/1000.0] [batch_t 0.338 (0.328)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-05 10:22:43,049 - Train: 12.31% [597000/4848000] [123.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 10:23:15,880 - Train: 12.32% [597100/4848000] [123.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 10:23:48,721 - Train: 12.32% [597200/4848000] [123.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 10:24:21,690 - Train: 12.32% [597300/4848000] [123.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 10:24:54,596 - Train: 12.32% [597400/4848000] [123.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 10:25:27,427 - Train: 12.32% [597500/4848000] [123.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 10:26:00,313 - Train: 12.33% [597600/4848000] [123.3/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 10:26:33,168 - Train: 12.33% [597700/4848000] [123.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 10:27:06,010 - Train: 12.33% [597800/4848000] [123.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 10:27:38,858 - Train: 12.33% [597900/4848000] [123.3/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 10:28:11,713 - Train: 12.33% [598000/4848000] [123.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 10:28:44,642 - Train: 12.34% [598100/4848000] [123.4/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 10:29:17,515 - Train: 12.34% [598200/4848000] [123.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 10:29:50,391 - Train: 12.34% [598300/4848000] [123.4/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 10:30:23,232 - Train: 12.34% [598400/4848000] [123.4/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 10:30:56,095 - Train: 12.35% [598500/4848000] [123.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 10:31:29,014 - Train: 12.35% [598600/4848000] [123.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 10:32:01,884 - Train: 12.35% [598700/4848000] [123.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 10:32:34,848 - Train: 12.35% [598800/4848000] [123.5/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-05 10:33:07,762 - Train: 12.35% [598900/4848000] [123.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 10:33:40,646 - Train: 12.36% [599000/4848000] [123.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 10:34:13,551 - Train: 12.36% [599100/4848000] [123.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 10:34:46,409 - Train: 12.36% [599200/4848000] [123.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 10:35:19,298 - Train: 12.36% [599300/4848000] [123.6/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 10:35:52,153 - Train: 12.36% [599400/4848000] [123.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 10:36:25,005 - Train: 12.37% [599500/4848000] [123.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 10:36:57,889 - Train: 12.37% [599600/4848000] [123.7/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 10:37:30,734 - Train: 12.37% [599700/4848000] [123.7/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 10:38:03,516 - Train: 12.37% [599800/4848000] [123.7/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 10:38:36,352 - Train: 12.37% [599900/4848000] [123.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 10:39:09,120 - Train: 12.38% [600000/4848000] [123.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 10:39:41,966 - Train: 12.38% [600100/4848000] [123.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 10:40:15,025 - Train: 12.38% [600200/4848000] [123.8/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 10:40:47,819 - Train: 12.38% [600300/4848000] [123.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 10:41:20,660 - Train: 12.38% [600400/4848000] [123.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 10:41:53,487 - Train: 12.39% [600500/4848000] [123.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 10:42:26,333 - Train: 12.39% [600600/4848000] [123.9/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 10:43:00,215 - Train: 12.39% [600700/4848000] [123.9/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 10:43:33,927 - Train: 12.39% [600800/4848000] [123.9/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 10:44:11,704 - Train: 12.39% [600900/4848000] [123.9/1000.0] [batch_t 1.253 (0.378)] [data_t 0.930] [optim_t 0.323] [lr 0.005000] 2024-04-05 10:44:44,568 - Train: 12.40% [601000/4848000] [124.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 10:45:18,802 - Train: 12.40% [601100/4848000] [124.0/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 10:45:35,888 - ==> Total time: 2 days, 11:19:34 Eta: 17 days, 11:06:39 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 10:45:53,792 - Train: 12.40% [601200/4848000] [124.0/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 10:46:26,587 - Train: 12.40% [601300/4848000] [124.0/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 10:46:59,483 - Train: 12.41% [601400/4848000] [124.1/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 10:47:32,275 - Train: 12.41% [601500/4848000] [124.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 10:48:05,215 - Train: 12.41% [601600/4848000] [124.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 10:48:38,009 - Train: 12.41% [601700/4848000] [124.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 10:49:11,267 - Train: 12.41% [601800/4848000] [124.1/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 10:49:44,080 - Train: 12.42% [601900/4848000] [124.2/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 10:50:16,875 - Train: 12.42% [602000/4848000] [124.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 10:50:49,690 - Train: 12.42% [602100/4848000] [124.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 10:51:23,407 - Train: 12.42% [602200/4848000] [124.2/1000.0] [batch_t 0.326 (0.337)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 10:51:56,221 - Train: 12.42% [602300/4848000] [124.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 10:52:31,249 - Train: 12.43% [602400/4848000] [124.3/1000.0] [batch_t 0.326 (0.350)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 10:53:04,080 - Train: 12.43% [602500/4848000] [124.3/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 10:53:37,553 - Train: 12.43% [602600/4848000] [124.3/1000.0] [batch_t 0.326 (0.335)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 10:54:11,969 - Train: 12.43% [602700/4848000] [124.3/1000.0] [batch_t 0.328 (0.344)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 10:54:44,762 - Train: 12.43% [602800/4848000] [124.3/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 10:55:17,594 - Train: 12.44% [602900/4848000] [124.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 10:55:50,636 - Train: 12.44% [603000/4848000] [124.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 10:56:25,625 - Train: 12.44% [603100/4848000] [124.4/1000.0] [batch_t 0.328 (0.350)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 10:56:58,516 - Train: 12.44% [603200/4848000] [124.4/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 10:57:32,519 - Train: 12.44% [603300/4848000] [124.4/1000.0] [batch_t 0.325 (0.340)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 10:58:07,840 - Train: 12.45% [603400/4848000] [124.5/1000.0] [batch_t 0.385 (0.353)] [data_t 0.050] [optim_t 0.335] [lr 0.005000] 2024-04-05 10:58:40,789 - Train: 12.45% [603500/4848000] [124.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 10:59:13,659 - Train: 12.45% [603600/4848000] [124.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 10:59:46,495 - Train: 12.45% [603700/4848000] [124.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 11:00:19,310 - Train: 12.45% [603800/4848000] [124.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 11:00:52,135 - Train: 12.46% [603900/4848000] [124.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 11:01:24,934 - Train: 12.46% [604000/4848000] [124.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 11:01:57,810 - Train: 12.46% [604100/4848000] [124.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 11:02:30,665 - Train: 12.46% [604200/4848000] [124.6/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 11:03:03,476 - Train: 12.46% [604300/4848000] [124.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 11:03:36,392 - Train: 12.47% [604400/4848000] [124.7/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-05 11:04:09,167 - Train: 12.47% [604500/4848000] [124.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 11:04:41,991 - Train: 12.47% [604600/4848000] [124.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 11:05:14,818 - Train: 12.47% [604700/4848000] [124.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 11:05:47,667 - Train: 12.48% [604800/4848000] [124.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 11:06:21,101 - Train: 12.48% [604900/4848000] [124.8/1000.0] [batch_t 0.325 (0.334)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 11:06:53,931 - Train: 12.48% [605000/4848000] [124.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 11:07:26,766 - Train: 12.48% [605100/4848000] [124.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 11:07:59,599 - Train: 12.48% [605200/4848000] [124.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 11:08:32,520 - Train: 12.49% [605300/4848000] [124.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 11:09:05,385 - Train: 12.49% [605400/4848000] [124.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 11:09:38,303 - Train: 12.49% [605500/4848000] [124.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 11:10:11,169 - Train: 12.49% [605600/4848000] [124.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 11:10:43,969 - Train: 12.49% [605700/4848000] [124.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 11:11:16,767 - Train: 12.50% [605800/4848000] [125.0/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 11:11:49,700 - Train: 12.50% [605900/4848000] [125.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 11:12:23,215 - Train: 12.50% [606000/4848000] [125.0/1000.0] [batch_t 0.331 (0.335)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 11:12:23,221 - ==> Total time: 2 days, 11:46:21 Eta: 17 days, 10:24:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 11:12:58,118 - Train: 12.50% [606100/4848000] [125.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 11:13:31,374 - Train: 12.50% [606200/4848000] [125.0/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 11:14:04,172 - Train: 12.51% [606300/4848000] [125.1/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 11:14:37,833 - Train: 12.51% [606400/4848000] [125.1/1000.0] [batch_t 0.332 (0.337)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 11:15:10,817 - Train: 12.51% [606500/4848000] [125.1/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 11:15:43,695 - Train: 12.51% [606600/4848000] [125.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 11:16:19,420 - Train: 12.51% [606700/4848000] [125.1/1000.0] [batch_t 0.325 (0.357)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 11:16:52,331 - Train: 12.52% [606800/4848000] [125.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 11:17:28,189 - Train: 12.52% [606900/4848000] [125.2/1000.0] [batch_t 0.329 (0.358)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 11:18:01,082 - Train: 12.52% [607000/4848000] [125.2/1000.0] [batch_t 0.336 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-05 11:18:34,626 - Train: 12.52% [607100/4848000] [125.2/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 11:19:08,450 - Train: 12.52% [607200/4848000] [125.2/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 11:19:41,250 - Train: 12.53% [607300/4848000] [125.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 11:20:15,486 - Train: 12.53% [607400/4848000] [125.3/1000.0] [batch_t 0.337 (0.342)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-05 11:20:48,330 - Train: 12.53% [607500/4848000] [125.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 11:21:21,448 - Train: 12.53% [607600/4848000] [125.3/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 11:21:54,310 - Train: 12.54% [607700/4848000] [125.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 11:22:28,380 - Train: 12.54% [607800/4848000] [125.4/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 11:23:01,303 - Train: 12.54% [607900/4848000] [125.4/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 11:23:34,243 - Train: 12.54% [608000/4848000] [125.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 11:24:07,533 - Train: 12.54% [608100/4848000] [125.4/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 11:24:40,401 - Train: 12.55% [608200/4848000] [125.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 11:25:13,329 - Train: 12.55% [608300/4848000] [125.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 11:25:46,238 - Train: 12.55% [608400/4848000] [125.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 11:26:20,035 - Train: 12.55% [608500/4848000] [125.5/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 11:26:52,939 - Train: 12.55% [608600/4848000] [125.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 11:27:26,043 - Train: 12.56% [608700/4848000] [125.6/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 11:27:58,905 - Train: 12.56% [608800/4848000] [125.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 11:28:31,761 - Train: 12.56% [608900/4848000] [125.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 11:29:04,649 - Train: 12.56% [609000/4848000] [125.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 11:29:37,481 - Train: 12.56% [609100/4848000] [125.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 11:30:10,353 - Train: 12.57% [609200/4848000] [125.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 11:30:43,226 - Train: 12.57% [609300/4848000] [125.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 11:31:16,148 - Train: 12.57% [609400/4848000] [125.7/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 11:31:48,995 - Train: 12.57% [609500/4848000] [125.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 11:32:21,874 - Train: 12.57% [609600/4848000] [125.7/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 11:32:54,724 - Train: 12.58% [609700/4848000] [125.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 11:33:27,594 - Train: 12.58% [609800/4848000] [125.8/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 11:34:00,431 - Train: 12.58% [609900/4848000] [125.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 11:34:33,365 - Train: 12.58% [610000/4848000] [125.8/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 11:35:07,368 - Train: 12.58% [610100/4848000] [125.8/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 11:35:40,241 - Train: 12.59% [610200/4848000] [125.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 11:36:14,153 - Train: 12.59% [610300/4848000] [125.9/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 11:36:46,994 - Train: 12.59% [610400/4848000] [125.9/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 11:37:19,997 - Train: 12.59% [610500/4848000] [125.9/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 11:37:52,952 - Train: 12.59% [610600/4848000] [125.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 11:38:26,899 - Train: 12.60% [610700/4848000] [126.0/1000.0] [batch_t 0.331 (0.339)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 11:38:59,760 - Train: 12.60% [610800/4848000] [126.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 11:39:15,553 - ==> Total time: 2 days, 12:13:14 Eta: 17 days, 9:43:13 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 11:39:34,821 - Train: 12.60% [610900/4848000] [126.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 11:40:08,932 - Train: 12.60% [611000/4848000] [126.0/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 11:40:41,774 - Train: 12.61% [611100/4848000] [126.1/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 11:41:14,630 - Train: 12.61% [611200/4848000] [126.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 11:41:47,484 - Train: 12.61% [611300/4848000] [126.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 11:42:20,539 - Train: 12.61% [611400/4848000] [126.1/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 11:42:53,474 - Train: 12.61% [611500/4848000] [126.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 11:43:26,317 - Train: 12.62% [611600/4848000] [126.2/1000.0] [batch_t 0.336 (0.328)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-05 11:43:59,172 - Train: 12.62% [611700/4848000] [126.2/1000.0] [batch_t 0.336 (0.328)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-05 11:44:32,824 - Train: 12.62% [611800/4848000] [126.2/1000.0] [batch_t 0.326 (0.336)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 11:45:05,819 - Train: 12.62% [611900/4848000] [126.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 11:45:38,647 - Train: 12.62% [612000/4848000] [126.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 11:46:11,437 - Train: 12.63% [612100/4848000] [126.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 11:46:44,234 - Train: 12.63% [612200/4848000] [126.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 11:47:17,037 - Train: 12.63% [612300/4848000] [126.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 11:47:49,876 - Train: 12.63% [612400/4848000] [126.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 11:48:22,679 - Train: 12.63% [612500/4848000] [126.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 11:48:55,510 - Train: 12.64% [612600/4848000] [126.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 11:49:29,336 - Train: 12.64% [612700/4848000] [126.4/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 11:50:02,269 - Train: 12.64% [612800/4848000] [126.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 11:50:35,051 - Train: 12.64% [612900/4848000] [126.4/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 11:51:08,122 - Train: 12.64% [613000/4848000] [126.4/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 11:51:40,910 - Train: 12.65% [613100/4848000] [126.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 11:52:15,047 - Train: 12.65% [613200/4848000] [126.5/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 11:52:47,881 - Train: 12.65% [613300/4848000] [126.5/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 11:53:21,564 - Train: 12.65% [613400/4848000] [126.5/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 11:53:54,416 - Train: 12.65% [613500/4848000] [126.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 11:54:27,429 - Train: 12.66% [613600/4848000] [126.6/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 11:55:00,241 - Train: 12.66% [613700/4848000] [126.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 11:55:34,224 - Train: 12.66% [613800/4848000] [126.6/1000.0] [batch_t 0.330 (0.340)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 11:56:07,059 - Train: 12.66% [613900/4848000] [126.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 11:56:39,831 - Train: 12.67% [614000/4848000] [126.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 11:57:13,656 - Train: 12.67% [614100/4848000] [126.7/1000.0] [batch_t 0.325 (0.338)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 11:57:46,450 - Train: 12.67% [614200/4848000] [126.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 11:58:20,417 - Train: 12.67% [614300/4848000] [126.7/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 11:58:53,256 - Train: 12.67% [614400/4848000] [126.7/1000.0] [batch_t 0.343 (0.328)] [data_t 0.002] [optim_t 0.341] [lr 0.005000] 2024-04-05 11:59:26,799 - Train: 12.68% [614500/4848000] [126.8/1000.0] [batch_t 0.331 (0.335)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 11:59:59,712 - Train: 12.68% [614600/4848000] [126.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 12:00:32,532 - Train: 12.68% [614700/4848000] [126.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 12:01:05,355 - Train: 12.68% [614800/4848000] [126.8/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 12:01:38,196 - Train: 12.68% [614900/4848000] [126.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 12:02:11,017 - Train: 12.69% [615000/4848000] [126.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 12:02:43,858 - Train: 12.69% [615100/4848000] [126.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 12:03:17,610 - Train: 12.69% [615200/4848000] [126.9/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 12:03:50,393 - Train: 12.69% [615300/4848000] [126.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 12:04:24,214 - Train: 12.69% [615400/4848000] [126.9/1000.0] [batch_t 0.325 (0.338)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 12:04:57,063 - Train: 12.70% [615500/4848000] [127.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 12:05:30,820 - Train: 12.70% [615600/4848000] [127.0/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 12:06:02,489 - ==> Total time: 2 days, 12:40:00 Eta: 17 days, 9:01:31 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 12:06:07,681 - Train: 12.70% [615700/4848000] [127.0/1000.0] [batch_t 0.327 (0.346)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 12:06:40,537 - Train: 12.70% [615800/4848000] [127.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 12:07:15,197 - Train: 12.70% [615900/4848000] [127.0/1000.0] [batch_t 0.329 (0.347)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 12:07:47,988 - Train: 12.71% [616000/4848000] [127.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 12:08:20,798 - Train: 12.71% [616100/4848000] [127.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 12:08:53,631 - Train: 12.71% [616200/4848000] [127.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 12:09:28,111 - Train: 12.71% [616300/4848000] [127.1/1000.0] [batch_t 0.325 (0.345)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 12:10:00,924 - Train: 12.71% [616400/4848000] [127.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 12:10:33,767 - Train: 12.72% [616500/4848000] [127.2/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 12:11:08,435 - Train: 12.72% [616600/4848000] [127.2/1000.0] [batch_t 0.322 (0.347)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 12:11:41,217 - Train: 12.72% [616700/4848000] [127.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 12:12:14,581 - Train: 12.72% [616800/4848000] [127.2/1000.0] [batch_t 0.326 (0.334)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 12:12:47,378 - Train: 12.72% [616900/4848000] [127.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 12:13:20,340 - Train: 12.73% [617000/4848000] [127.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 12:13:53,189 - Train: 12.73% [617100/4848000] [127.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 12:14:27,072 - Train: 12.73% [617200/4848000] [127.3/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 12:14:59,881 - Train: 12.73% [617300/4848000] [127.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 12:15:33,861 - Train: 12.74% [617400/4848000] [127.4/1000.0] [batch_t 0.330 (0.340)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 12:16:06,669 - Train: 12.74% [617500/4848000] [127.4/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 12:16:39,433 - Train: 12.74% [617600/4848000] [127.4/1000.0] [batch_t 0.333 (0.328)] [data_t 0.005] [optim_t 0.329] [lr 0.005000] 2024-04-05 12:17:13,597 - Train: 12.74% [617700/4848000] [127.4/1000.0] [batch_t 0.332 (0.342)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 12:17:46,434 - Train: 12.74% [617800/4848000] [127.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 12:18:20,504 - Train: 12.75% [617900/4848000] [127.5/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 12:18:53,368 - Train: 12.75% [618000/4848000] [127.5/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 12:19:26,222 - Train: 12.75% [618100/4848000] [127.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 12:19:59,056 - Train: 12.75% [618200/4848000] [127.5/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 12:20:31,908 - Train: 12.75% [618300/4848000] [127.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 12:21:05,395 - Train: 12.76% [618400/4848000] [127.6/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 12:21:38,245 - Train: 12.76% [618500/4848000] [127.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 12:22:11,111 - Train: 12.76% [618600/4848000] [127.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 12:22:43,911 - Train: 12.76% [618700/4848000] [127.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 12:23:16,687 - Train: 12.76% [618800/4848000] [127.6/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 12:23:49,468 - Train: 12.77% [618900/4848000] [127.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 12:24:23,026 - Train: 12.77% [619000/4848000] [127.7/1000.0] [batch_t 0.333 (0.335)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 12:24:55,804 - Train: 12.77% [619100/4848000] [127.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 12:25:29,304 - Train: 12.77% [619200/4848000] [127.7/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 12:26:02,108 - Train: 12.77% [619300/4848000] [127.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 12:26:35,448 - Train: 12.78% [619400/4848000] [127.8/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 12:27:09,389 - Train: 12.78% [619500/4848000] [127.8/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 12:27:42,295 - Train: 12.78% [619600/4848000] [127.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 12:28:16,469 - Train: 12.78% [619700/4848000] [127.8/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 12:28:49,402 - Train: 12.78% [619800/4848000] [127.8/1000.0] [batch_t 0.321 (0.329)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-05 12:29:23,047 - Train: 12.79% [619900/4848000] [127.9/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 12:29:55,822 - Train: 12.79% [620000/4848000] [127.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 12:30:28,685 - Train: 12.79% [620100/4848000] [127.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 12:31:01,490 - Train: 12.79% [620200/4848000] [127.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 12:31:35,506 - Train: 12.79% [620300/4848000] [127.9/1000.0] [batch_t 0.326 (0.340)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 12:32:08,352 - Train: 12.80% [620400/4848000] [128.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 12:32:41,192 - Train: 12.80% [620500/4848000] [128.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 12:32:55,633 - ==> Total time: 2 days, 13:06:54 Eta: 17 days, 8:20:46 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 12:33:17,554 - Train: 12.80% [620600/4848000] [128.0/1000.0] [batch_t 0.332 (0.350)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 12:33:50,407 - Train: 12.80% [620700/4848000] [128.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 12:34:25,299 - Train: 12.81% [620800/4848000] [128.1/1000.0] [batch_t 0.326 (0.349)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 12:34:58,128 - Train: 12.81% [620900/4848000] [128.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 12:35:31,958 - Train: 12.81% [621000/4848000] [128.1/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 12:36:05,661 - Train: 12.81% [621100/4848000] [128.1/1000.0] [batch_t 1.126 (0.337)] [data_t 0.787] [optim_t 0.339] [lr 0.005000] 2024-04-05 12:36:38,747 - Train: 12.81% [621200/4848000] [128.1/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 12:37:11,943 - Train: 12.82% [621300/4848000] [128.2/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 12:37:44,801 - Train: 12.82% [621400/4848000] [128.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 12:38:17,760 - Train: 12.82% [621500/4848000] [128.2/1000.0] [batch_t 0.321 (0.329)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-05 12:38:50,521 - Train: 12.82% [621600/4848000] [128.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 12:39:23,935 - Train: 12.82% [621700/4848000] [128.2/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 12:39:56,837 - Train: 12.83% [621800/4848000] [128.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 12:40:29,701 - Train: 12.83% [621900/4848000] [128.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 12:41:02,541 - Train: 12.83% [622000/4848000] [128.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 12:41:36,467 - Train: 12.83% [622100/4848000] [128.3/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 12:42:10,358 - Train: 12.83% [622200/4848000] [128.3/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 12:42:43,156 - Train: 12.84% [622300/4848000] [128.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 12:43:19,054 - Train: 12.84% [622400/4848000] [128.4/1000.0] [batch_t 0.329 (0.359)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 12:43:51,838 - Train: 12.84% [622500/4848000] [128.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 12:44:24,839 - Train: 12.84% [622600/4848000] [128.4/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 12:44:57,646 - Train: 12.84% [622700/4848000] [128.4/1000.0] [batch_t 0.340 (0.328)] [data_t 0.002] [optim_t 0.338] [lr 0.005000] 2024-04-05 12:45:30,568 - Train: 12.85% [622800/4848000] [128.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 12:46:03,482 - Train: 12.85% [622900/4848000] [128.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 12:46:36,764 - Train: 12.85% [623000/4848000] [128.5/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 12:47:09,698 - Train: 12.85% [623100/4848000] [128.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 12:47:42,530 - Train: 12.85% [623200/4848000] [128.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 12:48:16,030 - Train: 12.86% [623300/4848000] [128.6/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 12:48:48,883 - Train: 12.86% [623400/4848000] [128.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 12:49:22,741 - Train: 12.86% [623500/4848000] [128.6/1000.0] [batch_t 0.322 (0.338)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-05 12:49:55,641 - Train: 12.86% [623600/4848000] [128.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 12:50:28,545 - Train: 12.87% [623700/4848000] [128.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 12:51:01,412 - Train: 12.87% [623800/4848000] [128.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 12:51:34,760 - Train: 12.87% [623900/4848000] [128.7/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 12:52:07,837 - Train: 12.87% [624000/4848000] [128.7/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 12:52:40,617 - Train: 12.87% [624100/4848000] [128.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 12:53:13,894 - Train: 12.88% [624200/4848000] [128.8/1000.0] [batch_t 0.324 (0.333)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 12:53:46,721 - Train: 12.88% [624300/4848000] [128.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 12:54:20,360 - Train: 12.88% [624400/4848000] [128.8/1000.0] [batch_t 0.331 (0.336)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 12:54:53,139 - Train: 12.88% [624500/4848000] [128.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 12:55:26,872 - Train: 12.88% [624600/4848000] [128.8/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 12:55:59,707 - Train: 12.89% [624700/4848000] [128.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 12:56:32,812 - Train: 12.89% [624800/4848000] [128.9/1000.0] [batch_t 0.333 (0.331)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 12:57:05,624 - Train: 12.89% [624900/4848000] [128.9/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 12:57:38,354 - Train: 12.89% [625000/4848000] [128.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 12:58:11,138 - Train: 12.89% [625100/4848000] [128.9/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 12:58:43,944 - Train: 12.90% [625200/4848000] [129.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 12:59:16,764 - Train: 12.90% [625300/4848000] [129.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 12:59:46,941 - ==> Total time: 2 days, 13:33:45 Eta: 17 days, 7:40:01 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 12:59:51,873 - Train: 12.90% [625400/4848000] [129.0/1000.0] [batch_t 0.330 (0.340)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 13:00:26,729 - Train: 12.90% [625500/4848000] [129.0/1000.0] [batch_t 0.327 (0.348)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 13:00:59,557 - Train: 12.90% [625600/4848000] [129.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 13:01:36,010 - Train: 12.91% [625700/4848000] [129.1/1000.0] [batch_t 0.328 (0.364)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 13:02:08,875 - Train: 12.91% [625800/4848000] [129.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 13:02:41,693 - Train: 12.91% [625900/4848000] [129.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 13:03:16,569 - Train: 12.91% [626000/4848000] [129.1/1000.0] [batch_t 0.326 (0.349)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 13:03:49,400 - Train: 12.91% [626100/4848000] [129.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 13:04:22,729 - Train: 12.92% [626200/4848000] [129.2/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 13:04:55,596 - Train: 12.92% [626300/4848000] [129.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 13:05:28,439 - Train: 12.92% [626400/4848000] [129.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 13:06:01,261 - Train: 12.92% [626500/4848000] [129.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 13:06:36,345 - Train: 12.92% [626600/4848000] [129.2/1000.0] [batch_t 0.330 (0.351)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 13:07:10,055 - Train: 12.93% [626700/4848000] [129.3/1000.0] [batch_t 0.326 (0.337)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 13:07:43,043 - Train: 12.93% [626800/4848000] [129.3/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 13:08:16,147 - Train: 12.93% [626900/4848000] [129.3/1000.0] [batch_t 0.325 (0.331)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 13:08:49,002 - Train: 12.93% [627000/4848000] [129.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 13:09:22,358 - Train: 12.94% [627100/4848000] [129.4/1000.0] [batch_t 0.325 (0.333)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 13:09:55,218 - Train: 12.94% [627200/4848000] [129.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 13:10:28,093 - Train: 12.94% [627300/4848000] [129.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 13:11:00,962 - Train: 12.94% [627400/4848000] [129.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 13:11:34,643 - Train: 12.94% [627500/4848000] [129.4/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 13:12:07,523 - Train: 12.95% [627600/4848000] [129.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 13:12:40,373 - Train: 12.95% [627700/4848000] [129.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 13:13:13,253 - Train: 12.95% [627800/4848000] [129.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 13:13:46,094 - Train: 12.95% [627900/4848000] [129.5/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 13:14:19,485 - Train: 12.95% [628000/4848000] [129.5/1000.0] [batch_t 0.330 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 13:14:52,312 - Train: 12.96% [628100/4848000] [129.6/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 13:15:25,293 - Train: 12.96% [628200/4848000] [129.6/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 13:15:58,088 - Train: 12.96% [628300/4848000] [129.6/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 13:16:31,425 - Train: 12.96% [628400/4848000] [129.6/1000.0] [batch_t 0.333 (0.333)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 13:17:04,458 - Train: 12.96% [628500/4848000] [129.6/1000.0] [batch_t 0.521 (0.330)] [data_t 0.195] [optim_t 0.326] [lr 0.005000] 2024-04-05 13:17:37,413 - Train: 12.97% [628600/4848000] [129.7/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 13:18:10,171 - Train: 12.97% [628700/4848000] [129.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 13:18:42,943 - Train: 12.97% [628800/4848000] [129.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 13:19:16,637 - Train: 12.97% [628900/4848000] [129.7/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 13:19:49,445 - Train: 12.97% [629000/4848000] [129.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 13:20:22,381 - Train: 12.98% [629100/4848000] [129.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 13:20:55,183 - Train: 12.98% [629200/4848000] [129.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 13:21:28,520 - Train: 12.98% [629300/4848000] [129.8/1000.0] [batch_t 0.334 (0.333)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 13:22:01,370 - Train: 12.98% [629400/4848000] [129.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 13:22:34,531 - Train: 12.98% [629500/4848000] [129.8/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 13:23:07,669 - Train: 12.99% [629600/4848000] [129.9/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 13:23:40,691 - Train: 12.99% [629700/4848000] [129.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 13:24:14,218 - Train: 12.99% [629800/4848000] [129.9/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 13:24:47,047 - Train: 12.99% [629900/4848000] [129.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 13:25:21,267 - Train: 13.00% [630000/4848000] [130.0/1000.0] [batch_t 0.330 (0.342)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 13:25:54,157 - Train: 13.00% [630100/4848000] [130.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 13:26:28,148 - Train: 13.00% [630200/4848000] [130.0/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 13:26:41,269 - ==> Total time: 2 days, 14:00:39 Eta: 17 days, 6:59:49 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 13:27:02,988 - Train: 13.00% [630300/4848000] [130.0/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 13:27:36,302 - Train: 13.00% [630400/4848000] [130.0/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 13:28:09,581 - Train: 13.01% [630500/4848000] [130.1/1000.0] [batch_t 0.331 (0.333)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 13:28:42,430 - Train: 13.01% [630600/4848000] [130.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 13:29:15,708 - Train: 13.01% [630700/4848000] [130.1/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 13:29:48,588 - Train: 13.01% [630800/4848000] [130.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 13:30:25,053 - Train: 13.01% [630900/4848000] [130.1/1000.0] [batch_t 0.328 (0.365)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 13:30:58,038 - Train: 13.02% [631000/4848000] [130.2/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 13:31:36,757 - Train: 13.02% [631100/4848000] [130.2/1000.0] [batch_t 0.329 (0.387)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 13:32:09,942 - Train: 13.02% [631200/4848000] [130.2/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 13:32:42,715 - Train: 13.02% [631300/4848000] [130.2/1000.0] [batch_t 0.321 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-05 13:33:15,541 - Train: 13.02% [631400/4848000] [130.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 13:33:48,317 - Train: 13.03% [631500/4848000] [130.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 13:34:21,074 - Train: 13.03% [631600/4848000] [130.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 13:34:53,881 - Train: 13.03% [631700/4848000] [130.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 13:35:30,573 - Train: 13.03% [631800/4848000] [130.3/1000.0] [batch_t 0.329 (0.367)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 13:36:03,410 - Train: 13.03% [631900/4848000] [130.3/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 13:36:36,332 - Train: 13.04% [632000/4848000] [130.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 13:37:09,237 - Train: 13.04% [632100/4848000] [130.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 13:37:42,126 - Train: 13.04% [632200/4848000] [130.4/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 13:38:15,015 - Train: 13.04% [632300/4848000] [130.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 13:38:47,920 - Train: 13.04% [632400/4848000] [130.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 13:39:20,674 - Train: 13.05% [632500/4848000] [130.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 13:39:53,754 - Train: 13.05% [632600/4848000] [130.5/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 13:40:27,576 - Train: 13.05% [632700/4848000] [130.5/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 13:41:00,412 - Train: 13.05% [632800/4848000] [130.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 13:41:35,832 - Train: 13.05% [632900/4848000] [130.5/1000.0] [batch_t 0.321 (0.354)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-05 13:42:08,677 - Train: 13.06% [633000/4848000] [130.6/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 13:42:41,464 - Train: 13.06% [633100/4848000] [130.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 13:43:15,934 - Train: 13.06% [633200/4848000] [130.6/1000.0] [batch_t 0.333 (0.345)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 13:43:48,789 - Train: 13.06% [633300/4848000] [130.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 13:44:22,554 - Train: 13.07% [633400/4848000] [130.7/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 13:44:55,354 - Train: 13.07% [633500/4848000] [130.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 13:45:28,126 - Train: 13.07% [633600/4848000] [130.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 13:46:00,951 - Train: 13.07% [633700/4848000] [130.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 13:46:33,729 - Train: 13.07% [633800/4848000] [130.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 13:47:06,664 - Train: 13.08% [633900/4848000] [130.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 13:47:39,419 - Train: 13.08% [634000/4848000] [130.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 13:48:12,132 - Train: 13.08% [634100/4848000] [130.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 13:48:44,907 - Train: 13.08% [634200/4848000] [130.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 13:49:17,715 - Train: 13.08% [634300/4848000] [130.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 13:49:50,579 - Train: 13.09% [634400/4848000] [130.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 13:50:23,359 - Train: 13.09% [634500/4848000] [130.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 13:50:56,182 - Train: 13.09% [634600/4848000] [130.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 13:51:29,045 - Train: 13.09% [634700/4848000] [130.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 13:52:01,938 - Train: 13.09% [634800/4848000] [130.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 13:52:34,790 - Train: 13.10% [634900/4848000] [131.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 13:53:07,664 - Train: 13.10% [635000/4848000] [131.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 13:53:36,624 - ==> Total time: 2 days, 14:27:35 Eta: 17 days, 6:19:56 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 13:53:42,676 - Train: 13.10% [635100/4848000] [131.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 13:54:16,566 - Train: 13.10% [635200/4848000] [131.0/1000.0] [batch_t 0.324 (0.339)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 13:54:49,382 - Train: 13.10% [635300/4848000] [131.0/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 13:55:22,224 - Train: 13.11% [635400/4848000] [131.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 13:55:55,039 - Train: 13.11% [635500/4848000] [131.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 13:56:27,887 - Train: 13.11% [635600/4848000] [131.1/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 13:57:00,665 - Train: 13.11% [635700/4848000] [131.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 13:57:33,927 - Train: 13.11% [635800/4848000] [131.1/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 13:58:07,101 - Train: 13.12% [635900/4848000] [131.2/1000.0] [batch_t 0.333 (0.332)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 13:58:40,045 - Train: 13.12% [636000/4848000] [131.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 13:59:12,861 - Train: 13.12% [636100/4848000] [131.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 13:59:45,658 - Train: 13.12% [636200/4848000] [131.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 14:00:19,151 - Train: 13.12% [636300/4848000] [131.2/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 14:00:52,064 - Train: 13.13% [636400/4848000] [131.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 14:01:24,803 - Train: 13.13% [636500/4848000] [131.3/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 14:01:57,778 - Train: 13.13% [636600/4848000] [131.3/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 14:02:32,840 - Train: 13.13% [636700/4848000] [131.3/1000.0] [batch_t 0.326 (0.351)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 14:03:07,718 - Train: 13.14% [636800/4848000] [131.4/1000.0] [batch_t 0.330 (0.349)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 14:03:40,594 - Train: 13.14% [636900/4848000] [131.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 14:04:14,207 - Train: 13.14% [637000/4848000] [131.4/1000.0] [batch_t 0.326 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 14:04:46,994 - Train: 13.14% [637100/4848000] [131.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 14:05:21,070 - Train: 13.14% [637200/4848000] [131.4/1000.0] [batch_t 0.324 (0.341)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 14:05:53,909 - Train: 13.15% [637300/4848000] [131.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 14:06:26,695 - Train: 13.15% [637400/4848000] [131.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 14:06:59,507 - Train: 13.15% [637500/4848000] [131.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 14:07:32,361 - Train: 13.15% [637600/4848000] [131.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 14:08:05,161 - Train: 13.15% [637700/4848000] [131.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 14:08:38,059 - Train: 13.16% [637800/4848000] [131.6/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 14:09:10,920 - Train: 13.16% [637900/4848000] [131.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 14:09:43,690 - Train: 13.16% [638000/4848000] [131.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 14:10:16,668 - Train: 13.16% [638100/4848000] [131.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 14:10:49,462 - Train: 13.16% [638200/4848000] [131.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 14:11:23,601 - Train: 13.17% [638300/4848000] [131.7/1000.0] [batch_t 0.326 (0.341)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 14:11:56,406 - Train: 13.17% [638400/4848000] [131.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 14:12:30,222 - Train: 13.17% [638500/4848000] [131.7/1000.0] [batch_t 0.325 (0.338)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 14:13:03,031 - Train: 13.17% [638600/4848000] [131.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 14:13:35,807 - Train: 13.17% [638700/4848000] [131.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 14:14:08,977 - Train: 13.18% [638800/4848000] [131.8/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 14:14:41,812 - Train: 13.18% [638900/4848000] [131.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 14:15:17,589 - Train: 13.18% [639000/4848000] [131.8/1000.0] [batch_t 0.327 (0.358)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 14:15:50,495 - Train: 13.18% [639100/4848000] [131.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 14:16:25,741 - Train: 13.18% [639200/4848000] [131.8/1000.0] [batch_t 0.331 (0.352)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 14:16:58,617 - Train: 13.19% [639300/4848000] [131.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 14:17:33,502 - Train: 13.19% [639400/4848000] [131.9/1000.0] [batch_t 0.329 (0.349)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 14:18:06,488 - Train: 13.19% [639500/4848000] [131.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 14:18:39,342 - Train: 13.19% [639600/4848000] [131.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 14:19:13,335 - Train: 13.20% [639700/4848000] [132.0/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 14:19:46,128 - Train: 13.20% [639800/4848000] [132.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 14:20:18,930 - Train: 13.20% [639900/4848000] [132.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 14:20:30,744 - ==> Total time: 2 days, 14:54:29 Eta: 17 days, 5:40:06 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 14:20:53,754 - Train: 13.20% [640000/4848000] [132.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 14:21:27,677 - Train: 13.20% [640100/4848000] [132.0/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 14:22:00,489 - Train: 13.21% [640200/4848000] [132.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 14:22:35,454 - Train: 13.21% [640300/4848000] [132.1/1000.0] [batch_t 0.328 (0.350)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 14:23:08,973 - Train: 13.21% [640400/4848000] [132.1/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 14:23:41,909 - Train: 13.21% [640500/4848000] [132.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 14:24:15,400 - Train: 13.21% [640600/4848000] [132.1/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 14:24:48,193 - Train: 13.22% [640700/4848000] [132.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 14:25:20,982 - Train: 13.22% [640800/4848000] [132.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 14:25:53,984 - Train: 13.22% [640900/4848000] [132.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 14:26:29,300 - Train: 13.22% [641000/4848000] [132.2/1000.0] [batch_t 0.330 (0.353)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 14:27:02,142 - Train: 13.22% [641100/4848000] [132.2/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 14:27:36,131 - Train: 13.23% [641200/4848000] [132.3/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 14:28:08,927 - Train: 13.23% [641300/4848000] [132.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 14:28:41,813 - Train: 13.23% [641400/4848000] [132.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 14:29:14,883 - Train: 13.23% [641500/4848000] [132.3/1000.0] [batch_t 0.323 (0.331)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 14:29:47,729 - Train: 13.23% [641600/4848000] [132.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 14:30:21,517 - Train: 13.24% [641700/4848000] [132.4/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 14:30:54,332 - Train: 13.24% [641800/4848000] [132.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 14:31:28,257 - Train: 13.24% [641900/4848000] [132.4/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 14:32:01,113 - Train: 13.24% [642000/4848000] [132.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 14:32:33,923 - Train: 13.24% [642100/4848000] [132.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 14:33:07,874 - Train: 13.25% [642200/4848000] [132.5/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 14:33:40,747 - Train: 13.25% [642300/4848000] [132.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 14:34:13,583 - Train: 13.25% [642400/4848000] [132.5/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 14:34:46,388 - Train: 13.25% [642500/4848000] [132.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 14:35:19,174 - Train: 13.25% [642600/4848000] [132.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 14:35:52,007 - Train: 13.26% [642700/4848000] [132.6/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 14:36:24,805 - Train: 13.26% [642800/4848000] [132.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 14:36:57,638 - Train: 13.26% [642900/4848000] [132.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 14:37:30,405 - Train: 13.26% [643000/4848000] [132.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 14:38:03,227 - Train: 13.27% [643100/4848000] [132.7/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 14:38:36,054 - Train: 13.27% [643200/4848000] [132.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 14:39:08,971 - Train: 13.27% [643300/4848000] [132.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 14:39:41,768 - Train: 13.27% [643400/4848000] [132.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 14:40:14,569 - Train: 13.27% [643500/4848000] [132.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 14:40:47,461 - Train: 13.28% [643600/4848000] [132.8/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 14:41:20,357 - Train: 13.28% [643700/4848000] [132.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 14:41:53,129 - Train: 13.28% [643800/4848000] [132.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 14:42:26,063 - Train: 13.28% [643900/4848000] [132.8/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 14:42:58,823 - Train: 13.28% [644000/4848000] [132.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 14:43:32,391 - Train: 13.29% [644100/4848000] [132.9/1000.0] [batch_t 0.326 (0.336)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 14:44:06,100 - Train: 13.29% [644200/4848000] [132.9/1000.0] [batch_t 1.190 (0.337)] [data_t 0.861] [optim_t 0.329] [lr 0.005000] 2024-04-05 14:44:38,930 - Train: 13.29% [644300/4848000] [132.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 14:45:12,277 - Train: 13.29% [644400/4848000] [132.9/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 14:45:45,129 - Train: 13.29% [644500/4848000] [132.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 14:46:17,956 - Train: 13.30% [644600/4848000] [133.0/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 14:46:50,832 - Train: 13.30% [644700/4848000] [133.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 14:47:18,407 - ==> Total time: 2 days, 15:21:16 Eta: 17 days, 4:59:47 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 14:47:25,816 - Train: 13.30% [644800/4848000] [133.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 14:47:58,659 - Train: 13.30% [644900/4848000] [133.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 14:48:31,532 - Train: 13.30% [645000/4848000] [133.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 14:49:04,527 - Train: 13.31% [645100/4848000] [133.1/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 14:49:37,337 - Train: 13.31% [645200/4848000] [133.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 14:50:10,116 - Train: 13.31% [645300/4848000] [133.1/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 14:50:42,877 - Train: 13.31% [645400/4848000] [133.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 14:51:15,659 - Train: 13.31% [645500/4848000] [133.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 14:51:48,473 - Train: 13.32% [645600/4848000] [133.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 14:52:22,408 - Train: 13.32% [645700/4848000] [133.2/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 14:52:55,246 - Train: 13.32% [645800/4848000] [133.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 14:53:28,033 - Train: 13.32% [645900/4848000] [133.2/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 14:54:00,909 - Train: 13.33% [646000/4848000] [133.3/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 14:54:34,251 - Train: 13.33% [646100/4848000] [133.3/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 14:55:07,098 - Train: 13.33% [646200/4848000] [133.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 14:55:39,960 - Train: 13.33% [646300/4848000] [133.3/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 14:56:12,767 - Train: 13.33% [646400/4848000] [133.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 14:56:45,777 - Train: 13.34% [646500/4848000] [133.4/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 14:57:18,641 - Train: 13.34% [646600/4848000] [133.4/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 14:57:51,549 - Train: 13.34% [646700/4848000] [133.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 14:58:24,449 - Train: 13.34% [646800/4848000] [133.4/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-05 14:58:57,306 - Train: 13.34% [646900/4848000] [133.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 14:59:31,102 - Train: 13.35% [647000/4848000] [133.5/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 15:00:04,030 - Train: 13.35% [647100/4848000] [133.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 15:00:38,321 - Train: 13.35% [647200/4848000] [133.5/1000.0] [batch_t 0.330 (0.343)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 15:01:11,644 - Train: 13.35% [647300/4848000] [133.5/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 15:01:44,528 - Train: 13.35% [647400/4848000] [133.5/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 15:02:17,398 - Train: 13.36% [647500/4848000] [133.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 15:02:50,212 - Train: 13.36% [647600/4848000] [133.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 15:03:23,743 - Train: 13.36% [647700/4848000] [133.6/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 15:03:56,620 - Train: 13.36% [647800/4848000] [133.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 15:04:29,986 - Train: 13.36% [647900/4848000] [133.6/1000.0] [batch_t 0.331 (0.334)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 15:05:02,864 - Train: 13.37% [648000/4848000] [133.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 15:05:35,675 - Train: 13.37% [648100/4848000] [133.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 15:06:09,412 - Train: 13.37% [648200/4848000] [133.7/1000.0] [batch_t 0.336 (0.337)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-05 15:06:42,220 - Train: 13.37% [648300/4848000] [133.7/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 15:07:15,738 - Train: 13.37% [648400/4848000] [133.7/1000.0] [batch_t 0.335 (0.335)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-05 15:07:48,572 - Train: 13.38% [648500/4848000] [133.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 15:08:21,400 - Train: 13.38% [648600/4848000] [133.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 15:08:54,218 - Train: 13.38% [648700/4848000] [133.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 15:09:27,077 - Train: 13.38% [648800/4848000] [133.8/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 15:09:59,900 - Train: 13.38% [648900/4848000] [133.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 15:10:32,774 - Train: 13.39% [649000/4848000] [133.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 15:11:06,677 - Train: 13.39% [649100/4848000] [133.9/1000.0] [batch_t 0.332 (0.339)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 15:11:39,475 - Train: 13.39% [649200/4848000] [133.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 15:12:12,385 - Train: 13.39% [649300/4848000] [133.9/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 15:12:45,206 - Train: 13.40% [649400/4848000] [134.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 15:13:18,551 - Train: 13.40% [649500/4848000] [134.0/1000.0] [batch_t 0.325 (0.333)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 15:13:51,368 - Train: 13.40% [649600/4848000] [134.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 15:14:01,871 - ==> Total time: 2 days, 15:48:00 Eta: 17 days, 4:19:12 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 15:14:28,436 - Train: 13.40% [649700/4848000] [134.0/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 15:15:01,375 - Train: 13.40% [649800/4848000] [134.0/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 15:15:34,148 - Train: 13.41% [649900/4848000] [134.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 15:16:07,104 - Train: 13.41% [650000/4848000] [134.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 15:16:39,858 - Train: 13.41% [650100/4848000] [134.1/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 15:17:12,678 - Train: 13.41% [650200/4848000] [134.1/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 15:17:45,465 - Train: 13.41% [650300/4848000] [134.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 15:18:20,002 - Train: 13.42% [650400/4848000] [134.2/1000.0] [batch_t 0.327 (0.345)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 15:18:52,849 - Train: 13.42% [650500/4848000] [134.2/1000.0] [batch_t 0.336 (0.328)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-05 15:19:28,319 - Train: 13.42% [650600/4848000] [134.2/1000.0] [batch_t 0.330 (0.355)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 15:20:01,309 - Train: 13.42% [650700/4848000] [134.2/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 15:20:34,214 - Train: 13.42% [650800/4848000] [134.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 15:21:07,606 - Train: 13.43% [650900/4848000] [134.3/1000.0] [batch_t 0.330 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 15:21:40,456 - Train: 13.43% [651000/4848000] [134.3/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 15:22:13,745 - Train: 13.43% [651100/4848000] [134.3/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 15:22:46,650 - Train: 13.43% [651200/4848000] [134.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 15:23:19,605 - Train: 13.43% [651300/4848000] [134.3/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 15:23:52,486 - Train: 13.44% [651400/4848000] [134.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 15:24:25,886 - Train: 13.44% [651500/4848000] [134.4/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 15:24:58,686 - Train: 13.44% [651600/4848000] [134.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 15:25:31,453 - Train: 13.44% [651700/4848000] [134.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 15:26:04,272 - Train: 13.44% [651800/4848000] [134.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 15:26:37,094 - Train: 13.45% [651900/4848000] [134.5/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 15:27:09,967 - Train: 13.45% [652000/4848000] [134.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 15:27:42,897 - Train: 13.45% [652100/4848000] [134.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 15:28:15,777 - Train: 13.45% [652200/4848000] [134.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 15:28:48,646 - Train: 13.46% [652300/4848000] [134.6/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 15:29:21,563 - Train: 13.46% [652400/4848000] [134.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 15:29:54,398 - Train: 13.46% [652500/4848000] [134.6/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-05 15:30:27,320 - Train: 13.46% [652600/4848000] [134.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 15:31:00,168 - Train: 13.46% [652700/4848000] [134.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 15:31:34,516 - Train: 13.47% [652800/4848000] [134.7/1000.0] [batch_t 0.328 (0.343)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 15:32:07,414 - Train: 13.47% [652900/4848000] [134.7/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 15:32:40,259 - Train: 13.47% [653000/4848000] [134.7/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 15:33:13,547 - Train: 13.47% [653100/4848000] [134.7/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 15:33:46,401 - Train: 13.47% [653200/4848000] [134.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 15:34:20,049 - Train: 13.48% [653300/4848000] [134.8/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 15:34:52,968 - Train: 13.48% [653400/4848000] [134.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 15:35:26,503 - Train: 13.48% [653500/4848000] [134.8/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 15:35:59,332 - Train: 13.48% [653600/4848000] [134.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 15:36:33,215 - Train: 13.48% [653700/4848000] [134.8/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 15:37:07,284 - Train: 13.49% [653800/4848000] [134.9/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 15:37:40,151 - Train: 13.49% [653900/4848000] [134.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 15:38:13,552 - Train: 13.49% [654000/4848000] [134.9/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 15:38:46,424 - Train: 13.49% [654100/4848000] [134.9/1000.0] [batch_t 0.322 (0.329)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-05 15:39:19,298 - Train: 13.49% [654200/4848000] [134.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 15:39:52,150 - Train: 13.50% [654300/4848000] [135.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 15:40:26,314 - Train: 13.50% [654400/4848000] [135.0/1000.0] [batch_t 0.330 (0.342)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 15:40:52,591 - ==> Total time: 2 days, 16:14:51 Eta: 17 days, 3:39:36 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 15:41:01,356 - Train: 13.50% [654500/4848000] [135.0/1000.0] [batch_t 0.320 (0.329)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-05 15:41:35,250 - Train: 13.50% [654600/4848000] [135.0/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 15:42:09,745 - Train: 13.50% [654700/4848000] [135.0/1000.0] [batch_t 0.326 (0.345)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 15:42:42,588 - Train: 13.51% [654800/4848000] [135.1/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 15:43:17,008 - Train: 13.51% [654900/4848000] [135.1/1000.0] [batch_t 0.322 (0.344)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-05 15:43:49,850 - Train: 13.51% [655000/4848000] [135.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 15:44:23,284 - Train: 13.51% [655100/4848000] [135.1/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 15:44:56,128 - Train: 13.51% [655200/4848000] [135.1/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 15:45:31,542 - Train: 13.52% [655300/4848000] [135.2/1000.0] [batch_t 0.326 (0.354)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 15:46:04,362 - Train: 13.52% [655400/4848000] [135.2/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 15:46:37,870 - Train: 13.52% [655500/4848000] [135.2/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 15:47:12,425 - Train: 13.52% [655600/4848000] [135.2/1000.0] [batch_t 0.328 (0.345)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 15:47:45,266 - Train: 13.53% [655700/4848000] [135.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 15:48:19,073 - Train: 13.53% [655800/4848000] [135.3/1000.0] [batch_t 0.335 (0.338)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-05 15:48:51,903 - Train: 13.53% [655900/4848000] [135.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 15:49:24,736 - Train: 13.53% [656000/4848000] [135.3/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 15:49:57,538 - Train: 13.53% [656100/4848000] [135.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 15:50:30,362 - Train: 13.54% [656200/4848000] [135.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 15:51:03,425 - Train: 13.54% [656300/4848000] [135.4/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 15:51:36,233 - Train: 13.54% [656400/4848000] [135.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 15:52:10,088 - Train: 13.54% [656500/4848000] [135.4/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 15:52:42,915 - Train: 13.54% [656600/4848000] [135.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 15:53:17,068 - Train: 13.55% [656700/4848000] [135.5/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 15:53:49,933 - Train: 13.55% [656800/4848000] [135.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 15:54:22,754 - Train: 13.55% [656900/4848000] [135.5/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 15:54:55,598 - Train: 13.55% [657000/4848000] [135.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 15:55:28,401 - Train: 13.55% [657100/4848000] [135.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 15:56:01,259 - Train: 13.56% [657200/4848000] [135.6/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 15:56:36,585 - Train: 13.56% [657300/4848000] [135.6/1000.0] [batch_t 0.331 (0.353)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 15:57:09,368 - Train: 13.56% [657400/4848000] [135.6/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 15:57:42,168 - Train: 13.56% [657500/4848000] [135.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 15:58:15,292 - Train: 13.56% [657600/4848000] [135.6/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 15:58:48,221 - Train: 13.57% [657700/4848000] [135.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 15:59:21,043 - Train: 13.57% [657800/4848000] [135.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 15:59:53,844 - Train: 13.57% [657900/4848000] [135.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 16:00:27,748 - Train: 13.57% [658000/4848000] [135.7/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 16:01:00,540 - Train: 13.57% [658100/4848000] [135.7/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 16:01:34,860 - Train: 13.58% [658200/4848000] [135.8/1000.0] [batch_t 0.330 (0.343)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 16:02:08,022 - Train: 13.58% [658300/4848000] [135.8/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 16:02:40,863 - Train: 13.58% [658400/4848000] [135.8/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 16:03:13,658 - Train: 13.58% [658500/4848000] [135.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 16:03:46,448 - Train: 13.58% [658600/4848000] [135.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 16:04:20,006 - Train: 13.59% [658700/4848000] [135.9/1000.0] [batch_t 0.324 (0.335)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 16:04:52,817 - Train: 13.59% [658800/4848000] [135.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 16:05:25,883 - Train: 13.59% [658900/4848000] [135.9/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 16:05:58,676 - Train: 13.59% [659000/4848000] [135.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 16:06:32,351 - Train: 13.60% [659100/4848000] [136.0/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 16:07:05,336 - Train: 13.60% [659200/4848000] [136.0/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 16:07:38,224 - Train: 13.60% [659300/4848000] [136.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 16:07:47,433 - ==> Total time: 2 days, 16:41:45 Eta: 17 days, 3:00:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 16:08:12,971 - Train: 13.60% [659400/4848000] [136.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 16:08:45,789 - Train: 13.60% [659500/4848000] [136.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 16:09:18,603 - Train: 13.61% [659600/4848000] [136.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 16:09:51,382 - Train: 13.61% [659700/4848000] [136.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 16:10:24,213 - Train: 13.61% [659800/4848000] [136.1/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 16:10:57,021 - Train: 13.61% [659900/4848000] [136.1/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 16:11:29,859 - Train: 13.61% [660000/4848000] [136.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 16:12:02,697 - Train: 13.62% [660100/4848000] [136.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 16:12:35,473 - Train: 13.62% [660200/4848000] [136.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 16:13:09,264 - Train: 13.62% [660300/4848000] [136.2/1000.0] [batch_t 0.334 (0.338)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 16:13:42,183 - Train: 13.62% [660400/4848000] [136.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 16:14:15,918 - Train: 13.62% [660500/4848000] [136.2/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 16:14:48,797 - Train: 13.63% [660600/4848000] [136.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 16:15:22,577 - Train: 13.63% [660700/4848000] [136.3/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 16:15:55,434 - Train: 13.63% [660800/4848000] [136.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 16:16:28,783 - Train: 13.63% [660900/4848000] [136.3/1000.0] [batch_t 0.326 (0.333)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 16:17:01,584 - Train: 13.63% [661000/4848000] [136.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 16:17:35,928 - Train: 13.64% [661100/4848000] [136.4/1000.0] [batch_t 0.326 (0.343)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 16:18:08,968 - Train: 13.64% [661200/4848000] [136.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 16:18:41,767 - Train: 13.64% [661300/4848000] [136.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 16:19:15,522 - Train: 13.64% [661400/4848000] [136.4/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 16:19:48,330 - Train: 13.64% [661500/4848000] [136.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 16:20:21,622 - Train: 13.65% [661600/4848000] [136.5/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 16:20:54,437 - Train: 13.65% [661700/4848000] [136.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 16:21:28,415 - Train: 13.65% [661800/4848000] [136.5/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 16:22:01,243 - Train: 13.65% [661900/4848000] [136.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 16:22:34,252 - Train: 13.66% [662000/4848000] [136.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 16:23:07,968 - Train: 13.66% [662100/4848000] [136.6/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 16:23:40,775 - Train: 13.66% [662200/4848000] [136.6/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 16:24:13,563 - Train: 13.66% [662300/4848000] [136.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 16:24:46,368 - Train: 13.66% [662400/4848000] [136.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 16:25:19,306 - Train: 13.67% [662500/4848000] [136.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 16:25:52,072 - Train: 13.67% [662600/4848000] [136.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 16:26:25,581 - Train: 13.67% [662700/4848000] [136.7/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 16:26:58,439 - Train: 13.67% [662800/4848000] [136.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 16:27:32,197 - Train: 13.67% [662900/4848000] [136.7/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 16:28:05,095 - Train: 13.68% [663000/4848000] [136.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 16:28:37,951 - Train: 13.68% [663100/4848000] [136.8/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 16:29:11,544 - Train: 13.68% [663200/4848000] [136.8/1000.0] [batch_t 0.344 (0.336)] [data_t 0.002] [optim_t 0.342] [lr 0.005000] 2024-04-05 16:29:44,386 - Train: 13.68% [663300/4848000] [136.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 16:30:17,601 - Train: 13.68% [663400/4848000] [136.8/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 16:30:50,466 - Train: 13.69% [663500/4848000] [136.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 16:31:24,460 - Train: 13.69% [663600/4848000] [136.9/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 16:31:57,229 - Train: 13.69% [663700/4848000] [136.9/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-05 16:32:30,064 - Train: 13.69% [663800/4848000] [136.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 16:33:02,890 - Train: 13.69% [663900/4848000] [136.9/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 16:33:35,687 - Train: 13.70% [664000/4848000] [137.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 16:34:09,890 - Train: 13.70% [664100/4848000] [137.0/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 16:34:34,867 - ==> Total time: 2 days, 17:08:33 Eta: 17 days, 2:21:03 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 16:34:45,075 - Train: 13.70% [664200/4848000] [137.0/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 16:35:17,896 - Train: 13.70% [664300/4848000] [137.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 16:35:50,660 - Train: 13.70% [664400/4848000] [137.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 16:36:24,257 - Train: 13.71% [664500/4848000] [137.1/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 16:36:57,002 - Train: 13.71% [664600/4848000] [137.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 16:37:30,358 - Train: 13.71% [664700/4848000] [137.1/1000.0] [batch_t 0.326 (0.333)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 16:38:03,194 - Train: 13.71% [664800/4848000] [137.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 16:38:36,073 - Train: 13.71% [664900/4848000] [137.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 16:39:08,954 - Train: 13.72% [665000/4848000] [137.2/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 16:39:41,820 - Train: 13.72% [665100/4848000] [137.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 16:40:15,209 - Train: 13.72% [665200/4848000] [137.2/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 16:40:48,046 - Train: 13.72% [665300/4848000] [137.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 16:41:20,972 - Train: 13.73% [665400/4848000] [137.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 16:41:53,812 - Train: 13.73% [665500/4848000] [137.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 16:42:28,031 - Train: 13.73% [665600/4848000] [137.3/1000.0] [batch_t 0.327 (0.342)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 16:43:00,875 - Train: 13.73% [665700/4848000] [137.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 16:43:33,880 - Train: 13.73% [665800/4848000] [137.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 16:44:06,824 - Train: 13.74% [665900/4848000] [137.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 16:44:39,678 - Train: 13.74% [666000/4848000] [137.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 16:45:12,621 - Train: 13.74% [666100/4848000] [137.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 16:45:45,682 - Train: 13.74% [666200/4848000] [137.4/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 16:46:20,408 - Train: 13.74% [666300/4848000] [137.4/1000.0] [batch_t 0.330 (0.347)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 16:46:53,270 - Train: 13.75% [666400/4848000] [137.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 16:47:27,276 - Train: 13.75% [666500/4848000] [137.5/1000.0] [batch_t 0.335 (0.340)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-05 16:48:00,202 - Train: 13.75% [666600/4848000] [137.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 16:48:33,796 - Train: 13.75% [666700/4848000] [137.5/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 16:49:06,697 - Train: 13.75% [666800/4848000] [137.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 16:49:39,567 - Train: 13.76% [666900/4848000] [137.6/1000.0] [batch_t 0.321 (0.329)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-05 16:50:12,351 - Train: 13.76% [667000/4848000] [137.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 16:50:45,198 - Train: 13.76% [667100/4848000] [137.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 16:51:18,597 - Train: 13.76% [667200/4848000] [137.6/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 16:51:51,423 - Train: 13.76% [667300/4848000] [137.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 16:52:24,182 - Train: 13.77% [667400/4848000] [137.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 16:52:56,989 - Train: 13.77% [667500/4848000] [137.7/1000.0] [batch_t 0.339 (0.328)] [data_t 0.002] [optim_t 0.337] [lr 0.005000] 2024-04-05 16:53:29,902 - Train: 13.77% [667600/4848000] [137.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 16:54:02,717 - Train: 13.77% [667700/4848000] [137.7/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 16:54:35,526 - Train: 13.77% [667800/4848000] [137.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 16:55:08,305 - Train: 13.78% [667900/4848000] [137.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 16:55:41,120 - Train: 13.78% [668000/4848000] [137.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 16:56:14,683 - Train: 13.78% [668100/4848000] [137.8/1000.0] [batch_t 0.325 (0.336)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 16:56:47,478 - Train: 13.78% [668200/4848000] [137.8/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 16:57:20,292 - Train: 13.79% [668300/4848000] [137.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 16:57:53,058 - Train: 13.79% [668400/4848000] [137.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 16:58:25,834 - Train: 13.79% [668500/4848000] [137.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 16:58:58,615 - Train: 13.79% [668600/4848000] [137.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 16:59:31,777 - Train: 13.79% [668700/4848000] [137.9/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 17:00:04,614 - Train: 13.80% [668800/4848000] [138.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 17:00:38,111 - Train: 13.80% [668900/4848000] [138.0/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 17:01:10,976 - Train: 13.80% [669000/4848000] [138.0/1000.0] [batch_t 0.424 (0.329)] [data_t 0.002] [optim_t 0.423] [lr 0.005000] 2024-04-05 17:01:18,865 - ==> Total time: 2 days, 17:35:17 Eta: 17 days, 1:41:18 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 17:01:45,837 - Train: 13.80% [669100/4848000] [138.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 17:02:19,773 - Train: 13.80% [669200/4848000] [138.0/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 17:02:52,587 - Train: 13.81% [669300/4848000] [138.1/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 17:03:25,967 - Train: 13.81% [669400/4848000] [138.1/1000.0] [batch_t 0.330 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 17:03:58,738 - Train: 13.81% [669500/4848000] [138.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 17:04:32,428 - Train: 13.81% [669600/4848000] [138.1/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 17:05:06,297 - Train: 13.81% [669700/4848000] [138.1/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 17:05:39,069 - Train: 13.82% [669800/4848000] [138.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 17:06:12,433 - Train: 13.82% [669900/4848000] [138.2/1000.0] [batch_t 0.324 (0.334)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 17:06:45,320 - Train: 13.82% [670000/4848000] [138.2/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 17:07:19,273 - Train: 13.82% [670100/4848000] [138.2/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 17:07:52,073 - Train: 13.82% [670200/4848000] [138.2/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 17:08:25,839 - Train: 13.83% [670300/4848000] [138.3/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 17:08:58,769 - Train: 13.83% [670400/4848000] [138.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 17:09:31,740 - Train: 13.83% [670500/4848000] [138.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 17:10:04,572 - Train: 13.83% [670600/4848000] [138.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 17:10:38,532 - Train: 13.83% [670700/4848000] [138.3/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 17:11:12,529 - Train: 13.84% [670800/4848000] [138.4/1000.0] [batch_t 0.330 (0.340)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 17:11:45,300 - Train: 13.84% [670900/4848000] [138.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 17:12:19,578 - Train: 13.84% [671000/4848000] [138.4/1000.0] [batch_t 0.328 (0.343)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 17:12:52,357 - Train: 13.84% [671100/4848000] [138.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 17:13:26,352 - Train: 13.84% [671200/4848000] [138.4/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 17:13:59,140 - Train: 13.85% [671300/4848000] [138.5/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-05 17:14:33,492 - Train: 13.85% [671400/4848000] [138.5/1000.0] [batch_t 0.328 (0.343)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 17:15:07,032 - Train: 13.85% [671500/4848000] [138.5/1000.0] [batch_t 0.325 (0.335)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 17:15:39,872 - Train: 13.85% [671600/4848000] [138.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 17:16:14,799 - Train: 13.86% [671700/4848000] [138.6/1000.0] [batch_t 0.329 (0.349)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 17:16:47,748 - Train: 13.86% [671800/4848000] [138.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 17:17:20,592 - Train: 13.86% [671900/4848000] [138.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 17:17:53,378 - Train: 13.86% [672000/4848000] [138.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 17:18:29,023 - Train: 13.86% [672100/4848000] [138.6/1000.0] [batch_t 0.326 (0.356)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 17:19:01,747 - Train: 13.87% [672200/4848000] [138.7/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 17:19:34,518 - Train: 13.87% [672300/4848000] [138.7/1000.0] [batch_t 0.318 (0.328)] [data_t 0.002] [optim_t 0.316] [lr 0.005000] 2024-04-05 17:20:07,314 - Train: 13.87% [672400/4848000] [138.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 17:20:40,116 - Train: 13.87% [672500/4848000] [138.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 17:21:12,898 - Train: 13.87% [672600/4848000] [138.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 17:21:45,639 - Train: 13.88% [672700/4848000] [138.8/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 17:22:18,494 - Train: 13.88% [672800/4848000] [138.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 17:22:51,272 - Train: 13.88% [672900/4848000] [138.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 17:23:25,593 - Train: 13.88% [673000/4848000] [138.8/1000.0] [batch_t 0.326 (0.343)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 17:23:58,388 - Train: 13.88% [673100/4848000] [138.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 17:24:31,204 - Train: 13.89% [673200/4848000] [138.9/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 17:25:04,056 - Train: 13.89% [673300/4848000] [138.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 17:25:36,895 - Train: 13.89% [673400/4848000] [138.9/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 17:26:09,977 - Train: 13.89% [673500/4848000] [138.9/1000.0] [batch_t 0.322 (0.331)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-05 17:26:42,773 - Train: 13.89% [673600/4848000] [138.9/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 17:27:15,921 - Train: 13.90% [673700/4848000] [139.0/1000.0] [batch_t 0.319 (0.331)] [data_t 0.002] [optim_t 0.317] [lr 0.005000] 2024-04-05 17:27:48,709 - Train: 13.90% [673800/4848000] [139.0/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-05 17:28:13,322 - ==> Total time: 2 days, 18:02:11 Eta: 17 days, 1:02:49 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 17:28:24,558 - Train: 13.90% [673900/4848000] [139.0/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-05 17:28:57,455 - Train: 13.90% [674000/4848000] [139.0/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 17:29:30,252 - Train: 13.90% [674100/4848000] [139.0/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 17:30:03,119 - Train: 13.91% [674200/4848000] [139.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 17:30:37,144 - Train: 13.91% [674300/4848000] [139.1/1000.0] [batch_t 0.326 (0.340)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 17:31:10,293 - Train: 13.91% [674400/4848000] [139.1/1000.0] [batch_t 0.322 (0.331)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-05 17:31:43,141 - Train: 13.91% [674500/4848000] [139.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 17:32:16,113 - Train: 13.92% [674600/4848000] [139.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 17:32:48,970 - Train: 13.92% [674700/4848000] [139.2/1000.0] [batch_t 0.337 (0.328)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-05 17:33:21,845 - Train: 13.92% [674800/4848000] [139.2/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 17:33:54,706 - Train: 13.92% [674900/4848000] [139.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 17:34:28,172 - Train: 13.92% [675000/4848000] [139.2/1000.0] [batch_t 0.325 (0.335)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 17:35:00,985 - Train: 13.93% [675100/4848000] [139.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 17:35:33,878 - Train: 13.93% [675200/4848000] [139.3/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 17:36:07,717 - Train: 13.93% [675300/4848000] [139.3/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 17:36:40,487 - Train: 13.93% [675400/4848000] [139.3/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 17:37:14,314 - Train: 13.93% [675500/4848000] [139.3/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 17:37:47,101 - Train: 13.94% [675600/4848000] [139.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 17:38:19,956 - Train: 13.94% [675700/4848000] [139.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 17:38:52,719 - Train: 13.94% [675800/4848000] [139.4/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 17:39:25,992 - Train: 13.94% [675900/4848000] [139.4/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 17:39:58,796 - Train: 13.94% [676000/4848000] [139.4/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 17:40:31,987 - Train: 13.95% [676100/4848000] [139.5/1000.0] [batch_t 0.326 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 17:41:04,764 - Train: 13.95% [676200/4848000] [139.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 17:41:37,588 - Train: 13.95% [676300/4848000] [139.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 17:42:11,418 - Train: 13.95% [676400/4848000] [139.5/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 17:42:44,219 - Train: 13.95% [676500/4848000] [139.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 17:43:17,631 - Train: 13.96% [676600/4848000] [139.6/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 17:43:50,494 - Train: 13.96% [676700/4848000] [139.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 17:44:24,019 - Train: 13.96% [676800/4848000] [139.6/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 17:44:56,918 - Train: 13.96% [676900/4848000] [139.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 17:45:29,691 - Train: 13.96% [677000/4848000] [139.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 17:46:02,436 - Train: 13.97% [677100/4848000] [139.7/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 17:46:36,577 - Train: 13.97% [677200/4848000] [139.7/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 17:47:09,486 - Train: 13.97% [677300/4848000] [139.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 17:47:42,266 - Train: 13.97% [677400/4848000] [139.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 17:48:15,292 - Train: 13.97% [677500/4848000] [139.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 17:48:48,157 - Train: 13.98% [677600/4848000] [139.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 17:49:20,967 - Train: 13.98% [677700/4848000] [139.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 17:49:53,714 - Train: 13.98% [677800/4848000] [139.8/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 17:50:27,424 - Train: 13.98% [677900/4848000] [139.8/1000.0] [batch_t 0.326 (0.337)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 17:51:00,246 - Train: 13.99% [678000/4848000] [139.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 17:51:33,029 - Train: 13.99% [678100/4848000] [139.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 17:52:06,230 - Train: 13.99% [678200/4848000] [139.9/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 17:52:39,213 - Train: 13.99% [678300/4848000] [139.9/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 17:53:12,057 - Train: 13.99% [678400/4848000] [139.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 17:53:44,905 - Train: 14.00% [678500/4848000] [140.0/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 17:54:18,219 - Train: 14.00% [678600/4848000] [140.0/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 17:54:51,047 - Train: 14.00% [678700/4848000] [140.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 17:54:57,622 - ==> Total time: 2 days, 18:28:56 Eta: 17 days, 0:23:27 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 17:55:26,911 - Train: 14.00% [678800/4848000] [140.0/1000.0] [batch_t 0.328 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 17:55:59,665 - Train: 14.00% [678900/4848000] [140.0/1000.0] [batch_t 0.334 (0.327)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 17:56:32,447 - Train: 14.01% [679000/4848000] [140.1/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 17:57:07,993 - Train: 14.01% [679100/4848000] [140.1/1000.0] [batch_t 3.049 (0.355)] [data_t 2.725] [optim_t 0.324] [lr 0.005000] 2024-04-05 17:57:40,790 - Train: 14.01% [679200/4848000] [140.1/1000.0] [batch_t 0.337 (0.328)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-05 17:58:13,627 - Train: 14.01% [679300/4848000] [140.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 17:58:46,474 - Train: 14.01% [679400/4848000] [140.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 17:59:20,172 - Train: 14.02% [679500/4848000] [140.2/1000.0] [batch_t 0.326 (0.337)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 17:59:52,958 - Train: 14.02% [679600/4848000] [140.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 18:00:25,789 - Train: 14.02% [679700/4848000] [140.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 18:00:58,576 - Train: 14.02% [679800/4848000] [140.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 18:01:31,710 - Train: 14.02% [679900/4848000] [140.2/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 18:02:04,521 - Train: 14.03% [680000/4848000] [140.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 18:02:37,348 - Train: 14.03% [680100/4848000] [140.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 18:03:10,101 - Train: 14.03% [680200/4848000] [140.3/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 18:03:42,952 - Train: 14.03% [680300/4848000] [140.3/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 18:04:15,760 - Train: 14.03% [680400/4848000] [140.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 18:04:48,564 - Train: 14.04% [680500/4848000] [140.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 18:05:22,652 - Train: 14.04% [680600/4848000] [140.4/1000.0] [batch_t 0.323 (0.341)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 18:05:55,444 - Train: 14.04% [680700/4848000] [140.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 18:06:29,510 - Train: 14.04% [680800/4848000] [140.4/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 18:07:02,292 - Train: 14.04% [680900/4848000] [140.4/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 18:07:36,180 - Train: 14.05% [681000/4848000] [140.5/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 18:08:09,580 - Train: 14.05% [681100/4848000] [140.5/1000.0] [batch_t 0.337 (0.334)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-05 18:08:42,337 - Train: 14.05% [681200/4848000] [140.5/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 18:09:15,092 - Train: 14.05% [681300/4848000] [140.5/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 18:09:47,857 - Train: 14.06% [681400/4848000] [140.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 18:10:22,458 - Train: 14.06% [681500/4848000] [140.6/1000.0] [batch_t 0.327 (0.346)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 18:10:55,288 - Train: 14.06% [681600/4848000] [140.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 18:11:28,911 - Train: 14.06% [681700/4848000] [140.6/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 18:12:01,796 - Train: 14.06% [681800/4848000] [140.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 18:12:35,834 - Train: 14.07% [681900/4848000] [140.7/1000.0] [batch_t 0.334 (0.340)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 18:13:09,370 - Train: 14.07% [682000/4848000] [140.7/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 18:13:42,254 - Train: 14.07% [682100/4848000] [140.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 18:14:15,965 - Train: 14.07% [682200/4848000] [140.7/1000.0] [batch_t 0.336 (0.337)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-05 18:14:48,860 - Train: 14.07% [682300/4848000] [140.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 18:15:21,683 - Train: 14.08% [682400/4848000] [140.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 18:15:54,601 - Train: 14.08% [682500/4848000] [140.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 18:16:27,565 - Train: 14.08% [682600/4848000] [140.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 18:17:00,437 - Train: 14.08% [682700/4848000] [140.8/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 18:17:35,449 - Train: 14.08% [682800/4848000] [140.8/1000.0] [batch_t 0.329 (0.350)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 18:18:09,930 - Train: 14.09% [682900/4848000] [140.9/1000.0] [batch_t 0.328 (0.345)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 18:18:42,823 - Train: 14.09% [683000/4848000] [140.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 18:19:16,689 - Train: 14.09% [683100/4848000] [140.9/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 18:19:49,707 - Train: 14.09% [683200/4848000] [140.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 18:20:23,018 - Train: 14.09% [683300/4848000] [140.9/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 18:20:55,890 - Train: 14.10% [683400/4848000] [141.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 18:21:30,120 - Train: 14.10% [683500/4848000] [141.0/1000.0] [batch_t 0.323 (0.342)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 18:21:52,946 - ==> Total time: 2 days, 18:55:51 Eta: 16 days, 23:45:24 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 18:22:18,113 - Train: 14.10% [683600/4848000] [141.0/1000.0] [batch_t 0.330 (0.706)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 18:22:54,036 - Train: 14.10% [683700/4848000] [141.0/1000.0] [batch_t 0.326 (0.359)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 18:23:41,189 - Train: 14.10% [683800/4848000] [141.0/1000.0] [batch_t 0.671 (0.471)] [data_t 0.345] [optim_t 0.326] [lr 0.005000] 2024-04-05 18:24:15,424 - Train: 14.11% [683900/4848000] [141.1/1000.0] [batch_t 0.326 (0.342)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 18:24:48,288 - Train: 14.11% [684000/4848000] [141.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 18:25:35,661 - Train: 14.11% [684100/4848000] [141.1/1000.0] [batch_t 0.327 (0.474)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 18:26:15,794 - Train: 14.11% [684200/4848000] [141.1/1000.0] [batch_t 0.330 (0.401)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 18:26:57,092 - Train: 14.12% [684300/4848000] [141.2/1000.0] [batch_t 0.328 (0.413)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 18:27:30,979 - Train: 14.12% [684400/4848000] [141.2/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 18:28:07,180 - Train: 14.12% [684500/4848000] [141.2/1000.0] [batch_t 0.326 (0.362)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 18:28:42,766 - Train: 14.12% [684600/4848000] [141.2/1000.0] [batch_t 0.327 (0.356)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 18:29:22,860 - Train: 14.12% [684700/4848000] [141.2/1000.0] [batch_t 0.327 (0.401)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 18:29:55,701 - Train: 14.13% [684800/4848000] [141.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 18:30:28,919 - Train: 14.13% [684900/4848000] [141.3/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 18:31:01,747 - Train: 14.13% [685000/4848000] [141.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 18:31:36,534 - Train: 14.13% [685100/4848000] [141.3/1000.0] [batch_t 0.328 (0.348)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 18:32:09,344 - Train: 14.13% [685200/4848000] [141.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 18:32:42,130 - Train: 14.14% [685300/4848000] [141.4/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 18:33:14,895 - Train: 14.14% [685400/4848000] [141.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 18:33:47,710 - Train: 14.14% [685500/4848000] [141.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 18:34:20,564 - Train: 14.14% [685600/4848000] [141.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 18:34:53,337 - Train: 14.14% [685700/4848000] [141.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 18:35:26,109 - Train: 14.15% [685800/4848000] [141.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 18:35:58,880 - Train: 14.15% [685900/4848000] [141.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 18:36:32,176 - Train: 14.15% [686000/4848000] [141.5/1000.0] [batch_t 0.325 (0.333)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 18:37:05,025 - Train: 14.15% [686100/4848000] [141.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 18:37:37,853 - Train: 14.15% [686200/4848000] [141.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 18:38:11,355 - Train: 14.16% [686300/4848000] [141.6/1000.0] [batch_t 0.326 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 18:38:44,182 - Train: 14.16% [686400/4848000] [141.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 18:39:16,965 - Train: 14.16% [686500/4848000] [141.6/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 18:39:49,767 - Train: 14.16% [686600/4848000] [141.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 18:40:22,537 - Train: 14.16% [686700/4848000] [141.6/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 18:40:55,303 - Train: 14.17% [686800/4848000] [141.7/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 18:41:28,099 - Train: 14.17% [686900/4848000] [141.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 18:42:00,861 - Train: 14.17% [687000/4848000] [141.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 18:42:34,123 - Train: 14.17% [687100/4848000] [141.7/1000.0] [batch_t 0.326 (0.333)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 18:43:09,929 - Train: 14.17% [687200/4848000] [141.7/1000.0] [batch_t 0.325 (0.358)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 18:43:42,709 - Train: 14.18% [687300/4848000] [141.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 18:44:18,690 - Train: 14.18% [687400/4848000] [141.8/1000.0] [batch_t 0.325 (0.360)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 18:44:51,529 - Train: 14.18% [687500/4848000] [141.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 18:45:27,072 - Train: 14.18% [687600/4848000] [141.8/1000.0] [batch_t 0.330 (0.355)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 18:45:59,846 - Train: 14.19% [687700/4848000] [141.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 18:46:32,898 - Train: 14.19% [687800/4848000] [141.9/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 18:47:05,760 - Train: 14.19% [687900/4848000] [141.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 18:47:38,566 - Train: 14.19% [688000/4848000] [141.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 18:48:12,003 - Train: 14.19% [688100/4848000] [141.9/1000.0] [batch_t 0.323 (0.334)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 18:48:44,832 - Train: 14.20% [688200/4848000] [142.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 18:49:19,830 - Train: 14.20% [688300/4848000] [142.0/1000.0] [batch_t 0.327 (0.350)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 18:49:52,651 - Train: 14.20% [688400/4848000] [142.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 18:49:57,913 - ==> Total time: 2 days, 19:23:56 Eta: 16 days, 23:14:30 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 18:50:28,285 - Train: 14.20% [688500/4848000] [142.0/1000.0] [batch_t 0.324 (0.333)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 18:51:01,126 - Train: 14.20% [688600/4848000] [142.0/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 18:51:35,165 - Train: 14.21% [688700/4848000] [142.1/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 18:52:08,568 - Train: 14.21% [688800/4848000] [142.1/1000.0] [batch_t 0.325 (0.334)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 18:52:41,342 - Train: 14.21% [688900/4848000] [142.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 18:53:14,183 - Train: 14.21% [689000/4848000] [142.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 18:53:46,973 - Train: 14.21% [689100/4848000] [142.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 18:54:19,691 - Train: 14.22% [689200/4848000] [142.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 18:54:52,462 - Train: 14.22% [689300/4848000] [142.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 18:55:28,911 - Train: 14.22% [689400/4848000] [142.2/1000.0] [batch_t 0.328 (0.364)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 18:56:01,647 - Train: 14.22% [689500/4848000] [142.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 18:56:34,434 - Train: 14.22% [689600/4848000] [142.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 18:57:07,866 - Train: 14.23% [689700/4848000] [142.3/1000.0] [batch_t 0.331 (0.334)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 18:57:40,623 - Train: 14.23% [689800/4848000] [142.3/1000.0] [batch_t 0.334 (0.327)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 18:58:13,360 - Train: 14.23% [689900/4848000] [142.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 18:58:46,115 - Train: 14.23% [690000/4848000] [142.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 18:59:19,184 - Train: 14.23% [690100/4848000] [142.3/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 18:59:52,149 - Train: 14.24% [690200/4848000] [142.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 19:00:25,014 - Train: 14.24% [690300/4848000] [142.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 19:00:57,849 - Train: 14.24% [690400/4848000] [142.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 19:01:30,683 - Train: 14.24% [690500/4848000] [142.4/1000.0] [batch_t 0.321 (0.328)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-05 19:02:03,545 - Train: 14.25% [690600/4848000] [142.5/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 19:02:39,074 - Train: 14.25% [690700/4848000] [142.5/1000.0] [batch_t 0.328 (0.355)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 19:03:24,009 - Train: 14.25% [690800/4848000] [142.5/1000.0] [batch_t 0.325 (0.449)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 19:04:02,825 - Train: 14.25% [690900/4848000] [142.5/1000.0] [batch_t 0.327 (0.388)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 19:04:41,448 - Train: 14.25% [691000/4848000] [142.5/1000.0] [batch_t 0.330 (0.386)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 19:05:17,071 - Train: 14.26% [691100/4848000] [142.6/1000.0] [batch_t 0.329 (0.356)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 19:05:54,153 - Train: 14.26% [691200/4848000] [142.6/1000.0] [batch_t 0.328 (0.371)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 19:06:31,157 - Train: 14.26% [691300/4848000] [142.6/1000.0] [batch_t 0.323 (0.370)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 19:07:07,564 - Train: 14.26% [691400/4848000] [142.6/1000.0] [batch_t 0.334 (0.364)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 19:07:41,462 - Train: 14.26% [691500/4848000] [142.6/1000.0] [batch_t 0.335 (0.339)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-05 19:08:16,145 - Train: 14.27% [691600/4848000] [142.7/1000.0] [batch_t 0.334 (0.347)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 19:08:52,404 - Train: 14.27% [691700/4848000] [142.7/1000.0] [batch_t 0.328 (0.362)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 19:09:25,209 - Train: 14.27% [691800/4848000] [142.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 19:09:59,891 - Train: 14.27% [691900/4848000] [142.7/1000.0] [batch_t 0.335 (0.347)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-05 19:10:38,174 - Train: 14.27% [692000/4848000] [142.7/1000.0] [batch_t 0.332 (0.383)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 19:11:42,869 - Train: 14.28% [692100/4848000] [142.8/1000.0] [batch_t 0.328 (0.647)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 19:12:37,802 - Train: 14.28% [692200/4848000] [142.8/1000.0] [batch_t 0.330 (0.549)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 19:13:28,172 - Train: 14.28% [692300/4848000] [142.8/1000.0] [batch_t 0.331 (0.504)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 19:14:02,805 - Train: 14.28% [692400/4848000] [142.8/1000.0] [batch_t 0.329 (0.346)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 19:14:50,097 - Train: 14.28% [692500/4848000] [142.8/1000.0] [batch_t 0.330 (0.473)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 19:15:40,839 - Train: 14.29% [692600/4848000] [142.9/1000.0] [batch_t 0.330 (0.507)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 19:16:14,224 - Train: 14.29% [692700/4848000] [142.9/1000.0] [batch_t 0.332 (0.334)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 19:16:46,976 - Train: 14.29% [692800/4848000] [142.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 19:17:20,601 - Train: 14.29% [692900/4848000] [142.9/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 19:17:53,373 - Train: 14.29% [693000/4848000] [142.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 19:18:28,277 - Train: 14.30% [693100/4848000] [143.0/1000.0] [batch_t 0.328 (0.349)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 19:19:01,080 - Train: 14.30% [693200/4848000] [143.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 19:19:23,402 - ==> Total time: 2 days, 19:53:21 Eta: 16 days, 22:51:42 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 19:19:39,375 - Train: 14.30% [693300/4848000] [143.0/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 19:20:13,227 - Train: 14.30% [693400/4848000] [143.0/1000.0] [batch_t 0.336 (0.338)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-05 19:20:46,045 - Train: 14.30% [693500/4848000] [143.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 19:21:20,575 - Train: 14.31% [693600/4848000] [143.1/1000.0] [batch_t 0.330 (0.345)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 19:21:53,373 - Train: 14.31% [693700/4848000] [143.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 19:22:28,795 - Train: 14.31% [693800/4848000] [143.1/1000.0] [batch_t 0.330 (0.354)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 19:23:04,157 - Train: 14.31% [693900/4848000] [143.1/1000.0] [batch_t 0.329 (0.354)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 19:23:37,010 - Train: 14.32% [694000/4848000] [143.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 19:24:10,983 - Train: 14.32% [694100/4848000] [143.2/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 19:24:46,805 - Train: 14.32% [694200/4848000] [143.2/1000.0] [batch_t 0.331 (0.358)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 19:25:39,397 - Train: 14.32% [694300/4848000] [143.2/1000.0] [batch_t 0.327 (0.526)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 19:26:32,652 - Train: 14.32% [694400/4848000] [143.2/1000.0] [batch_t 0.328 (0.532)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 19:27:15,435 - Train: 14.33% [694500/4848000] [143.3/1000.0] [batch_t 0.324 (0.428)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 19:27:59,884 - Train: 14.33% [694600/4848000] [143.3/1000.0] [batch_t 0.327 (0.444)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 19:29:21,428 - Train: 14.33% [694700/4848000] [143.3/1000.0] [batch_t 0.326 (0.815)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 19:30:45,334 - Train: 14.33% [694800/4848000] [143.3/1000.0] [batch_t 0.328 (0.839)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 19:32:12,941 - Train: 14.33% [694900/4848000] [143.3/1000.0] [batch_t 0.476 (0.876)] [data_t 0.148] [optim_t 0.328] [lr 0.005000] 2024-04-05 19:33:01,031 - Train: 14.34% [695000/4848000] [143.4/1000.0] [batch_t 0.329 (0.481)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 19:33:59,853 - Train: 14.34% [695100/4848000] [143.4/1000.0] [batch_t 0.329 (0.588)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 19:35:31,603 - Train: 14.34% [695200/4848000] [143.4/1000.0] [batch_t 0.326 (0.917)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 19:36:50,978 - Train: 14.34% [695300/4848000] [143.4/1000.0] [batch_t 0.329 (0.794)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 19:38:04,788 - Train: 14.34% [695400/4848000] [143.4/1000.0] [batch_t 0.330 (0.738)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 19:38:45,183 - Train: 14.35% [695500/4848000] [143.5/1000.0] [batch_t 0.326 (0.404)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 19:39:36,094 - Train: 14.35% [695600/4848000] [143.5/1000.0] [batch_t 0.335 (0.509)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 19:41:00,287 - Train: 14.35% [695700/4848000] [143.5/1000.0] [batch_t 0.327 (0.842)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 19:41:34,166 - Train: 14.35% [695800/4848000] [143.5/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 19:42:06,939 - Train: 14.35% [695900/4848000] [143.5/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 19:42:39,873 - Train: 14.36% [696000/4848000] [143.6/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 19:43:13,068 - Train: 14.36% [696100/4848000] [143.6/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 19:43:45,822 - Train: 14.36% [696200/4848000] [143.6/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 19:44:18,677 - Train: 14.36% [696300/4848000] [143.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 19:44:51,468 - Train: 14.36% [696400/4848000] [143.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 19:45:24,285 - Train: 14.37% [696500/4848000] [143.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 19:45:57,193 - Train: 14.37% [696600/4848000] [143.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 19:46:30,037 - Train: 14.37% [696700/4848000] [143.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 19:47:02,873 - Train: 14.37% [696800/4848000] [143.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 19:47:35,671 - Train: 14.37% [696900/4848000] [143.8/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 19:48:08,493 - Train: 14.38% [697000/4848000] [143.8/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 19:48:41,300 - Train: 14.38% [697100/4848000] [143.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 19:49:14,101 - Train: 14.38% [697200/4848000] [143.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 19:49:46,886 - Train: 14.38% [697300/4848000] [143.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 19:50:19,869 - Train: 14.39% [697400/4848000] [143.9/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 19:50:52,957 - Train: 14.39% [697500/4848000] [143.9/1000.0] [batch_t 0.323 (0.331)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 19:51:25,770 - Train: 14.39% [697600/4848000] [143.9/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 19:51:58,636 - Train: 14.39% [697700/4848000] [143.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 19:52:31,505 - Train: 14.39% [697800/4848000] [143.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 19:53:04,335 - Train: 14.40% [697900/4848000] [144.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 19:53:37,185 - Train: 14.40% [698000/4848000] [144.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 19:54:10,019 - Train: 14.40% [698100/4848000] [144.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 19:54:13,957 - ==> Total time: 2 days, 20:28:12 Eta: 16 days, 23:01:00 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 19:54:45,594 - Train: 14.40% [698200/4848000] [144.0/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 19:55:18,361 - Train: 14.40% [698300/4848000] [144.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 19:55:51,186 - Train: 14.41% [698400/4848000] [144.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 19:56:23,990 - Train: 14.41% [698500/4848000] [144.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 19:56:56,815 - Train: 14.41% [698600/4848000] [144.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 19:57:29,722 - Train: 14.41% [698700/4848000] [144.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 19:58:02,520 - Train: 14.41% [698800/4848000] [144.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 19:58:35,336 - Train: 14.42% [698900/4848000] [144.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 19:59:08,759 - Train: 14.42% [699000/4848000] [144.2/1000.0] [batch_t 0.334 (0.334)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 19:59:41,601 - Train: 14.42% [699100/4848000] [144.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 20:00:14,439 - Train: 14.42% [699200/4848000] [144.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 20:00:47,270 - Train: 14.42% [699300/4848000] [144.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 20:01:21,334 - Train: 14.43% [699400/4848000] [144.3/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 20:01:54,190 - Train: 14.43% [699500/4848000] [144.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 20:02:27,969 - Train: 14.43% [699600/4848000] [144.3/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 20:03:00,837 - Train: 14.43% [699700/4848000] [144.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 20:03:34,288 - Train: 14.43% [699800/4848000] [144.3/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 20:04:08,970 - Train: 14.44% [699900/4848000] [144.4/1000.0] [batch_t 0.326 (0.347)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 20:04:41,804 - Train: 14.44% [700000/4848000] [144.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 20:05:16,737 - Train: 14.44% [700100/4848000] [144.4/1000.0] [batch_t 0.328 (0.349)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 20:05:49,741 - Train: 14.44% [700200/4848000] [144.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 20:06:26,033 - Train: 14.45% [700300/4848000] [144.5/1000.0] [batch_t 0.328 (0.363)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 20:06:58,857 - Train: 14.45% [700400/4848000] [144.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 20:07:32,982 - Train: 14.45% [700500/4848000] [144.5/1000.0] [batch_t 0.335 (0.341)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-05 20:08:05,769 - Train: 14.45% [700600/4848000] [144.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 20:08:38,664 - Train: 14.45% [700700/4848000] [144.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 20:09:11,801 - Train: 14.46% [700800/4848000] [144.6/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 20:09:44,609 - Train: 14.46% [700900/4848000] [144.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 20:10:17,507 - Train: 14.46% [701000/4848000] [144.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 20:10:50,369 - Train: 14.46% [701100/4848000] [144.6/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 20:11:23,307 - Train: 14.46% [701200/4848000] [144.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 20:11:56,083 - Train: 14.47% [701300/4848000] [144.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 20:12:28,974 - Train: 14.47% [701400/4848000] [144.7/1000.0] [batch_t 0.337 (0.329)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-05 20:13:01,911 - Train: 14.47% [701500/4848000] [144.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 20:13:34,961 - Train: 14.47% [701600/4848000] [144.7/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 20:14:08,082 - Train: 14.47% [701700/4848000] [144.7/1000.0] [batch_t 0.324 (0.331)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 20:14:40,883 - Train: 14.48% [701800/4848000] [144.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 20:15:14,677 - Train: 14.48% [701900/4848000] [144.8/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 20:15:47,452 - Train: 14.48% [702000/4848000] [144.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 20:16:21,695 - Train: 14.48% [702100/4848000] [144.8/1000.0] [batch_t 0.327 (0.342)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 20:16:54,519 - Train: 14.48% [702200/4848000] [144.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 20:17:27,487 - Train: 14.49% [702300/4848000] [144.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 20:18:00,321 - Train: 14.49% [702400/4848000] [144.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 20:18:34,993 - Train: 14.49% [702500/4848000] [144.9/1000.0] [batch_t 0.325 (0.347)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 20:19:09,778 - Train: 14.49% [702600/4848000] [144.9/1000.0] [batch_t 0.323 (0.348)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 20:19:42,644 - Train: 14.49% [702700/4848000] [144.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 20:20:18,738 - Train: 14.50% [702800/4848000] [145.0/1000.0] [batch_t 0.328 (0.361)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 20:20:51,594 - Train: 14.50% [702900/4848000] [145.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 20:21:12,549 - ==> Total time: 2 days, 20:55:11 Eta: 16 days, 22:23:19 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 20:21:27,910 - Train: 14.50% [703000/4848000] [145.0/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-05 20:22:00,704 - Train: 14.50% [703100/4848000] [145.0/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 20:22:34,971 - Train: 14.50% [703200/4848000] [145.0/1000.0] [batch_t 0.335 (0.343)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-05 20:23:07,743 - Train: 14.51% [703300/4848000] [145.1/1000.0] [batch_t 0.320 (0.328)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-05 20:23:40,560 - Train: 14.51% [703400/4848000] [145.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 20:24:14,155 - Train: 14.51% [703500/4848000] [145.1/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 20:24:46,997 - Train: 14.51% [703600/4848000] [145.1/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 20:25:20,334 - Train: 14.52% [703700/4848000] [145.2/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 20:25:53,146 - Train: 14.52% [703800/4848000] [145.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 20:26:27,079 - Train: 14.52% [703900/4848000] [145.2/1000.0] [batch_t 0.324 (0.339)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 20:26:59,932 - Train: 14.52% [704000/4848000] [145.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 20:27:33,965 - Train: 14.52% [704100/4848000] [145.2/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 20:28:08,945 - Train: 14.53% [704200/4848000] [145.3/1000.0] [batch_t 0.328 (0.350)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 20:28:41,856 - Train: 14.53% [704300/4848000] [145.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 20:29:15,363 - Train: 14.53% [704400/4848000] [145.3/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 20:29:48,309 - Train: 14.53% [704500/4848000] [145.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 20:30:21,143 - Train: 14.53% [704600/4848000] [145.3/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 20:30:53,926 - Train: 14.54% [704700/4848000] [145.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 20:31:30,358 - Train: 14.54% [704800/4848000] [145.4/1000.0] [batch_t 0.329 (0.364)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 20:32:03,133 - Train: 14.54% [704900/4848000] [145.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 20:32:35,882 - Train: 14.54% [705000/4848000] [145.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 20:33:08,672 - Train: 14.54% [705100/4848000] [145.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 20:33:41,459 - Train: 14.55% [705200/4848000] [145.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 20:34:14,395 - Train: 14.55% [705300/4848000] [145.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 20:34:47,184 - Train: 14.55% [705400/4848000] [145.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 20:35:20,805 - Train: 14.55% [705500/4848000] [145.5/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 20:35:53,621 - Train: 14.55% [705600/4848000] [145.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 20:36:26,464 - Train: 14.56% [705700/4848000] [145.6/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 20:36:59,256 - Train: 14.56% [705800/4848000] [145.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 20:37:32,203 - Train: 14.56% [705900/4848000] [145.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 20:38:05,023 - Train: 14.56% [706000/4848000] [145.6/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 20:38:37,812 - Train: 14.56% [706100/4848000] [145.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 20:39:13,118 - Train: 14.57% [706200/4848000] [145.7/1000.0] [batch_t 0.329 (0.353)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 20:39:45,964 - Train: 14.57% [706300/4848000] [145.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 20:40:18,726 - Train: 14.57% [706400/4848000] [145.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 20:40:51,522 - Train: 14.57% [706500/4848000] [145.7/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-05 20:41:24,902 - Train: 14.58% [706600/4848000] [145.8/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 20:41:57,684 - Train: 14.58% [706700/4848000] [145.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 20:42:31,104 - Train: 14.58% [706800/4848000] [145.8/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 20:43:03,894 - Train: 14.58% [706900/4848000] [145.8/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 20:43:37,138 - Train: 14.58% [707000/4848000] [145.8/1000.0] [batch_t 0.336 (0.332)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-05 20:44:10,963 - Train: 14.59% [707100/4848000] [145.9/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 20:44:43,829 - Train: 14.59% [707200/4848000] [145.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 20:45:17,494 - Train: 14.59% [707300/4848000] [145.9/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 20:45:50,268 - Train: 14.59% [707400/4848000] [145.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 20:46:24,115 - Train: 14.59% [707500/4848000] [145.9/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 20:46:56,928 - Train: 14.60% [707600/4848000] [146.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 20:47:31,249 - Train: 14.60% [707700/4848000] [146.0/1000.0] [batch_t 0.325 (0.343)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 20:48:04,060 - Train: 14.60% [707800/4848000] [146.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 20:48:07,578 - ==> Total time: 2 days, 21:22:06 Eta: 16 days, 21:45:26 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 20:48:39,648 - Train: 14.60% [707900/4848000] [146.0/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 20:49:13,258 - Train: 14.60% [708000/4848000] [146.0/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 20:49:46,017 - Train: 14.61% [708100/4848000] [146.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 20:50:18,770 - Train: 14.61% [708200/4848000] [146.1/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 20:50:51,576 - Train: 14.61% [708300/4848000] [146.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 20:51:25,119 - Train: 14.61% [708400/4848000] [146.1/1000.0] [batch_t 0.332 (0.335)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 20:51:57,933 - Train: 14.61% [708500/4848000] [146.1/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 20:52:31,775 - Train: 14.62% [708600/4848000] [146.2/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 20:53:04,505 - Train: 14.62% [708700/4848000] [146.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 20:53:38,436 - Train: 14.62% [708800/4848000] [146.2/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 20:54:12,620 - Train: 14.62% [708900/4848000] [146.2/1000.0] [batch_t 0.325 (0.342)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 20:54:45,408 - Train: 14.62% [709000/4848000] [146.2/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 20:55:18,211 - Train: 14.63% [709100/4848000] [146.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 20:55:51,043 - Train: 14.63% [709200/4848000] [146.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 20:56:23,807 - Train: 14.63% [709300/4848000] [146.3/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 20:56:56,558 - Train: 14.63% [709400/4848000] [146.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 20:57:29,343 - Train: 14.63% [709500/4848000] [146.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 20:58:02,111 - Train: 14.64% [709600/4848000] [146.4/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 20:58:36,317 - Train: 14.64% [709700/4848000] [146.4/1000.0] [batch_t 0.326 (0.342)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 20:59:15,114 - Train: 14.64% [709800/4848000] [146.4/1000.0] [batch_t 0.324 (0.388)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 20:59:47,894 - Train: 14.64% [709900/4848000] [146.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 21:00:21,371 - Train: 14.65% [710000/4848000] [146.5/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 21:00:54,292 - Train: 14.65% [710100/4848000] [146.5/1000.0] [batch_t 0.318 (0.329)] [data_t 0.002] [optim_t 0.316] [lr 0.005000] 2024-04-05 21:01:28,716 - Train: 14.65% [710200/4848000] [146.5/1000.0] [batch_t 0.330 (0.344)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 21:02:01,464 - Train: 14.65% [710300/4848000] [146.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 21:02:34,341 - Train: 14.65% [710400/4848000] [146.5/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 21:03:08,421 - Train: 14.66% [710500/4848000] [146.6/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 21:03:41,194 - Train: 14.66% [710600/4848000] [146.6/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 21:04:16,561 - Train: 14.66% [710700/4848000] [146.6/1000.0] [batch_t 0.327 (0.354)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 21:04:49,456 - Train: 14.66% [710800/4848000] [146.6/1000.0] [batch_t 0.340 (0.329)] [data_t 0.002] [optim_t 0.338] [lr 0.005000] 2024-04-05 21:05:23,619 - Train: 14.66% [710900/4848000] [146.6/1000.0] [batch_t 0.331 (0.342)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 21:05:56,419 - Train: 14.67% [711000/4848000] [146.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 21:06:29,191 - Train: 14.67% [711100/4848000] [146.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 21:07:02,021 - Train: 14.67% [711200/4848000] [146.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 21:07:36,558 - Train: 14.67% [711300/4848000] [146.7/1000.0] [batch_t 0.327 (0.345)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 21:08:09,952 - Train: 14.67% [711400/4848000] [146.7/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 21:08:42,925 - Train: 14.68% [711500/4848000] [146.8/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 21:09:18,218 - Train: 14.68% [711600/4848000] [146.8/1000.0] [batch_t 0.328 (0.353)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 21:09:51,055 - Train: 14.68% [711700/4848000] [146.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 21:10:25,106 - Train: 14.68% [711800/4848000] [146.8/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 21:10:57,903 - Train: 14.68% [711900/4848000] [146.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 21:11:34,103 - Train: 14.69% [712000/4848000] [146.9/1000.0] [batch_t 0.326 (0.362)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 21:12:08,562 - Train: 14.69% [712100/4848000] [146.9/1000.0] [batch_t 0.327 (0.344)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 21:12:41,353 - Train: 14.69% [712200/4848000] [146.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 21:13:14,992 - Train: 14.69% [712300/4848000] [146.9/1000.0] [batch_t 0.326 (0.336)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 21:13:47,823 - Train: 14.69% [712400/4848000] [146.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 21:14:22,076 - Train: 14.70% [712500/4848000] [147.0/1000.0] [batch_t 0.326 (0.342)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 21:14:54,925 - Train: 14.70% [712600/4848000] [147.0/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 21:15:14,062 - ==> Total time: 2 days, 21:49:12 Eta: 16 days, 21:08:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 21:15:30,684 - Train: 14.70% [712700/4848000] [147.0/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 21:16:03,660 - Train: 14.70% [712800/4848000] [147.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 21:16:37,530 - Train: 14.71% [712900/4848000] [147.1/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 21:17:10,402 - Train: 14.71% [713000/4848000] [147.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 21:17:43,195 - Train: 14.71% [713100/4848000] [147.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 21:18:17,976 - Train: 14.71% [713200/4848000] [147.1/1000.0] [batch_t 0.327 (0.348)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 21:18:50,777 - Train: 14.71% [713300/4848000] [147.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 21:19:25,123 - Train: 14.72% [713400/4848000] [147.2/1000.0] [batch_t 0.329 (0.343)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 21:19:57,966 - Train: 14.72% [713500/4848000] [147.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 21:20:30,924 - Train: 14.72% [713600/4848000] [147.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 21:21:03,763 - Train: 14.72% [713700/4848000] [147.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 21:21:37,534 - Train: 14.72% [713800/4848000] [147.2/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 21:22:11,605 - Train: 14.73% [713900/4848000] [147.3/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 21:22:44,399 - Train: 14.73% [714000/4848000] [147.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 21:23:17,314 - Train: 14.73% [714100/4848000] [147.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 21:23:50,187 - Train: 14.73% [714200/4848000] [147.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 21:24:24,202 - Train: 14.73% [714300/4848000] [147.3/1000.0] [batch_t 0.334 (0.340)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 21:24:57,053 - Train: 14.74% [714400/4848000] [147.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 21:25:32,715 - Train: 14.74% [714500/4848000] [147.4/1000.0] [batch_t 0.327 (0.357)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 21:26:06,848 - Train: 14.74% [714600/4848000] [147.4/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 21:26:39,794 - Train: 14.74% [714700/4848000] [147.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 21:27:12,601 - Train: 14.74% [714800/4848000] [147.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 21:27:45,387 - Train: 14.75% [714900/4848000] [147.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 21:28:19,576 - Train: 14.75% [715000/4848000] [147.5/1000.0] [batch_t 0.332 (0.342)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 21:28:52,351 - Train: 14.75% [715100/4848000] [147.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 21:29:25,135 - Train: 14.75% [715200/4848000] [147.5/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 21:29:57,937 - Train: 14.75% [715300/4848000] [147.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 21:30:31,531 - Train: 14.76% [715400/4848000] [147.6/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 21:31:04,358 - Train: 14.76% [715500/4848000] [147.6/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 21:31:37,430 - Train: 14.76% [715600/4848000] [147.6/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 21:32:13,460 - Train: 14.76% [715700/4848000] [147.6/1000.0] [batch_t 0.328 (0.360)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 21:32:46,269 - Train: 14.76% [715800/4848000] [147.6/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 21:33:19,038 - Train: 14.77% [715900/4848000] [147.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 21:33:51,796 - Train: 14.77% [716000/4848000] [147.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 21:34:25,806 - Train: 14.77% [716100/4848000] [147.7/1000.0] [batch_t 0.323 (0.340)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-05 21:34:58,612 - Train: 14.77% [716200/4848000] [147.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 21:35:31,432 - Train: 14.78% [716300/4848000] [147.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 21:36:04,211 - Train: 14.78% [716400/4848000] [147.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 21:36:37,016 - Train: 14.78% [716500/4848000] [147.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 21:37:11,307 - Train: 14.78% [716600/4848000] [147.8/1000.0] [batch_t 0.327 (0.343)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 21:37:44,088 - Train: 14.78% [716700/4848000] [147.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 21:38:18,775 - Train: 14.79% [716800/4848000] [147.9/1000.0] [batch_t 0.330 (0.347)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 21:38:51,621 - Train: 14.79% [716900/4848000] [147.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 21:39:24,475 - Train: 14.79% [717000/4848000] [147.9/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 21:39:57,389 - Train: 14.79% [717100/4848000] [147.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 21:40:31,167 - Train: 14.79% [717200/4848000] [147.9/1000.0] [batch_t 0.334 (0.338)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 21:41:03,889 - Train: 14.80% [717300/4848000] [148.0/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 21:41:36,721 - Train: 14.80% [717400/4848000] [148.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 21:42:09,551 - Train: 14.80% [717500/4848000] [148.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 21:42:10,865 - ==> Total time: 2 days, 22:16:09 Eta: 16 days, 20:31:23 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 21:42:44,241 - Train: 14.80% [717600/4848000] [148.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 21:43:20,246 - Train: 14.80% [717700/4848000] [148.0/1000.0] [batch_t 0.324 (0.360)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 21:43:53,024 - Train: 14.81% [717800/4848000] [148.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 21:44:27,231 - Train: 14.81% [717900/4848000] [148.1/1000.0] [batch_t 0.325 (0.342)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 21:45:00,075 - Train: 14.81% [718000/4848000] [148.1/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 21:45:32,834 - Train: 14.81% [718100/4848000] [148.1/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 21:46:06,662 - Train: 14.81% [718200/4848000] [148.1/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 21:46:39,478 - Train: 14.82% [718300/4848000] [148.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 21:47:13,847 - Train: 14.82% [718400/4848000] [148.2/1000.0] [batch_t 0.328 (0.344)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 21:47:46,662 - Train: 14.82% [718500/4848000] [148.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 21:48:21,016 - Train: 14.82% [718600/4848000] [148.2/1000.0] [batch_t 0.330 (0.343)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 21:48:53,803 - Train: 14.82% [718700/4848000] [148.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 21:49:27,725 - Train: 14.83% [718800/4848000] [148.3/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 21:50:00,532 - Train: 14.83% [718900/4848000] [148.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 21:50:34,762 - Train: 14.83% [719000/4848000] [148.3/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 21:51:08,600 - Train: 14.83% [719100/4848000] [148.3/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 21:51:41,432 - Train: 14.83% [719200/4848000] [148.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 21:52:15,884 - Train: 14.84% [719300/4848000] [148.4/1000.0] [batch_t 0.328 (0.344)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 21:52:48,665 - Train: 14.84% [719400/4848000] [148.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 21:53:24,010 - Train: 14.84% [719500/4848000] [148.4/1000.0] [batch_t 0.332 (0.353)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 21:53:56,796 - Train: 14.84% [719600/4848000] [148.4/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 21:54:31,800 - Train: 14.85% [719700/4848000] [148.5/1000.0] [batch_t 0.332 (0.350)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 21:55:04,606 - Train: 14.85% [719800/4848000] [148.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 21:55:38,479 - Train: 14.85% [719900/4848000] [148.5/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 21:56:12,650 - Train: 14.85% [720000/4848000] [148.5/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 21:56:45,468 - Train: 14.85% [720100/4848000] [148.5/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 21:57:19,850 - Train: 14.86% [720200/4848000] [148.6/1000.0] [batch_t 0.327 (0.344)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 21:57:52,711 - Train: 14.86% [720300/4848000] [148.6/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 21:58:26,132 - Train: 14.86% [720400/4848000] [148.6/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 21:58:58,995 - Train: 14.86% [720500/4848000] [148.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 21:59:31,845 - Train: 14.86% [720600/4848000] [148.6/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 22:00:06,378 - Train: 14.87% [720700/4848000] [148.7/1000.0] [batch_t 2.058 (0.345)] [data_t 1.735] [optim_t 0.323] [lr 0.005000] 2024-04-05 22:00:39,184 - Train: 14.87% [720800/4848000] [148.7/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 22:01:11,923 - Train: 14.87% [720900/4848000] [148.7/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 22:01:44,719 - Train: 14.87% [721000/4848000] [148.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 22:02:18,875 - Train: 14.87% [721100/4848000] [148.7/1000.0] [batch_t 0.333 (0.341)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-05 22:02:51,751 - Train: 14.88% [721200/4848000] [148.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 22:03:26,707 - Train: 14.88% [721300/4848000] [148.8/1000.0] [batch_t 0.330 (0.349)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 22:03:59,554 - Train: 14.88% [721400/4848000] [148.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 22:04:34,493 - Train: 14.88% [721500/4848000] [148.8/1000.0] [batch_t 0.328 (0.349)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 22:05:07,283 - Train: 14.88% [721600/4848000] [148.8/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 22:05:40,133 - Train: 14.89% [721700/4848000] [148.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 22:06:12,936 - Train: 14.89% [721800/4848000] [148.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 22:06:45,759 - Train: 14.89% [721900/4848000] [148.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 22:07:20,605 - Train: 14.89% [722000/4848000] [148.9/1000.0] [batch_t 0.329 (0.348)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 22:07:53,413 - Train: 14.89% [722100/4848000] [148.9/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 22:08:26,249 - Train: 14.90% [722200/4848000] [149.0/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-05 22:08:59,110 - Train: 14.90% [722300/4848000] [149.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 22:09:17,089 - ==> Total time: 2 days, 22:43:15 Eta: 16 days, 19:54:59 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 22:09:34,810 - Train: 14.90% [722400/4848000] [149.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 22:10:08,925 - Train: 14.90% [722500/4848000] [149.0/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 22:10:41,883 - Train: 14.91% [722600/4848000] [149.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 22:11:17,551 - Train: 14.91% [722700/4848000] [149.1/1000.0] [batch_t 0.324 (0.357)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 22:11:50,358 - Train: 14.91% [722800/4848000] [149.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 22:12:23,995 - Train: 14.91% [722900/4848000] [149.1/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 22:12:56,791 - Train: 14.91% [723000/4848000] [149.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 22:13:30,713 - Train: 14.92% [723100/4848000] [149.2/1000.0] [batch_t 0.324 (0.339)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 22:14:03,551 - Train: 14.92% [723200/4848000] [149.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 22:14:37,819 - Train: 14.92% [723300/4848000] [149.2/1000.0] [batch_t 0.329 (0.343)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 22:15:12,341 - Train: 14.92% [723400/4848000] [149.2/1000.0] [batch_t 0.330 (0.345)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 22:15:45,169 - Train: 14.92% [723500/4848000] [149.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 22:16:19,289 - Train: 14.93% [723600/4848000] [149.3/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 22:16:52,109 - Train: 14.93% [723700/4848000] [149.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-05 22:17:26,541 - Train: 14.93% [723800/4848000] [149.3/1000.0] [batch_t 0.326 (0.344)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 22:17:59,282 - Train: 14.93% [723900/4848000] [149.3/1000.0] [batch_t 0.321 (0.327)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-05 22:18:32,052 - Train: 14.93% [724000/4848000] [149.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 22:19:06,142 - Train: 14.94% [724100/4848000] [149.4/1000.0] [batch_t 0.326 (0.341)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 22:19:38,935 - Train: 14.94% [724200/4848000] [149.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 22:20:12,677 - Train: 14.94% [724300/4848000] [149.4/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 22:20:45,473 - Train: 14.94% [724400/4848000] [149.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 22:21:18,269 - Train: 14.94% [724500/4848000] [149.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 22:21:51,105 - Train: 14.95% [724600/4848000] [149.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 22:22:23,933 - Train: 14.95% [724700/4848000] [149.5/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 22:22:56,881 - Train: 14.95% [724800/4848000] [149.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 22:23:31,211 - Train: 14.95% [724900/4848000] [149.5/1000.0] [batch_t 0.326 (0.343)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 22:24:04,056 - Train: 14.95% [725000/4848000] [149.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 22:24:36,937 - Train: 14.96% [725100/4848000] [149.6/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 22:25:09,743 - Train: 14.96% [725200/4848000] [149.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 22:25:42,550 - Train: 14.96% [725300/4848000] [149.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 22:26:15,484 - Train: 14.96% [725400/4848000] [149.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-05 22:26:48,463 - Train: 14.96% [725500/4848000] [149.6/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 22:27:21,294 - Train: 14.97% [725600/4848000] [149.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 22:27:54,129 - Train: 14.97% [725700/4848000] [149.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 22:28:28,305 - Train: 14.97% [725800/4848000] [149.7/1000.0] [batch_t 0.324 (0.342)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-05 22:29:01,118 - Train: 14.97% [725900/4848000] [149.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 22:29:34,586 - Train: 14.98% [726000/4848000] [149.8/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 22:30:08,202 - Train: 14.98% [726100/4848000] [149.8/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 22:30:41,007 - Train: 14.98% [726200/4848000] [149.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 22:31:14,326 - Train: 14.98% [726300/4848000] [149.8/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 22:31:47,155 - Train: 14.98% [726400/4848000] [149.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-05 22:32:20,741 - Train: 14.99% [726500/4848000] [149.9/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-05 22:32:53,573 - Train: 14.99% [726600/4848000] [149.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 22:33:27,347 - Train: 14.99% [726700/4848000] [149.9/1000.0] [batch_t 0.325 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 22:34:00,148 - Train: 14.99% [726800/4848000] [149.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 22:34:34,231 - Train: 14.99% [726900/4848000] [149.9/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-05 22:35:07,671 - Train: 15.00% [727000/4848000] [150.0/1000.0] [batch_t 0.326 (0.334)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-05 22:35:40,501 - Train: 15.00% [727100/4848000] [150.0/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-05 22:36:14,184 - Train: 15.00% [727200/4848000] [150.0/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-05 22:36:23,195 - Test: 16.13% [50/310] [batch_t 0.151 (0.162)] 2024-04-05 22:36:30,970 - Test: 32.26% [100/310] [batch_t 0.161 (0.159)] 2024-04-05 22:36:38,674 - Test: 48.39% [150/310] [batch_t 0.156 (0.157)] 2024-04-05 22:36:46,357 - Test: 64.52% [200/310] [batch_t 0.155 (0.156)] 2024-04-05 22:36:54,028 - Test: 80.65% [250/310] [batch_t 0.157 (0.156)] 2024-04-05 22:37:01,767 - Test: 96.77% [300/310] [batch_t 0.154 (0.156)] 2024-04-05 22:37:04,831 - Test: 100.00% [310/310] [batch_t 0.084 (0.160)] 2024-04-05 23:02:34,930 - ==> Metric Time for coco : 0.004 (mAUROC_sp_max) 0.001 (mAP_sp_max) 0.001 (mF1_max_sp_max) 342.941 (mAUROC_px) 271.933 (mAP_px) 32.072 (mF1_max_px) 804.671 (mAUPRO_px) 11.762 (mF1_px_0.2_0.8_0.1) 11.739 (mAcc_px_0.2_0.8_0.1) 11.678 (mIoU_px_0.2_0.8_0.1) 34.062 (mIoU_max_px) 2024-04-05 23:02:35,366 - | Name | mAUROC_sp_max | mAUROC_sp_max (Max) | mAP_sp_max | mAP_sp_max (Max) | mF1_max_sp_max | mF1_max_sp_max (Max) | mAUROC_px | mAUROC_px (Max) | mAP_px | mAP_px (Max) | mF1_max_px | mF1_max_px (Max) | mAUPRO_px | mAUPRO_px (Max) | mF1_px_0.2_0.8_0.1 | mF1_px_0.2_0.8_0.1 (Max) | mAcc_px_0.2_0.8_0.1 | mAcc_px_0.2_0.8_0.1 (Max) | mIoU_px_0.2_0.8_0.1 | mIoU_px_0.2_0.8_0.1 (Max) | mIoU_max_px | mIoU_max_px (Max) | |:------:|:---------------:|:---------------------:|:------------:|:------------------:|:----------------:|:----------------------:|:-----------:|:------------------:|:--------:|:-----------------:|:------------:|:------------------:|:-----------:|:------------------:|:--------------------:|:--------------------------:|:---------------------:|:---------------------------:|:---------------------:|:---------------------------:|:-------------:|:-------------------:| | coco | 55.522 | 58.404 (50 epoch) | 37.354 | 40.479 (50 epoch) | 51.574 | 51.858 (50 epoch) | 59.361 | 59.488 (100 epoch) | 8.631 | 8.714 (100 epoch) | 14.435 | 14.504 (100 epoch) | 30.490 | 34.736 (50 epoch) | 8.555 | 9.008 (50 epoch) | 37.639 | 38.999 (50 epoch) | 4.568 | 4.810 (50 epoch) | 7.779 | 7.819 (100 epoch) | 2024-04-05 23:02:35,987 - ==> Total time: 2 days, 23:36:34 Eta: 16 days, 21:47:15 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-05 23:03:54,371 - Train: 15.00% [727300/4848000] [150.0/1000.0] [batch_t 0.749 (0.757)] [data_t 0.002] [optim_t 0.747] [lr 0.005000] 2024-04-05 23:05:10,117 - Train: 15.00% [727400/4848000] [150.0/1000.0] [batch_t 0.753 (0.757)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-05 23:06:25,885 - Train: 15.01% [727500/4848000] [150.1/1000.0] [batch_t 0.763 (0.758)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-05 23:07:41,663 - Train: 15.01% [727600/4848000] [150.1/1000.0] [batch_t 0.753 (0.758)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-05 23:08:57,538 - Train: 15.01% [727700/4848000] [150.1/1000.0] [batch_t 0.764 (0.759)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-05 23:10:13,347 - Train: 15.01% [727800/4848000] [150.1/1000.0] [batch_t 0.762 (0.758)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-05 23:11:29,101 - Train: 15.01% [727900/4848000] [150.1/1000.0] [batch_t 0.754 (0.757)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-05 23:12:44,900 - Train: 15.02% [728000/4848000] [150.2/1000.0] [batch_t 0.755 (0.758)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-05 23:14:00,595 - Train: 15.02% [728100/4848000] [150.2/1000.0] [batch_t 0.759 (0.757)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-05 23:15:16,377 - Train: 15.02% [728200/4848000] [150.2/1000.0] [batch_t 0.758 (0.758)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-05 23:16:32,162 - Train: 15.02% [728300/4848000] [150.2/1000.0] [batch_t 0.750 (0.758)] [data_t 0.002] [optim_t 0.748] [lr 0.005000] 2024-04-05 23:17:47,846 - Train: 15.02% [728400/4848000] [150.2/1000.0] [batch_t 0.755 (0.757)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-05 23:19:03,547 - Train: 15.03% [728500/4848000] [150.3/1000.0] [batch_t 0.764 (0.757)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-05 23:20:19,257 - Train: 15.03% [728600/4848000] [150.3/1000.0] [batch_t 0.745 (0.757)] [data_t 0.002] [optim_t 0.742] [lr 0.005000] 2024-04-05 23:21:35,085 - Train: 15.03% [728700/4848000] [150.3/1000.0] [batch_t 0.754 (0.758)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-05 23:22:50,901 - Train: 15.03% [728800/4848000] [150.3/1000.0] [batch_t 0.768 (0.758)] [data_t 0.002] [optim_t 0.766] [lr 0.005000] 2024-04-05 23:24:06,725 - Train: 15.04% [728900/4848000] [150.4/1000.0] [batch_t 0.764 (0.758)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-05 23:25:22,520 - Train: 15.04% [729000/4848000] [150.4/1000.0] [batch_t 0.760 (0.758)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-05 23:26:38,151 - Train: 15.04% [729100/4848000] [150.4/1000.0] [batch_t 0.755 (0.756)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-05 23:27:53,948 - Train: 15.04% [729200/4848000] [150.4/1000.0] [batch_t 0.759 (0.758)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-05 23:29:09,743 - Train: 15.04% [729300/4848000] [150.4/1000.0] [batch_t 0.754 (0.758)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-05 23:30:25,470 - Train: 15.05% [729400/4848000] [150.5/1000.0] [batch_t 0.759 (0.757)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-05 23:31:41,206 - Train: 15.05% [729500/4848000] [150.5/1000.0] [batch_t 0.750 (0.757)] [data_t 0.002] [optim_t 0.747] [lr 0.005000] 2024-04-05 23:32:57,045 - Train: 15.05% [729600/4848000] [150.5/1000.0] [batch_t 0.754 (0.758)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-05 23:34:12,810 - Train: 15.05% [729700/4848000] [150.5/1000.0] [batch_t 0.754 (0.758)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-05 23:35:28,575 - Train: 15.05% [729800/4848000] [150.5/1000.0] [batch_t 0.768 (0.758)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-05 23:36:44,380 - Train: 15.06% [729900/4848000] [150.6/1000.0] [batch_t 0.758 (0.758)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-05 23:38:00,095 - Train: 15.06% [730000/4848000] [150.6/1000.0] [batch_t 0.755 (0.757)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-05 23:39:15,802 - Train: 15.06% [730100/4848000] [150.6/1000.0] [batch_t 0.757 (0.757)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-05 23:40:31,603 - Train: 15.06% [730200/4848000] [150.6/1000.0] [batch_t 0.756 (0.758)] [data_t 0.002] [optim_t 0.754] [lr 0.005000] 2024-04-05 23:41:47,274 - Train: 15.06% [730300/4848000] [150.6/1000.0] [batch_t 0.755 (0.757)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-05 23:43:03,032 - Train: 15.07% [730400/4848000] [150.7/1000.0] [batch_t 0.758 (0.757)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-05 23:44:18,851 - Train: 15.07% [730500/4848000] [150.7/1000.0] [batch_t 0.749 (0.758)] [data_t 0.002] [optim_t 0.747] [lr 0.005000] 2024-04-05 23:45:34,585 - Train: 15.07% [730600/4848000] [150.7/1000.0] [batch_t 0.755 (0.757)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-05 23:46:50,460 - Train: 15.07% [730700/4848000] [150.7/1000.0] [batch_t 0.767 (0.759)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-05 23:48:06,113 - Train: 15.07% [730800/4848000] [150.7/1000.0] [batch_t 0.753 (0.756)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-05 23:49:21,899 - Train: 15.08% [730900/4848000] [150.8/1000.0] [batch_t 0.758 (0.758)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-05 23:50:37,704 - Train: 15.08% [731000/4848000] [150.8/1000.0] [batch_t 0.759 (0.758)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-05 23:51:53,437 - Train: 15.08% [731100/4848000] [150.8/1000.0] [batch_t 0.747 (0.757)] [data_t 0.002] [optim_t 0.745] [lr 0.005000] 2024-04-05 23:53:09,243 - Train: 15.08% [731200/4848000] [150.8/1000.0] [batch_t 0.759 (0.758)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-05 23:54:24,952 - Train: 15.08% [731300/4848000] [150.8/1000.0] [batch_t 0.750 (0.757)] [data_t 0.002] [optim_t 0.748] [lr 0.005000] 2024-04-05 23:55:40,804 - Train: 15.09% [731400/4848000] [150.9/1000.0] [batch_t 0.762 (0.758)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-05 23:56:56,624 - Train: 15.09% [731500/4848000] [150.9/1000.0] [batch_t 0.758 (0.758)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-05 23:58:12,237 - Train: 15.09% [731600/4848000] [150.9/1000.0] [batch_t 0.764 (0.756)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-05 23:59:27,917 - Train: 15.09% [731700/4848000] [150.9/1000.0] [batch_t 0.760 (0.757)] [data_t 0.002] [optim_t 0.758] [lr 0.005000] 2024-04-06 00:00:43,680 - Train: 15.09% [731800/4848000] [150.9/1000.0] [batch_t 0.758 (0.758)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-06 00:01:59,461 - Train: 15.10% [731900/4848000] [151.0/1000.0] [batch_t 0.753 (0.758)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-06 00:03:15,197 - Train: 15.10% [732000/4848000] [151.0/1000.0] [batch_t 0.764 (0.757)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 00:03:51,629 - ==> Total time: 3 days, 0:37:50 Eta: 17 days, 0:22:00 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 00:04:33,103 - Train: 15.10% [732100/4848000] [151.0/1000.0] [batch_t 0.760 (0.757)] [data_t 0.002] [optim_t 0.758] [lr 0.005000] 2024-04-06 00:05:48,916 - Train: 15.10% [732200/4848000] [151.0/1000.0] [batch_t 0.759 (0.758)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-06 00:07:04,722 - Train: 15.11% [732300/4848000] [151.1/1000.0] [batch_t 0.759 (0.758)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-06 00:08:20,583 - Train: 15.11% [732400/4848000] [151.1/1000.0] [batch_t 0.756 (0.759)] [data_t 0.002] [optim_t 0.754] [lr 0.005000] 2024-04-06 00:09:36,450 - Train: 15.11% [732500/4848000] [151.1/1000.0] [batch_t 0.759 (0.759)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-06 00:10:52,238 - Train: 15.11% [732600/4848000] [151.1/1000.0] [batch_t 0.763 (0.758)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-06 00:12:08,072 - Train: 15.11% [732700/4848000] [151.1/1000.0] [batch_t 0.759 (0.758)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-06 00:13:23,786 - Train: 15.12% [732800/4848000] [151.2/1000.0] [batch_t 0.755 (0.757)] [data_t 0.002] [optim_t 0.753] [lr 0.005000] 2024-04-06 00:14:39,609 - Train: 15.12% [732900/4848000] [151.2/1000.0] [batch_t 0.758 (0.758)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-06 00:15:55,393 - Train: 15.12% [733000/4848000] [151.2/1000.0] [batch_t 0.767 (0.758)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-06 00:17:11,122 - Train: 15.12% [733100/4848000] [151.2/1000.0] [batch_t 0.758 (0.757)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-06 00:18:26,747 - Train: 15.12% [733200/4848000] [151.2/1000.0] [batch_t 0.753 (0.756)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-06 00:19:42,548 - Train: 15.13% [733300/4848000] [151.3/1000.0] [batch_t 0.756 (0.758)] [data_t 0.002] [optim_t 0.754] [lr 0.005000] 2024-04-06 00:20:58,264 - Train: 15.13% [733400/4848000] [151.3/1000.0] [batch_t 0.764 (0.757)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-06 00:22:14,047 - Train: 15.13% [733500/4848000] [151.3/1000.0] [batch_t 0.763 (0.758)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-06 00:23:29,678 - Train: 15.13% [733600/4848000] [151.3/1000.0] [batch_t 0.741 (0.756)] [data_t 0.002] [optim_t 0.739] [lr 0.005000] 2024-04-06 00:24:45,446 - Train: 15.13% [733700/4848000] [151.3/1000.0] [batch_t 0.743 (0.758)] [data_t 0.002] [optim_t 0.741] [lr 0.005000] 2024-04-06 00:26:01,236 - Train: 15.14% [733800/4848000] [151.4/1000.0] [batch_t 0.760 (0.758)] [data_t 0.002] [optim_t 0.758] [lr 0.005000] 2024-04-06 00:27:17,008 - Train: 15.14% [733900/4848000] [151.4/1000.0] [batch_t 0.759 (0.758)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-06 00:28:32,823 - Train: 15.14% [734000/4848000] [151.4/1000.0] [batch_t 0.753 (0.758)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-06 00:29:48,531 - Train: 15.14% [734100/4848000] [151.4/1000.0] [batch_t 0.748 (0.757)] [data_t 0.002] [optim_t 0.746] [lr 0.005000] 2024-04-06 00:30:51,289 - Train: 15.14% [734200/4848000] [151.4/1000.0] [batch_t 0.329 (0.627)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 00:31:24,098 - Train: 15.15% [734300/4848000] [151.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 00:31:56,880 - Train: 15.15% [734400/4848000] [151.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 00:32:29,633 - Train: 15.15% [734500/4848000] [151.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 00:33:02,464 - Train: 15.15% [734600/4848000] [151.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 00:33:37,101 - Train: 15.15% [734700/4848000] [151.5/1000.0] [batch_t 0.331 (0.346)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 00:34:11,411 - Train: 15.16% [734800/4848000] [151.6/1000.0] [batch_t 0.327 (0.343)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 00:34:44,241 - Train: 15.16% [734900/4848000] [151.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 00:35:17,596 - Train: 15.16% [735000/4848000] [151.6/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 00:35:50,339 - Train: 15.16% [735100/4848000] [151.6/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 00:36:23,112 - Train: 15.17% [735200/4848000] [151.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 00:36:55,969 - Train: 15.17% [735300/4848000] [151.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 00:37:30,020 - Train: 15.17% [735400/4848000] [151.7/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 00:38:02,764 - Train: 15.17% [735500/4848000] [151.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 00:38:36,993 - Train: 15.17% [735600/4848000] [151.7/1000.0] [batch_t 0.327 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 00:39:10,908 - Train: 15.18% [735700/4848000] [151.8/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 00:39:43,794 - Train: 15.18% [735800/4848000] [151.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 00:40:17,944 - Train: 15.18% [735900/4848000] [151.8/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 00:40:50,753 - Train: 15.18% [736000/4848000] [151.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 00:41:25,010 - Train: 15.18% [736100/4848000] [151.8/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 00:41:57,834 - Train: 15.19% [736200/4848000] [151.9/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 00:42:32,054 - Train: 15.19% [736300/4848000] [151.9/1000.0] [batch_t 0.325 (0.342)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 00:43:06,171 - Train: 15.19% [736400/4848000] [151.9/1000.0] [batch_t 0.325 (0.341)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 00:43:38,976 - Train: 15.19% [736500/4848000] [151.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 00:44:12,383 - Train: 15.19% [736600/4848000] [151.9/1000.0] [batch_t 0.323 (0.334)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 00:44:45,461 - Train: 15.20% [736700/4848000] [152.0/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 00:45:19,709 - Train: 15.20% [736800/4848000] [152.0/1000.0] [batch_t 0.332 (0.342)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 00:45:51,264 - ==> Total time: 3 days, 1:19:49 Eta: 17 days, 1:06:24 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 00:45:54,604 - Train: 15.20% [736900/4848000] [152.0/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 00:46:29,114 - Train: 15.20% [737000/4848000] [152.0/1000.0] [batch_t 0.329 (0.345)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 00:47:01,993 - Train: 15.20% [737100/4848000] [152.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 00:47:36,501 - Train: 15.21% [737200/4848000] [152.1/1000.0] [batch_t 0.329 (0.345)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 00:48:11,945 - Train: 15.21% [737300/4848000] [152.1/1000.0] [batch_t 0.323 (0.354)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 00:48:44,759 - Train: 15.21% [737400/4848000] [152.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 00:49:18,390 - Train: 15.21% [737500/4848000] [152.1/1000.0] [batch_t 0.325 (0.336)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 00:49:51,266 - Train: 15.21% [737600/4848000] [152.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 00:50:25,011 - Train: 15.22% [737700/4848000] [152.2/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 00:50:57,860 - Train: 15.22% [737800/4848000] [152.2/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-06 00:51:31,797 - Train: 15.22% [737900/4848000] [152.2/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 00:52:07,248 - Train: 15.22% [738000/4848000] [152.2/1000.0] [batch_t 2.945 (0.354)] [data_t 2.617] [optim_t 0.328] [lr 0.005000] 2024-04-06 00:52:40,313 - Train: 15.22% [738100/4848000] [152.2/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 00:53:14,394 - Train: 15.23% [738200/4848000] [152.3/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 00:53:47,236 - Train: 15.23% [738300/4848000] [152.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 00:54:21,647 - Train: 15.23% [738400/4848000] [152.3/1000.0] [batch_t 0.330 (0.344)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 00:54:54,445 - Train: 15.23% [738500/4848000] [152.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 00:55:29,363 - Train: 15.24% [738600/4848000] [152.4/1000.0] [batch_t 0.332 (0.349)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 00:56:02,206 - Train: 15.24% [738700/4848000] [152.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 00:56:36,287 - Train: 15.24% [738800/4848000] [152.4/1000.0] [batch_t 0.325 (0.341)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 00:57:09,243 - Train: 15.24% [738900/4848000] [152.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 00:57:42,116 - Train: 15.24% [739000/4848000] [152.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 00:58:15,903 - Train: 15.25% [739100/4848000] [152.5/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 00:58:48,727 - Train: 15.25% [739200/4848000] [152.5/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 00:59:22,337 - Train: 15.25% [739300/4848000] [152.5/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 00:59:55,158 - Train: 15.25% [739400/4848000] [152.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 01:00:28,138 - Train: 15.25% [739500/4848000] [152.5/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 01:01:00,927 - Train: 15.26% [739600/4848000] [152.6/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 01:01:35,053 - Train: 15.26% [739700/4848000] [152.6/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 01:02:09,418 - Train: 15.26% [739800/4848000] [152.6/1000.0] [batch_t 0.325 (0.344)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 01:02:42,267 - Train: 15.26% [739900/4848000] [152.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 01:03:16,583 - Train: 15.26% [740000/4848000] [152.6/1000.0] [batch_t 0.327 (0.343)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 01:03:49,351 - Train: 15.27% [740100/4848000] [152.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 01:04:24,985 - Train: 15.27% [740200/4848000] [152.7/1000.0] [batch_t 0.324 (0.356)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 01:04:57,727 - Train: 15.27% [740300/4848000] [152.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 01:05:30,476 - Train: 15.27% [740400/4848000] [152.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 01:06:03,290 - Train: 15.27% [740500/4848000] [152.7/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-06 01:06:37,241 - Train: 15.28% [740600/4848000] [152.8/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 01:07:11,711 - Train: 15.28% [740700/4848000] [152.8/1000.0] [batch_t 0.328 (0.345)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 01:07:44,498 - Train: 15.28% [740800/4848000] [152.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 01:08:17,456 - Train: 15.28% [740900/4848000] [152.8/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 01:08:50,241 - Train: 15.28% [741000/4848000] [152.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 01:09:24,211 - Train: 15.29% [741100/4848000] [152.9/1000.0] [batch_t 0.331 (0.340)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 01:09:56,983 - Train: 15.29% [741200/4848000] [152.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 01:10:32,215 - Train: 15.29% [741300/4848000] [152.9/1000.0] [batch_t 0.328 (0.352)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 01:11:06,565 - Train: 15.29% [741400/4848000] [152.9/1000.0] [batch_t 0.328 (0.343)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 01:11:39,615 - Train: 15.29% [741500/4848000] [152.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 01:12:13,943 - Train: 15.30% [741600/4848000] [153.0/1000.0] [batch_t 0.326 (0.343)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 01:12:46,725 - Train: 15.30% [741700/4848000] [153.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 01:13:01,128 - ==> Total time: 3 days, 1:46:59 Eta: 17 days, 0:27:36 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 01:13:24,918 - Train: 15.30% [741800/4848000] [153.0/1000.0] [batch_t 0.327 (0.388)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 01:13:57,766 - Train: 15.30% [741900/4848000] [153.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 01:14:33,183 - Train: 15.31% [742000/4848000] [153.1/1000.0] [batch_t 0.330 (0.354)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 01:15:07,017 - Train: 15.31% [742100/4848000] [153.1/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 01:15:39,984 - Train: 15.31% [742200/4848000] [153.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 01:16:14,143 - Train: 15.31% [742300/4848000] [153.1/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 01:16:47,052 - Train: 15.31% [742400/4848000] [153.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 01:17:20,887 - Train: 15.32% [742500/4848000] [153.2/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 01:17:53,742 - Train: 15.32% [742600/4848000] [153.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 01:18:27,873 - Train: 15.32% [742700/4848000] [153.2/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 01:19:00,739 - Train: 15.32% [742800/4848000] [153.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 01:19:34,695 - Train: 15.32% [742900/4848000] [153.2/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 01:20:08,759 - Train: 15.33% [743000/4848000] [153.3/1000.0] [batch_t 0.332 (0.341)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 01:20:41,535 - Train: 15.33% [743100/4848000] [153.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 01:21:15,918 - Train: 15.33% [743200/4848000] [153.3/1000.0] [batch_t 0.328 (0.344)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 01:21:48,679 - Train: 15.33% [743300/4848000] [153.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 01:22:22,958 - Train: 15.33% [743400/4848000] [153.3/1000.0] [batch_t 0.330 (0.343)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 01:22:55,712 - Train: 15.34% [743500/4848000] [153.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 01:23:30,200 - Train: 15.34% [743600/4848000] [153.4/1000.0] [batch_t 0.330 (0.345)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 01:24:03,142 - Train: 15.34% [743700/4848000] [153.4/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 01:24:36,814 - Train: 15.34% [743800/4848000] [153.4/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 01:25:10,344 - Train: 15.34% [743900/4848000] [153.4/1000.0] [batch_t 0.331 (0.335)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 01:25:43,155 - Train: 15.35% [744000/4848000] [153.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 01:26:17,514 - Train: 15.35% [744100/4848000] [153.5/1000.0] [batch_t 0.329 (0.344)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 01:26:50,333 - Train: 15.35% [744200/4848000] [153.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 01:27:24,613 - Train: 15.35% [744300/4848000] [153.5/1000.0] [batch_t 0.329 (0.343)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 01:27:57,362 - Train: 15.35% [744400/4848000] [153.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 01:28:32,835 - Train: 15.36% [744500/4848000] [153.6/1000.0] [batch_t 0.330 (0.355)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 01:29:08,370 - Train: 15.36% [744600/4848000] [153.6/1000.0] [batch_t 0.327 (0.355)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 01:29:41,149 - Train: 15.36% [744700/4848000] [153.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 01:30:16,579 - Train: 15.36% [744800/4848000] [153.6/1000.0] [batch_t 0.330 (0.354)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 01:30:49,377 - Train: 15.37% [744900/4848000] [153.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 01:31:23,077 - Train: 15.37% [745000/4848000] [153.7/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 01:31:56,001 - Train: 15.37% [745100/4848000] [153.7/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 01:32:28,753 - Train: 15.37% [745200/4848000] [153.7/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 01:33:01,514 - Train: 15.37% [745300/4848000] [153.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 01:33:35,034 - Train: 15.38% [745400/4848000] [153.8/1000.0] [batch_t 0.332 (0.335)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 01:34:08,711 - Train: 15.38% [745500/4848000] [153.8/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 01:34:41,509 - Train: 15.38% [745600/4848000] [153.8/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 01:35:15,130 - Train: 15.38% [745700/4848000] [153.8/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 01:35:47,961 - Train: 15.38% [745800/4848000] [153.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 01:36:21,839 - Train: 15.39% [745900/4848000] [153.9/1000.0] [batch_t 0.331 (0.339)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 01:36:54,677 - Train: 15.39% [746000/4848000] [153.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 01:37:28,755 - Train: 15.39% [746100/4848000] [153.9/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 01:38:01,622 - Train: 15.39% [746200/4848000] [153.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 01:38:34,542 - Train: 15.39% [746300/4848000] [153.9/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 01:39:08,655 - Train: 15.40% [746400/4848000] [154.0/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 01:39:41,635 - Train: 15.40% [746500/4848000] [154.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 01:40:13,068 - ==> Total time: 3 days, 2:14:11 Eta: 16 days, 23:49:08 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 01:40:17,904 - Train: 15.40% [746600/4848000] [154.0/1000.0] [batch_t 0.330 (0.352)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 01:40:50,635 - Train: 15.40% [746700/4848000] [154.0/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 01:41:25,112 - Train: 15.40% [746800/4848000] [154.0/1000.0] [batch_t 0.328 (0.345)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 01:41:57,874 - Train: 15.41% [746900/4848000] [154.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 01:42:30,677 - Train: 15.41% [747000/4848000] [154.1/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 01:43:03,496 - Train: 15.41% [747100/4848000] [154.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 01:43:37,484 - Train: 15.41% [747200/4848000] [154.1/1000.0] [batch_t 0.323 (0.340)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 01:44:11,158 - Train: 15.41% [747300/4848000] [154.1/1000.0] [batch_t 0.325 (0.337)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 01:44:43,953 - Train: 15.42% [747400/4848000] [154.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 01:45:17,686 - Train: 15.42% [747500/4848000] [154.2/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 01:45:50,483 - Train: 15.42% [747600/4848000] [154.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 01:46:24,436 - Train: 15.42% [747700/4848000] [154.2/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 01:46:57,372 - Train: 15.42% [747800/4848000] [154.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 01:47:30,556 - Train: 15.43% [747900/4848000] [154.3/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 01:48:03,319 - Train: 15.43% [748000/4848000] [154.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 01:48:38,067 - Train: 15.43% [748100/4848000] [154.3/1000.0] [batch_t 0.330 (0.347)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 01:49:11,720 - Train: 15.43% [748200/4848000] [154.3/1000.0] [batch_t 0.326 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 01:49:44,529 - Train: 15.44% [748300/4848000] [154.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 01:50:18,239 - Train: 15.44% [748400/4848000] [154.4/1000.0] [batch_t 0.326 (0.337)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 01:50:51,021 - Train: 15.44% [748500/4848000] [154.4/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 01:51:24,703 - Train: 15.44% [748600/4848000] [154.4/1000.0] [batch_t 0.326 (0.337)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 01:51:57,573 - Train: 15.44% [748700/4848000] [154.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 01:52:30,431 - Train: 15.45% [748800/4848000] [154.5/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-06 01:53:03,230 - Train: 15.45% [748900/4848000] [154.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 01:53:37,056 - Train: 15.45% [749000/4848000] [154.5/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 01:54:09,807 - Train: 15.45% [749100/4848000] [154.5/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 01:54:42,579 - Train: 15.45% [749200/4848000] [154.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 01:55:16,333 - Train: 15.46% [749300/4848000] [154.6/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 01:55:49,100 - Train: 15.46% [749400/4848000] [154.6/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 01:56:21,867 - Train: 15.46% [749500/4848000] [154.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 01:56:54,611 - Train: 15.46% [749600/4848000] [154.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 01:57:28,636 - Train: 15.46% [749700/4848000] [154.6/1000.0] [batch_t 0.325 (0.340)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 01:58:01,431 - Train: 15.47% [749800/4848000] [154.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 01:58:34,249 - Train: 15.47% [749900/4848000] [154.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 01:59:07,019 - Train: 15.47% [750000/4848000] [154.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 01:59:39,813 - Train: 15.47% [750100/4848000] [154.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 02:00:12,773 - Train: 15.47% [750200/4848000] [154.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 02:00:45,594 - Train: 15.48% [750300/4848000] [154.8/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 02:01:19,346 - Train: 15.48% [750400/4848000] [154.8/1000.0] [batch_t 0.325 (0.337)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 02:01:52,222 - Train: 15.48% [750500/4848000] [154.8/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 02:02:25,096 - Train: 15.48% [750600/4848000] [154.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 02:02:58,104 - Train: 15.48% [750700/4848000] [154.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 02:03:31,827 - Train: 15.49% [750800/4848000] [154.9/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 02:04:04,653 - Train: 15.49% [750900/4848000] [154.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 02:04:37,542 - Train: 15.49% [751000/4848000] [154.9/1000.0] [batch_t 0.336 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-06 02:05:10,331 - Train: 15.49% [751100/4848000] [154.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 02:05:43,136 - Train: 15.50% [751200/4848000] [155.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 02:06:18,373 - Train: 15.50% [751300/4848000] [155.0/1000.0] [batch_t 0.327 (0.352)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 02:06:51,123 - Train: 15.50% [751400/4848000] [155.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 02:07:04,219 - ==> Total time: 3 days, 2:41:02 Eta: 16 days, 23:08:55 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 02:07:26,227 - Train: 15.50% [751500/4848000] [155.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 02:07:59,122 - Train: 15.50% [751600/4848000] [155.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 02:08:31,966 - Train: 15.51% [751700/4848000] [155.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 02:09:06,027 - Train: 15.51% [751800/4848000] [155.1/1000.0] [batch_t 0.331 (0.341)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 02:09:38,825 - Train: 15.51% [751900/4848000] [155.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 02:10:12,764 - Train: 15.51% [752000/4848000] [155.1/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 02:10:45,599 - Train: 15.51% [752100/4848000] [155.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 02:11:19,128 - Train: 15.52% [752200/4848000] [155.2/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 02:11:51,996 - Train: 15.52% [752300/4848000] [155.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 02:12:25,500 - Train: 15.52% [752400/4848000] [155.2/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 02:12:58,338 - Train: 15.52% [752500/4848000] [155.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 02:13:32,762 - Train: 15.52% [752600/4848000] [155.2/1000.0] [batch_t 0.328 (0.344)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 02:14:05,784 - Train: 15.53% [752700/4848000] [155.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 02:14:38,680 - Train: 15.53% [752800/4848000] [155.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 02:15:13,533 - Train: 15.53% [752900/4848000] [155.3/1000.0] [batch_t 0.328 (0.348)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 02:15:46,347 - Train: 15.53% [753000/4848000] [155.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 02:16:20,100 - Train: 15.53% [753100/4848000] [155.3/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 02:16:52,951 - Train: 15.54% [753200/4848000] [155.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 02:17:26,301 - Train: 15.54% [753300/4848000] [155.4/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 02:17:59,149 - Train: 15.54% [753400/4848000] [155.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 02:18:32,175 - Train: 15.54% [753500/4848000] [155.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 02:19:06,318 - Train: 15.54% [753600/4848000] [155.4/1000.0] [batch_t 1.635 (0.341)] [data_t 1.308] [optim_t 0.327] [lr 0.005000] 2024-04-06 02:19:39,163 - Train: 15.55% [753700/4848000] [155.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 02:20:12,856 - Train: 15.55% [753800/4848000] [155.5/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 02:20:45,708 - Train: 15.55% [753900/4848000] [155.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 02:21:19,651 - Train: 15.55% [754000/4848000] [155.5/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 02:21:52,542 - Train: 15.55% [754100/4848000] [155.5/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 02:22:26,423 - Train: 15.56% [754200/4848000] [155.6/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 02:22:59,262 - Train: 15.56% [754300/4848000] [155.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 02:23:33,193 - Train: 15.56% [754400/4848000] [155.6/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 02:24:07,248 - Train: 15.56% [754500/4848000] [155.6/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 02:24:39,991 - Train: 15.57% [754600/4848000] [155.7/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 02:25:13,522 - Train: 15.57% [754700/4848000] [155.7/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 02:25:46,275 - Train: 15.57% [754800/4848000] [155.7/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 02:26:19,243 - Train: 15.57% [754900/4848000] [155.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 02:26:52,010 - Train: 15.57% [755000/4848000] [155.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 02:27:25,157 - Train: 15.58% [755100/4848000] [155.8/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 02:27:57,892 - Train: 15.58% [755200/4848000] [155.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 02:28:32,008 - Train: 15.58% [755300/4848000] [155.8/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 02:29:07,064 - Train: 15.58% [755400/4848000] [155.8/1000.0] [batch_t 2.659 (0.350)] [data_t 2.329] [optim_t 0.329] [lr 0.005000] 2024-04-06 02:29:39,831 - Train: 15.58% [755500/4848000] [155.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 02:30:12,630 - Train: 15.59% [755600/4848000] [155.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 02:30:45,416 - Train: 15.59% [755700/4848000] [155.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 02:31:18,305 - Train: 15.59% [755800/4848000] [155.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 02:31:51,043 - Train: 15.59% [755900/4848000] [155.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 02:32:23,882 - Train: 15.59% [756000/4848000] [155.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 02:32:56,600 - Train: 15.60% [756100/4848000] [156.0/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 02:33:29,457 - Train: 15.60% [756200/4848000] [156.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 02:33:58,477 - ==> Total time: 3 days, 3:07:56 Eta: 16 days, 22:29:09 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 02:34:04,763 - Train: 15.60% [756300/4848000] [156.0/1000.0] [batch_t 0.694 (0.368)] [data_t 0.367] [optim_t 0.327] [lr 0.005000] 2024-04-06 02:34:37,597 - Train: 15.60% [756400/4848000] [156.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 02:35:11,804 - Train: 15.60% [756500/4848000] [156.0/1000.0] [batch_t 0.330 (0.342)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 02:35:44,591 - Train: 15.61% [756600/4848000] [156.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 02:36:18,122 - Train: 15.61% [756700/4848000] [156.1/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 02:36:50,941 - Train: 15.61% [756800/4848000] [156.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 02:37:25,217 - Train: 15.61% [756900/4848000] [156.1/1000.0] [batch_t 0.327 (0.343)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 02:37:58,024 - Train: 15.61% [757000/4848000] [156.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 02:38:31,801 - Train: 15.62% [757100/4848000] [156.2/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 02:39:04,642 - Train: 15.62% [757200/4848000] [156.2/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 02:39:39,215 - Train: 15.62% [757300/4848000] [156.2/1000.0] [batch_t 0.328 (0.346)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 02:40:12,762 - Train: 15.62% [757400/4848000] [156.2/1000.0] [batch_t 0.326 (0.335)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 02:40:45,589 - Train: 15.62% [757500/4848000] [156.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 02:41:20,902 - Train: 15.63% [757600/4848000] [156.3/1000.0] [batch_t 0.325 (0.353)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 02:41:53,734 - Train: 15.63% [757700/4848000] [156.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 02:42:28,895 - Train: 15.63% [757800/4848000] [156.3/1000.0] [batch_t 0.326 (0.352)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 02:43:01,660 - Train: 15.63% [757900/4848000] [156.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 02:43:35,455 - Train: 15.64% [758000/4848000] [156.4/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 02:44:08,719 - Train: 15.64% [758100/4848000] [156.4/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 02:44:41,517 - Train: 15.64% [758200/4848000] [156.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 02:45:14,741 - Train: 15.64% [758300/4848000] [156.4/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 02:45:47,587 - Train: 15.64% [758400/4848000] [156.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 02:46:20,327 - Train: 15.65% [758500/4848000] [156.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 02:46:53,055 - Train: 15.65% [758600/4848000] [156.5/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 02:47:27,405 - Train: 15.65% [758700/4848000] [156.5/1000.0] [batch_t 0.328 (0.343)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 02:48:00,209 - Train: 15.65% [758800/4848000] [156.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 02:48:34,232 - Train: 15.65% [758900/4848000] [156.5/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 02:49:07,013 - Train: 15.66% [759000/4848000] [156.6/1000.0] [batch_t 0.336 (0.328)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-06 02:49:39,975 - Train: 15.66% [759100/4848000] [156.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 02:50:13,556 - Train: 15.66% [759200/4848000] [156.6/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 02:50:46,343 - Train: 15.66% [759300/4848000] [156.6/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 02:51:20,177 - Train: 15.66% [759400/4848000] [156.6/1000.0] [batch_t 0.325 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 02:51:53,033 - Train: 15.67% [759500/4848000] [156.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 02:52:28,263 - Train: 15.67% [759600/4848000] [156.7/1000.0] [batch_t 0.326 (0.352)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 02:53:01,090 - Train: 15.67% [759700/4848000] [156.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 02:53:35,301 - Train: 15.67% [759800/4848000] [156.7/1000.0] [batch_t 0.328 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 02:54:09,534 - Train: 15.67% [759900/4848000] [156.7/1000.0] [batch_t 0.325 (0.342)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 02:54:42,337 - Train: 15.68% [760000/4848000] [156.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 02:55:16,584 - Train: 15.68% [760100/4848000] [156.8/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 02:55:49,413 - Train: 15.68% [760200/4848000] [156.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 02:56:23,384 - Train: 15.68% [760300/4848000] [156.8/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 02:56:56,169 - Train: 15.68% [760400/4848000] [156.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 02:57:30,510 - Train: 15.69% [760500/4848000] [156.9/1000.0] [batch_t 0.326 (0.343)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 02:58:03,349 - Train: 15.69% [760600/4848000] [156.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 02:58:37,258 - Train: 15.69% [760700/4848000] [156.9/1000.0] [batch_t 0.324 (0.339)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 02:59:10,927 - Train: 15.69% [760800/4848000] [156.9/1000.0] [batch_t 0.325 (0.337)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 02:59:43,708 - Train: 15.70% [760900/4848000] [157.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 03:00:18,040 - Train: 15.70% [761000/4848000] [157.0/1000.0] [batch_t 0.325 (0.343)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 03:00:50,858 - Train: 15.70% [761100/4848000] [157.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 03:01:02,702 - ==> Total time: 3 days, 3:35:01 Eta: 16 days, 21:50:27 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 03:01:28,305 - Train: 15.70% [761200/4848000] [157.0/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 03:02:01,066 - Train: 15.70% [761300/4848000] [157.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 03:02:35,134 - Train: 15.71% [761400/4848000] [157.1/1000.0] [batch_t 0.326 (0.341)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 03:03:08,342 - Train: 15.71% [761500/4848000] [157.1/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 03:03:41,107 - Train: 15.71% [761600/4848000] [157.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 03:04:16,058 - Train: 15.71% [761700/4848000] [157.1/1000.0] [batch_t 0.329 (0.349)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 03:04:49,036 - Train: 15.71% [761800/4848000] [157.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 03:05:23,078 - Train: 15.72% [761900/4848000] [157.2/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 03:05:55,903 - Train: 15.72% [762000/4848000] [157.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 03:06:28,705 - Train: 15.72% [762100/4848000] [157.2/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 03:07:01,514 - Train: 15.72% [762200/4848000] [157.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 03:07:35,686 - Train: 15.72% [762300/4848000] [157.2/1000.0] [batch_t 0.328 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 03:08:08,490 - Train: 15.73% [762400/4848000] [157.3/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 03:08:41,362 - Train: 15.73% [762500/4848000] [157.3/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 03:09:14,237 - Train: 15.73% [762600/4848000] [157.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 03:09:47,046 - Train: 15.73% [762700/4848000] [157.3/1000.0] [batch_t 0.336 (0.328)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-06 03:10:20,829 - Train: 15.73% [762800/4848000] [157.3/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 03:10:53,620 - Train: 15.74% [762900/4848000] [157.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 03:11:27,157 - Train: 15.74% [763000/4848000] [157.4/1000.0] [batch_t 0.324 (0.335)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 03:12:00,002 - Train: 15.74% [763100/4848000] [157.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 03:12:34,303 - Train: 15.74% [763200/4848000] [157.4/1000.0] [batch_t 0.326 (0.343)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 03:13:07,241 - Train: 15.74% [763300/4848000] [157.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 03:13:40,095 - Train: 15.75% [763400/4848000] [157.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 03:14:12,971 - Train: 15.75% [763500/4848000] [157.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 03:14:45,857 - Train: 15.75% [763600/4848000] [157.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 03:15:18,596 - Train: 15.75% [763700/4848000] [157.5/1000.0] [batch_t 0.326 (0.327)] [data_t 0.004] [optim_t 0.322] [lr 0.005000] 2024-04-06 03:15:51,378 - Train: 15.75% [763800/4848000] [157.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 03:16:24,183 - Train: 15.76% [763900/4848000] [157.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 03:16:56,955 - Train: 15.76% [764000/4848000] [157.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 03:17:30,196 - Train: 15.76% [764100/4848000] [157.6/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 03:18:02,952 - Train: 15.76% [764200/4848000] [157.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 03:18:35,788 - Train: 15.77% [764300/4848000] [157.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 03:19:09,659 - Train: 15.77% [764400/4848000] [157.7/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 03:19:42,522 - Train: 15.77% [764500/4848000] [157.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 03:20:15,417 - Train: 15.77% [764600/4848000] [157.7/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 03:20:48,392 - Train: 15.77% [764700/4848000] [157.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 03:21:23,667 - Train: 15.78% [764800/4848000] [157.8/1000.0] [batch_t 0.331 (0.353)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 03:21:56,569 - Train: 15.78% [764900/4848000] [157.8/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 03:22:30,893 - Train: 15.78% [765000/4848000] [157.8/1000.0] [batch_t 0.328 (0.343)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 03:23:03,729 - Train: 15.78% [765100/4848000] [157.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 03:23:37,433 - Train: 15.78% [765200/4848000] [157.8/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 03:24:11,065 - Train: 15.79% [765300/4848000] [157.9/1000.0] [batch_t 0.333 (0.336)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 03:24:43,903 - Train: 15.79% [765400/4848000] [157.9/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 03:25:17,906 - Train: 15.79% [765500/4848000] [157.9/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 03:25:50,702 - Train: 15.79% [765600/4848000] [157.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 03:26:23,586 - Train: 15.79% [765700/4848000] [157.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 03:26:56,404 - Train: 15.80% [765800/4848000] [158.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 03:27:30,274 - Train: 15.80% [765900/4848000] [158.0/1000.0] [batch_t 0.324 (0.339)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 03:27:57,964 - ==> Total time: 3 days, 4:01:56 Eta: 16 days, 21:11:06 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 03:28:05,594 - Train: 15.80% [766000/4848000] [158.0/1000.0] [batch_t 0.327 (0.355)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 03:28:38,387 - Train: 15.80% [766100/4848000] [158.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 03:29:13,614 - Train: 15.80% [766200/4848000] [158.0/1000.0] [batch_t 0.338 (0.352)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-06 03:29:46,458 - Train: 15.81% [766300/4848000] [158.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 03:30:19,486 - Train: 15.81% [766400/4848000] [158.1/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 03:30:52,290 - Train: 15.81% [766500/4848000] [158.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 03:31:26,228 - Train: 15.81% [766600/4848000] [158.1/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 03:31:58,985 - Train: 15.81% [766700/4848000] [158.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 03:32:33,373 - Train: 15.82% [766800/4848000] [158.2/1000.0] [batch_t 0.330 (0.344)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 03:33:07,796 - Train: 15.82% [766900/4848000] [158.2/1000.0] [batch_t 0.339 (0.344)] [data_t 0.002] [optim_t 0.337] [lr 0.005000] 2024-04-06 03:33:40,707 - Train: 15.82% [767000/4848000] [158.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 03:34:13,636 - Train: 15.82% [767100/4848000] [158.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 03:34:46,527 - Train: 15.83% [767200/4848000] [158.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 03:35:19,408 - Train: 15.83% [767300/4848000] [158.3/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 03:35:52,277 - Train: 15.83% [767400/4848000] [158.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 03:36:25,470 - Train: 15.83% [767500/4848000] [158.3/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 03:36:58,300 - Train: 15.83% [767600/4848000] [158.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 03:37:31,100 - Train: 15.84% [767700/4848000] [158.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 03:38:03,876 - Train: 15.84% [767800/4848000] [158.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 03:38:38,272 - Train: 15.84% [767900/4848000] [158.4/1000.0] [batch_t 0.326 (0.344)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 03:39:11,856 - Train: 15.84% [768000/4848000] [158.4/1000.0] [batch_t 0.325 (0.336)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 03:39:44,599 - Train: 15.84% [768100/4848000] [158.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 03:40:17,573 - Train: 15.85% [768200/4848000] [158.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 03:40:50,353 - Train: 15.85% [768300/4848000] [158.5/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 03:41:23,790 - Train: 15.85% [768400/4848000] [158.5/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 03:41:56,569 - Train: 15.85% [768500/4848000] [158.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 03:42:32,489 - Train: 15.85% [768600/4848000] [158.5/1000.0] [batch_t 0.328 (0.359)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 03:43:08,246 - Train: 15.86% [768700/4848000] [158.6/1000.0] [batch_t 3.294 (0.357)] [data_t 2.968] [optim_t 0.326] [lr 0.005000] 2024-04-06 03:43:40,996 - Train: 15.86% [768800/4848000] [158.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 03:44:15,548 - Train: 15.86% [768900/4848000] [158.6/1000.0] [batch_t 0.329 (0.345)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 03:44:48,275 - Train: 15.86% [769000/4848000] [158.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 03:45:22,787 - Train: 15.86% [769100/4848000] [158.6/1000.0] [batch_t 0.328 (0.345)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 03:45:55,562 - Train: 15.87% [769200/4848000] [158.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 03:46:31,790 - Train: 15.87% [769300/4848000] [158.7/1000.0] [batch_t 0.328 (0.362)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 03:47:04,773 - Train: 15.87% [769400/4848000] [158.7/1000.0] [batch_t 0.552 (0.330)] [data_t 0.224] [optim_t 0.328] [lr 0.005000] 2024-04-06 03:47:38,566 - Train: 15.87% [769500/4848000] [158.7/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 03:48:12,043 - Train: 15.87% [769600/4848000] [158.7/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 03:48:44,819 - Train: 15.88% [769700/4848000] [158.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 03:49:17,720 - Train: 15.88% [769800/4848000] [158.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 03:49:50,508 - Train: 15.88% [769900/4848000] [158.8/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 03:50:25,819 - Train: 15.88% [770000/4848000] [158.8/1000.0] [batch_t 0.328 (0.353)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 03:50:58,679 - Train: 15.88% [770100/4848000] [158.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 03:51:31,790 - Train: 15.89% [770200/4848000] [158.9/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 03:52:04,858 - Train: 15.89% [770300/4848000] [158.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 03:52:39,721 - Train: 15.89% [770400/4848000] [158.9/1000.0] [batch_t 0.328 (0.349)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 03:53:14,899 - Train: 15.89% [770500/4848000] [158.9/1000.0] [batch_t 0.328 (0.352)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 03:53:47,700 - Train: 15.90% [770600/4848000] [159.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 03:54:23,260 - Train: 15.90% [770700/4848000] [159.0/1000.0] [batch_t 0.327 (0.355)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 03:54:56,154 - Train: 15.90% [770800/4848000] [159.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 03:55:07,579 - ==> Total time: 3 days, 4:29:06 Eta: 16 days, 20:33:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 03:55:31,831 - Train: 15.90% [770900/4848000] [159.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 03:56:05,268 - Train: 15.90% [771000/4848000] [159.0/1000.0] [batch_t 0.964 (0.334)] [data_t 0.640] [optim_t 0.324] [lr 0.005000] 2024-04-06 03:56:38,041 - Train: 15.91% [771100/4848000] [159.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 03:57:11,844 - Train: 15.91% [771200/4848000] [159.1/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 03:57:44,613 - Train: 15.91% [771300/4848000] [159.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 03:58:18,151 - Train: 15.91% [771400/4848000] [159.1/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 03:58:50,909 - Train: 15.91% [771500/4848000] [159.1/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 03:59:26,527 - Train: 15.92% [771600/4848000] [159.2/1000.0] [batch_t 0.327 (0.356)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 03:59:59,376 - Train: 15.92% [771700/4848000] [159.2/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 04:00:33,248 - Train: 15.92% [771800/4848000] [159.2/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 04:01:07,031 - Train: 15.92% [771900/4848000] [159.2/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 04:01:39,847 - Train: 15.92% [772000/4848000] [159.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 04:02:15,099 - Train: 15.93% [772100/4848000] [159.3/1000.0] [batch_t 0.333 (0.352)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 04:02:47,911 - Train: 15.93% [772200/4848000] [159.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 04:03:22,834 - Train: 15.93% [772300/4848000] [159.3/1000.0] [batch_t 0.326 (0.349)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 04:03:55,699 - Train: 15.93% [772400/4848000] [159.3/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 04:04:29,151 - Train: 15.93% [772500/4848000] [159.3/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 04:05:02,033 - Train: 15.94% [772600/4848000] [159.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 04:05:36,065 - Train: 15.94% [772700/4848000] [159.4/1000.0] [batch_t 0.326 (0.340)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 04:06:10,154 - Train: 15.94% [772800/4848000] [159.4/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 04:06:42,930 - Train: 15.94% [772900/4848000] [159.4/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 04:07:17,090 - Train: 15.94% [773000/4848000] [159.4/1000.0] [batch_t 0.330 (0.342)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 04:07:50,036 - Train: 15.95% [773100/4848000] [159.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 04:08:23,613 - Train: 15.95% [773200/4848000] [159.5/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 04:08:56,474 - Train: 15.95% [773300/4848000] [159.5/1000.0] [batch_t 0.339 (0.329)] [data_t 0.002] [optim_t 0.337] [lr 0.005000] 2024-04-06 04:09:30,257 - Train: 15.95% [773400/4848000] [159.5/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 04:10:03,067 - Train: 15.96% [773500/4848000] [159.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 04:10:36,778 - Train: 15.96% [773600/4848000] [159.6/1000.0] [batch_t 0.323 (0.337)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 04:11:09,928 - Train: 15.96% [773700/4848000] [159.6/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 04:11:42,741 - Train: 15.96% [773800/4848000] [159.6/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 04:12:18,036 - Train: 15.96% [773900/4848000] [159.6/1000.0] [batch_t 0.328 (0.353)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 04:12:50,813 - Train: 15.97% [774000/4848000] [159.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 04:13:23,554 - Train: 15.97% [774100/4848000] [159.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 04:13:56,361 - Train: 15.97% [774200/4848000] [159.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 04:14:31,719 - Train: 15.97% [774300/4848000] [159.7/1000.0] [batch_t 0.328 (0.353)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 04:15:04,509 - Train: 15.97% [774400/4848000] [159.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 04:15:37,432 - Train: 15.98% [774500/4848000] [159.8/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 04:16:10,265 - Train: 15.98% [774600/4848000] [159.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 04:16:43,165 - Train: 15.98% [774700/4848000] [159.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 04:17:17,155 - Train: 15.98% [774800/4848000] [159.8/1000.0] [batch_t 0.326 (0.340)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 04:17:50,009 - Train: 15.98% [774900/4848000] [159.8/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-06 04:18:22,940 - Train: 15.99% [775000/4848000] [159.9/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 04:18:55,868 - Train: 15.99% [775100/4848000] [159.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 04:19:29,213 - Train: 15.99% [775200/4848000] [159.9/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 04:20:02,081 - Train: 15.99% [775300/4848000] [159.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 04:20:37,570 - Train: 15.99% [775400/4848000] [159.9/1000.0] [batch_t 0.328 (0.355)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 04:21:11,980 - Train: 16.00% [775500/4848000] [160.0/1000.0] [batch_t 0.328 (0.344)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 04:21:44,751 - Train: 16.00% [775600/4848000] [160.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 04:22:14,094 - ==> Total time: 3 days, 4:56:12 Eta: 16 days, 19:55:06 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 04:22:22,860 - Train: 16.00% [775700/4848000] [160.0/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 04:22:55,815 - Train: 16.00% [775800/4848000] [160.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 04:23:29,557 - Train: 16.00% [775900/4848000] [160.0/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 04:24:02,329 - Train: 16.01% [776000/4848000] [160.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 04:24:36,660 - Train: 16.01% [776100/4848000] [160.1/1000.0] [batch_t 0.329 (0.343)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 04:25:10,315 - Train: 16.01% [776200/4848000] [160.1/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 04:25:43,102 - Train: 16.01% [776300/4848000] [160.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 04:26:16,757 - Train: 16.01% [776400/4848000] [160.1/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 04:26:49,630 - Train: 16.02% [776500/4848000] [160.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 04:27:23,399 - Train: 16.02% [776600/4848000] [160.2/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 04:27:56,230 - Train: 16.02% [776700/4848000] [160.2/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 04:28:29,764 - Train: 16.02% [776800/4848000] [160.2/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 04:29:02,523 - Train: 16.03% [776900/4848000] [160.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 04:29:36,406 - Train: 16.03% [777000/4848000] [160.3/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 04:30:09,463 - Train: 16.03% [777100/4848000] [160.3/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 04:30:42,278 - Train: 16.03% [777200/4848000] [160.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 04:31:15,858 - Train: 16.03% [777300/4848000] [160.3/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 04:31:48,660 - Train: 16.04% [777400/4848000] [160.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 04:32:22,760 - Train: 16.04% [777500/4848000] [160.4/1000.0] [batch_t 0.326 (0.341)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 04:32:55,546 - Train: 16.04% [777600/4848000] [160.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 04:33:29,465 - Train: 16.04% [777700/4848000] [160.4/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 04:34:02,275 - Train: 16.04% [777800/4848000] [160.4/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 04:34:36,427 - Train: 16.05% [777900/4848000] [160.5/1000.0] [batch_t 0.325 (0.341)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 04:35:09,923 - Train: 16.05% [778000/4848000] [160.5/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 04:35:42,667 - Train: 16.05% [778100/4848000] [160.5/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 04:36:16,638 - Train: 16.05% [778200/4848000] [160.5/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 04:36:49,457 - Train: 16.05% [778300/4848000] [160.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 04:37:24,585 - Train: 16.06% [778400/4848000] [160.6/1000.0] [batch_t 0.328 (0.351)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 04:37:57,380 - Train: 16.06% [778500/4848000] [160.6/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 04:38:30,856 - Train: 16.06% [778600/4848000] [160.6/1000.0] [batch_t 0.326 (0.335)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 04:39:03,812 - Train: 16.06% [778700/4848000] [160.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 04:39:38,608 - Train: 16.06% [778800/4848000] [160.6/1000.0] [batch_t 0.330 (0.348)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 04:40:12,708 - Train: 16.07% [778900/4848000] [160.7/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 04:40:45,481 - Train: 16.07% [779000/4848000] [160.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 04:41:19,522 - Train: 16.07% [779100/4848000] [160.7/1000.0] [batch_t 0.325 (0.340)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 04:41:52,297 - Train: 16.07% [779200/4848000] [160.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 04:42:25,918 - Train: 16.07% [779300/4848000] [160.7/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 04:42:58,743 - Train: 16.08% [779400/4848000] [160.8/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 04:43:32,658 - Train: 16.08% [779500/4848000] [160.8/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 04:44:05,485 - Train: 16.08% [779600/4848000] [160.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 04:44:38,278 - Train: 16.08% [779700/4848000] [160.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 04:45:12,623 - Train: 16.08% [779800/4848000] [160.8/1000.0] [batch_t 0.327 (0.343)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 04:45:45,419 - Train: 16.09% [779900/4848000] [160.9/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 04:46:18,267 - Train: 16.09% [780000/4848000] [160.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 04:46:51,221 - Train: 16.09% [780100/4848000] [160.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 04:47:24,216 - Train: 16.09% [780200/4848000] [160.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 04:47:57,072 - Train: 16.10% [780300/4848000] [161.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 04:48:29,862 - Train: 16.10% [780400/4848000] [161.0/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 04:49:02,685 - Train: 16.10% [780500/4848000] [161.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 04:49:12,830 - ==> Total time: 3 days, 5:23:11 Eta: 16 days, 19:16:29 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 04:49:38,391 - Train: 16.10% [780600/4848000] [161.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 04:50:11,470 - Train: 16.10% [780700/4848000] [161.0/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 04:50:44,229 - Train: 16.11% [780800/4848000] [161.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 04:51:17,304 - Train: 16.11% [780900/4848000] [161.1/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 04:51:50,070 - Train: 16.11% [781000/4848000] [161.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 04:52:24,830 - Train: 16.11% [781100/4848000] [161.1/1000.0] [batch_t 0.327 (0.348)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 04:52:57,584 - Train: 16.11% [781200/4848000] [161.1/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 04:53:31,317 - Train: 16.12% [781300/4848000] [161.2/1000.0] [batch_t 0.324 (0.337)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 04:54:04,107 - Train: 16.12% [781400/4848000] [161.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 04:54:39,346 - Train: 16.12% [781500/4848000] [161.2/1000.0] [batch_t 0.323 (0.352)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 04:55:12,127 - Train: 16.12% [781600/4848000] [161.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 04:55:44,905 - Train: 16.12% [781700/4848000] [161.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 04:56:17,716 - Train: 16.13% [781800/4848000] [161.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 04:56:50,478 - Train: 16.13% [781900/4848000] [161.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 04:57:23,279 - Train: 16.13% [782000/4848000] [161.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 04:57:56,055 - Train: 16.13% [782100/4848000] [161.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 04:58:30,519 - Train: 16.13% [782200/4848000] [161.3/1000.0] [batch_t 0.326 (0.345)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 04:59:03,335 - Train: 16.14% [782300/4848000] [161.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 04:59:38,481 - Train: 16.14% [782400/4848000] [161.4/1000.0] [batch_t 0.327 (0.351)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 05:00:11,240 - Train: 16.14% [782500/4848000] [161.4/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 05:00:44,029 - Train: 16.14% [782600/4848000] [161.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 05:01:16,851 - Train: 16.14% [782700/4848000] [161.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 05:01:49,673 - Train: 16.15% [782800/4848000] [161.5/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 05:02:22,604 - Train: 16.15% [782900/4848000] [161.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 05:02:55,366 - Train: 16.15% [783000/4848000] [161.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 05:03:28,241 - Train: 16.15% [783100/4848000] [161.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 05:04:01,079 - Train: 16.16% [783200/4848000] [161.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 05:04:34,936 - Train: 16.16% [783300/4848000] [161.6/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 05:05:08,277 - Train: 16.16% [783400/4848000] [161.6/1000.0] [batch_t 0.331 (0.333)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 05:05:41,045 - Train: 16.16% [783500/4848000] [161.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 05:06:15,052 - Train: 16.16% [783600/4848000] [161.6/1000.0] [batch_t 0.325 (0.340)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 05:06:47,847 - Train: 16.17% [783700/4848000] [161.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 05:07:21,270 - Train: 16.17% [783800/4848000] [161.7/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 05:07:54,060 - Train: 16.17% [783900/4848000] [161.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 05:08:27,760 - Train: 16.17% [784000/4848000] [161.7/1000.0] [batch_t 0.323 (0.337)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 05:09:00,583 - Train: 16.17% [784100/4848000] [161.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 05:09:34,489 - Train: 16.18% [784200/4848000] [161.8/1000.0] [batch_t 0.322 (0.339)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-06 05:10:08,301 - Train: 16.18% [784300/4848000] [161.8/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 05:10:41,099 - Train: 16.18% [784400/4848000] [161.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 05:11:15,122 - Train: 16.18% [784500/4848000] [161.8/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 05:11:47,908 - Train: 16.18% [784600/4848000] [161.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 05:12:21,259 - Train: 16.19% [784700/4848000] [161.9/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 05:12:54,019 - Train: 16.19% [784800/4848000] [161.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 05:13:28,261 - Train: 16.19% [784900/4848000] [161.9/1000.0] [batch_t 0.323 (0.342)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 05:14:01,055 - Train: 16.19% [785000/4848000] [161.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 05:14:35,282 - Train: 16.19% [785100/4848000] [161.9/1000.0] [batch_t 0.326 (0.342)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 05:15:09,223 - Train: 16.20% [785200/4848000] [162.0/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 05:15:41,958 - Train: 16.20% [785300/4848000] [162.0/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 05:16:08,261 - ==> Total time: 3 days, 5:50:06 Eta: 16 days, 18:37:44 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 05:16:18,265 - Train: 16.20% [785400/4848000] [162.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 05:16:51,026 - Train: 16.20% [785500/4848000] [162.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 05:17:24,650 - Train: 16.20% [785600/4848000] [162.0/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 05:17:57,634 - Train: 16.21% [785700/4848000] [162.1/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 05:18:31,288 - Train: 16.21% [785800/4848000] [162.1/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 05:19:04,110 - Train: 16.21% [785900/4848000] [162.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 05:19:38,127 - Train: 16.21% [786000/4848000] [162.1/1000.0] [batch_t 0.323 (0.340)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 05:20:11,058 - Train: 16.21% [786100/4848000] [162.1/1000.0] [batch_t 0.319 (0.329)] [data_t 0.002] [optim_t 0.318] [lr 0.005000] 2024-04-06 05:20:43,886 - Train: 16.22% [786200/4848000] [162.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 05:21:17,314 - Train: 16.22% [786300/4848000] [162.2/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 05:21:50,146 - Train: 16.22% [786400/4848000] [162.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 05:22:24,827 - Train: 16.22% [786500/4848000] [162.2/1000.0] [batch_t 0.331 (0.347)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 05:22:57,694 - Train: 16.23% [786600/4848000] [162.3/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 05:23:31,630 - Train: 16.23% [786700/4848000] [162.3/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 05:24:04,416 - Train: 16.23% [786800/4848000] [162.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 05:24:37,243 - Train: 16.23% [786900/4848000] [162.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 05:25:10,087 - Train: 16.23% [787000/4848000] [162.3/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 05:25:42,987 - Train: 16.24% [787100/4848000] [162.4/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 05:26:16,783 - Train: 16.24% [787200/4848000] [162.4/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 05:26:49,601 - Train: 16.24% [787300/4848000] [162.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 05:27:22,949 - Train: 16.24% [787400/4848000] [162.4/1000.0] [batch_t 0.331 (0.333)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 05:27:55,754 - Train: 16.24% [787500/4848000] [162.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 05:28:28,583 - Train: 16.25% [787600/4848000] [162.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 05:29:01,388 - Train: 16.25% [787700/4848000] [162.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 05:29:34,195 - Train: 16.25% [787800/4848000] [162.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 05:30:07,866 - Train: 16.25% [787900/4848000] [162.5/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 05:30:40,746 - Train: 16.25% [788000/4848000] [162.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 05:31:14,934 - Train: 16.26% [788100/4848000] [162.6/1000.0] [batch_t 0.328 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 05:31:47,705 - Train: 16.26% [788200/4848000] [162.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 05:32:20,509 - Train: 16.26% [788300/4848000] [162.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 05:32:53,351 - Train: 16.26% [788400/4848000] [162.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 05:33:27,456 - Train: 16.26% [788500/4848000] [162.6/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 05:34:00,378 - Train: 16.27% [788600/4848000] [162.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 05:34:34,780 - Train: 16.27% [788700/4848000] [162.7/1000.0] [batch_t 0.330 (0.344)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 05:35:08,995 - Train: 16.27% [788800/4848000] [162.7/1000.0] [batch_t 0.327 (0.342)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 05:35:41,860 - Train: 16.27% [788900/4848000] [162.7/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-06 05:36:15,539 - Train: 16.27% [789000/4848000] [162.7/1000.0] [batch_t 0.324 (0.337)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 05:36:48,353 - Train: 16.28% [789100/4848000] [162.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 05:37:22,131 - Train: 16.28% [789200/4848000] [162.8/1000.0] [batch_t 0.331 (0.338)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 05:37:54,925 - Train: 16.28% [789300/4848000] [162.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 05:38:28,937 - Train: 16.28% [789400/4848000] [162.8/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 05:39:01,719 - Train: 16.29% [789500/4848000] [162.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 05:39:35,522 - Train: 16.29% [789600/4848000] [162.9/1000.0] [batch_t 0.324 (0.338)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 05:40:08,267 - Train: 16.29% [789700/4848000] [162.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 05:40:41,024 - Train: 16.29% [789800/4848000] [162.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 05:41:14,787 - Train: 16.29% [789900/4848000] [162.9/1000.0] [batch_t 0.331 (0.338)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 05:41:47,720 - Train: 16.30% [790000/4848000] [163.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 05:42:20,911 - Train: 16.30% [790100/4848000] [163.0/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 05:42:53,738 - Train: 16.30% [790200/4848000] [163.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 05:43:01,631 - ==> Total time: 3 days, 6:17:00 Eta: 16 days, 17:58:57 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 05:43:30,855 - Train: 16.30% [790300/4848000] [163.0/1000.0] [batch_t 0.326 (0.335)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 05:44:03,639 - Train: 16.30% [790400/4848000] [163.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 05:44:37,624 - Train: 16.31% [790500/4848000] [163.1/1000.0] [batch_t 0.326 (0.340)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 05:45:10,416 - Train: 16.31% [790600/4848000] [163.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 05:45:43,163 - Train: 16.31% [790700/4848000] [163.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 05:46:17,022 - Train: 16.31% [790800/4848000] [163.1/1000.0] [batch_t 0.331 (0.338)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 05:46:49,798 - Train: 16.31% [790900/4848000] [163.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 05:47:23,830 - Train: 16.32% [791000/4848000] [163.2/1000.0] [batch_t 0.330 (0.340)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 05:47:56,642 - Train: 16.32% [791100/4848000] [163.2/1000.0] [batch_t 0.337 (0.328)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-06 05:48:30,410 - Train: 16.32% [791200/4848000] [163.2/1000.0] [batch_t 0.331 (0.338)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 05:49:03,240 - Train: 16.32% [791300/4848000] [163.2/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-06 05:49:38,356 - Train: 16.32% [791400/4848000] [163.2/1000.0] [batch_t 0.324 (0.351)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 05:50:12,274 - Train: 16.33% [791500/4848000] [163.3/1000.0] [batch_t 0.325 (0.339)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 05:50:45,009 - Train: 16.33% [791600/4848000] [163.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 05:51:17,792 - Train: 16.33% [791700/4848000] [163.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 05:51:50,549 - Train: 16.33% [791800/4848000] [163.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 05:52:23,288 - Train: 16.33% [791900/4848000] [163.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 05:52:56,090 - Train: 16.34% [792000/4848000] [163.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 05:53:29,103 - Train: 16.34% [792100/4848000] [163.4/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 05:54:01,914 - Train: 16.34% [792200/4848000] [163.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 05:54:34,741 - Train: 16.34% [792300/4848000] [163.4/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 05:55:08,948 - Train: 16.34% [792400/4848000] [163.4/1000.0] [batch_t 0.334 (0.342)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-06 05:55:41,765 - Train: 16.35% [792500/4848000] [163.5/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-06 05:56:15,995 - Train: 16.35% [792600/4848000] [163.5/1000.0] [batch_t 0.328 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 05:56:49,003 - Train: 16.35% [792700/4848000] [163.5/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 05:57:22,447 - Train: 16.35% [792800/4848000] [163.5/1000.0] [batch_t 0.326 (0.334)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 05:57:55,246 - Train: 16.36% [792900/4848000] [163.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 05:58:30,490 - Train: 16.36% [793000/4848000] [163.6/1000.0] [batch_t 0.329 (0.352)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 05:59:03,238 - Train: 16.36% [793100/4848000] [163.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 05:59:36,198 - Train: 16.36% [793200/4848000] [163.6/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 06:00:09,760 - Train: 16.36% [793300/4848000] [163.6/1000.0] [batch_t 0.323 (0.336)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 06:00:42,534 - Train: 16.37% [793400/4848000] [163.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 06:01:15,295 - Train: 16.37% [793500/4848000] [163.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 06:01:48,086 - Train: 16.37% [793600/4848000] [163.7/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 06:02:21,741 - Train: 16.37% [793700/4848000] [163.7/1000.0] [batch_t 0.325 (0.336)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 06:02:54,516 - Train: 16.37% [793800/4848000] [163.7/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 06:03:27,992 - Train: 16.38% [793900/4848000] [163.8/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 06:04:00,745 - Train: 16.38% [794000/4848000] [163.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 06:04:34,611 - Train: 16.38% [794100/4848000] [163.8/1000.0] [batch_t 0.333 (0.339)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 06:05:08,550 - Train: 16.38% [794200/4848000] [163.8/1000.0] [batch_t 0.323 (0.339)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 06:05:41,371 - Train: 16.38% [794300/4848000] [163.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 06:06:15,681 - Train: 16.39% [794400/4848000] [163.9/1000.0] [batch_t 0.329 (0.343)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 06:06:48,483 - Train: 16.39% [794500/4848000] [163.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 06:07:21,933 - Train: 16.39% [794600/4848000] [163.9/1000.0] [batch_t 0.326 (0.334)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 06:07:54,690 - Train: 16.39% [794700/4848000] [163.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 06:08:28,530 - Train: 16.39% [794800/4848000] [163.9/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 06:09:01,337 - Train: 16.40% [794900/4848000] [164.0/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 06:09:35,140 - Train: 16.40% [795000/4848000] [164.0/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 06:09:58,754 - ==> Total time: 3 days, 6:43:57 Eta: 16 days, 17:20:38 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 06:10:11,284 - Train: 16.40% [795100/4848000] [164.0/1000.0] [batch_t 0.326 (0.373)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 06:10:44,074 - Train: 16.40% [795200/4848000] [164.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 06:11:17,945 - Train: 16.40% [795300/4848000] [164.0/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 06:11:50,698 - Train: 16.41% [795400/4848000] [164.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 06:12:25,812 - Train: 16.41% [795500/4848000] [164.1/1000.0] [batch_t 0.329 (0.351)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 06:12:58,568 - Train: 16.41% [795600/4848000] [164.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 06:13:32,703 - Train: 16.41% [795700/4848000] [164.1/1000.0] [batch_t 0.326 (0.341)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 06:14:06,610 - Train: 16.42% [795800/4848000] [164.2/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 06:14:39,373 - Train: 16.42% [795900/4848000] [164.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 06:15:13,468 - Train: 16.42% [796000/4848000] [164.2/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 06:15:46,225 - Train: 16.42% [796100/4848000] [164.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 06:16:20,176 - Train: 16.42% [796200/4848000] [164.2/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 06:16:52,908 - Train: 16.43% [796300/4848000] [164.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 06:17:26,285 - Train: 16.43% [796400/4848000] [164.3/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 06:17:59,036 - Train: 16.43% [796500/4848000] [164.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 06:18:32,765 - Train: 16.43% [796600/4848000] [164.3/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 06:19:07,770 - Train: 16.43% [796700/4848000] [164.3/1000.0] [batch_t 0.328 (0.350)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 06:19:40,513 - Train: 16.44% [796800/4848000] [164.4/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 06:20:15,504 - Train: 16.44% [796900/4848000] [164.4/1000.0] [batch_t 0.328 (0.350)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 06:20:48,396 - Train: 16.44% [797000/4848000] [164.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 06:21:22,737 - Train: 16.44% [797100/4848000] [164.4/1000.0] [batch_t 0.327 (0.343)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 06:21:55,593 - Train: 16.44% [797200/4848000] [164.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 06:22:28,415 - Train: 16.45% [797300/4848000] [164.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 06:23:01,248 - Train: 16.45% [797400/4848000] [164.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 06:23:35,702 - Train: 16.45% [797500/4848000] [164.5/1000.0] [batch_t 0.328 (0.344)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 06:24:09,659 - Train: 16.45% [797600/4848000] [164.5/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 06:24:42,517 - Train: 16.45% [797700/4848000] [164.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 06:25:16,190 - Train: 16.46% [797800/4848000] [164.6/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 06:25:48,972 - Train: 16.46% [797900/4848000] [164.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 06:26:22,313 - Train: 16.46% [798000/4848000] [164.6/1000.0] [batch_t 0.334 (0.333)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-06 06:26:55,158 - Train: 16.46% [798100/4848000] [164.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 06:27:28,980 - Train: 16.46% [798200/4848000] [164.6/1000.0] [batch_t 0.332 (0.338)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 06:28:01,908 - Train: 16.47% [798300/4848000] [164.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 06:28:34,787 - Train: 16.47% [798400/4848000] [164.7/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 06:29:07,616 - Train: 16.47% [798500/4848000] [164.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 06:29:40,384 - Train: 16.47% [798600/4848000] [164.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 06:30:13,154 - Train: 16.47% [798700/4848000] [164.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 06:30:45,932 - Train: 16.48% [798800/4848000] [164.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 06:31:18,722 - Train: 16.48% [798900/4848000] [164.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 06:31:51,550 - Train: 16.48% [799000/4848000] [164.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 06:32:24,469 - Train: 16.48% [799100/4848000] [164.8/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 06:32:57,243 - Train: 16.49% [799200/4848000] [164.9/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 06:33:30,117 - Train: 16.49% [799300/4848000] [164.9/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 06:34:02,857 - Train: 16.49% [799400/4848000] [164.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 06:34:36,655 - Train: 16.49% [799500/4848000] [164.9/1000.0] [batch_t 0.323 (0.338)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 06:35:09,743 - Train: 16.49% [799600/4848000] [164.9/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 06:35:42,599 - Train: 16.50% [799700/4848000] [165.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 06:36:17,961 - Train: 16.50% [799800/4848000] [165.0/1000.0] [batch_t 0.326 (0.354)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 06:36:50,802 - Train: 16.50% [799900/4848000] [165.0/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 06:36:57,372 - ==> Total time: 3 days, 7:10:55 Eta: 16 days, 16:42:35 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 06:37:25,607 - Train: 16.50% [800000/4848000] [165.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 06:37:58,424 - Train: 16.50% [800100/4848000] [165.0/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 06:38:31,268 - Train: 16.51% [800200/4848000] [165.1/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 06:39:04,460 - Train: 16.51% [800300/4848000] [165.1/1000.0] [batch_t 0.468 (0.332)] [data_t 0.145] [optim_t 0.323] [lr 0.005000] 2024-04-06 06:39:38,798 - Train: 16.51% [800400/4848000] [165.1/1000.0] [batch_t 0.329 (0.343)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 06:40:12,609 - Train: 16.51% [800500/4848000] [165.1/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 06:40:45,398 - Train: 16.51% [800600/4848000] [165.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 06:41:18,184 - Train: 16.52% [800700/4848000] [165.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 06:41:51,002 - Train: 16.52% [800800/4848000] [165.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 06:42:27,000 - Train: 16.52% [800900/4848000] [165.2/1000.0] [batch_t 0.328 (0.360)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 06:42:59,807 - Train: 16.52% [801000/4848000] [165.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 06:43:34,012 - Train: 16.52% [801100/4848000] [165.2/1000.0] [batch_t 0.328 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 06:44:09,707 - Train: 16.53% [801200/4848000] [165.3/1000.0] [batch_t 0.327 (0.357)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 06:44:42,464 - Train: 16.53% [801300/4848000] [165.3/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 06:45:15,833 - Train: 16.53% [801400/4848000] [165.3/1000.0] [batch_t 0.326 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 06:45:48,599 - Train: 16.53% [801500/4848000] [165.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 06:46:22,989 - Train: 16.53% [801600/4848000] [165.3/1000.0] [batch_t 0.325 (0.344)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 06:46:55,752 - Train: 16.54% [801700/4848000] [165.4/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 06:47:30,097 - Train: 16.54% [801800/4848000] [165.4/1000.0] [batch_t 0.333 (0.343)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 06:48:02,980 - Train: 16.54% [801900/4848000] [165.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 06:48:37,035 - Train: 16.54% [802000/4848000] [165.4/1000.0] [batch_t 0.330 (0.340)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 06:49:11,227 - Train: 16.54% [802100/4848000] [165.4/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 06:49:44,061 - Train: 16.55% [802200/4848000] [165.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 06:50:19,521 - Train: 16.55% [802300/4848000] [165.5/1000.0] [batch_t 0.329 (0.354)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 06:50:52,338 - Train: 16.55% [802400/4848000] [165.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 06:51:26,414 - Train: 16.55% [802500/4848000] [165.5/1000.0] [batch_t 0.325 (0.341)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 06:51:59,383 - Train: 16.56% [802600/4848000] [165.6/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 06:52:33,786 - Train: 16.56% [802700/4848000] [165.6/1000.0] [batch_t 0.329 (0.344)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 06:53:08,016 - Train: 16.56% [802800/4848000] [165.6/1000.0] [batch_t 0.328 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 06:53:40,877 - Train: 16.56% [802900/4848000] [165.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 06:54:13,729 - Train: 16.56% [803000/4848000] [165.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 06:54:46,556 - Train: 16.57% [803100/4848000] [165.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 06:55:19,890 - Train: 16.57% [803200/4848000] [165.7/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 06:55:52,693 - Train: 16.57% [803300/4848000] [165.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 06:56:25,547 - Train: 16.57% [803400/4848000] [165.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 06:56:58,365 - Train: 16.57% [803500/4848000] [165.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 06:57:32,952 - Train: 16.58% [803600/4848000] [165.8/1000.0] [batch_t 0.334 (0.346)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-06 06:58:07,269 - Train: 16.58% [803700/4848000] [165.8/1000.0] [batch_t 0.325 (0.343)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 06:58:40,064 - Train: 16.58% [803800/4848000] [165.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 06:59:13,020 - Train: 16.58% [803900/4848000] [165.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 06:59:45,985 - Train: 16.58% [804000/4848000] [165.8/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 07:00:19,971 - Train: 16.59% [804100/4848000] [165.9/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 07:00:52,761 - Train: 16.59% [804200/4848000] [165.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 07:01:26,577 - Train: 16.59% [804300/4848000] [165.9/1000.0] [batch_t 0.325 (0.338)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 07:01:59,354 - Train: 16.59% [804400/4848000] [165.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 07:02:34,992 - Train: 16.59% [804500/4848000] [165.9/1000.0] [batch_t 0.332 (0.356)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 07:03:08,564 - Train: 16.60% [804600/4848000] [166.0/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 07:03:41,341 - Train: 16.60% [804700/4848000] [166.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 07:04:03,644 - ==> Total time: 3 days, 7:38:02 Eta: 16 days, 16:05:18 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 07:04:17,817 - Train: 16.60% [804800/4848000] [166.0/1000.0] [batch_t 0.326 (0.336)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 07:04:50,629 - Train: 16.60% [804900/4848000] [166.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 07:05:23,473 - Train: 16.60% [805000/4848000] [166.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 07:05:56,384 - Train: 16.61% [805100/4848000] [166.1/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 07:06:30,178 - Train: 16.61% [805200/4848000] [166.1/1000.0] [batch_t 0.325 (0.338)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 07:07:03,172 - Train: 16.61% [805300/4848000] [166.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 07:07:36,037 - Train: 16.61% [805400/4848000] [166.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 07:08:09,000 - Train: 16.62% [805500/4848000] [166.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 07:08:41,866 - Train: 16.62% [805600/4848000] [166.2/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 07:09:15,171 - Train: 16.62% [805700/4848000] [166.2/1000.0] [batch_t 0.333 (0.333)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 07:09:47,962 - Train: 16.62% [805800/4848000] [166.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 07:10:20,858 - Train: 16.62% [805900/4848000] [166.2/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 07:10:53,711 - Train: 16.63% [806000/4848000] [166.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 07:11:28,453 - Train: 16.63% [806100/4848000] [166.3/1000.0] [batch_t 0.326 (0.347)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 07:12:01,225 - Train: 16.63% [806200/4848000] [166.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 07:12:35,993 - Train: 16.63% [806300/4848000] [166.3/1000.0] [batch_t 0.326 (0.348)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 07:13:09,833 - Train: 16.63% [806400/4848000] [166.3/1000.0] [batch_t 0.324 (0.338)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 07:13:42,652 - Train: 16.64% [806500/4848000] [166.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 07:14:15,857 - Train: 16.64% [806600/4848000] [166.4/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 07:14:48,652 - Train: 16.64% [806700/4848000] [166.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 07:15:23,173 - Train: 16.64% [806800/4848000] [166.4/1000.0] [batch_t 0.330 (0.345)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 07:15:55,964 - Train: 16.64% [806900/4848000] [166.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 07:16:30,232 - Train: 16.65% [807000/4848000] [166.5/1000.0] [batch_t 0.330 (0.343)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 07:17:03,032 - Train: 16.65% [807100/4848000] [166.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 07:17:36,853 - Train: 16.65% [807200/4848000] [166.5/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 07:18:09,949 - Train: 16.65% [807300/4848000] [166.5/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 07:18:42,801 - Train: 16.65% [807400/4848000] [166.5/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 07:19:16,499 - Train: 16.66% [807500/4848000] [166.6/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 07:19:49,309 - Train: 16.66% [807600/4848000] [166.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 07:20:23,866 - Train: 16.66% [807700/4848000] [166.6/1000.0] [batch_t 0.326 (0.345)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 07:20:56,642 - Train: 16.66% [807800/4848000] [166.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 07:21:30,671 - Train: 16.66% [807900/4848000] [166.6/1000.0] [batch_t 0.325 (0.340)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 07:22:03,433 - Train: 16.67% [808000/4848000] [166.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 07:22:38,008 - Train: 16.67% [808100/4848000] [166.7/1000.0] [batch_t 0.327 (0.346)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 07:23:12,076 - Train: 16.67% [808200/4848000] [166.7/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 07:23:44,883 - Train: 16.67% [808300/4848000] [166.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 07:24:18,853 - Train: 16.67% [808400/4848000] [166.7/1000.0] [batch_t 0.330 (0.340)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 07:24:51,582 - Train: 16.68% [808500/4848000] [166.8/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 07:25:24,555 - Train: 16.68% [808600/4848000] [166.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 07:25:57,429 - Train: 16.68% [808700/4848000] [166.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 07:26:30,315 - Train: 16.68% [808800/4848000] [166.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 07:27:03,101 - Train: 16.69% [808900/4848000] [166.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 07:27:35,894 - Train: 16.69% [809000/4848000] [166.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 07:28:08,768 - Train: 16.69% [809100/4848000] [166.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 07:28:41,570 - Train: 16.69% [809200/4848000] [166.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 07:29:15,541 - Train: 16.69% [809300/4848000] [166.9/1000.0] [batch_t 0.321 (0.340)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-06 07:29:48,377 - Train: 16.70% [809400/4848000] [167.0/1000.0] [batch_t 0.321 (0.328)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-06 07:30:22,301 - Train: 16.70% [809500/4848000] [167.0/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 07:30:55,230 - Train: 16.70% [809600/4848000] [167.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 07:31:00,495 - ==> Total time: 3 days, 8:04:58 Eta: 16 days, 15:27:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 07:31:31,818 - Train: 16.70% [809700/4848000] [167.0/1000.0] [batch_t 0.326 (0.346)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 07:32:04,633 - Train: 16.70% [809800/4848000] [167.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 07:32:37,404 - Train: 16.71% [809900/4848000] [167.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 07:33:10,265 - Train: 16.71% [810000/4848000] [167.1/1000.0] [batch_t 0.336 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-06 07:33:43,058 - Train: 16.71% [810100/4848000] [167.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 07:34:15,897 - Train: 16.71% [810200/4848000] [167.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 07:34:48,739 - Train: 16.71% [810300/4848000] [167.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 07:35:21,587 - Train: 16.72% [810400/4848000] [167.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 07:35:54,408 - Train: 16.72% [810500/4848000] [167.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 07:36:27,838 - Train: 16.72% [810600/4848000] [167.2/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 07:37:00,609 - Train: 16.72% [810700/4848000] [167.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 07:37:33,454 - Train: 16.72% [810800/4848000] [167.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 07:38:07,778 - Train: 16.73% [810900/4848000] [167.3/1000.0] [batch_t 0.756 (0.343)] [data_t 0.432] [optim_t 0.325] [lr 0.005000] 2024-04-06 07:38:40,571 - Train: 16.73% [811000/4848000] [167.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 07:39:13,724 - Train: 16.73% [811100/4848000] [167.3/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 07:39:46,492 - Train: 16.73% [811200/4848000] [167.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 07:40:19,251 - Train: 16.73% [811300/4848000] [167.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 07:40:52,017 - Train: 16.74% [811400/4848000] [167.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 07:41:25,783 - Train: 16.74% [811500/4848000] [167.4/1000.0] [batch_t 0.323 (0.338)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 07:41:58,575 - Train: 16.74% [811600/4848000] [167.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 07:42:31,998 - Train: 16.74% [811700/4848000] [167.4/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 07:43:04,853 - Train: 16.75% [811800/4848000] [167.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 07:43:38,365 - Train: 16.75% [811900/4848000] [167.5/1000.0] [batch_t 0.326 (0.335)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 07:44:12,481 - Train: 16.75% [812000/4848000] [167.5/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 07:44:45,257 - Train: 16.75% [812100/4848000] [167.5/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 07:45:19,195 - Train: 16.75% [812200/4848000] [167.5/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 07:45:52,023 - Train: 16.76% [812300/4848000] [167.6/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 07:46:26,626 - Train: 16.76% [812400/4848000] [167.6/1000.0] [batch_t 0.324 (0.346)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 07:46:59,480 - Train: 16.76% [812500/4848000] [167.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 07:47:32,398 - Train: 16.76% [812600/4848000] [167.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 07:48:06,189 - Train: 16.76% [812700/4848000] [167.6/1000.0] [batch_t 0.324 (0.338)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 07:48:39,025 - Train: 16.77% [812800/4848000] [167.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 07:49:12,347 - Train: 16.77% [812900/4848000] [167.7/1000.0] [batch_t 0.324 (0.333)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 07:49:45,146 - Train: 16.77% [813000/4848000] [167.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 07:50:17,951 - Train: 16.77% [813100/4848000] [167.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 07:50:50,835 - Train: 16.77% [813200/4848000] [167.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 07:51:24,953 - Train: 16.78% [813300/4848000] [167.8/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 07:51:57,709 - Train: 16.78% [813400/4848000] [167.8/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 07:52:30,758 - Train: 16.78% [813500/4848000] [167.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 07:53:03,560 - Train: 16.78% [813600/4848000] [167.8/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-06 07:53:37,884 - Train: 16.78% [813700/4848000] [167.8/1000.0] [batch_t 0.325 (0.343)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 07:54:11,934 - Train: 16.79% [813800/4848000] [167.9/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 07:54:44,776 - Train: 16.79% [813900/4848000] [167.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 07:55:17,653 - Train: 16.79% [814000/4848000] [167.9/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 07:55:50,422 - Train: 16.79% [814100/4848000] [167.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 07:56:23,156 - Train: 16.79% [814200/4848000] [167.9/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 07:56:55,964 - Train: 16.80% [814300/4848000] [168.0/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 07:57:30,120 - Train: 16.80% [814400/4848000] [168.0/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 07:57:51,088 - ==> Total time: 3 days, 8:31:49 Eta: 16 days, 14:49:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 07:58:06,509 - Train: 16.80% [814500/4848000] [168.0/1000.0] [batch_t 1.791 (0.372)] [data_t 1.466] [optim_t 0.325] [lr 0.005000] 2024-04-06 07:58:39,790 - Train: 16.80% [814600/4848000] [168.0/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 07:59:12,630 - Train: 16.80% [814700/4848000] [168.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 07:59:45,499 - Train: 16.81% [814800/4848000] [168.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 08:00:18,295 - Train: 16.81% [814900/4848000] [168.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 08:00:51,081 - Train: 16.81% [815000/4848000] [168.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 08:01:24,437 - Train: 16.81% [815100/4848000] [168.1/1000.0] [batch_t 0.324 (0.333)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 08:01:57,233 - Train: 16.82% [815200/4848000] [168.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 08:02:31,649 - Train: 16.82% [815300/4848000] [168.2/1000.0] [batch_t 0.329 (0.344)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 08:03:04,498 - Train: 16.82% [815400/4848000] [168.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 08:03:37,910 - Train: 16.82% [815500/4848000] [168.2/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 08:04:12,959 - Train: 16.82% [815600/4848000] [168.2/1000.0] [batch_t 0.325 (0.350)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 08:04:45,739 - Train: 16.83% [815700/4848000] [168.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 08:05:18,552 - Train: 16.83% [815800/4848000] [168.3/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 08:05:51,298 - Train: 16.83% [815900/4848000] [168.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 08:06:24,256 - Train: 16.83% [816000/4848000] [168.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 08:06:57,028 - Train: 16.83% [816100/4848000] [168.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 08:07:30,527 - Train: 16.84% [816200/4848000] [168.4/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 08:08:03,312 - Train: 16.84% [816300/4848000] [168.4/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 08:08:36,226 - Train: 16.84% [816400/4848000] [168.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 08:09:10,271 - Train: 16.84% [816500/4848000] [168.4/1000.0] [batch_t 0.333 (0.340)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 08:09:43,156 - Train: 16.84% [816600/4848000] [168.4/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 08:10:17,061 - Train: 16.85% [816700/4848000] [168.5/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 08:10:49,806 - Train: 16.85% [816800/4848000] [168.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 08:11:24,168 - Train: 16.85% [816900/4848000] [168.5/1000.0] [batch_t 0.325 (0.344)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 08:11:56,970 - Train: 16.85% [817000/4848000] [168.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 08:12:30,818 - Train: 16.85% [817100/4848000] [168.5/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 08:13:03,591 - Train: 16.86% [817200/4848000] [168.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 08:13:37,812 - Train: 16.86% [817300/4848000] [168.6/1000.0] [batch_t 0.328 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 08:14:12,164 - Train: 16.86% [817400/4848000] [168.6/1000.0] [batch_t 0.330 (0.343)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 08:14:44,962 - Train: 16.86% [817500/4848000] [168.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 08:15:18,845 - Train: 16.86% [817600/4848000] [168.6/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 08:15:51,698 - Train: 16.87% [817700/4848000] [168.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 08:16:24,551 - Train: 16.87% [817800/4848000] [168.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 08:16:57,400 - Train: 16.87% [817900/4848000] [168.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 08:17:32,089 - Train: 16.87% [818000/4848000] [168.7/1000.0] [batch_t 0.328 (0.347)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 08:18:04,861 - Train: 16.88% [818100/4848000] [168.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 08:18:38,733 - Train: 16.88% [818200/4848000] [168.8/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 08:19:12,813 - Train: 16.88% [818300/4848000] [168.8/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 08:19:45,596 - Train: 16.88% [818400/4848000] [168.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 08:20:19,477 - Train: 16.88% [818500/4848000] [168.8/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 08:20:52,225 - Train: 16.89% [818600/4848000] [168.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 08:21:26,280 - Train: 16.89% [818700/4848000] [168.9/1000.0] [batch_t 0.333 (0.340)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 08:21:59,087 - Train: 16.89% [818800/4848000] [168.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 08:22:33,262 - Train: 16.89% [818900/4848000] [168.9/1000.0] [batch_t 0.328 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 08:23:06,587 - Train: 16.89% [819000/4848000] [168.9/1000.0] [batch_t 0.324 (0.333)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 08:23:39,412 - Train: 16.90% [819100/4848000] [169.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 08:24:12,948 - Train: 16.90% [819200/4848000] [169.0/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 08:24:45,746 - Train: 16.90% [819300/4848000] [169.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 08:24:49,691 - ==> Total time: 3 days, 8:58:48 Eta: 16 days, 14:11:30 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 08:25:21,926 - Train: 16.90% [819400/4848000] [169.0/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 08:25:54,794 - Train: 16.90% [819500/4848000] [169.0/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 08:26:27,795 - Train: 16.91% [819600/4848000] [169.1/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 08:27:00,575 - Train: 16.91% [819700/4848000] [169.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 08:27:33,407 - Train: 16.91% [819800/4848000] [169.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 08:28:06,897 - Train: 16.91% [819900/4848000] [169.1/1000.0] [batch_t 0.743 (0.335)] [data_t 0.417] [optim_t 0.326] [lr 0.005000] 2024-04-06 08:28:39,701 - Train: 16.91% [820000/4848000] [169.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 08:29:13,447 - Train: 16.92% [820100/4848000] [169.2/1000.0] [batch_t 0.323 (0.337)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 08:29:46,272 - Train: 16.92% [820200/4848000] [169.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 08:30:19,058 - Train: 16.92% [820300/4848000] [169.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 08:30:51,926 - Train: 16.92% [820400/4848000] [169.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 08:31:25,717 - Train: 16.92% [820500/4848000] [169.2/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 08:31:58,503 - Train: 16.93% [820600/4848000] [169.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 08:32:31,697 - Train: 16.93% [820700/4848000] [169.3/1000.0] [batch_t 0.332 (0.332)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 08:33:05,294 - Train: 16.93% [820800/4848000] [169.3/1000.0] [batch_t 0.964 (0.336)] [data_t 0.639] [optim_t 0.325] [lr 0.005000] 2024-04-06 08:33:38,083 - Train: 16.93% [820900/4848000] [169.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 08:34:10,795 - Train: 16.93% [821000/4848000] [169.3/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 08:34:43,560 - Train: 16.94% [821100/4848000] [169.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 08:35:17,632 - Train: 16.94% [821200/4848000] [169.4/1000.0] [batch_t 0.326 (0.341)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 08:35:50,348 - Train: 16.94% [821300/4848000] [169.4/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 08:36:24,376 - Train: 16.94% [821400/4848000] [169.4/1000.0] [batch_t 0.326 (0.340)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 08:36:57,233 - Train: 16.95% [821500/4848000] [169.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 08:37:30,050 - Train: 16.95% [821600/4848000] [169.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 08:38:02,856 - Train: 16.95% [821700/4848000] [169.5/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 08:38:37,147 - Train: 16.95% [821800/4848000] [169.5/1000.0] [batch_t 0.327 (0.343)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 08:39:11,295 - Train: 16.95% [821900/4848000] [169.5/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 08:39:44,182 - Train: 16.96% [822000/4848000] [169.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 08:40:17,327 - Train: 16.96% [822100/4848000] [169.6/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 08:40:50,122 - Train: 16.96% [822200/4848000] [169.6/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 08:41:24,165 - Train: 16.96% [822300/4848000] [169.6/1000.0] [batch_t 0.325 (0.340)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 08:41:56,936 - Train: 16.96% [822400/4848000] [169.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 08:42:29,765 - Train: 16.97% [822500/4848000] [169.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 08:43:02,485 - Train: 16.97% [822600/4848000] [169.7/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 08:43:35,241 - Train: 16.97% [822700/4848000] [169.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 08:44:09,110 - Train: 16.97% [822800/4848000] [169.7/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 08:44:41,974 - Train: 16.97% [822900/4848000] [169.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 08:45:15,555 - Train: 16.98% [823000/4848000] [169.8/1000.0] [batch_t 0.331 (0.336)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 08:45:48,332 - Train: 16.98% [823100/4848000] [169.8/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 08:46:22,284 - Train: 16.98% [823200/4848000] [169.8/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 08:46:55,061 - Train: 16.98% [823300/4848000] [169.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 08:47:28,835 - Train: 16.98% [823400/4848000] [169.8/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 08:48:01,676 - Train: 16.99% [823500/4848000] [169.9/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-06 08:48:35,734 - Train: 16.99% [823600/4848000] [169.9/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 08:49:09,269 - Train: 16.99% [823700/4848000] [169.9/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 08:49:42,093 - Train: 16.99% [823800/4848000] [169.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 08:50:17,851 - Train: 16.99% [823900/4848000] [169.9/1000.0] [batch_t 0.327 (0.357)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 08:50:50,666 - Train: 17.00% [824000/4848000] [170.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 08:51:25,826 - Train: 17.00% [824100/4848000] [170.0/1000.0] [batch_t 0.327 (0.351)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 08:51:45,529 - ==> Total time: 3 days, 9:25:44 Eta: 16 days, 13:33:52 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 08:52:00,740 - Train: 17.00% [824200/4848000] [170.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 08:52:34,916 - Train: 17.00% [824300/4848000] [170.0/1000.0] [batch_t 0.327 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 08:53:09,483 - Train: 17.00% [824400/4848000] [170.0/1000.0] [batch_t 0.326 (0.346)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 08:53:42,241 - Train: 17.01% [824500/4848000] [170.1/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 08:54:16,365 - Train: 17.01% [824600/4848000] [170.1/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 08:54:49,191 - Train: 17.01% [824700/4848000] [170.1/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 08:55:22,741 - Train: 17.01% [824800/4848000] [170.1/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 08:55:55,479 - Train: 17.02% [824900/4848000] [170.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 08:56:28,478 - Train: 17.02% [825000/4848000] [170.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 08:57:01,437 - Train: 17.02% [825100/4848000] [170.2/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 08:57:35,368 - Train: 17.02% [825200/4848000] [170.2/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 08:58:10,508 - Train: 17.02% [825300/4848000] [170.2/1000.0] [batch_t 0.325 (0.351)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 08:58:43,260 - Train: 17.03% [825400/4848000] [170.3/1000.0] [batch_t 0.334 (0.327)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-06 08:59:16,957 - Train: 17.03% [825500/4848000] [170.3/1000.0] [batch_t 0.323 (0.337)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 08:59:49,840 - Train: 17.03% [825600/4848000] [170.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 09:00:27,428 - Train: 17.03% [825700/4848000] [170.3/1000.0] [batch_t 0.327 (0.376)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 09:01:00,282 - Train: 17.03% [825800/4848000] [170.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 09:01:34,029 - Train: 17.04% [825900/4848000] [170.4/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 09:02:08,239 - Train: 17.04% [826000/4848000] [170.4/1000.0] [batch_t 0.331 (0.342)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 09:02:41,018 - Train: 17.04% [826100/4848000] [170.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 09:03:14,214 - Train: 17.04% [826200/4848000] [170.4/1000.0] [batch_t 0.323 (0.332)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 09:03:46,996 - Train: 17.04% [826300/4848000] [170.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 09:04:20,845 - Train: 17.05% [826400/4848000] [170.5/1000.0] [batch_t 0.325 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 09:04:53,605 - Train: 17.05% [826500/4848000] [170.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 09:05:27,802 - Train: 17.05% [826600/4848000] [170.5/1000.0] [batch_t 0.331 (0.342)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 09:06:00,573 - Train: 17.05% [826700/4848000] [170.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 09:06:35,160 - Train: 17.05% [826800/4848000] [170.5/1000.0] [batch_t 0.325 (0.346)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 09:07:11,651 - Train: 17.06% [826900/4848000] [170.6/1000.0] [batch_t 0.330 (0.365)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 09:07:44,425 - Train: 17.06% [827000/4848000] [170.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 09:08:17,151 - Train: 17.06% [827100/4848000] [170.6/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 09:08:49,916 - Train: 17.06% [827200/4848000] [170.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 09:09:23,501 - Train: 17.06% [827300/4848000] [170.6/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 09:09:56,290 - Train: 17.07% [827400/4848000] [170.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 09:10:30,011 - Train: 17.07% [827500/4848000] [170.7/1000.0] [batch_t 0.326 (0.337)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 09:11:02,860 - Train: 17.07% [827600/4848000] [170.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 09:11:35,894 - Train: 17.07% [827700/4848000] [170.7/1000.0] [batch_t 0.323 (0.330)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 09:12:11,665 - Train: 17.08% [827800/4848000] [170.8/1000.0] [batch_t 0.324 (0.358)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 09:12:44,534 - Train: 17.08% [827900/4848000] [170.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 09:13:18,250 - Train: 17.08% [828000/4848000] [170.8/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 09:13:51,057 - Train: 17.08% [828100/4848000] [170.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 09:14:25,383 - Train: 17.08% [828200/4848000] [170.8/1000.0] [batch_t 0.334 (0.343)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-06 09:14:58,223 - Train: 17.09% [828300/4848000] [170.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 09:15:32,020 - Train: 17.09% [828400/4848000] [170.9/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 09:16:04,917 - Train: 17.09% [828500/4848000] [170.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 09:16:37,833 - Train: 17.09% [828600/4848000] [170.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 09:17:13,634 - Train: 17.09% [828700/4848000] [170.9/1000.0] [batch_t 0.330 (0.358)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 09:17:46,452 - Train: 17.10% [828800/4848000] [171.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 09:18:20,202 - Train: 17.10% [828900/4848000] [171.0/1000.0] [batch_t 0.326 (0.337)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 09:18:53,098 - Train: 17.10% [829000/4848000] [171.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 09:18:55,730 - ==> Total time: 3 days, 9:52:54 Eta: 16 days, 12:57:31 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 09:19:28,883 - Train: 17.10% [829100/4848000] [171.0/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 09:20:01,908 - Train: 17.10% [829200/4848000] [171.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 09:20:36,136 - Train: 17.11% [829300/4848000] [171.1/1000.0] [batch_t 0.330 (0.342)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 09:21:09,673 - Train: 17.11% [829400/4848000] [171.1/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 09:21:42,533 - Train: 17.11% [829500/4848000] [171.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 09:22:16,046 - Train: 17.11% [829600/4848000] [171.1/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 09:22:48,829 - Train: 17.11% [829700/4848000] [171.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 09:23:22,879 - Train: 17.12% [829800/4848000] [171.2/1000.0] [batch_t 0.334 (0.340)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-06 09:23:55,801 - Train: 17.12% [829900/4848000] [171.2/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 09:24:30,027 - Train: 17.12% [830000/4848000] [171.2/1000.0] [batch_t 0.335 (0.342)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-06 09:25:02,835 - Train: 17.12% [830100/4848000] [171.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 09:25:36,856 - Train: 17.12% [830200/4848000] [171.2/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 09:26:09,746 - Train: 17.13% [830300/4848000] [171.3/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-06 09:26:42,554 - Train: 17.13% [830400/4848000] [171.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 09:27:16,340 - Train: 17.13% [830500/4848000] [171.3/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 09:27:49,292 - Train: 17.13% [830600/4848000] [171.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 09:28:23,133 - Train: 17.13% [830700/4848000] [171.3/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 09:28:55,932 - Train: 17.14% [830800/4848000] [171.4/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 09:29:28,808 - Train: 17.14% [830900/4848000] [171.4/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-06 09:30:01,601 - Train: 17.14% [831000/4848000] [171.4/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 09:30:35,178 - Train: 17.14% [831100/4848000] [171.4/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 09:31:08,265 - Train: 17.15% [831200/4848000] [171.5/1000.0] [batch_t 0.324 (0.331)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 09:31:41,053 - Train: 17.15% [831300/4848000] [171.5/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 09:32:14,828 - Train: 17.15% [831400/4848000] [171.5/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 09:32:47,627 - Train: 17.15% [831500/4848000] [171.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 09:33:20,466 - Train: 17.15% [831600/4848000] [171.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 09:33:53,264 - Train: 17.16% [831700/4848000] [171.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 09:34:27,533 - Train: 17.16% [831800/4848000] [171.6/1000.0] [batch_t 0.328 (0.343)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 09:35:00,382 - Train: 17.16% [831900/4848000] [171.6/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 09:35:33,191 - Train: 17.16% [832000/4848000] [171.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 09:36:06,155 - Train: 17.16% [832100/4848000] [171.6/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 09:36:38,927 - Train: 17.17% [832200/4848000] [171.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 09:37:11,747 - Train: 17.17% [832300/4848000] [171.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 09:37:44,524 - Train: 17.17% [832400/4848000] [171.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 09:38:17,320 - Train: 17.17% [832500/4848000] [171.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 09:38:50,125 - Train: 17.17% [832600/4848000] [171.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 09:39:22,889 - Train: 17.18% [832700/4848000] [171.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 09:39:55,699 - Train: 17.18% [832800/4848000] [171.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 09:40:29,623 - Train: 17.18% [832900/4848000] [171.8/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 09:41:02,405 - Train: 17.18% [833000/4848000] [171.8/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 09:41:36,519 - Train: 17.18% [833100/4848000] [171.8/1000.0] [batch_t 0.325 (0.341)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 09:42:10,555 - Train: 17.19% [833200/4848000] [171.9/1000.0] [batch_t 0.332 (0.340)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 09:42:43,389 - Train: 17.19% [833300/4848000] [171.9/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 09:43:17,880 - Train: 17.19% [833400/4848000] [171.9/1000.0] [batch_t 0.327 (0.345)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 09:43:50,757 - Train: 17.19% [833500/4848000] [171.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 09:44:23,584 - Train: 17.19% [833600/4848000] [171.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 09:44:56,428 - Train: 17.20% [833700/4848000] [172.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 09:45:30,574 - Train: 17.20% [833800/4848000] [172.0/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 09:45:48,969 - ==> Total time: 3 days, 10:19:47 Eta: 16 days, 12:19:55 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 09:46:05,956 - Train: 17.20% [833900/4848000] [172.0/1000.0] [batch_t 0.326 (0.341)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 09:46:38,738 - Train: 17.20% [834000/4848000] [172.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 09:47:13,203 - Train: 17.21% [834100/4848000] [172.1/1000.0] [batch_t 0.324 (0.345)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 09:47:46,027 - Train: 17.21% [834200/4848000] [172.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 09:48:20,067 - Train: 17.21% [834300/4848000] [172.1/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 09:48:52,841 - Train: 17.21% [834400/4848000] [172.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 09:49:26,647 - Train: 17.21% [834500/4848000] [172.1/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 09:49:59,418 - Train: 17.22% [834600/4848000] [172.2/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 09:50:33,712 - Train: 17.22% [834700/4848000] [172.2/1000.0] [batch_t 0.327 (0.343)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 09:51:07,557 - Train: 17.22% [834800/4848000] [172.2/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 09:51:40,326 - Train: 17.22% [834900/4848000] [172.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 09:52:14,528 - Train: 17.22% [835000/4848000] [172.2/1000.0] [batch_t 0.328 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 09:52:47,351 - Train: 17.23% [835100/4848000] [172.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 09:53:21,845 - Train: 17.23% [835200/4848000] [172.3/1000.0] [batch_t 0.326 (0.345)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 09:53:54,697 - Train: 17.23% [835300/4848000] [172.3/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 09:54:28,099 - Train: 17.23% [835400/4848000] [172.3/1000.0] [batch_t 0.321 (0.334)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-06 09:55:00,963 - Train: 17.23% [835500/4848000] [172.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 09:55:34,599 - Train: 17.24% [835600/4848000] [172.4/1000.0] [batch_t 0.331 (0.336)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 09:56:08,729 - Train: 17.24% [835700/4848000] [172.4/1000.0] [batch_t 0.332 (0.341)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 09:56:41,587 - Train: 17.24% [835800/4848000] [172.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 09:57:14,340 - Train: 17.24% [835900/4848000] [172.4/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 09:57:47,121 - Train: 17.24% [836000/4848000] [172.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 09:58:20,021 - Train: 17.25% [836100/4848000] [172.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 09:58:52,913 - Train: 17.25% [836200/4848000] [172.5/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 09:59:26,645 - Train: 17.25% [836300/4848000] [172.5/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 09:59:59,459 - Train: 17.25% [836400/4848000] [172.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 10:00:32,261 - Train: 17.25% [836500/4848000] [172.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 10:01:05,045 - Train: 17.26% [836600/4848000] [172.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 10:01:37,826 - Train: 17.26% [836700/4848000] [172.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 10:02:11,521 - Train: 17.26% [836800/4848000] [172.6/1000.0] [batch_t 0.333 (0.337)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 10:02:44,257 - Train: 17.26% [836900/4848000] [172.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 10:03:18,533 - Train: 17.26% [837000/4848000] [172.6/1000.0] [batch_t 0.328 (0.343)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 10:03:51,375 - Train: 17.27% [837100/4848000] [172.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 10:04:24,161 - Train: 17.27% [837200/4848000] [172.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 10:04:56,989 - Train: 17.27% [837300/4848000] [172.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 10:05:30,904 - Train: 17.27% [837400/4848000] [172.7/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 10:06:03,684 - Train: 17.28% [837500/4848000] [172.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 10:06:37,754 - Train: 17.28% [837600/4848000] [172.8/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 10:07:12,186 - Train: 17.28% [837700/4848000] [172.8/1000.0] [batch_t 0.327 (0.344)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 10:07:45,057 - Train: 17.28% [837800/4848000] [172.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 10:08:17,821 - Train: 17.28% [837900/4848000] [172.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 10:08:50,554 - Train: 17.29% [838000/4848000] [172.9/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 10:09:24,006 - Train: 17.29% [838100/4848000] [172.9/1000.0] [batch_t 0.331 (0.334)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 10:09:56,775 - Train: 17.29% [838200/4848000] [172.9/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 10:10:29,848 - Train: 17.29% [838300/4848000] [172.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 10:11:02,643 - Train: 17.29% [838400/4848000] [172.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 10:11:36,741 - Train: 17.30% [838500/4848000] [173.0/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 10:12:11,500 - Train: 17.30% [838600/4848000] [173.0/1000.0] [batch_t 0.328 (0.347)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 10:12:44,298 - Train: 17.30% [838700/4848000] [173.0/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-06 10:12:45,629 - ==> Total time: 3 days, 10:46:44 Eta: 16 days, 11:42:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 10:13:20,314 - Train: 17.30% [838800/4848000] [173.0/1000.0] [batch_t 0.327 (0.343)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 10:13:53,063 - Train: 17.30% [838900/4848000] [173.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 10:14:27,163 - Train: 17.31% [839000/4848000] [173.1/1000.0] [batch_t 0.317 (0.341)] [data_t 0.002] [optim_t 0.315] [lr 0.005000] 2024-04-06 10:14:59,927 - Train: 17.31% [839100/4848000] [173.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 10:15:34,210 - Train: 17.31% [839200/4848000] [173.1/1000.0] [batch_t 0.325 (0.343)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 10:16:08,806 - Train: 17.31% [839300/4848000] [173.1/1000.0] [batch_t 0.326 (0.346)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 10:16:41,553 - Train: 17.31% [839400/4848000] [173.1/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 10:17:16,756 - Train: 17.32% [839500/4848000] [173.2/1000.0] [batch_t 0.330 (0.352)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 10:17:49,549 - Train: 17.32% [839600/4848000] [173.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 10:18:23,597 - Train: 17.32% [839700/4848000] [173.2/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 10:18:56,430 - Train: 17.32% [839800/4848000] [173.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 10:19:30,886 - Train: 17.32% [839900/4848000] [173.2/1000.0] [batch_t 0.328 (0.344)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 10:20:03,656 - Train: 17.33% [840000/4848000] [173.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 10:20:37,746 - Train: 17.33% [840100/4848000] [173.3/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 10:21:12,609 - Train: 17.33% [840200/4848000] [173.3/1000.0] [batch_t 0.327 (0.349)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 10:21:45,418 - Train: 17.33% [840300/4848000] [173.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 10:22:19,146 - Train: 17.33% [840400/4848000] [173.3/1000.0] [batch_t 0.331 (0.337)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 10:22:51,964 - Train: 17.34% [840500/4848000] [173.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 10:23:25,684 - Train: 17.34% [840600/4848000] [173.4/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 10:23:58,468 - Train: 17.34% [840700/4848000] [173.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 10:24:32,108 - Train: 17.34% [840800/4848000] [173.4/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 10:25:06,289 - Train: 17.35% [840900/4848000] [173.5/1000.0] [batch_t 0.327 (0.342)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 10:25:39,066 - Train: 17.35% [841000/4848000] [173.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 10:26:13,090 - Train: 17.35% [841100/4848000] [173.5/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 10:26:45,861 - Train: 17.35% [841200/4848000] [173.5/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 10:27:19,377 - Train: 17.35% [841300/4848000] [173.5/1000.0] [batch_t 0.326 (0.335)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 10:27:52,190 - Train: 17.36% [841400/4848000] [173.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 10:28:27,794 - Train: 17.36% [841500/4848000] [173.6/1000.0] [batch_t 0.331 (0.356)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 10:29:00,921 - Train: 17.36% [841600/4848000] [173.6/1000.0] [batch_t 0.323 (0.331)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 10:29:36,208 - Train: 17.36% [841700/4848000] [173.6/1000.0] [batch_t 0.325 (0.353)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 10:30:10,250 - Train: 17.36% [841800/4848000] [173.6/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 10:30:43,177 - Train: 17.37% [841900/4848000] [173.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 10:31:15,970 - Train: 17.37% [842000/4848000] [173.7/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 10:31:48,765 - Train: 17.37% [842100/4848000] [173.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 10:32:21,517 - Train: 17.37% [842200/4848000] [173.7/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 10:32:54,358 - Train: 17.37% [842300/4848000] [173.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 10:33:27,172 - Train: 17.38% [842400/4848000] [173.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 10:33:59,955 - Train: 17.38% [842500/4848000] [173.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 10:34:33,751 - Train: 17.38% [842600/4848000] [173.8/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 10:35:07,574 - Train: 17.38% [842700/4848000] [173.8/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 10:35:40,349 - Train: 17.38% [842800/4848000] [173.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 10:36:13,126 - Train: 17.39% [842900/4848000] [173.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 10:36:45,921 - Train: 17.39% [843000/4848000] [173.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 10:37:20,672 - Train: 17.39% [843100/4848000] [173.9/1000.0] [batch_t 0.328 (0.347)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 10:37:53,460 - Train: 17.39% [843200/4848000] [173.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 10:38:27,281 - Train: 17.39% [843300/4848000] [173.9/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 10:39:00,094 - Train: 17.40% [843400/4848000] [174.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 10:39:33,713 - Train: 17.40% [843500/4848000] [174.0/1000.0] [batch_t 0.325 (0.336)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 10:39:50,768 - ==> Total time: 3 days, 11:13:49 Eta: 16 days, 11:06:18 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 10:40:09,040 - Train: 17.40% [843600/4848000] [174.0/1000.0] [batch_t 0.325 (0.335)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 10:40:41,834 - Train: 17.40% [843700/4848000] [174.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 10:41:14,894 - Train: 17.41% [843800/4848000] [174.1/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 10:41:47,700 - Train: 17.41% [843900/4848000] [174.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 10:42:21,459 - Train: 17.41% [844000/4848000] [174.1/1000.0] [batch_t 0.337 (0.337)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-06 10:42:54,321 - Train: 17.41% [844100/4848000] [174.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 10:43:27,979 - Train: 17.41% [844200/4848000] [174.1/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 10:44:00,812 - Train: 17.42% [844300/4848000] [174.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 10:44:33,631 - Train: 17.42% [844400/4848000] [174.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 10:45:07,535 - Train: 17.42% [844500/4848000] [174.2/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 10:45:40,462 - Train: 17.42% [844600/4848000] [174.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 10:46:14,142 - Train: 17.42% [844700/4848000] [174.2/1000.0] [batch_t 0.333 (0.337)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 10:46:46,925 - Train: 17.43% [844800/4848000] [174.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 10:47:20,967 - Train: 17.43% [844900/4848000] [174.3/1000.0] [batch_t 0.321 (0.340)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-06 10:47:53,873 - Train: 17.43% [845000/4848000] [174.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 10:48:27,645 - Train: 17.43% [845100/4848000] [174.3/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 10:49:00,410 - Train: 17.43% [845200/4848000] [174.3/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-06 10:49:34,254 - Train: 17.44% [845300/4848000] [174.4/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 10:50:07,886 - Train: 17.44% [845400/4848000] [174.4/1000.0] [batch_t 0.322 (0.336)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 10:50:40,655 - Train: 17.44% [845500/4848000] [174.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 10:51:14,110 - Train: 17.44% [845600/4848000] [174.4/1000.0] [batch_t 0.325 (0.334)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 10:51:46,883 - Train: 17.44% [845700/4848000] [174.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 10:52:21,050 - Train: 17.45% [845800/4848000] [174.5/1000.0] [batch_t 0.326 (0.342)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 10:52:53,834 - Train: 17.45% [845900/4848000] [174.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 10:53:27,528 - Train: 17.45% [846000/4848000] [174.5/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 10:54:00,445 - Train: 17.45% [846100/4848000] [174.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 10:54:34,178 - Train: 17.45% [846200/4848000] [174.5/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 10:55:07,892 - Train: 17.46% [846300/4848000] [174.6/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 10:55:40,659 - Train: 17.46% [846400/4848000] [174.6/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 10:56:13,417 - Train: 17.46% [846500/4848000] [174.6/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 10:56:46,208 - Train: 17.46% [846600/4848000] [174.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 10:57:19,008 - Train: 17.46% [846700/4848000] [174.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 10:57:51,785 - Train: 17.47% [846800/4848000] [174.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 10:58:24,577 - Train: 17.47% [846900/4848000] [174.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 10:58:57,311 - Train: 17.47% [847000/4848000] [174.7/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 10:59:30,121 - Train: 17.47% [847100/4848000] [174.7/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 11:00:02,945 - Train: 17.48% [847200/4848000] [174.8/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 11:00:36,363 - Train: 17.48% [847300/4848000] [174.8/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 11:01:09,189 - Train: 17.48% [847400/4848000] [174.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 11:01:42,082 - Train: 17.48% [847500/4848000] [174.8/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 11:02:15,454 - Train: 17.48% [847600/4848000] [174.8/1000.0] [batch_t 0.326 (0.334)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 11:02:48,220 - Train: 17.49% [847700/4848000] [174.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 11:03:21,070 - Train: 17.49% [847800/4848000] [174.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 11:03:53,823 - Train: 17.49% [847900/4848000] [174.9/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 11:04:29,301 - Train: 17.49% [848000/4848000] [174.9/1000.0] [batch_t 0.327 (0.355)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 11:05:02,113 - Train: 17.49% [848100/4848000] [174.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 11:05:36,658 - Train: 17.50% [848200/4848000] [175.0/1000.0] [batch_t 0.328 (0.345)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 11:06:09,463 - Train: 17.50% [848300/4848000] [175.0/1000.0] [batch_t 0.334 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-06 11:06:42,227 - Train: 17.50% [848400/4848000] [175.0/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 11:06:42,231 - ==> Total time: 3 days, 11:40:40 Eta: 16 days, 10:28:54 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 11:07:18,798 - Train: 17.50% [848500/4848000] [175.0/1000.0] [batch_t 0.327 (0.346)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 11:07:51,594 - Train: 17.50% [848600/4848000] [175.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 11:08:25,790 - Train: 17.51% [848700/4848000] [175.1/1000.0] [batch_t 0.327 (0.342)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 11:08:58,667 - Train: 17.51% [848800/4848000] [175.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 11:09:32,942 - Train: 17.51% [848900/4848000] [175.1/1000.0] [batch_t 0.329 (0.343)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 11:10:06,983 - Train: 17.51% [849000/4848000] [175.1/1000.0] [batch_t 0.330 (0.340)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 11:10:39,785 - Train: 17.51% [849100/4848000] [175.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 11:11:15,872 - Train: 17.52% [849200/4848000] [175.2/1000.0] [batch_t 0.327 (0.361)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 11:11:48,723 - Train: 17.52% [849300/4848000] [175.2/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 11:12:22,513 - Train: 17.52% [849400/4848000] [175.2/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 11:12:55,384 - Train: 17.52% [849500/4848000] [175.2/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 11:13:30,852 - Train: 17.52% [849600/4848000] [175.2/1000.0] [batch_t 0.326 (0.355)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 11:14:03,641 - Train: 17.53% [849700/4848000] [175.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 11:14:37,658 - Train: 17.53% [849800/4848000] [175.3/1000.0] [batch_t 0.325 (0.340)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 11:15:12,041 - Train: 17.53% [849900/4848000] [175.3/1000.0] [batch_t 0.324 (0.344)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 11:15:44,881 - Train: 17.53% [850000/4848000] [175.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 11:16:19,434 - Train: 17.54% [850100/4848000] [175.4/1000.0] [batch_t 0.328 (0.345)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 11:16:52,370 - Train: 17.54% [850200/4848000] [175.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 11:17:25,372 - Train: 17.54% [850300/4848000] [175.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 11:17:58,241 - Train: 17.54% [850400/4848000] [175.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 11:18:31,975 - Train: 17.54% [850500/4848000] [175.4/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 11:19:04,706 - Train: 17.55% [850600/4848000] [175.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 11:19:37,507 - Train: 17.55% [850700/4848000] [175.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 11:20:10,861 - Train: 17.55% [850800/4848000] [175.5/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 11:20:43,637 - Train: 17.55% [850900/4848000] [175.5/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 11:21:18,078 - Train: 17.55% [851000/4848000] [175.5/1000.0] [batch_t 0.326 (0.344)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 11:21:50,856 - Train: 17.56% [851100/4848000] [175.6/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 11:22:25,396 - Train: 17.56% [851200/4848000] [175.6/1000.0] [batch_t 0.328 (0.345)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 11:22:58,268 - Train: 17.56% [851300/4848000] [175.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 11:23:32,744 - Train: 17.56% [851400/4848000] [175.6/1000.0] [batch_t 0.328 (0.345)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 11:24:06,682 - Train: 17.56% [851500/4848000] [175.6/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 11:24:39,451 - Train: 17.57% [851600/4848000] [175.7/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 11:25:14,094 - Train: 17.57% [851700/4848000] [175.7/1000.0] [batch_t 0.330 (0.346)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 11:25:47,095 - Train: 17.57% [851800/4848000] [175.7/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 11:26:21,721 - Train: 17.57% [851900/4848000] [175.7/1000.0] [batch_t 0.327 (0.346)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 11:26:54,527 - Train: 17.57% [852000/4848000] [175.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 11:27:27,376 - Train: 17.58% [852100/4848000] [175.8/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 11:28:00,246 - Train: 17.58% [852200/4848000] [175.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 11:28:33,095 - Train: 17.58% [852300/4848000] [175.8/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 11:29:07,513 - Train: 17.58% [852400/4848000] [175.8/1000.0] [batch_t 0.328 (0.344)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 11:29:40,371 - Train: 17.58% [852500/4848000] [175.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 11:30:15,000 - Train: 17.59% [852600/4848000] [175.9/1000.0] [batch_t 0.328 (0.346)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 11:30:47,835 - Train: 17.59% [852700/4848000] [175.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 11:31:20,690 - Train: 17.59% [852800/4848000] [175.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 11:31:53,592 - Train: 17.59% [852900/4848000] [175.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 11:32:27,973 - Train: 17.59% [853000/4848000] [175.9/1000.0] [batch_t 0.323 (0.344)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 11:33:00,843 - Train: 17.60% [853100/4848000] [176.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 11:33:34,828 - Train: 17.60% [853200/4848000] [176.0/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 11:33:50,627 - ==> Total time: 3 days, 12:07:49 Eta: 16 days, 9:52:58 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 11:34:10,092 - Train: 17.60% [853300/4848000] [176.0/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 11:34:42,911 - Train: 17.60% [853400/4848000] [176.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 11:35:15,730 - Train: 17.61% [853500/4848000] [176.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 11:35:48,523 - Train: 17.61% [853600/4848000] [176.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 11:36:21,311 - Train: 17.61% [853700/4848000] [176.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 11:36:54,102 - Train: 17.61% [853800/4848000] [176.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 11:37:27,285 - Train: 17.61% [853900/4848000] [176.1/1000.0] [batch_t 0.332 (0.332)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 11:38:00,101 - Train: 17.62% [854000/4848000] [176.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 11:38:32,836 - Train: 17.62% [854100/4848000] [176.2/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 11:39:06,963 - Train: 17.62% [854200/4848000] [176.2/1000.0] [batch_t 0.336 (0.341)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-06 11:39:39,770 - Train: 17.62% [854300/4848000] [176.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 11:40:13,366 - Train: 17.62% [854400/4848000] [176.2/1000.0] [batch_t 0.332 (0.336)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 11:40:46,364 - Train: 17.63% [854500/4848000] [176.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 11:41:19,212 - Train: 17.63% [854600/4848000] [176.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 11:41:52,057 - Train: 17.63% [854700/4848000] [176.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 11:42:24,859 - Train: 17.63% [854800/4848000] [176.3/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 11:42:57,677 - Train: 17.63% [854900/4848000] [176.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 11:43:30,499 - Train: 17.64% [855000/4848000] [176.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 11:44:03,293 - Train: 17.64% [855100/4848000] [176.4/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 11:44:36,080 - Train: 17.64% [855200/4848000] [176.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 11:45:11,217 - Train: 17.64% [855300/4848000] [176.4/1000.0] [batch_t 0.325 (0.351)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 11:45:51,414 - Train: 17.64% [855400/4848000] [176.4/1000.0] [batch_t 0.324 (0.402)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 11:46:36,276 - Train: 17.65% [855500/4848000] [176.5/1000.0] [batch_t 0.329 (0.449)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 11:47:16,839 - Train: 17.65% [855600/4848000] [176.5/1000.0] [batch_t 0.324 (0.406)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 11:48:00,276 - Train: 17.65% [855700/4848000] [176.5/1000.0] [batch_t 0.330 (0.434)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 11:49:28,186 - Train: 17.65% [855800/4848000] [176.5/1000.0] [batch_t 0.323 (0.879)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 11:50:30,624 - Train: 17.65% [855900/4848000] [176.5/1000.0] [batch_t 0.328 (0.624)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 11:51:12,150 - Train: 17.66% [856000/4848000] [176.6/1000.0] [batch_t 0.662 (0.415)] [data_t 0.341] [optim_t 0.322] [lr 0.005000] 2024-04-06 11:51:50,544 - Train: 17.66% [856100/4848000] [176.6/1000.0] [batch_t 0.331 (0.384)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 11:52:24,233 - Train: 17.66% [856200/4848000] [176.6/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 11:52:56,989 - Train: 17.66% [856300/4848000] [176.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 11:53:30,849 - Train: 17.67% [856400/4848000] [176.7/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 11:54:03,639 - Train: 17.67% [856500/4848000] [176.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 11:54:38,006 - Train: 17.67% [856600/4848000] [176.7/1000.0] [batch_t 0.327 (0.344)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 11:55:10,781 - Train: 17.67% [856700/4848000] [176.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 11:55:43,598 - Train: 17.67% [856800/4848000] [176.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 11:56:17,778 - Train: 17.68% [856900/4848000] [176.8/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 11:56:50,636 - Train: 17.68% [857000/4848000] [176.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 11:57:24,101 - Train: 17.68% [857100/4848000] [176.8/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 11:57:56,875 - Train: 17.68% [857200/4848000] [176.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-06 11:58:29,983 - Train: 17.68% [857300/4848000] [176.8/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 11:59:02,930 - Train: 17.69% [857400/4848000] [176.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 11:59:35,748 - Train: 17.69% [857500/4848000] [176.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 12:00:08,538 - Train: 17.69% [857600/4848000] [176.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 12:00:41,323 - Train: 17.69% [857700/4848000] [176.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 12:01:14,141 - Train: 17.69% [857800/4848000] [176.9/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 12:01:46,916 - Train: 17.70% [857900/4848000] [177.0/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 12:02:19,707 - Train: 17.70% [858000/4848000] [177.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 12:02:51,174 - ==> Total time: 3 days, 12:36:49 Eta: 16 days, 9:25:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 12:02:55,405 - Train: 17.70% [858100/4848000] [177.0/1000.0] [batch_t 0.330 (0.436)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 12:03:29,141 - Train: 17.70% [858200/4848000] [177.0/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 12:04:01,956 - Train: 17.70% [858300/4848000] [177.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 12:04:35,682 - Train: 17.71% [858400/4848000] [177.1/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 12:05:08,475 - Train: 17.71% [858500/4848000] [177.1/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 12:05:41,283 - Train: 17.71% [858600/4848000] [177.1/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 12:06:14,343 - Train: 17.71% [858700/4848000] [177.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 12:06:47,108 - Train: 17.71% [858800/4848000] [177.1/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 12:07:20,247 - Train: 17.72% [858900/4848000] [177.2/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 12:07:53,048 - Train: 17.72% [859000/4848000] [177.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 12:08:26,705 - Train: 17.72% [859100/4848000] [177.2/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 12:08:59,531 - Train: 17.72% [859200/4848000] [177.2/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 12:09:32,825 - Train: 17.72% [859300/4848000] [177.2/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 12:10:05,608 - Train: 17.73% [859400/4848000] [177.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 12:10:38,401 - Train: 17.73% [859500/4848000] [177.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 12:11:13,059 - Train: 17.73% [859600/4848000] [177.3/1000.0] [batch_t 0.331 (0.346)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 12:11:45,875 - Train: 17.73% [859700/4848000] [177.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 12:12:19,973 - Train: 17.74% [859800/4848000] [177.4/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 12:12:52,719 - Train: 17.74% [859900/4848000] [177.4/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 12:13:26,472 - Train: 17.74% [860000/4848000] [177.4/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 12:13:59,231 - Train: 17.74% [860100/4848000] [177.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 12:14:33,263 - Train: 17.74% [860200/4848000] [177.4/1000.0] [batch_t 0.326 (0.340)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 12:15:06,679 - Train: 17.75% [860300/4848000] [177.5/1000.0] [batch_t 0.324 (0.334)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 12:15:39,438 - Train: 17.75% [860400/4848000] [177.5/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-06 12:16:13,188 - Train: 17.75% [860500/4848000] [177.5/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 12:16:45,980 - Train: 17.75% [860600/4848000] [177.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 12:17:20,337 - Train: 17.75% [860700/4848000] [177.5/1000.0] [batch_t 0.327 (0.343)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 12:17:53,159 - Train: 17.76% [860800/4848000] [177.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 12:18:25,930 - Train: 17.76% [860900/4848000] [177.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 12:18:58,808 - Train: 17.76% [861000/4848000] [177.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 12:19:32,217 - Train: 17.76% [861100/4848000] [177.6/1000.0] [batch_t 0.325 (0.334)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 12:20:05,964 - Train: 17.76% [861200/4848000] [177.6/1000.0] [batch_t 1.278 (0.337)] [data_t 0.956] [optim_t 0.322] [lr 0.005000] 2024-04-06 12:20:38,753 - Train: 17.77% [861300/4848000] [177.7/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 12:21:11,551 - Train: 17.77% [861400/4848000] [177.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 12:21:44,423 - Train: 17.77% [861500/4848000] [177.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 12:22:17,581 - Train: 17.77% [861600/4848000] [177.7/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 12:22:50,347 - Train: 17.77% [861700/4848000] [177.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 12:23:23,887 - Train: 17.78% [861800/4848000] [177.8/1000.0] [batch_t 0.323 (0.335)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 12:23:56,672 - Train: 17.78% [861900/4848000] [177.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 12:24:29,804 - Train: 17.78% [862000/4848000] [177.8/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 12:25:02,601 - Train: 17.78% [862100/4848000] [177.8/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 12:25:35,419 - Train: 17.78% [862200/4848000] [177.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 12:26:08,190 - Train: 17.79% [862300/4848000] [177.9/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 12:26:41,023 - Train: 17.79% [862400/4848000] [177.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 12:27:14,197 - Train: 17.79% [862500/4848000] [177.9/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 12:27:46,981 - Train: 17.79% [862600/4848000] [177.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 12:28:19,811 - Train: 17.79% [862700/4848000] [177.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 12:28:52,560 - Train: 17.80% [862800/4848000] [178.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 12:29:26,835 - Train: 17.80% [862900/4848000] [178.0/1000.0] [batch_t 0.326 (0.343)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 12:29:41,261 - ==> Total time: 3 days, 13:03:39 Eta: 16 days, 8:48:35 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 12:30:02,968 - Train: 17.80% [863000/4848000] [178.0/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 12:30:36,456 - Train: 17.80% [863100/4848000] [178.0/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 12:31:10,730 - Train: 17.81% [863200/4848000] [178.1/1000.0] [batch_t 0.324 (0.343)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 12:31:43,557 - Train: 17.81% [863300/4848000] [178.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 12:32:17,446 - Train: 17.81% [863400/4848000] [178.1/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 12:32:50,214 - Train: 17.81% [863500/4848000] [178.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 12:33:22,967 - Train: 17.81% [863600/4848000] [178.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 12:33:55,769 - Train: 17.82% [863700/4848000] [178.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 12:34:29,392 - Train: 17.82% [863800/4848000] [178.2/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 12:35:02,186 - Train: 17.82% [863900/4848000] [178.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 12:35:35,740 - Train: 17.82% [864000/4848000] [178.2/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 12:36:08,537 - Train: 17.82% [864100/4848000] [178.2/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 12:36:41,287 - Train: 17.83% [864200/4848000] [178.3/1000.0] [batch_t 0.334 (0.327)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-06 12:37:14,077 - Train: 17.83% [864300/4848000] [178.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 12:37:47,042 - Train: 17.83% [864400/4848000] [178.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 12:38:20,267 - Train: 17.83% [864500/4848000] [178.3/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 12:38:53,108 - Train: 17.83% [864600/4848000] [178.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 12:39:26,251 - Train: 17.84% [864700/4848000] [178.4/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 12:39:59,091 - Train: 17.84% [864800/4848000] [178.4/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 12:40:31,932 - Train: 17.84% [864900/4848000] [178.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 12:41:05,971 - Train: 17.84% [865000/4848000] [178.4/1000.0] [batch_t 0.337 (0.340)] [data_t 0.010] [optim_t 0.327] [lr 0.005000] 2024-04-06 12:41:38,825 - Train: 17.84% [865100/4848000] [178.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 12:42:12,272 - Train: 17.85% [865200/4848000] [178.5/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 12:42:45,150 - Train: 17.85% [865300/4848000] [178.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 12:43:19,594 - Train: 17.85% [865400/4848000] [178.5/1000.0] [batch_t 0.328 (0.344)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 12:43:52,404 - Train: 17.85% [865500/4848000] [178.5/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 12:44:28,072 - Train: 17.85% [865600/4848000] [178.5/1000.0] [batch_t 0.329 (0.357)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 12:45:00,898 - Train: 17.86% [865700/4848000] [178.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 12:45:35,013 - Train: 17.86% [865800/4848000] [178.6/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 12:46:08,732 - Train: 17.86% [865900/4848000] [178.6/1000.0] [batch_t 0.324 (0.337)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 12:46:41,599 - Train: 17.86% [866000/4848000] [178.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 12:47:15,103 - Train: 17.87% [866100/4848000] [178.7/1000.0] [batch_t 0.331 (0.335)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 12:47:47,988 - Train: 17.87% [866200/4848000] [178.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 12:48:22,096 - Train: 17.87% [866300/4848000] [178.7/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 12:48:54,871 - Train: 17.87% [866400/4848000] [178.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 12:49:27,659 - Train: 17.87% [866500/4848000] [178.7/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 12:50:00,461 - Train: 17.88% [866600/4848000] [178.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 12:50:33,289 - Train: 17.88% [866700/4848000] [178.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 12:51:06,811 - Train: 17.88% [866800/4848000] [178.8/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 12:51:39,563 - Train: 17.88% [866900/4848000] [178.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 12:52:14,672 - Train: 17.88% [867000/4848000] [178.8/1000.0] [batch_t 0.329 (0.351)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 12:52:47,420 - Train: 17.89% [867100/4848000] [178.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 12:53:22,452 - Train: 17.89% [867200/4848000] [178.9/1000.0] [batch_t 0.330 (0.350)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 12:53:55,400 - Train: 17.89% [867300/4848000] [178.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 12:54:28,153 - Train: 17.89% [867400/4848000] [178.9/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 12:55:00,920 - Train: 17.89% [867500/4848000] [178.9/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 12:55:33,696 - Train: 17.90% [867600/4848000] [179.0/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 12:56:06,488 - Train: 17.90% [867700/4848000] [179.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 12:56:36,679 - ==> Total time: 3 days, 13:30:35 Eta: 16 days, 8:11:54 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 12:56:41,288 - Train: 17.90% [867800/4848000] [179.0/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 12:57:13,998 - Train: 17.90% [867900/4848000] [179.0/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 12:57:46,763 - Train: 17.90% [868000/4848000] [179.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 12:58:20,053 - Train: 17.91% [868100/4848000] [179.1/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 12:58:52,914 - Train: 17.91% [868200/4848000] [179.1/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 12:59:25,710 - Train: 17.91% [868300/4848000] [179.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 12:59:58,507 - Train: 17.91% [868400/4848000] [179.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 13:00:31,280 - Train: 17.91% [868500/4848000] [179.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 13:01:04,202 - Train: 17.92% [868600/4848000] [179.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 13:01:38,892 - Train: 17.92% [868700/4848000] [179.2/1000.0] [batch_t 0.328 (0.347)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 13:02:11,667 - Train: 17.92% [868800/4848000] [179.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 13:02:44,437 - Train: 17.92% [868900/4848000] [179.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 13:03:18,763 - Train: 17.92% [869000/4848000] [179.2/1000.0] [batch_t 0.327 (0.343)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 13:03:51,613 - Train: 17.93% [869100/4848000] [179.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 13:04:25,225 - Train: 17.93% [869200/4848000] [179.3/1000.0] [batch_t 0.333 (0.336)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 13:04:58,038 - Train: 17.93% [869300/4848000] [179.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 13:05:31,820 - Train: 17.93% [869400/4848000] [179.3/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 13:06:04,561 - Train: 17.94% [869500/4848000] [179.4/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 13:06:38,429 - Train: 17.94% [869600/4848000] [179.4/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 13:07:12,140 - Train: 17.94% [869700/4848000] [179.4/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 13:07:44,949 - Train: 17.94% [869800/4848000] [179.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 13:08:19,001 - Train: 17.94% [869900/4848000] [179.4/1000.0] [batch_t 0.324 (0.340)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 13:08:51,873 - Train: 17.95% [870000/4848000] [179.5/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 13:09:25,517 - Train: 17.95% [870100/4848000] [179.5/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 13:09:58,294 - Train: 17.95% [870200/4848000] [179.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 13:10:33,552 - Train: 17.95% [870300/4848000] [179.5/1000.0] [batch_t 0.330 (0.352)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 13:11:07,495 - Train: 17.95% [870400/4848000] [179.5/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 13:11:40,320 - Train: 17.96% [870500/4848000] [179.6/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-06 13:12:13,120 - Train: 17.96% [870600/4848000] [179.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 13:12:45,962 - Train: 17.96% [870700/4848000] [179.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 13:13:21,308 - Train: 17.96% [870800/4848000] [179.6/1000.0] [batch_t 0.327 (0.353)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 13:13:54,096 - Train: 17.96% [870900/4848000] [179.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 13:14:27,879 - Train: 17.97% [871000/4848000] [179.7/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 13:15:00,629 - Train: 17.97% [871100/4848000] [179.7/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 13:15:33,457 - Train: 17.97% [871200/4848000] [179.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 13:16:08,838 - Train: 17.97% [871300/4848000] [179.7/1000.0] [batch_t 0.330 (0.354)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 13:16:41,663 - Train: 17.97% [871400/4848000] [179.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 13:17:16,047 - Train: 17.98% [871500/4848000] [179.8/1000.0] [batch_t 0.330 (0.344)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 13:17:48,803 - Train: 17.98% [871600/4848000] [179.8/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 13:18:24,083 - Train: 17.98% [871700/4848000] [179.8/1000.0] [batch_t 0.328 (0.353)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 13:18:56,892 - Train: 17.98% [871800/4848000] [179.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 13:19:29,725 - Train: 17.98% [871900/4848000] [179.8/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 13:20:02,505 - Train: 17.99% [872000/4848000] [179.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 13:20:35,263 - Train: 17.99% [872100/4848000] [179.9/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 13:21:08,046 - Train: 17.99% [872200/4848000] [179.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 13:21:40,814 - Train: 17.99% [872300/4848000] [179.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 13:22:14,074 - Train: 18.00% [872400/4848000] [180.0/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 13:22:46,891 - Train: 18.00% [872500/4848000] [180.0/1000.0] [batch_t 0.321 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-06 13:23:20,633 - Train: 18.00% [872600/4848000] [180.0/1000.0] [batch_t 0.326 (0.337)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 13:23:33,817 - ==> Total time: 3 days, 13:57:32 Eta: 16 days, 7:35:27 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 13:23:55,335 - Train: 18.00% [872700/4848000] [180.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 13:24:29,245 - Train: 18.00% [872800/4848000] [180.0/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 13:25:02,082 - Train: 18.01% [872900/4848000] [180.1/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-06 13:25:35,318 - Train: 18.01% [873000/4848000] [180.1/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 13:26:08,092 - Train: 18.01% [873100/4848000] [180.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 13:26:40,859 - Train: 18.01% [873200/4848000] [180.1/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-06 13:27:13,705 - Train: 18.01% [873300/4848000] [180.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 13:27:46,579 - Train: 18.02% [873400/4848000] [180.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 13:28:19,370 - Train: 18.02% [873500/4848000] [180.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 13:28:52,174 - Train: 18.02% [873600/4848000] [180.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 13:29:26,079 - Train: 18.02% [873700/4848000] [180.2/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 13:29:58,919 - Train: 18.02% [873800/4848000] [180.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 13:30:32,412 - Train: 18.03% [873900/4848000] [180.3/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 13:31:07,982 - Train: 18.03% [874000/4848000] [180.3/1000.0] [batch_t 3.101 (0.356)] [data_t 2.774] [optim_t 0.327] [lr 0.005000] 2024-04-06 13:31:40,790 - Train: 18.03% [874100/4848000] [180.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 13:32:14,763 - Train: 18.03% [874200/4848000] [180.3/1000.0] [batch_t 0.330 (0.340)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 13:32:47,542 - Train: 18.03% [874300/4848000] [180.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 13:33:21,890 - Train: 18.04% [874400/4848000] [180.4/1000.0] [batch_t 0.324 (0.343)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 13:33:54,683 - Train: 18.04% [874500/4848000] [180.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 13:34:28,179 - Train: 18.04% [874600/4848000] [180.4/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 13:35:00,999 - Train: 18.04% [874700/4848000] [180.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 13:35:33,759 - Train: 18.04% [874800/4848000] [180.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 13:36:07,795 - Train: 18.05% [874900/4848000] [180.5/1000.0] [batch_t 0.332 (0.340)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 13:36:40,624 - Train: 18.05% [875000/4848000] [180.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 13:37:13,777 - Train: 18.05% [875100/4848000] [180.5/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 13:37:46,648 - Train: 18.05% [875200/4848000] [180.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 13:38:19,497 - Train: 18.05% [875300/4848000] [180.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 13:38:52,335 - Train: 18.06% [875400/4848000] [180.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 13:39:25,689 - Train: 18.06% [875500/4848000] [180.6/1000.0] [batch_t 0.325 (0.333)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 13:39:58,526 - Train: 18.06% [875600/4848000] [180.6/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 13:40:31,493 - Train: 18.06% [875700/4848000] [180.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 13:41:04,264 - Train: 18.07% [875800/4848000] [180.7/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 13:41:37,855 - Train: 18.07% [875900/4848000] [180.7/1000.0] [batch_t 0.326 (0.336)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 13:42:11,107 - Train: 18.07% [876000/4848000] [180.7/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 13:42:43,868 - Train: 18.07% [876100/4848000] [180.7/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 13:43:16,685 - Train: 18.07% [876200/4848000] [180.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 13:43:49,412 - Train: 18.08% [876300/4848000] [180.8/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 13:44:22,138 - Train: 18.08% [876400/4848000] [180.8/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 13:44:54,975 - Train: 18.08% [876500/4848000] [180.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 13:45:27,804 - Train: 18.08% [876600/4848000] [180.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 13:46:00,592 - Train: 18.08% [876700/4848000] [180.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 13:46:34,361 - Train: 18.09% [876800/4848000] [180.9/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 13:47:07,166 - Train: 18.09% [876900/4848000] [180.9/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 13:47:40,009 - Train: 18.09% [877000/4848000] [180.9/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 13:48:14,147 - Train: 18.09% [877100/4848000] [180.9/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 13:48:46,956 - Train: 18.09% [877200/4848000] [180.9/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-06 13:49:20,350 - Train: 18.10% [877300/4848000] [181.0/1000.0] [batch_t 0.326 (0.334)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 13:49:53,154 - Train: 18.10% [877400/4848000] [181.0/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 13:50:22,515 - ==> Total time: 3 days, 14:24:21 Eta: 16 days, 6:58:28 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 13:50:28,755 - Train: 18.10% [877500/4848000] [181.0/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 13:51:01,490 - Train: 18.10% [877600/4848000] [181.0/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 13:51:34,250 - Train: 18.10% [877700/4848000] [181.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 13:52:06,960 - Train: 18.11% [877800/4848000] [181.1/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 13:52:39,749 - Train: 18.11% [877900/4848000] [181.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 13:53:12,536 - Train: 18.11% [878000/4848000] [181.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 13:53:45,333 - Train: 18.11% [878100/4848000] [181.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 13:54:18,179 - Train: 18.11% [878200/4848000] [181.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 13:54:50,910 - Train: 18.12% [878300/4848000] [181.2/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 13:55:23,826 - Train: 18.12% [878400/4848000] [181.2/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 13:55:56,647 - Train: 18.12% [878500/4848000] [181.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 13:56:29,402 - Train: 18.12% [878600/4848000] [181.2/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 13:57:02,187 - Train: 18.12% [878700/4848000] [181.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 13:57:34,952 - Train: 18.13% [878800/4848000] [181.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 13:58:07,709 - Train: 18.13% [878900/4848000] [181.3/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 13:58:40,468 - Train: 18.13% [879000/4848000] [181.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 13:59:13,289 - Train: 18.13% [879100/4848000] [181.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 13:59:46,051 - Train: 18.14% [879200/4848000] [181.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 14:00:19,577 - Train: 18.14% [879300/4848000] [181.4/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 14:00:52,371 - Train: 18.14% [879400/4848000] [181.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 14:01:25,161 - Train: 18.14% [879500/4848000] [181.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 14:01:57,946 - Train: 18.14% [879600/4848000] [181.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 14:02:30,720 - Train: 18.15% [879700/4848000] [181.5/1000.0] [batch_t 0.321 (0.328)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-06 14:03:03,590 - Train: 18.15% [879800/4848000] [181.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 14:03:38,624 - Train: 18.15% [879900/4848000] [181.5/1000.0] [batch_t 0.329 (0.350)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 14:04:11,420 - Train: 18.15% [880000/4848000] [181.5/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 14:04:44,208 - Train: 18.15% [880100/4848000] [181.5/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 14:05:18,034 - Train: 18.16% [880200/4848000] [181.6/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 14:05:50,822 - Train: 18.16% [880300/4848000] [181.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 14:06:24,460 - Train: 18.16% [880400/4848000] [181.6/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 14:06:57,284 - Train: 18.16% [880500/4848000] [181.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 14:07:31,019 - Train: 18.16% [880600/4848000] [181.6/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 14:08:03,758 - Train: 18.17% [880700/4848000] [181.7/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 14:08:37,745 - Train: 18.17% [880800/4848000] [181.7/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 14:09:11,289 - Train: 18.17% [880900/4848000] [181.7/1000.0] [batch_t 0.326 (0.335)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 14:09:44,111 - Train: 18.17% [881000/4848000] [181.7/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 14:10:17,964 - Train: 18.17% [881100/4848000] [181.7/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 14:10:50,800 - Train: 18.18% [881200/4848000] [181.8/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 14:11:25,168 - Train: 18.18% [881300/4848000] [181.8/1000.0] [batch_t 0.329 (0.344)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 14:11:57,982 - Train: 18.18% [881400/4848000] [181.8/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 14:12:32,168 - Train: 18.18% [881500/4848000] [181.8/1000.0] [batch_t 0.334 (0.342)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-06 14:13:05,214 - Train: 18.18% [881600/4848000] [181.8/1000.0] [batch_t 0.534 (0.330)] [data_t 0.208] [optim_t 0.326] [lr 0.005000] 2024-04-06 14:13:37,992 - Train: 18.19% [881700/4848000] [181.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 14:14:11,879 - Train: 18.19% [881800/4848000] [181.9/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 14:14:44,681 - Train: 18.19% [881900/4848000] [181.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 14:15:18,693 - Train: 18.19% [882000/4848000] [181.9/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 14:15:51,505 - Train: 18.20% [882100/4848000] [182.0/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 14:16:25,420 - Train: 18.20% [882200/4848000] [182.0/1000.0] [batch_t 0.328 (0.339)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-06 14:16:58,310 - Train: 18.20% [882300/4848000] [182.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 14:17:11,887 - ==> Total time: 3 days, 14:51:10 Eta: 16 days, 6:21:38 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 14:17:34,726 - Train: 18.20% [882400/4848000] [182.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 14:18:09,080 - Train: 18.20% [882500/4848000] [182.0/1000.0] [batch_t 0.330 (0.343)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 14:18:42,030 - Train: 18.21% [882600/4848000] [182.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 14:19:16,019 - Train: 18.21% [882700/4848000] [182.1/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 14:19:48,855 - Train: 18.21% [882800/4848000] [182.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 14:20:22,723 - Train: 18.21% [882900/4848000] [182.1/1000.0] [batch_t 0.331 (0.339)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 14:20:55,472 - Train: 18.21% [883000/4848000] [182.1/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 14:21:29,594 - Train: 18.22% [883100/4848000] [182.2/1000.0] [batch_t 0.324 (0.341)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 14:22:02,389 - Train: 18.22% [883200/4848000] [182.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 14:22:35,175 - Train: 18.22% [883300/4848000] [182.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 14:23:09,362 - Train: 18.22% [883400/4848000] [182.2/1000.0] [batch_t 0.334 (0.342)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 14:23:42,227 - Train: 18.22% [883500/4848000] [182.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 14:24:15,069 - Train: 18.23% [883600/4848000] [182.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 14:24:47,862 - Train: 18.23% [883700/4848000] [182.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 14:25:22,900 - Train: 18.23% [883800/4848000] [182.3/1000.0] [batch_t 0.323 (0.350)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 14:25:55,647 - Train: 18.23% [883900/4848000] [182.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 14:26:28,556 - Train: 18.23% [884000/4848000] [182.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 14:27:01,384 - Train: 18.24% [884100/4848000] [182.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 14:27:34,231 - Train: 18.24% [884200/4848000] [182.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 14:28:07,086 - Train: 18.24% [884300/4848000] [182.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 14:28:39,929 - Train: 18.24% [884400/4848000] [182.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 14:29:12,749 - Train: 18.24% [884500/4848000] [182.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 14:29:45,550 - Train: 18.25% [884600/4848000] [182.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 14:30:20,714 - Train: 18.25% [884700/4848000] [182.5/1000.0] [batch_t 0.327 (0.352)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 14:30:53,574 - Train: 18.25% [884800/4848000] [182.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 14:31:27,583 - Train: 18.25% [884900/4848000] [182.5/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 14:32:00,347 - Train: 18.25% [885000/4848000] [182.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 14:32:33,957 - Train: 18.26% [885100/4848000] [182.6/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 14:33:06,769 - Train: 18.26% [885200/4848000] [182.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 14:33:39,491 - Train: 18.26% [885300/4848000] [182.6/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 14:34:12,230 - Train: 18.26% [885400/4848000] [182.6/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 14:34:45,156 - Train: 18.27% [885500/4848000] [182.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 14:35:18,005 - Train: 18.27% [885600/4848000] [182.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 14:35:50,821 - Train: 18.27% [885700/4848000] [182.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 14:36:23,622 - Train: 18.27% [885800/4848000] [182.7/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-06 14:36:56,499 - Train: 18.27% [885900/4848000] [182.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 14:37:29,267 - Train: 18.28% [886000/4848000] [182.8/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 14:38:02,045 - Train: 18.28% [886100/4848000] [182.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 14:38:36,232 - Train: 18.28% [886200/4848000] [182.8/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 14:39:08,994 - Train: 18.28% [886300/4848000] [182.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 14:39:41,766 - Train: 18.28% [886400/4848000] [182.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 14:40:15,623 - Train: 18.29% [886500/4848000] [182.9/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 14:40:48,447 - Train: 18.29% [886600/4848000] [182.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 14:41:21,916 - Train: 18.29% [886700/4848000] [182.9/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 14:41:54,703 - Train: 18.29% [886800/4848000] [182.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 14:42:29,176 - Train: 18.29% [886900/4848000] [182.9/1000.0] [batch_t 0.329 (0.345)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 14:43:01,928 - Train: 18.30% [887000/4848000] [183.0/1000.0] [batch_t 0.322 (0.327)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-06 14:43:35,653 - Train: 18.30% [887100/4848000] [183.0/1000.0] [batch_t 0.326 (0.337)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 14:44:03,199 - ==> Total time: 3 days, 15:18:01 Eta: 16 days, 5:45:04 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 14:44:11,530 - Train: 18.30% [887200/4848000] [183.0/1000.0] [batch_t 0.327 (0.351)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 14:44:44,322 - Train: 18.30% [887300/4848000] [183.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 14:45:17,402 - Train: 18.30% [887400/4848000] [183.0/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 14:45:50,198 - Train: 18.31% [887500/4848000] [183.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 14:46:23,018 - Train: 18.31% [887600/4848000] [183.1/1000.0] [batch_t 0.337 (0.328)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-06 14:46:55,881 - Train: 18.31% [887700/4848000] [183.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 14:47:28,708 - Train: 18.31% [887800/4848000] [183.1/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 14:48:01,541 - Train: 18.31% [887900/4848000] [183.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 14:48:35,659 - Train: 18.32% [888000/4848000] [183.2/1000.0] [batch_t 0.326 (0.341)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 14:49:09,724 - Train: 18.32% [888100/4848000] [183.2/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 14:49:42,615 - Train: 18.32% [888200/4848000] [183.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 14:50:16,082 - Train: 18.32% [888300/4848000] [183.2/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 14:50:48,924 - Train: 18.33% [888400/4848000] [183.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 14:51:21,940 - Train: 18.33% [888500/4848000] [183.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 14:51:54,721 - Train: 18.33% [888600/4848000] [183.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 14:52:27,529 - Train: 18.33% [888700/4848000] [183.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 14:53:00,279 - Train: 18.33% [888800/4848000] [183.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 14:53:33,072 - Train: 18.34% [888900/4848000] [183.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 14:54:05,922 - Train: 18.34% [889000/4848000] [183.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 14:54:38,698 - Train: 18.34% [889100/4848000] [183.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 14:55:11,477 - Train: 18.34% [889200/4848000] [183.4/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 14:55:44,222 - Train: 18.34% [889300/4848000] [183.4/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 14:56:17,416 - Train: 18.35% [889400/4848000] [183.5/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 14:56:50,173 - Train: 18.35% [889500/4848000] [183.5/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 14:57:23,027 - Train: 18.35% [889600/4848000] [183.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 14:57:55,791 - Train: 18.35% [889700/4848000] [183.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 14:58:28,556 - Train: 18.35% [889800/4848000] [183.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 14:59:01,285 - Train: 18.36% [889900/4848000] [183.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 14:59:34,049 - Train: 18.36% [890000/4848000] [183.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 15:00:06,856 - Train: 18.36% [890100/4848000] [183.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 15:00:39,629 - Train: 18.36% [890200/4848000] [183.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 15:01:12,408 - Train: 18.36% [890300/4848000] [183.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 15:01:45,197 - Train: 18.37% [890400/4848000] [183.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 15:02:18,027 - Train: 18.37% [890500/4848000] [183.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 15:02:50,810 - Train: 18.37% [890600/4848000] [183.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 15:03:23,605 - Train: 18.37% [890700/4848000] [183.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 15:03:56,413 - Train: 18.37% [890800/4848000] [183.7/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 15:04:29,182 - Train: 18.38% [890900/4848000] [183.8/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 15:05:01,972 - Train: 18.38% [891000/4848000] [183.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 15:05:34,903 - Train: 18.38% [891100/4848000] [183.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 15:06:07,709 - Train: 18.38% [891200/4848000] [183.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 15:06:40,480 - Train: 18.38% [891300/4848000] [183.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 15:07:13,299 - Train: 18.39% [891400/4848000] [183.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 15:07:46,140 - Train: 18.39% [891500/4848000] [183.9/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 15:08:18,913 - Train: 18.39% [891600/4848000] [183.9/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 15:08:51,673 - Train: 18.39% [891700/4848000] [183.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 15:09:24,496 - Train: 18.40% [891800/4848000] [184.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 15:09:57,290 - Train: 18.40% [891900/4848000] [184.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 15:10:33,165 - Train: 18.40% [892000/4848000] [184.0/1000.0] [batch_t 0.330 (0.359)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 15:10:43,698 - ==> Total time: 3 days, 15:44:42 Eta: 16 days, 5:07:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 15:11:12,559 - Train: 18.40% [892100/4848000] [184.0/1000.0] [batch_t 0.327 (0.381)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 15:11:45,335 - Train: 18.40% [892200/4848000] [184.0/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 15:12:20,563 - Train: 18.41% [892300/4848000] [184.1/1000.0] [batch_t 0.330 (0.352)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 15:12:54,376 - Train: 18.41% [892400/4848000] [184.1/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 15:13:27,982 - Train: 18.41% [892500/4848000] [184.1/1000.0] [batch_t 0.332 (0.336)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 15:14:01,431 - Train: 18.41% [892600/4848000] [184.1/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 15:14:35,065 - Train: 18.41% [892700/4848000] [184.1/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 15:15:08,966 - Train: 18.42% [892800/4848000] [184.2/1000.0] [batch_t 0.331 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 15:16:07,585 - Train: 18.42% [892900/4848000] [184.2/1000.0] [batch_t 0.323 (0.586)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 15:16:48,959 - Train: 18.42% [893000/4848000] [184.2/1000.0] [batch_t 0.328 (0.414)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 15:17:23,803 - Train: 18.42% [893100/4848000] [184.2/1000.0] [batch_t 0.331 (0.348)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 15:17:59,744 - Train: 18.42% [893200/4848000] [184.2/1000.0] [batch_t 0.327 (0.359)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 15:18:33,887 - Train: 18.43% [893300/4848000] [184.3/1000.0] [batch_t 0.336 (0.341)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-06 15:19:06,802 - Train: 18.43% [893400/4848000] [184.3/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 15:19:39,625 - Train: 18.43% [893500/4848000] [184.3/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 15:20:15,564 - Train: 18.43% [893600/4848000] [184.3/1000.0] [batch_t 0.330 (0.359)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 15:20:48,549 - Train: 18.43% [893700/4848000] [184.3/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 15:21:22,640 - Train: 18.44% [893800/4848000] [184.4/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 15:21:55,574 - Train: 18.44% [893900/4848000] [184.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 15:22:28,393 - Train: 18.44% [894000/4848000] [184.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 15:23:01,111 - Train: 18.44% [894100/4848000] [184.4/1000.0] [batch_t 0.330 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 15:23:35,437 - Train: 18.44% [894200/4848000] [184.4/1000.0] [batch_t 0.328 (0.343)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 15:24:08,938 - Train: 18.45% [894300/4848000] [184.5/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 15:24:41,695 - Train: 18.45% [894400/4848000] [184.5/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 15:25:15,145 - Train: 18.45% [894500/4848000] [184.5/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 15:25:47,973 - Train: 18.45% [894600/4848000] [184.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 15:26:21,638 - Train: 18.46% [894700/4848000] [184.6/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 15:26:54,393 - Train: 18.46% [894800/4848000] [184.6/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 15:27:30,115 - Train: 18.46% [894900/4848000] [184.6/1000.0] [batch_t 0.327 (0.357)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 15:28:04,078 - Train: 18.46% [895000/4848000] [184.6/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 15:28:39,575 - Train: 18.46% [895100/4848000] [184.6/1000.0] [batch_t 0.329 (0.355)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 15:29:12,519 - Train: 18.47% [895200/4848000] [184.7/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 15:29:45,489 - Train: 18.47% [895300/4848000] [184.7/1000.0] [batch_t 0.323 (0.330)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 15:30:20,481 - Train: 18.47% [895400/4848000] [184.7/1000.0] [batch_t 0.328 (0.350)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 15:30:53,857 - Train: 18.47% [895500/4848000] [184.7/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 15:31:27,154 - Train: 18.47% [895600/4848000] [184.7/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 15:31:59,993 - Train: 18.48% [895700/4848000] [184.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 15:32:34,175 - Train: 18.48% [895800/4848000] [184.8/1000.0] [batch_t 0.330 (0.342)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 15:33:07,589 - Train: 18.48% [895900/4848000] [184.8/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 15:33:40,684 - Train: 18.48% [896000/4848000] [184.8/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 15:34:14,472 - Train: 18.48% [896100/4848000] [184.8/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 15:34:47,321 - Train: 18.49% [896200/4848000] [184.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 15:35:20,161 - Train: 18.49% [896300/4848000] [184.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 15:35:52,963 - Train: 18.49% [896400/4848000] [184.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 15:36:26,585 - Train: 18.49% [896500/4848000] [184.9/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 15:36:59,371 - Train: 18.49% [896600/4848000] [184.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 15:37:32,694 - Train: 18.50% [896700/4848000] [185.0/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 15:38:05,600 - Train: 18.50% [896800/4848000] [185.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 15:38:31,817 - ==> Total time: 3 days, 16:12:30 Eta: 16 days, 4:35:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 15:38:41,067 - Train: 18.50% [896900/4848000] [185.0/1000.0] [batch_t 0.326 (0.340)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 15:39:14,748 - Train: 18.50% [897000/4848000] [185.0/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 15:39:47,542 - Train: 18.50% [897100/4848000] [185.0/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 15:40:21,564 - Train: 18.51% [897200/4848000] [185.1/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 15:40:54,337 - Train: 18.51% [897300/4848000] [185.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 15:41:27,819 - Train: 18.51% [897400/4848000] [185.1/1000.0] [batch_t 0.326 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 15:42:00,653 - Train: 18.51% [897500/4848000] [185.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 15:42:36,179 - Train: 18.51% [897600/4848000] [185.1/1000.0] [batch_t 0.327 (0.355)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 15:43:12,532 - Train: 18.52% [897700/4848000] [185.2/1000.0] [batch_t 0.326 (0.363)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 15:43:46,222 - Train: 18.52% [897800/4848000] [185.2/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 15:44:20,598 - Train: 18.52% [897900/4848000] [185.2/1000.0] [batch_t 0.329 (0.344)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 15:44:54,199 - Train: 18.52% [898000/4848000] [185.2/1000.0] [batch_t 0.326 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 15:45:29,040 - Train: 18.53% [898100/4848000] [185.3/1000.0] [batch_t 0.326 (0.348)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 15:46:01,905 - Train: 18.53% [898200/4848000] [185.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 15:46:35,393 - Train: 18.53% [898300/4848000] [185.3/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 15:47:08,546 - Train: 18.53% [898400/4848000] [185.3/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 15:47:41,313 - Train: 18.53% [898500/4848000] [185.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 15:48:16,339 - Train: 18.54% [898600/4848000] [185.4/1000.0] [batch_t 0.327 (0.350)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 15:48:49,090 - Train: 18.54% [898700/4848000] [185.4/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 15:49:22,131 - Train: 18.54% [898800/4848000] [185.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 15:49:58,162 - Train: 18.54% [898900/4848000] [185.4/1000.0] [batch_t 0.326 (0.360)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 15:50:35,165 - Train: 18.54% [899000/4848000] [185.4/1000.0] [batch_t 0.328 (0.370)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 15:51:10,466 - Train: 18.55% [899100/4848000] [185.5/1000.0] [batch_t 0.325 (0.353)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 15:51:44,052 - Train: 18.55% [899200/4848000] [185.5/1000.0] [batch_t 0.325 (0.336)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 15:52:17,726 - Train: 18.55% [899300/4848000] [185.5/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 15:52:51,660 - Train: 18.55% [899400/4848000] [185.5/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 15:53:25,774 - Train: 18.55% [899500/4848000] [185.5/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 15:54:01,707 - Train: 18.56% [899600/4848000] [185.6/1000.0] [batch_t 0.329 (0.359)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 15:54:40,361 - Train: 18.56% [899700/4848000] [185.6/1000.0] [batch_t 0.327 (0.386)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 15:55:17,692 - Train: 18.56% [899800/4848000] [185.6/1000.0] [batch_t 0.328 (0.373)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 15:55:53,659 - Train: 18.56% [899900/4848000] [185.6/1000.0] [batch_t 0.328 (0.360)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 15:56:26,547 - Train: 18.56% [900000/4848000] [185.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 15:56:59,257 - Train: 18.57% [900100/4848000] [185.7/1000.0] [batch_t 0.324 (0.327)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 15:57:36,743 - Train: 18.57% [900200/4848000] [185.7/1000.0] [batch_t 0.326 (0.375)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 15:58:11,736 - Train: 18.57% [900300/4848000] [185.7/1000.0] [batch_t 0.326 (0.350)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 15:58:47,309 - Train: 18.57% [900400/4848000] [185.7/1000.0] [batch_t 0.329 (0.356)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 16:01:57,941 - Train: 18.57% [900500/4848000] [185.7/1000.0] [batch_t 0.324 (1.906)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 16:02:32,985 - Train: 18.58% [900600/4848000] [185.8/1000.0] [batch_t 0.330 (0.350)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 16:03:06,546 - Train: 18.58% [900700/4848000] [185.8/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 16:03:39,332 - Train: 18.58% [900800/4848000] [185.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 16:04:12,111 - Train: 18.58% [900900/4848000] [185.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 16:04:44,987 - Train: 18.58% [901000/4848000] [185.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 16:05:18,220 - Train: 18.59% [901100/4848000] [185.9/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 16:05:51,014 - Train: 18.59% [901200/4848000] [185.9/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 16:06:24,924 - Train: 18.59% [901300/4848000] [185.9/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 16:06:57,712 - Train: 18.59% [901400/4848000] [185.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 16:07:32,987 - Train: 18.60% [901500/4848000] [186.0/1000.0] [batch_t 0.329 (0.353)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 16:08:06,169 - Train: 18.60% [901600/4848000] [186.0/1000.0] [batch_t 0.323 (0.332)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 16:08:38,956 - Train: 18.60% [901700/4848000] [186.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 16:08:48,148 - ==> Total time: 3 days, 16:42:46 Eta: 16 days, 4:14:18 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 16:09:13,885 - Train: 18.60% [901800/4848000] [186.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 16:09:46,763 - Train: 18.60% [901900/4848000] [186.0/1000.0] [batch_t 0.321 (0.329)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-06 16:10:19,588 - Train: 18.61% [902000/4848000] [186.1/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 16:10:52,412 - Train: 18.61% [902100/4848000] [186.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 16:11:25,279 - Train: 18.61% [902200/4848000] [186.1/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 16:11:58,165 - Train: 18.61% [902300/4848000] [186.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 16:12:31,578 - Train: 18.61% [902400/4848000] [186.1/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 16:13:04,993 - Train: 18.62% [902500/4848000] [186.2/1000.0] [batch_t 0.921 (0.334)] [data_t 0.597] [optim_t 0.324] [lr 0.005000] 2024-04-06 16:13:37,821 - Train: 18.62% [902600/4848000] [186.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 16:14:11,279 - Train: 18.62% [902700/4848000] [186.2/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 16:14:44,077 - Train: 18.62% [902800/4848000] [186.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 16:15:17,086 - Train: 18.62% [902900/4848000] [186.2/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 16:15:49,855 - Train: 18.63% [903000/4848000] [186.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 16:16:22,849 - Train: 18.63% [903100/4848000] [186.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 16:16:55,649 - Train: 18.63% [903200/4848000] [186.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 16:17:29,796 - Train: 18.63% [903300/4848000] [186.3/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 16:18:02,579 - Train: 18.63% [903400/4848000] [186.3/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 16:18:37,586 - Train: 18.64% [903500/4848000] [186.4/1000.0] [batch_t 0.330 (0.350)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 16:19:11,859 - Train: 18.64% [903600/4848000] [186.4/1000.0] [batch_t 0.325 (0.343)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 16:19:44,700 - Train: 18.64% [903700/4848000] [186.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 16:20:18,625 - Train: 18.64% [903800/4848000] [186.4/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 16:20:51,444 - Train: 18.64% [903900/4848000] [186.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 16:21:27,144 - Train: 18.65% [904000/4848000] [186.5/1000.0] [batch_t 0.331 (0.357)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 16:21:59,974 - Train: 18.65% [904100/4848000] [186.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 16:22:35,764 - Train: 18.65% [904200/4848000] [186.5/1000.0] [batch_t 0.329 (0.358)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 16:23:10,321 - Train: 18.65% [904300/4848000] [186.5/1000.0] [batch_t 0.328 (0.345)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 16:23:43,121 - Train: 18.66% [904400/4848000] [186.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 16:24:15,923 - Train: 18.66% [904500/4848000] [186.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 16:24:48,723 - Train: 18.66% [904600/4848000] [186.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 16:25:21,660 - Train: 18.66% [904700/4848000] [186.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 16:25:54,523 - Train: 18.66% [904800/4848000] [186.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 16:26:27,291 - Train: 18.67% [904900/4848000] [186.7/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 16:27:00,081 - Train: 18.67% [905000/4848000] [186.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 16:27:32,894 - Train: 18.67% [905100/4848000] [186.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 16:28:05,776 - Train: 18.67% [905200/4848000] [186.7/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 16:28:38,539 - Train: 18.67% [905300/4848000] [186.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 16:29:11,843 - Train: 18.68% [905400/4848000] [186.8/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 16:29:44,639 - Train: 18.68% [905500/4848000] [186.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 16:30:17,458 - Train: 18.68% [905600/4848000] [186.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 16:30:50,240 - Train: 18.68% [905700/4848000] [186.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 16:31:23,862 - Train: 18.68% [905800/4848000] [186.8/1000.0] [batch_t 0.326 (0.336)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 16:31:56,661 - Train: 18.69% [905900/4848000] [186.9/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 16:32:29,690 - Train: 18.69% [906000/4848000] [186.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 16:33:02,447 - Train: 18.69% [906100/4848000] [186.9/1000.0] [batch_t 0.332 (0.327)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 16:33:37,583 - Train: 18.69% [906200/4848000] [186.9/1000.0] [batch_t 0.329 (0.351)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 16:34:10,446 - Train: 18.69% [906300/4848000] [186.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 16:34:43,296 - Train: 18.70% [906400/4848000] [187.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 16:35:17,950 - Train: 18.70% [906500/4848000] [187.0/1000.0] [batch_t 0.327 (0.346)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 16:35:42,847 - ==> Total time: 3 days, 17:09:41 Eta: 16 days, 3:38:16 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 16:35:52,989 - Train: 18.70% [906600/4848000] [187.0/1000.0] [batch_t 0.335 (0.336)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-06 16:36:27,476 - Train: 18.70% [906700/4848000] [187.0/1000.0] [batch_t 0.329 (0.345)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 16:37:00,279 - Train: 18.70% [906800/4848000] [187.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 16:37:34,271 - Train: 18.71% [906900/4848000] [187.1/1000.0] [batch_t 0.324 (0.340)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 16:38:08,007 - Train: 18.71% [907000/4848000] [187.1/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 16:38:40,832 - Train: 18.71% [907100/4848000] [187.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 16:39:13,612 - Train: 18.71% [907200/4848000] [187.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 16:39:46,468 - Train: 18.71% [907300/4848000] [187.1/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 16:40:19,236 - Train: 18.72% [907400/4848000] [187.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 16:40:52,008 - Train: 18.72% [907500/4848000] [187.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 16:41:24,806 - Train: 18.72% [907600/4848000] [187.2/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 16:41:57,612 - Train: 18.72% [907700/4848000] [187.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 16:42:30,985 - Train: 18.73% [907800/4848000] [187.3/1000.0] [batch_t 0.324 (0.334)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 16:43:03,735 - Train: 18.73% [907900/4848000] [187.3/1000.0] [batch_t 0.328 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 16:43:37,315 - Train: 18.73% [908000/4848000] [187.3/1000.0] [batch_t 0.322 (0.336)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-06 16:44:11,433 - Train: 18.73% [908100/4848000] [187.3/1000.0] [batch_t 0.332 (0.341)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 16:44:44,220 - Train: 18.73% [908200/4848000] [187.3/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 16:45:18,804 - Train: 18.74% [908300/4848000] [187.4/1000.0] [batch_t 0.333 (0.346)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 16:45:51,633 - Train: 18.74% [908400/4848000] [187.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 16:46:25,385 - Train: 18.74% [908500/4848000] [187.4/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 16:46:58,193 - Train: 18.74% [908600/4848000] [187.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 16:47:31,922 - Train: 18.74% [908700/4848000] [187.4/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 16:48:05,987 - Train: 18.75% [908800/4848000] [187.5/1000.0] [batch_t 1.521 (0.341)] [data_t 1.193] [optim_t 0.328] [lr 0.005000] 2024-04-06 16:48:38,816 - Train: 18.75% [908900/4848000] [187.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 16:49:12,824 - Train: 18.75% [909000/4848000] [187.5/1000.0] [batch_t 0.326 (0.340)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 16:49:45,566 - Train: 18.75% [909100/4848000] [187.5/1000.0] [batch_t 0.329 (0.327)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 16:50:19,535 - Train: 18.75% [909200/4848000] [187.5/1000.0] [batch_t 0.324 (0.340)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 16:50:52,331 - Train: 18.76% [909300/4848000] [187.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 16:51:26,512 - Train: 18.76% [909400/4848000] [187.6/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 16:51:59,508 - Train: 18.76% [909500/4848000] [187.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 16:52:33,061 - Train: 18.76% [909600/4848000] [187.6/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 16:53:05,882 - Train: 18.76% [909700/4848000] [187.6/1000.0] [batch_t 0.335 (0.328)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-06 16:53:38,671 - Train: 18.77% [909800/4848000] [187.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 16:54:12,411 - Train: 18.77% [909900/4848000] [187.7/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 16:54:45,231 - Train: 18.77% [910000/4848000] [187.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 16:55:20,328 - Train: 18.77% [910100/4848000] [187.7/1000.0] [batch_t 0.327 (0.351)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 16:55:53,090 - Train: 18.77% [910200/4848000] [187.7/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 16:56:25,920 - Train: 18.78% [910300/4848000] [187.8/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 16:56:58,729 - Train: 18.78% [910400/4848000] [187.8/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 16:57:31,567 - Train: 18.78% [910500/4848000] [187.8/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-06 16:58:04,395 - Train: 18.78% [910600/4848000] [187.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 16:58:37,167 - Train: 18.79% [910700/4848000] [187.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 16:59:11,197 - Train: 18.79% [910800/4848000] [187.9/1000.0] [batch_t 0.335 (0.340)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-06 16:59:44,121 - Train: 18.79% [910900/4848000] [187.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 17:00:18,339 - Train: 18.79% [911000/4848000] [187.9/1000.0] [batch_t 0.331 (0.342)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 17:00:51,153 - Train: 18.79% [911100/4848000] [187.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 17:01:23,984 - Train: 18.80% [911200/4848000] [188.0/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 17:01:56,777 - Train: 18.80% [911300/4848000] [188.0/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 17:02:29,535 - Train: 18.80% [911400/4848000] [188.0/1000.0] [batch_t 0.331 (0.327)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 17:02:37,423 - ==> Total time: 3 days, 17:36:35 Eta: 16 days, 3:02:19 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 17:03:04,488 - Train: 18.80% [911500/4848000] [188.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 17:03:38,678 - Train: 18.80% [911600/4848000] [188.0/1000.0] [batch_t 0.328 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 17:04:11,458 - Train: 18.81% [911700/4848000] [188.1/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 17:04:44,292 - Train: 18.81% [911800/4848000] [188.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 17:05:17,095 - Train: 18.81% [911900/4848000] [188.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 17:05:49,916 - Train: 18.81% [912000/4848000] [188.1/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 17:06:23,730 - Train: 18.81% [912100/4848000] [188.1/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 17:06:56,528 - Train: 18.82% [912200/4848000] [188.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 17:07:29,860 - Train: 18.82% [912300/4848000] [188.2/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 17:08:02,666 - Train: 18.82% [912400/4848000] [188.2/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 17:08:35,687 - Train: 18.82% [912500/4848000] [188.2/1000.0] [batch_t 0.341 (0.330)] [data_t 0.002] [optim_t 0.339] [lr 0.005000] 2024-04-06 17:09:09,309 - Train: 18.82% [912600/4848000] [188.2/1000.0] [batch_t 0.325 (0.336)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 17:09:42,111 - Train: 18.83% [912700/4848000] [188.3/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 17:10:18,113 - Train: 18.83% [912800/4848000] [188.3/1000.0] [batch_t 0.325 (0.360)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 17:10:50,924 - Train: 18.83% [912900/4848000] [188.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 17:11:26,536 - Train: 18.83% [913000/4848000] [188.3/1000.0] [batch_t 0.327 (0.356)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 17:11:59,300 - Train: 18.83% [913100/4848000] [188.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 17:12:32,323 - Train: 18.84% [913200/4848000] [188.4/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 17:13:07,165 - Train: 18.84% [913300/4848000] [188.4/1000.0] [batch_t 0.331 (0.348)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 17:13:39,990 - Train: 18.84% [913400/4848000] [188.4/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 17:14:13,917 - Train: 18.84% [913500/4848000] [188.4/1000.0] [batch_t 0.325 (0.339)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 17:14:46,714 - Train: 18.84% [913600/4848000] [188.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 17:15:20,875 - Train: 18.85% [913700/4848000] [188.5/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 17:15:53,684 - Train: 18.85% [913800/4848000] [188.5/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 17:16:27,307 - Train: 18.85% [913900/4848000] [188.5/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 17:17:00,109 - Train: 18.85% [914000/4848000] [188.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 17:17:33,781 - Train: 18.86% [914100/4848000] [188.6/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 17:18:06,729 - Train: 18.86% [914200/4848000] [188.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 17:18:39,534 - Train: 18.86% [914300/4848000] [188.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 17:19:13,704 - Train: 18.86% [914400/4848000] [188.6/1000.0] [batch_t 0.328 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 17:19:46,540 - Train: 18.86% [914500/4848000] [188.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 17:20:19,919 - Train: 18.87% [914600/4848000] [188.7/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 17:20:52,690 - Train: 18.87% [914700/4848000] [188.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 17:21:26,224 - Train: 18.87% [914800/4848000] [188.7/1000.0] [batch_t 0.326 (0.335)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 17:21:59,032 - Train: 18.87% [914900/4848000] [188.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 17:22:32,961 - Train: 18.87% [915000/4848000] [188.7/1000.0] [batch_t 0.333 (0.339)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 17:23:06,429 - Train: 18.88% [915100/4848000] [188.8/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 17:23:39,368 - Train: 18.88% [915200/4848000] [188.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 17:24:13,249 - Train: 18.88% [915300/4848000] [188.8/1000.0] [batch_t 0.325 (0.339)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 17:24:46,030 - Train: 18.88% [915400/4848000] [188.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 17:25:20,553 - Train: 18.88% [915500/4848000] [188.8/1000.0] [batch_t 0.328 (0.345)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 17:25:53,368 - Train: 18.89% [915600/4848000] [188.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 17:26:27,433 - Train: 18.89% [915700/4848000] [188.9/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 17:27:00,234 - Train: 18.89% [915800/4848000] [188.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 17:27:33,895 - Train: 18.89% [915900/4848000] [188.9/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 17:28:10,046 - Train: 18.89% [916000/4848000] [188.9/1000.0] [batch_t 0.329 (0.361)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 17:28:42,826 - Train: 18.90% [916100/4848000] [189.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 17:29:20,917 - Train: 18.90% [916200/4848000] [189.0/1000.0] [batch_t 0.326 (0.381)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 17:29:44,583 - ==> Total time: 3 days, 18:03:43 Eta: 16 days, 2:27:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 17:29:57,146 - Train: 18.90% [916300/4848000] [189.0/1000.0] [batch_t 0.325 (0.354)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 17:30:31,593 - Train: 18.90% [916400/4848000] [189.0/1000.0] [batch_t 0.328 (0.344)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 17:31:04,567 - Train: 18.90% [916500/4848000] [189.0/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-06 17:31:37,649 - Train: 18.91% [916600/4848000] [189.1/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 17:32:11,645 - Train: 18.91% [916700/4848000] [189.1/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 17:32:44,441 - Train: 18.91% [916800/4848000] [189.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 17:33:18,138 - Train: 18.91% [916900/4848000] [189.1/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 17:33:50,902 - Train: 18.92% [917000/4848000] [189.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 17:34:24,100 - Train: 18.92% [917100/4848000] [189.2/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 17:34:56,855 - Train: 18.92% [917200/4848000] [189.2/1000.0] [batch_t 0.325 (0.327)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 17:35:30,794 - Train: 18.92% [917300/4848000] [189.2/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 17:36:03,602 - Train: 18.92% [917400/4848000] [189.2/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 17:36:37,614 - Train: 18.93% [917500/4848000] [189.3/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 17:37:11,857 - Train: 18.93% [917600/4848000] [189.3/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 17:37:44,667 - Train: 18.93% [917700/4848000] [189.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 17:38:17,402 - Train: 18.93% [917800/4848000] [189.3/1000.0] [batch_t 0.326 (0.327)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 17:38:50,252 - Train: 18.93% [917900/4848000] [189.3/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 17:39:23,254 - Train: 18.94% [918000/4848000] [189.4/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 17:39:56,089 - Train: 18.94% [918100/4848000] [189.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 17:40:29,340 - Train: 18.94% [918200/4848000] [189.4/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 17:41:02,171 - Train: 18.94% [918300/4848000] [189.4/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 17:41:36,151 - Train: 18.94% [918400/4848000] [189.4/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 17:42:09,755 - Train: 18.95% [918500/4848000] [189.5/1000.0] [batch_t 0.335 (0.336)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-06 17:42:42,626 - Train: 18.95% [918600/4848000] [189.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 17:43:16,147 - Train: 18.95% [918700/4848000] [189.5/1000.0] [batch_t 0.331 (0.335)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 17:43:49,009 - Train: 18.95% [918800/4848000] [189.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 17:44:21,826 - Train: 18.95% [918900/4848000] [189.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 17:44:54,640 - Train: 18.96% [919000/4848000] [189.6/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 17:45:27,530 - Train: 18.96% [919100/4848000] [189.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 17:46:00,328 - Train: 18.96% [919200/4848000] [189.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 17:46:34,383 - Train: 18.96% [919300/4848000] [189.6/1000.0] [batch_t 0.325 (0.340)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 17:47:09,539 - Train: 18.96% [919400/4848000] [189.6/1000.0] [batch_t 0.332 (0.351)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 17:47:42,377 - Train: 18.97% [919500/4848000] [189.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 17:48:15,238 - Train: 18.97% [919600/4848000] [189.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 17:48:48,096 - Train: 18.97% [919700/4848000] [189.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 17:49:22,248 - Train: 18.97% [919800/4848000] [189.7/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 17:49:55,081 - Train: 18.97% [919900/4848000] [189.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 17:50:29,175 - Train: 18.98% [920000/4848000] [189.8/1000.0] [batch_t 0.329 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 17:51:01,968 - Train: 18.98% [920100/4848000] [189.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 17:51:37,810 - Train: 18.98% [920200/4848000] [189.8/1000.0] [batch_t 0.329 (0.358)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 17:52:10,616 - Train: 18.98% [920300/4848000] [189.8/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-06 17:52:43,351 - Train: 18.99% [920400/4848000] [189.9/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 17:53:17,220 - Train: 18.99% [920500/4848000] [189.9/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 17:53:50,001 - Train: 18.99% [920600/4848000] [189.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 17:54:25,005 - Train: 18.99% [920700/4848000] [189.9/1000.0] [batch_t 0.333 (0.350)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 17:54:57,810 - Train: 18.99% [920800/4848000] [189.9/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 17:55:32,781 - Train: 19.00% [920900/4848000] [190.0/1000.0] [batch_t 0.324 (0.350)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 17:56:06,552 - Train: 19.00% [921000/4848000] [190.0/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 17:56:39,421 - Train: 19.00% [921100/4848000] [190.0/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 17:56:45,987 - ==> Total time: 3 days, 18:30:44 Eta: 16 days, 1:52:06 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 17:57:15,674 - Train: 19.00% [921200/4848000] [190.0/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 17:57:48,475 - Train: 19.00% [921300/4848000] [190.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 17:58:22,242 - Train: 19.01% [921400/4848000] [190.1/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 17:58:55,032 - Train: 19.01% [921500/4848000] [190.1/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 17:59:27,872 - Train: 19.01% [921600/4848000] [190.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 18:00:00,611 - Train: 19.01% [921700/4848000] [190.1/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 18:00:33,452 - Train: 19.01% [921800/4848000] [190.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 18:01:06,645 - Train: 19.02% [921900/4848000] [190.2/1000.0] [batch_t 0.332 (0.332)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 18:01:39,473 - Train: 19.02% [922000/4848000] [190.2/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 18:02:12,494 - Train: 19.02% [922100/4848000] [190.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 18:02:45,332 - Train: 19.02% [922200/4848000] [190.2/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 18:03:18,691 - Train: 19.02% [922300/4848000] [190.2/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 18:03:51,442 - Train: 19.03% [922400/4848000] [190.3/1000.0] [batch_t 0.327 (0.327)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 18:04:25,010 - Train: 19.03% [922500/4848000] [190.3/1000.0] [batch_t 0.327 (0.336)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 18:04:57,840 - Train: 19.03% [922600/4848000] [190.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 18:05:30,641 - Train: 19.03% [922700/4848000] [190.3/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 18:06:03,446 - Train: 19.03% [922800/4848000] [190.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 18:06:38,299 - Train: 19.04% [922900/4848000] [190.4/1000.0] [batch_t 0.328 (0.348)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 18:07:11,085 - Train: 19.04% [923000/4848000] [190.4/1000.0] [batch_t 0.322 (0.328)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-06 18:07:43,834 - Train: 19.04% [923100/4848000] [190.4/1000.0] [batch_t 0.323 (0.327)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-06 18:08:16,815 - Train: 19.04% [923200/4848000] [190.4/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 18:08:49,623 - Train: 19.04% [923300/4848000] [190.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 18:09:22,414 - Train: 19.05% [923400/4848000] [190.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 18:09:55,255 - Train: 19.05% [923500/4848000] [190.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 18:10:28,236 - Train: 19.05% [923600/4848000] [190.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 18:11:01,036 - Train: 19.05% [923700/4848000] [190.5/1000.0] [batch_t 0.336 (0.328)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-06 18:11:34,636 - Train: 19.06% [923800/4848000] [190.6/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 18:12:07,444 - Train: 19.06% [923900/4848000] [190.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 18:12:40,283 - Train: 19.06% [924000/4848000] [190.6/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 18:13:13,233 - Train: 19.06% [924100/4848000] [190.6/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 18:13:46,089 - Train: 19.06% [924200/4848000] [190.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 18:14:19,181 - Train: 19.07% [924300/4848000] [190.7/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 18:14:51,986 - Train: 19.07% [924400/4848000] [190.7/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 18:15:24,866 - Train: 19.07% [924500/4848000] [190.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 18:15:57,705 - Train: 19.07% [924600/4848000] [190.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 18:16:30,547 - Train: 19.07% [924700/4848000] [190.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 18:17:03,459 - Train: 19.08% [924800/4848000] [190.8/1000.0] [batch_t 0.321 (0.329)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-06 18:17:36,312 - Train: 19.08% [924900/4848000] [190.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 18:18:09,233 - Train: 19.08% [925000/4848000] [190.8/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-06 18:18:42,186 - Train: 19.08% [925100/4848000] [190.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 18:19:15,025 - Train: 19.08% [925200/4848000] [190.8/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 18:19:47,805 - Train: 19.09% [925300/4848000] [190.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 18:20:22,129 - Train: 19.09% [925400/4848000] [190.9/1000.0] [batch_t 0.327 (0.343)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 18:20:54,890 - Train: 19.09% [925500/4848000] [190.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 18:21:29,212 - Train: 19.09% [925600/4848000] [190.9/1000.0] [batch_t 0.336 (0.343)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-06 18:22:02,028 - Train: 19.09% [925700/4848000] [190.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 18:22:35,691 - Train: 19.10% [925800/4848000] [191.0/1000.0] [batch_t 0.337 (0.337)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-06 18:23:09,716 - Train: 19.10% [925900/4848000] [191.0/1000.0] [batch_t 0.332 (0.340)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 18:23:31,998 - ==> Total time: 3 days, 18:57:30 Eta: 16 days, 1:15:50 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 18:23:44,407 - Train: 19.10% [926000/4848000] [191.0/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 18:24:18,549 - Train: 19.10% [926100/4848000] [191.0/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 18:24:51,326 - Train: 19.10% [926200/4848000] [191.0/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 18:25:25,530 - Train: 19.11% [926300/4848000] [191.1/1000.0] [batch_t 0.327 (0.342)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 18:25:58,498 - Train: 19.11% [926400/4848000] [191.1/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 18:26:31,566 - Train: 19.11% [926500/4848000] [191.1/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 18:27:04,364 - Train: 19.11% [926600/4848000] [191.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 18:27:40,119 - Train: 19.12% [926700/4848000] [191.2/1000.0] [batch_t 0.327 (0.357)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 18:28:14,809 - Train: 19.12% [926800/4848000] [191.2/1000.0] [batch_t 0.330 (0.347)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 18:28:47,634 - Train: 19.12% [926900/4848000] [191.2/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 18:29:21,932 - Train: 19.12% [927000/4848000] [191.2/1000.0] [batch_t 0.331 (0.343)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 18:29:54,695 - Train: 19.12% [927100/4848000] [191.2/1000.0] [batch_t 0.323 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 18:30:28,294 - Train: 19.13% [927200/4848000] [191.3/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 18:31:01,096 - Train: 19.13% [927300/4848000] [191.3/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 18:31:34,652 - Train: 19.13% [927400/4848000] [191.3/1000.0] [batch_t 0.329 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 18:32:07,427 - Train: 19.13% [927500/4848000] [191.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 18:32:40,380 - Train: 19.13% [927600/4848000] [191.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 18:33:13,311 - Train: 19.14% [927700/4848000] [191.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 18:33:46,199 - Train: 19.14% [927800/4848000] [191.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 18:34:19,812 - Train: 19.14% [927900/4848000] [191.4/1000.0] [batch_t 0.326 (0.336)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 18:34:52,650 - Train: 19.14% [928000/4848000] [191.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 18:35:25,469 - Train: 19.14% [928100/4848000] [191.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 18:35:58,252 - Train: 19.15% [928200/4848000] [191.5/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 18:36:31,415 - Train: 19.15% [928300/4848000] [191.5/1000.0] [batch_t 0.648 (0.332)] [data_t 0.319] [optim_t 0.329] [lr 0.005000] 2024-04-06 18:37:07,791 - Train: 19.15% [928400/4848000] [191.5/1000.0] [batch_t 0.327 (0.364)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 18:37:40,646 - Train: 19.15% [928500/4848000] [191.5/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 18:38:13,484 - Train: 19.15% [928600/4848000] [191.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 18:38:46,387 - Train: 19.16% [928700/4848000] [191.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-06 18:39:20,233 - Train: 19.16% [928800/4848000] [191.6/1000.0] [batch_t 0.323 (0.338)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-06 18:39:53,094 - Train: 19.16% [928900/4848000] [191.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 18:40:25,988 - Train: 19.16% [929000/4848000] [191.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-06 18:40:58,851 - Train: 19.16% [929100/4848000] [191.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-06 18:41:31,739 - Train: 19.17% [929200/4848000] [191.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 18:42:04,761 - Train: 19.17% [929300/4848000] [191.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-06 18:42:37,639 - Train: 19.17% [929400/4848000] [191.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-06 18:43:10,576 - Train: 19.17% [929500/4848000] [191.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 18:43:43,526 - Train: 19.17% [929600/4848000] [191.7/1000.0] [batch_t 0.336 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-06 18:44:16,456 - Train: 19.18% [929700/4848000] [191.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-06 18:44:49,393 - Train: 19.18% [929800/4848000] [191.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-06 18:45:23,087 - Train: 19.18% [929900/4848000] [191.8/1000.0] [batch_t 0.326 (0.337)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-06 18:45:56,052 - Train: 19.18% [930000/4848000] [191.8/1000.0] [batch_t 0.325 (0.330)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-06 18:46:29,047 - Train: 19.19% [930100/4848000] [191.9/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 18:47:01,952 - Train: 19.19% [930200/4848000] [191.9/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 18:47:35,371 - Train: 19.19% [930300/4848000] [191.9/1000.0] [batch_t 0.327 (0.334)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-06 18:48:08,361 - Train: 19.19% [930400/4848000] [191.9/1000.0] [batch_t 0.325 (0.330)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-06 18:48:41,265 - Train: 19.19% [930500/4848000] [191.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-06 18:49:14,870 - Train: 19.20% [930600/4848000] [192.0/1000.0] [batch_t 0.333 (0.336)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-06 18:49:50,601 - Train: 19.20% [930700/4848000] [192.0/1000.0] [batch_t 0.330 (0.357)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 18:50:26,262 - Train: 19.20% [930800/4848000] [192.0/1000.0] [batch_t 0.329 (0.357)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 18:50:31,562 - ==> Total time: 3 days, 19:24:30 Eta: 16 days, 0:40:36 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 18:51:01,955 - Train: 19.20% [930900/4848000] [192.0/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 18:52:29,000 - Train: 19.20% [931000/4848000] [192.0/1000.0] [batch_t 0.950 (0.870)] [data_t 0.624] [optim_t 0.326] [lr 0.005000] 2024-04-06 18:53:45,732 - Train: 19.21% [931100/4848000] [192.1/1000.0] [batch_t 0.328 (0.767)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-06 18:55:06,432 - Train: 19.21% [931200/4848000] [192.1/1000.0] [batch_t 0.334 (0.807)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-06 18:55:45,964 - Train: 19.21% [931300/4848000] [192.1/1000.0] [batch_t 0.330 (0.395)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 18:56:56,663 - Train: 19.21% [931400/4848000] [192.1/1000.0] [batch_t 0.331 (0.707)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-06 18:58:17,798 - Train: 19.21% [931500/4848000] [192.1/1000.0] [batch_t 0.332 (0.811)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-06 18:59:07,006 - Train: 19.22% [931600/4848000] [192.2/1000.0] [batch_t 0.327 (0.492)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 19:00:10,528 - Train: 19.22% [931700/4848000] [192.2/1000.0] [batch_t 0.334 (0.635)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-06 19:02:27,057 - Train: 19.22% [931800/4848000] [192.2/1000.0] [batch_t 3.487 (1.365)] [data_t 2.710] [optim_t 0.777] [lr 0.005000] 2024-04-06 19:04:44,146 - Train: 19.22% [931900/4848000] [192.2/1000.0] [batch_t 0.773 (1.371)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 19:06:37,597 - Train: 19.22% [932000/4848000] [192.2/1000.0] [batch_t 0.776 (1.134)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-06 19:08:43,402 - Train: 19.23% [932100/4848000] [192.3/1000.0] [batch_t 0.944 (1.258)] [data_t 0.169] [optim_t 0.775] [lr 0.005000] 2024-04-06 19:10:24,543 - Train: 19.23% [932200/4848000] [192.3/1000.0] [batch_t 0.777 (1.011)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 19:11:42,317 - Train: 19.23% [932300/4848000] [192.3/1000.0] [batch_t 0.773 (0.778)] [data_t 0.004] [optim_t 0.770] [lr 0.005000] 2024-04-06 19:13:00,158 - Train: 19.23% [932400/4848000] [192.3/1000.0] [batch_t 0.780 (0.778)] [data_t 0.003] [optim_t 0.777] [lr 0.005000] 2024-04-06 19:14:17,904 - Train: 19.23% [932500/4848000] [192.3/1000.0] [batch_t 0.779 (0.777)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 19:15:35,765 - Train: 19.24% [932600/4848000] [192.4/1000.0] [batch_t 0.777 (0.779)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 19:16:53,436 - Train: 19.24% [932700/4848000] [192.4/1000.0] [batch_t 0.787 (0.777)] [data_t 0.003] [optim_t 0.784] [lr 0.005000] 2024-04-06 19:18:11,270 - Train: 19.24% [932800/4848000] [192.4/1000.0] [batch_t 0.778 (0.778)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 19:19:29,100 - Train: 19.24% [932900/4848000] [192.4/1000.0] [batch_t 0.776 (0.778)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-06 19:20:46,839 - Train: 19.25% [933000/4848000] [192.5/1000.0] [batch_t 0.783 (0.777)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-06 19:22:04,678 - Train: 19.25% [933100/4848000] [192.5/1000.0] [batch_t 0.778 (0.778)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 19:23:22,479 - Train: 19.25% [933200/4848000] [192.5/1000.0] [batch_t 0.782 (0.778)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-06 19:24:40,349 - Train: 19.25% [933300/4848000] [192.5/1000.0] [batch_t 0.787 (0.779)] [data_t 0.003] [optim_t 0.784] [lr 0.005000] 2024-04-06 19:25:58,007 - Train: 19.25% [933400/4848000] [192.5/1000.0] [batch_t 0.785 (0.776)] [data_t 0.003] [optim_t 0.781] [lr 0.005000] 2024-04-06 19:27:15,826 - Train: 19.26% [933500/4848000] [192.6/1000.0] [batch_t 0.777 (0.778)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-06 19:28:33,589 - Train: 19.26% [933600/4848000] [192.6/1000.0] [batch_t 0.788 (0.778)] [data_t 0.003] [optim_t 0.785] [lr 0.005000] 2024-04-06 19:29:51,469 - Train: 19.26% [933700/4848000] [192.6/1000.0] [batch_t 0.778 (0.779)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-06 19:31:09,303 - Train: 19.26% [933800/4848000] [192.6/1000.0] [batch_t 0.773 (0.778)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 19:32:27,177 - Train: 19.26% [933900/4848000] [192.6/1000.0] [batch_t 0.778 (0.779)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 19:33:44,920 - Train: 19.27% [934000/4848000] [192.7/1000.0] [batch_t 0.777 (0.777)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 19:35:02,732 - Train: 19.27% [934100/4848000] [192.7/1000.0] [batch_t 0.783 (0.778)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-06 19:36:20,536 - Train: 19.27% [934200/4848000] [192.7/1000.0] [batch_t 0.784 (0.778)] [data_t 0.003] [optim_t 0.781] [lr 0.005000] 2024-04-06 19:37:38,296 - Train: 19.27% [934300/4848000] [192.7/1000.0] [batch_t 0.782 (0.778)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-06 19:38:56,122 - Train: 19.27% [934400/4848000] [192.7/1000.0] [batch_t 0.762 (0.778)] [data_t 0.003] [optim_t 0.759] [lr 0.005000] 2024-04-06 19:40:13,848 - Train: 19.28% [934500/4848000] [192.8/1000.0] [batch_t 0.776 (0.777)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-06 19:41:31,591 - Train: 19.28% [934600/4848000] [192.8/1000.0] [batch_t 0.776 (0.777)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-06 19:42:49,389 - Train: 19.28% [934700/4848000] [192.8/1000.0] [batch_t 0.778 (0.778)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 19:44:07,236 - Train: 19.28% [934800/4848000] [192.8/1000.0] [batch_t 0.782 (0.778)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-06 19:45:24,995 - Train: 19.28% [934900/4848000] [192.8/1000.0] [batch_t 0.783 (0.777)] [data_t 0.003] [optim_t 0.780] [lr 0.005000] 2024-04-06 19:46:43,993 - Train: 19.29% [935000/4848000] [192.9/1000.0] [batch_t 0.778 (0.790)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 19:48:01,759 - Train: 19.29% [935100/4848000] [192.9/1000.0] [batch_t 0.774 (0.778)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-06 19:49:19,564 - Train: 19.29% [935200/4848000] [192.9/1000.0] [batch_t 0.777 (0.778)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-06 19:50:37,209 - Train: 19.29% [935300/4848000] [192.9/1000.0] [batch_t 0.785 (0.776)] [data_t 0.003] [optim_t 0.782] [lr 0.005000] 2024-04-06 19:51:55,003 - Train: 19.29% [935400/4848000] [192.9/1000.0] [batch_t 0.788 (0.778)] [data_t 0.003] [optim_t 0.785] [lr 0.005000] 2024-04-06 19:53:12,807 - Train: 19.30% [935500/4848000] [193.0/1000.0] [batch_t 0.777 (0.778)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-06 19:54:30,674 - Train: 19.30% [935600/4848000] [193.0/1000.0] [batch_t 0.782 (0.779)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-06 19:55:20,479 - ==> Total time: 3 days, 20:29:18 Eta: 16 days, 2:43:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 19:55:50,576 - Train: 19.30% [935700/4848000] [193.0/1000.0] [batch_t 0.777 (0.778)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-06 19:57:08,295 - Train: 19.30% [935800/4848000] [193.0/1000.0] [batch_t 0.772 (0.777)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 19:58:26,088 - Train: 19.30% [935900/4848000] [193.0/1000.0] [batch_t 0.781 (0.778)] [data_t 0.003] [optim_t 0.778] [lr 0.005000] 2024-04-06 19:59:43,827 - Train: 19.31% [936000/4848000] [193.1/1000.0] [batch_t 0.773 (0.777)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 20:01:01,596 - Train: 19.31% [936100/4848000] [193.1/1000.0] [batch_t 0.778 (0.778)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 20:02:19,340 - Train: 19.31% [936200/4848000] [193.1/1000.0] [batch_t 0.773 (0.777)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 20:03:37,183 - Train: 19.31% [936300/4848000] [193.1/1000.0] [batch_t 0.769 (0.778)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-06 20:04:54,963 - Train: 19.32% [936400/4848000] [193.2/1000.0] [batch_t 0.778 (0.778)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 20:06:12,788 - Train: 19.32% [936500/4848000] [193.2/1000.0] [batch_t 0.778 (0.778)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-06 20:07:30,532 - Train: 19.32% [936600/4848000] [193.2/1000.0] [batch_t 0.772 (0.777)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 20:08:48,389 - Train: 19.32% [936700/4848000] [193.2/1000.0] [batch_t 0.778 (0.778)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 20:10:06,029 - Train: 19.32% [936800/4848000] [193.2/1000.0] [batch_t 0.777 (0.776)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 20:11:23,803 - Train: 19.33% [936900/4848000] [193.3/1000.0] [batch_t 0.787 (0.778)] [data_t 0.003] [optim_t 0.784] [lr 0.005000] 2024-04-06 20:12:41,601 - Train: 19.33% [937000/4848000] [193.3/1000.0] [batch_t 0.778 (0.778)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 20:13:59,312 - Train: 19.33% [937100/4848000] [193.3/1000.0] [batch_t 0.782 (0.777)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-06 20:15:17,041 - Train: 19.33% [937200/4848000] [193.3/1000.0] [batch_t 0.781 (0.777)] [data_t 0.003] [optim_t 0.778] [lr 0.005000] 2024-04-06 20:16:34,873 - Train: 19.33% [937300/4848000] [193.3/1000.0] [batch_t 0.773 (0.778)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 20:17:52,596 - Train: 19.34% [937400/4848000] [193.4/1000.0] [batch_t 0.777 (0.777)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 20:19:10,339 - Train: 19.34% [937500/4848000] [193.4/1000.0] [batch_t 0.777 (0.777)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 20:20:28,256 - Train: 19.34% [937600/4848000] [193.4/1000.0] [batch_t 0.789 (0.779)] [data_t 0.003] [optim_t 0.786] [lr 0.005000] 2024-04-06 20:21:45,968 - Train: 19.34% [937700/4848000] [193.4/1000.0] [batch_t 0.773 (0.777)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 20:23:03,803 - Train: 19.34% [937800/4848000] [193.4/1000.0] [batch_t 0.783 (0.778)] [data_t 0.003] [optim_t 0.780] [lr 0.005000] 2024-04-06 20:24:21,625 - Train: 19.35% [937900/4848000] [193.5/1000.0] [batch_t 0.772 (0.778)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 20:25:39,443 - Train: 19.35% [938000/4848000] [193.5/1000.0] [batch_t 0.772 (0.778)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 20:26:57,174 - Train: 19.35% [938100/4848000] [193.5/1000.0] [batch_t 0.767 (0.777)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 20:28:14,937 - Train: 19.35% [938200/4848000] [193.5/1000.0] [batch_t 0.772 (0.778)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 20:29:32,765 - Train: 19.35% [938300/4848000] [193.5/1000.0] [batch_t 0.777 (0.778)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 20:30:50,617 - Train: 19.36% [938400/4848000] [193.6/1000.0] [batch_t 0.772 (0.778)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 20:32:08,302 - Train: 19.36% [938500/4848000] [193.6/1000.0] [batch_t 0.777 (0.777)] [data_t 0.004] [optim_t 0.773] [lr 0.005000] 2024-04-06 20:33:26,090 - Train: 19.36% [938600/4848000] [193.6/1000.0] [batch_t 0.788 (0.778)] [data_t 0.003] [optim_t 0.785] [lr 0.005000] 2024-04-06 20:34:43,974 - Train: 19.36% [938700/4848000] [193.6/1000.0] [batch_t 0.781 (0.779)] [data_t 0.003] [optim_t 0.778] [lr 0.005000] 2024-04-06 20:36:01,792 - Train: 19.36% [938800/4848000] [193.6/1000.0] [batch_t 0.773 (0.778)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 20:37:19,550 - Train: 19.37% [938900/4848000] [193.7/1000.0] [batch_t 0.774 (0.777)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-06 20:38:37,292 - Train: 19.37% [939000/4848000] [193.7/1000.0] [batch_t 0.778 (0.777)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 20:39:55,151 - Train: 19.37% [939100/4848000] [193.7/1000.0] [batch_t 0.784 (0.778)] [data_t 0.003] [optim_t 0.780] [lr 0.005000] 2024-04-06 20:41:12,975 - Train: 19.37% [939200/4848000] [193.7/1000.0] [batch_t 0.788 (0.778)] [data_t 0.003] [optim_t 0.785] [lr 0.005000] 2024-04-06 20:42:30,743 - Train: 19.38% [939300/4848000] [193.8/1000.0] [batch_t 0.782 (0.778)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-06 20:43:48,549 - Train: 19.38% [939400/4848000] [193.8/1000.0] [batch_t 0.785 (0.778)] [data_t 0.003] [optim_t 0.782] [lr 0.005000] 2024-04-06 20:45:06,312 - Train: 19.38% [939500/4848000] [193.8/1000.0] [batch_t 0.781 (0.778)] [data_t 0.003] [optim_t 0.778] [lr 0.005000] 2024-04-06 20:46:24,129 - Train: 19.38% [939600/4848000] [193.8/1000.0] [batch_t 0.772 (0.778)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 20:47:41,942 - Train: 19.38% [939700/4848000] [193.8/1000.0] [batch_t 0.781 (0.778)] [data_t 0.003] [optim_t 0.778] [lr 0.005000] 2024-04-06 20:48:59,745 - Train: 19.39% [939800/4848000] [193.9/1000.0] [batch_t 0.777 (0.778)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 20:50:17,481 - Train: 19.39% [939900/4848000] [193.9/1000.0] [batch_t 0.775 (0.777)] [data_t 0.003] [optim_t 0.772] [lr 0.005000] 2024-04-06 20:51:35,300 - Train: 19.39% [940000/4848000] [193.9/1000.0] [batch_t 0.777 (0.778)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-06 20:52:53,146 - Train: 19.39% [940100/4848000] [193.9/1000.0] [batch_t 0.771 (0.778)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 20:54:10,959 - Train: 19.39% [940200/4848000] [193.9/1000.0] [batch_t 0.779 (0.778)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-06 20:55:28,828 - Train: 19.40% [940300/4848000] [194.0/1000.0] [batch_t 0.776 (0.779)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-06 20:56:46,615 - Train: 19.40% [940400/4848000] [194.0/1000.0] [batch_t 0.767 (0.778)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-06 20:58:04,400 - Train: 19.40% [940500/4848000] [194.0/1000.0] [batch_t 0.773 (0.778)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-06 20:58:13,740 - ==> Total time: 3 days, 21:32:12 Eta: 16 days, 4:36:40 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 20:59:24,502 - Train: 19.40% [940600/4848000] [194.0/1000.0] [batch_t 0.782 (0.780)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-06 21:00:42,370 - Train: 19.40% [940700/4848000] [194.0/1000.0] [batch_t 0.772 (0.779)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 21:02:00,114 - Train: 19.41% [940800/4848000] [194.1/1000.0] [batch_t 0.779 (0.777)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-06 21:03:17,832 - Train: 19.41% [940900/4848000] [194.1/1000.0] [batch_t 0.783 (0.777)] [data_t 0.003] [optim_t 0.780] [lr 0.005000] 2024-04-06 21:04:35,628 - Train: 19.41% [941000/4848000] [194.1/1000.0] [batch_t 0.781 (0.778)] [data_t 0.003] [optim_t 0.778] [lr 0.005000] 2024-04-06 21:05:53,458 - Train: 19.41% [941100/4848000] [194.1/1000.0] [batch_t 0.781 (0.778)] [data_t 0.003] [optim_t 0.778] [lr 0.005000] 2024-04-06 21:07:11,179 - Train: 19.41% [941200/4848000] [194.1/1000.0] [batch_t 0.785 (0.777)] [data_t 0.003] [optim_t 0.781] [lr 0.005000] 2024-04-06 21:08:28,962 - Train: 19.42% [941300/4848000] [194.2/1000.0] [batch_t 0.778 (0.778)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 21:09:46,575 - Train: 19.42% [941400/4848000] [194.2/1000.0] [batch_t 0.773 (0.776)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 21:11:04,457 - Train: 19.42% [941500/4848000] [194.2/1000.0] [batch_t 0.781 (0.779)] [data_t 0.003] [optim_t 0.778] [lr 0.005000] 2024-04-06 21:12:22,313 - Train: 19.42% [941600/4848000] [194.2/1000.0] [batch_t 0.778 (0.778)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 21:13:40,111 - Train: 19.42% [941700/4848000] [194.2/1000.0] [batch_t 0.777 (0.778)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 21:14:57,833 - Train: 19.43% [941800/4848000] [194.3/1000.0] [batch_t 0.772 (0.777)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-06 21:16:15,539 - Train: 19.43% [941900/4848000] [194.3/1000.0] [batch_t 0.781 (0.777)] [data_t 0.004] [optim_t 0.778] [lr 0.005000] 2024-04-06 21:17:33,284 - Train: 19.43% [942000/4848000] [194.3/1000.0] [batch_t 0.777 (0.777)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 21:18:51,074 - Train: 19.43% [942100/4848000] [194.3/1000.0] [batch_t 0.778 (0.778)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 21:20:08,794 - Train: 19.43% [942200/4848000] [194.3/1000.0] [batch_t 0.788 (0.777)] [data_t 0.003] [optim_t 0.785] [lr 0.005000] 2024-04-06 21:21:26,581 - Train: 19.44% [942300/4848000] [194.4/1000.0] [batch_t 0.780 (0.778)] [data_t 0.003] [optim_t 0.777] [lr 0.005000] 2024-04-06 21:22:44,367 - Train: 19.44% [942400/4848000] [194.4/1000.0] [batch_t 0.778 (0.778)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 21:24:02,011 - Train: 19.44% [942500/4848000] [194.4/1000.0] [batch_t 0.782 (0.776)] [data_t 0.003] [optim_t 0.778] [lr 0.005000] 2024-04-06 21:25:19,646 - Train: 19.44% [942600/4848000] [194.4/1000.0] [batch_t 0.782 (0.776)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-06 21:26:37,373 - Train: 19.45% [942700/4848000] [194.5/1000.0] [batch_t 0.781 (0.777)] [data_t 0.003] [optim_t 0.778] [lr 0.005000] 2024-04-06 21:27:55,079 - Train: 19.45% [942800/4848000] [194.5/1000.0] [batch_t 0.777 (0.777)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-06 21:29:12,789 - Train: 19.45% [942900/4848000] [194.5/1000.0] [batch_t 0.779 (0.777)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-06 21:30:30,527 - Train: 19.45% [943000/4848000] [194.5/1000.0] [batch_t 0.769 (0.777)] [data_t 0.003] [optim_t 0.766] [lr 0.005000] 2024-04-06 21:31:45,104 - Train: 19.45% [943100/4848000] [194.5/1000.0] [batch_t 0.328 (0.746)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 21:32:17,933 - Train: 19.46% [943200/4848000] [194.6/1000.0] [batch_t 0.337 (0.328)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-06 21:32:50,845 - Train: 19.46% [943300/4848000] [194.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 21:33:23,796 - Train: 19.46% [943400/4848000] [194.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 21:33:56,635 - Train: 19.46% [943500/4848000] [194.6/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 21:34:29,504 - Train: 19.46% [943600/4848000] [194.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 21:35:02,472 - Train: 19.47% [943700/4848000] [194.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 21:35:35,400 - Train: 19.47% [943800/4848000] [194.7/1000.0] [batch_t 0.323 (0.329)] [data_t 0.003] [optim_t 0.320] [lr 0.005000] 2024-04-06 21:36:08,321 - Train: 19.47% [943900/4848000] [194.7/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-06 21:36:41,230 - Train: 19.47% [944000/4848000] [194.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 21:37:14,155 - Train: 19.47% [944100/4848000] [194.7/1000.0] [batch_t 0.326 (0.329)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-06 21:37:47,141 - Train: 19.48% [944200/4848000] [194.8/1000.0] [batch_t 0.337 (0.330)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-06 21:38:20,164 - Train: 19.48% [944300/4848000] [194.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 21:38:53,180 - Train: 19.48% [944400/4848000] [194.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-06 21:39:26,337 - Train: 19.48% [944500/4848000] [194.8/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-06 21:39:59,367 - Train: 19.48% [944600/4848000] [194.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-06 21:40:32,415 - Train: 19.49% [944700/4848000] [194.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-06 21:41:05,429 - Train: 19.49% [944800/4848000] [194.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 21:41:38,392 - Train: 19.49% [944900/4848000] [194.9/1000.0] [batch_t 0.333 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-06 21:42:11,412 - Train: 19.49% [945000/4848000] [194.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-06 21:42:44,485 - Train: 19.49% [945100/4848000] [194.9/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-06 21:43:17,549 - Train: 19.50% [945200/4848000] [195.0/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 21:43:50,549 - Train: 19.50% [945300/4848000] [195.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-06 21:44:10,422 - ==> Total time: 3 days, 22:18:08 Eta: 16 days, 5:17:59 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 21:44:25,523 - Train: 19.50% [945400/4848000] [195.0/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-06 21:44:58,525 - Train: 19.50% [945500/4848000] [195.0/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-06 21:45:31,551 - Train: 19.50% [945600/4848000] [195.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 21:46:04,495 - Train: 19.51% [945700/4848000] [195.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 21:46:37,460 - Train: 19.51% [945800/4848000] [195.1/1000.0] [batch_t 0.333 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-06 21:47:10,377 - Train: 19.51% [945900/4848000] [195.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 21:47:43,397 - Train: 19.51% [946000/4848000] [195.1/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-06 21:48:16,284 - Train: 19.52% [946100/4848000] [195.2/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 21:48:49,223 - Train: 19.52% [946200/4848000] [195.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-06 21:49:22,110 - Train: 19.52% [946300/4848000] [195.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 21:49:55,016 - Train: 19.52% [946400/4848000] [195.2/1000.0] [batch_t 0.326 (0.329)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-06 21:50:27,952 - Train: 19.52% [946500/4848000] [195.2/1000.0] [batch_t 0.325 (0.329)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-06 21:51:00,829 - Train: 19.53% [946600/4848000] [195.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-06 21:51:33,746 - Train: 19.53% [946700/4848000] [195.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-06 21:52:06,635 - Train: 19.53% [946800/4848000] [195.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-06 21:52:39,527 - Train: 19.53% [946900/4848000] [195.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 21:53:12,409 - Train: 19.53% [947000/4848000] [195.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 21:53:45,269 - Train: 19.54% [947100/4848000] [195.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-06 21:54:18,209 - Train: 19.54% [947200/4848000] [195.4/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-06 21:54:51,083 - Train: 19.54% [947300/4848000] [195.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-06 21:55:24,122 - Train: 19.54% [947400/4848000] [195.4/1000.0] [batch_t 0.326 (0.330)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-06 21:55:57,033 - Train: 19.54% [947500/4848000] [195.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 21:56:29,960 - Train: 19.55% [947600/4848000] [195.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 21:57:02,874 - Train: 19.55% [947700/4848000] [195.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-06 21:57:36,240 - Train: 19.55% [947800/4848000] [195.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-06 21:58:09,165 - Train: 19.55% [947900/4848000] [195.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-06 21:58:42,086 - Train: 19.55% [948000/4848000] [195.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 21:59:14,946 - Train: 19.56% [948100/4848000] [195.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 21:59:47,848 - Train: 19.56% [948200/4848000] [195.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-06 22:00:20,736 - Train: 19.56% [948300/4848000] [195.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-06 22:00:53,678 - Train: 19.56% [948400/4848000] [195.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-06 22:01:26,635 - Train: 19.56% [948500/4848000] [195.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 22:01:59,554 - Train: 19.57% [948600/4848000] [195.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 22:02:32,439 - Train: 19.57% [948700/4848000] [195.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-06 22:03:05,345 - Train: 19.57% [948800/4848000] [195.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 22:03:38,340 - Train: 19.57% [948900/4848000] [195.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-06 22:04:11,222 - Train: 19.58% [949000/4848000] [195.8/1000.0] [batch_t 0.324 (0.329)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-06 22:04:44,117 - Train: 19.58% [949100/4848000] [195.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 22:05:17,020 - Train: 19.58% [949200/4848000] [195.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-06 22:05:49,857 - Train: 19.58% [949300/4848000] [195.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 22:06:22,781 - Train: 19.58% [949400/4848000] [195.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 22:06:55,692 - Train: 19.59% [949500/4848000] [195.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-06 22:07:28,571 - Train: 19.59% [949600/4848000] [195.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 22:08:01,505 - Train: 19.59% [949700/4848000] [195.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-06 22:08:34,405 - Train: 19.59% [949800/4848000] [195.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-06 22:09:07,329 - Train: 19.59% [949900/4848000] [195.9/1000.0] [batch_t 0.324 (0.329)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-06 22:09:40,175 - Train: 19.60% [950000/4848000] [196.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-06 22:10:13,129 - Train: 19.60% [950100/4848000] [196.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 22:10:46,050 - Train: 19.60% [950200/4848000] [196.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-06 22:10:48,692 - ==> Total time: 3 days, 22:44:47 Eta: 16 days, 4:39:13 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 22:11:21,116 - Train: 19.60% [950300/4848000] [196.0/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-06 22:11:54,052 - Train: 19.60% [950400/4848000] [196.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-06 22:12:26,984 - Train: 19.61% [950500/4848000] [196.1/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-06 22:12:59,891 - Train: 19.61% [950600/4848000] [196.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 22:13:32,852 - Train: 19.61% [950700/4848000] [196.1/1000.0] [batch_t 0.326 (0.330)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-06 22:14:05,739 - Train: 19.61% [950800/4848000] [196.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-06 22:14:38,642 - Train: 19.61% [950900/4848000] [196.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-06 22:15:11,566 - Train: 19.62% [951000/4848000] [196.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 22:15:44,497 - Train: 19.62% [951100/4848000] [196.2/1000.0] [batch_t 0.325 (0.329)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-06 22:16:17,441 - Train: 19.62% [951200/4848000] [196.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-06 22:16:50,345 - Train: 19.62% [951300/4848000] [196.2/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-06 22:17:23,189 - Train: 19.62% [951400/4848000] [196.2/1000.0] [batch_t 0.325 (0.328)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-06 22:17:56,094 - Train: 19.63% [951500/4848000] [196.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-06 22:18:29,984 - Train: 19.63% [951600/4848000] [196.3/1000.0] [batch_t 0.340 (0.339)] [data_t 0.003] [optim_t 0.337] [lr 0.005000] 2024-04-06 22:19:02,988 - Train: 19.63% [951700/4848000] [196.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 22:19:35,956 - Train: 19.63% [951800/4848000] [196.3/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-06 22:20:08,841 - Train: 19.63% [951900/4848000] [196.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 22:20:41,759 - Train: 19.64% [952000/4848000] [196.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 22:21:14,646 - Train: 19.64% [952100/4848000] [196.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 22:21:47,522 - Train: 19.64% [952200/4848000] [196.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 22:22:20,377 - Train: 19.64% [952300/4848000] [196.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 22:22:53,236 - Train: 19.65% [952400/4848000] [196.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-06 22:23:26,209 - Train: 19.65% [952500/4848000] [196.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 22:23:59,179 - Train: 19.65% [952600/4848000] [196.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 22:24:32,113 - Train: 19.65% [952700/4848000] [196.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-06 22:25:05,854 - Train: 19.65% [952800/4848000] [196.5/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 22:25:38,812 - Train: 19.66% [952900/4848000] [196.6/1000.0] [batch_t 0.336 (0.329)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-06 22:26:11,711 - Train: 19.66% [953000/4848000] [196.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-06 22:26:44,609 - Train: 19.66% [953100/4848000] [196.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-06 22:27:17,583 - Train: 19.66% [953200/4848000] [196.6/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 22:27:50,451 - Train: 19.66% [953300/4848000] [196.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 22:28:23,370 - Train: 19.67% [953400/4848000] [196.7/1000.0] [batch_t 0.334 (0.329)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-06 22:28:56,284 - Train: 19.67% [953500/4848000] [196.7/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 22:29:30,144 - Train: 19.67% [953600/4848000] [196.7/1000.0] [batch_t 0.331 (0.338)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 22:30:03,065 - Train: 19.67% [953700/4848000] [196.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 22:30:36,014 - Train: 19.67% [953800/4848000] [196.7/1000.0] [batch_t 0.334 (0.329)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-06 22:31:08,887 - Train: 19.68% [953900/4848000] [196.8/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 22:31:41,837 - Train: 19.68% [954000/4848000] [196.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-06 22:32:14,755 - Train: 19.68% [954100/4848000] [196.8/1000.0] [batch_t 0.326 (0.329)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-06 22:32:47,651 - Train: 19.68% [954200/4848000] [196.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-06 22:33:20,577 - Train: 19.68% [954300/4848000] [196.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-06 22:33:53,451 - Train: 19.69% [954400/4848000] [196.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 22:34:26,345 - Train: 19.69% [954500/4848000] [196.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 22:34:59,312 - Train: 19.69% [954600/4848000] [196.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 22:35:32,210 - Train: 19.69% [954700/4848000] [196.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-06 22:36:05,111 - Train: 19.69% [954800/4848000] [196.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-06 22:36:37,953 - Train: 19.70% [954900/4848000] [197.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-06 22:37:10,839 - Train: 19.70% [955000/4848000] [197.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 22:37:29,258 - ==> Total time: 3 days, 23:11:27 Eta: 16 days, 4:00:44 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 22:37:45,461 - Train: 19.70% [955100/4848000] [197.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-06 22:38:18,281 - Train: 19.70% [955200/4848000] [197.0/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 22:38:51,119 - Train: 19.71% [955300/4848000] [197.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 22:39:24,003 - Train: 19.71% [955400/4848000] [197.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 22:39:56,879 - Train: 19.71% [955500/4848000] [197.1/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 22:40:29,782 - Train: 19.71% [955600/4848000] [197.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 22:41:02,707 - Train: 19.71% [955700/4848000] [197.1/1000.0] [batch_t 0.335 (0.329)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-06 22:41:35,601 - Train: 19.72% [955800/4848000] [197.2/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 22:42:08,619 - Train: 19.72% [955900/4848000] [197.2/1000.0] [batch_t 0.325 (0.330)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-06 22:42:41,560 - Train: 19.72% [956000/4848000] [197.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-06 22:43:14,554 - Train: 19.72% [956100/4848000] [197.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 22:43:47,558 - Train: 19.72% [956200/4848000] [197.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-06 22:44:20,490 - Train: 19.73% [956300/4848000] [197.3/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 22:44:53,413 - Train: 19.73% [956400/4848000] [197.3/1000.0] [batch_t 0.342 (0.329)] [data_t 0.003] [optim_t 0.339] [lr 0.005000] 2024-04-06 22:45:26,388 - Train: 19.73% [956500/4848000] [197.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-06 22:45:59,325 - Train: 19.73% [956600/4848000] [197.3/1000.0] [batch_t 0.326 (0.329)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-06 22:46:32,249 - Train: 19.73% [956700/4848000] [197.3/1000.0] [batch_t 0.335 (0.329)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-06 22:47:05,148 - Train: 19.74% [956800/4848000] [197.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-06 22:47:38,051 - Train: 19.74% [956900/4848000] [197.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-06 22:48:11,022 - Train: 19.74% [957000/4848000] [197.4/1000.0] [batch_t 0.333 (0.330)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-06 22:48:43,880 - Train: 19.74% [957100/4848000] [197.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 22:49:16,830 - Train: 19.74% [957200/4848000] [197.4/1000.0] [batch_t 0.333 (0.329)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-06 22:49:49,879 - Train: 19.75% [957300/4848000] [197.5/1000.0] [batch_t 0.335 (0.330)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-06 22:50:22,763 - Train: 19.75% [957400/4848000] [197.5/1000.0] [batch_t 0.326 (0.329)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-06 22:50:55,652 - Train: 19.75% [957500/4848000] [197.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-06 22:51:28,528 - Train: 19.75% [957600/4848000] [197.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-06 22:52:01,378 - Train: 19.75% [957700/4848000] [197.5/1000.0] [batch_t 0.331 (0.328)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-06 22:52:34,250 - Train: 19.76% [957800/4848000] [197.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 22:53:07,182 - Train: 19.76% [957900/4848000] [197.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-06 22:53:40,146 - Train: 19.76% [958000/4848000] [197.6/1000.0] [batch_t 0.335 (0.330)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-06 22:54:13,075 - Train: 19.76% [958100/4848000] [197.6/1000.0] [batch_t 0.325 (0.329)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-06 22:54:46,038 - Train: 19.76% [958200/4848000] [197.6/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 22:55:18,920 - Train: 19.77% [958300/4848000] [197.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 22:55:51,809 - Train: 19.77% [958400/4848000] [197.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-06 22:56:24,719 - Train: 19.77% [958500/4848000] [197.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 22:56:57,623 - Train: 19.77% [958600/4848000] [197.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-06 22:57:30,524 - Train: 19.78% [958700/4848000] [197.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 22:58:03,598 - Train: 19.78% [958800/4848000] [197.8/1000.0] [batch_t 0.327 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-06 22:58:36,495 - Train: 19.78% [958900/4848000] [197.8/1000.0] [batch_t 0.333 (0.329)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-06 22:59:09,397 - Train: 19.78% [959000/4848000] [197.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 22:59:42,407 - Train: 19.78% [959100/4848000] [197.8/1000.0] [batch_t 0.334 (0.330)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-06 23:00:15,341 - Train: 19.79% [959200/4848000] [197.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-06 23:00:48,294 - Train: 19.79% [959300/4848000] [197.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 23:01:21,236 - Train: 19.79% [959400/4848000] [197.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 23:01:54,145 - Train: 19.79% [959500/4848000] [197.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-06 23:02:27,044 - Train: 19.79% [959600/4848000] [197.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-06 23:02:59,916 - Train: 19.80% [959700/4848000] [198.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-06 23:03:32,772 - Train: 19.80% [959800/4848000] [198.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-06 23:04:05,681 - Train: 19.80% [959900/4848000] [198.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 23:04:07,003 - ==> Total time: 3 days, 23:38:05 Eta: 16 days, 3:22:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 23:04:40,595 - Train: 19.80% [960000/4848000] [198.0/1000.0] [batch_t 0.327 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-06 23:05:13,647 - Train: 19.80% [960100/4848000] [198.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-06 23:05:46,511 - Train: 19.81% [960200/4848000] [198.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 23:06:19,419 - Train: 19.81% [960300/4848000] [198.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 23:06:52,313 - Train: 19.81% [960400/4848000] [198.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 23:07:25,224 - Train: 19.81% [960500/4848000] [198.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 23:07:58,125 - Train: 19.81% [960600/4848000] [198.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 23:08:31,013 - Train: 19.82% [960700/4848000] [198.2/1000.0] [batch_t 0.326 (0.329)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-06 23:09:03,912 - Train: 19.82% [960800/4848000] [198.2/1000.0] [batch_t 0.326 (0.329)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-06 23:09:36,880 - Train: 19.82% [960900/4848000] [198.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-06 23:10:09,801 - Train: 19.82% [961000/4848000] [198.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-06 23:10:42,714 - Train: 19.82% [961100/4848000] [198.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 23:11:15,804 - Train: 19.83% [961200/4848000] [198.3/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 23:11:48,754 - Train: 19.83% [961300/4848000] [198.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-06 23:12:21,716 - Train: 19.83% [961400/4848000] [198.3/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 23:12:54,663 - Train: 19.83% [961500/4848000] [198.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 23:13:27,519 - Train: 19.83% [961600/4848000] [198.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 23:14:00,385 - Train: 19.84% [961700/4848000] [198.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-06 23:14:33,272 - Train: 19.84% [961800/4848000] [198.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-06 23:15:06,153 - Train: 19.84% [961900/4848000] [198.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-06 23:15:39,095 - Train: 19.84% [962000/4848000] [198.4/1000.0] [batch_t 0.336 (0.329)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-06 23:16:11,985 - Train: 19.85% [962100/4848000] [198.5/1000.0] [batch_t 0.333 (0.329)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-06 23:16:44,874 - Train: 19.85% [962200/4848000] [198.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 23:17:17,811 - Train: 19.85% [962300/4848000] [198.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 23:17:50,710 - Train: 19.85% [962400/4848000] [198.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-06 23:18:23,611 - Train: 19.85% [962500/4848000] [198.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 23:18:56,492 - Train: 19.86% [962600/4848000] [198.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-06 23:19:29,349 - Train: 19.86% [962700/4848000] [198.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 23:20:02,287 - Train: 19.86% [962800/4848000] [198.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 23:20:35,231 - Train: 19.86% [962900/4848000] [198.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-06 23:21:08,276 - Train: 19.86% [963000/4848000] [198.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 23:21:41,137 - Train: 19.87% [963100/4848000] [198.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-06 23:22:14,054 - Train: 19.87% [963200/4848000] [198.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-06 23:22:46,983 - Train: 19.87% [963300/4848000] [198.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 23:23:19,866 - Train: 19.87% [963400/4848000] [198.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 23:23:52,775 - Train: 19.87% [963500/4848000] [198.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 23:24:25,730 - Train: 19.88% [963600/4848000] [198.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 23:24:58,639 - Train: 19.88% [963700/4848000] [198.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 23:25:31,600 - Train: 19.88% [963800/4848000] [198.8/1000.0] [batch_t 0.334 (0.330)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-06 23:26:04,601 - Train: 19.88% [963900/4848000] [198.8/1000.0] [batch_t 0.336 (0.330)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-06 23:26:38,630 - Train: 19.88% [964000/4848000] [198.8/1000.0] [batch_t 0.329 (0.340)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 23:27:12,856 - Train: 19.89% [964100/4848000] [198.9/1000.0] [batch_t 0.329 (0.342)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 23:27:45,753 - Train: 19.89% [964200/4848000] [198.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-06 23:28:18,879 - Train: 19.89% [964300/4848000] [198.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 23:28:51,942 - Train: 19.89% [964400/4848000] [198.9/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-06 23:29:24,839 - Train: 19.89% [964500/4848000] [198.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-06 23:29:57,749 - Train: 19.90% [964600/4848000] [199.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-06 23:30:30,639 - Train: 19.90% [964700/4848000] [199.0/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 23:30:47,751 - ==> Total time: 4 days, 0:04:46 Eta: 16 days, 2:43:55 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-06 23:31:05,627 - Train: 19.90% [964800/4848000] [199.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-06 23:31:38,529 - Train: 19.90% [964900/4848000] [199.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 23:32:11,431 - Train: 19.91% [965000/4848000] [199.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-06 23:32:44,393 - Train: 19.91% [965100/4848000] [199.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 23:33:17,329 - Train: 19.91% [965200/4848000] [199.1/1000.0] [batch_t 0.335 (0.329)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-06 23:33:50,154 - Train: 19.91% [965300/4848000] [199.1/1000.0] [batch_t 0.327 (0.328)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-06 23:34:23,007 - Train: 19.91% [965400/4848000] [199.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-06 23:34:55,946 - Train: 19.92% [965500/4848000] [199.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-06 23:35:28,830 - Train: 19.92% [965600/4848000] [199.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-06 23:36:01,836 - Train: 19.92% [965700/4848000] [199.2/1000.0] [batch_t 0.324 (0.330)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-06 23:36:34,726 - Train: 19.92% [965800/4848000] [199.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-06 23:37:07,605 - Train: 19.92% [965900/4848000] [199.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-06 23:37:40,480 - Train: 19.93% [966000/4848000] [199.3/1000.0] [batch_t 0.325 (0.329)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-06 23:38:13,443 - Train: 19.93% [966100/4848000] [199.3/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-06 23:38:46,388 - Train: 19.93% [966200/4848000] [199.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 23:39:19,294 - Train: 19.93% [966300/4848000] [199.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-06 23:39:52,178 - Train: 19.93% [966400/4848000] [199.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-06 23:40:25,102 - Train: 19.94% [966500/4848000] [199.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-06 23:40:58,007 - Train: 19.94% [966600/4848000] [199.4/1000.0] [batch_t 0.325 (0.329)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-06 23:41:30,897 - Train: 19.94% [966700/4848000] [199.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-06 23:42:03,839 - Train: 19.94% [966800/4848000] [199.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 23:42:36,700 - Train: 19.94% [966900/4848000] [199.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-06 23:43:09,654 - Train: 19.95% [967000/4848000] [199.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-06 23:43:42,648 - Train: 19.95% [967100/4848000] [199.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-06 23:44:15,524 - Train: 19.95% [967200/4848000] [199.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 23:44:48,424 - Train: 19.95% [967300/4848000] [199.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-06 23:45:21,280 - Train: 19.95% [967400/4848000] [199.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-06 23:45:54,131 - Train: 19.96% [967500/4848000] [199.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-06 23:46:27,015 - Train: 19.96% [967600/4848000] [199.6/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-06 23:46:59,907 - Train: 19.96% [967700/4848000] [199.6/1000.0] [batch_t 0.334 (0.329)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-06 23:47:32,824 - Train: 19.96% [967800/4848000] [199.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 23:48:05,724 - Train: 19.96% [967900/4848000] [199.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 23:48:38,605 - Train: 19.97% [968000/4848000] [199.7/1000.0] [batch_t 0.333 (0.329)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-06 23:49:11,489 - Train: 19.97% [968100/4848000] [199.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-06 23:49:44,357 - Train: 19.97% [968200/4848000] [199.7/1000.0] [batch_t 0.333 (0.329)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-06 23:50:17,263 - Train: 19.97% [968300/4848000] [199.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 23:50:50,143 - Train: 19.98% [968400/4848000] [199.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-06 23:51:23,060 - Train: 19.98% [968500/4848000] [199.8/1000.0] [batch_t 0.426 (0.329)] [data_t 0.003] [optim_t 0.423] [lr 0.005000] 2024-04-06 23:51:55,905 - Train: 19.98% [968600/4848000] [199.8/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-06 23:52:28,825 - Train: 19.98% [968700/4848000] [199.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 23:53:01,708 - Train: 19.98% [968800/4848000] [199.8/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-06 23:53:34,625 - Train: 19.99% [968900/4848000] [199.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-06 23:54:07,491 - Train: 19.99% [969000/4848000] [199.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-06 23:54:40,378 - Train: 19.99% [969100/4848000] [199.9/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-06 23:55:13,256 - Train: 19.99% [969200/4848000] [199.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-06 23:55:46,096 - Train: 19.99% [969300/4848000] [199.9/1000.0] [batch_t 0.331 (0.328)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-06 23:56:18,969 - Train: 20.00% [969400/4848000] [200.0/1000.0] [batch_t 0.334 (0.329)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-06 23:56:51,843 - Train: 20.00% [969500/4848000] [200.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-06 23:57:24,724 - Train: 20.00% [969600/4848000] [200.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-06 23:57:33,546 - Test: 16.13% [50/310] [batch_t 0.154 (0.159)] 2024-04-06 23:57:41,194 - Test: 32.26% [100/310] [batch_t 0.153 (0.156)] 2024-04-06 23:57:48,849 - Test: 48.39% [150/310] [batch_t 0.155 (0.155)] 2024-04-06 23:57:56,450 - Test: 64.52% [200/310] [batch_t 0.150 (0.154)] 2024-04-06 23:58:04,987 - Test: 80.65% [250/310] [batch_t 0.352 (0.158)] 2024-04-06 23:58:12,643 - Test: 96.77% [300/310] [batch_t 0.157 (0.157)] 2024-04-06 23:58:14,105 - Test: 100.00% [310/310] [batch_t 0.084 (0.156)] 2024-04-07 00:24:28,709 - ==> Metric Time for coco : 0.004 (mAUROC_sp_max) 0.001 (mAP_sp_max) 0.001 (mF1_max_sp_max) 359.721 (mAUROC_px) 290.995 (mAP_px) 31.290 (mF1_max_px) 813.175 (mAUPRO_px) 11.887 (mF1_px_0.2_0.8_0.1) 11.898 (mAcc_px_0.2_0.8_0.1) 11.882 (mIoU_px_0.2_0.8_0.1) 34.452 (mIoU_max_px) 2024-04-07 00:24:29,169 - | Name | mAUROC_sp_max | mAUROC_sp_max (Max) | mAP_sp_max | mAP_sp_max (Max) | mF1_max_sp_max | mF1_max_sp_max (Max) | mAUROC_px | mAUROC_px (Max) | mAP_px | mAP_px (Max) | mF1_max_px | mF1_max_px (Max) | mAUPRO_px | mAUPRO_px (Max) | mF1_px_0.2_0.8_0.1 | mF1_px_0.2_0.8_0.1 (Max) | mAcc_px_0.2_0.8_0.1 | mAcc_px_0.2_0.8_0.1 (Max) | mIoU_px_0.2_0.8_0.1 | mIoU_px_0.2_0.8_0.1 (Max) | mIoU_max_px | mIoU_max_px (Max) | |:------:|:---------------:|:---------------------:|:------------:|:------------------:|:----------------:|:----------------------:|:-----------:|:------------------:|:--------:|:-----------------:|:------------:|:------------------:|:-----------:|:------------------:|:--------------------:|:--------------------------:|:---------------------:|:---------------------------:|:---------------------:|:---------------------------:|:-------------:|:-------------------:| | coco | 55.759 | 58.404 (50 epoch) | 37.566 | 40.479 (50 epoch) | 51.727 | 51.858 (50 epoch) | 59.646 | 59.646 (200 epoch) | 8.668 | 8.714 (100 epoch) | 14.519 | 14.519 (200 epoch) | 29.968 | 34.736 (50 epoch) | 8.693 | 9.008 (50 epoch) | 38.750 | 38.999 (50 epoch) | 4.644 | 4.810 (50 epoch) | 7.828 | 7.828 (200 epoch) | 2024-04-07 00:24:29,784 - ==> Total time: 4 days, 0:58:28 Eta: 16 days, 3:53:53 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 00:25:50,332 - Train: 20.00% [969700/4848000] [200.0/1000.0] [batch_t 0.782 (0.780)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-07 00:27:08,345 - Train: 20.00% [969800/4848000] [200.0/1000.0] [batch_t 0.782 (0.780)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-07 00:28:26,237 - Train: 20.01% [969900/4848000] [200.1/1000.0] [batch_t 0.784 (0.779)] [data_t 0.003] [optim_t 0.780] [lr 0.005000] 2024-04-07 00:29:44,136 - Train: 20.01% [970000/4848000] [200.1/1000.0] [batch_t 0.781 (0.779)] [data_t 0.003] [optim_t 0.778] [lr 0.005000] 2024-04-07 00:31:01,851 - Train: 20.01% [970100/4848000] [200.1/1000.0] [batch_t 0.773 (0.777)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-07 00:32:06,343 - Train: 20.01% [970200/4848000] [200.1/1000.0] [batch_t 0.326 (0.645)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 00:32:39,377 - Train: 20.01% [970300/4848000] [200.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 00:33:12,474 - Train: 20.02% [970400/4848000] [200.2/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 00:33:45,379 - Train: 20.02% [970500/4848000] [200.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 00:34:18,465 - Train: 20.02% [970600/4848000] [200.2/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 00:34:51,521 - Train: 20.02% [970700/4848000] [200.2/1000.0] [batch_t 0.336 (0.330)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-07 00:35:24,486 - Train: 20.02% [970800/4848000] [200.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 00:35:57,512 - Train: 20.03% [970900/4848000] [200.3/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 00:36:30,489 - Train: 20.03% [971000/4848000] [200.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 00:37:03,537 - Train: 20.03% [971100/4848000] [200.3/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 00:37:36,557 - Train: 20.03% [971200/4848000] [200.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 00:38:09,784 - Train: 20.04% [971300/4848000] [200.4/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 00:38:42,735 - Train: 20.04% [971400/4848000] [200.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 00:39:15,735 - Train: 20.04% [971500/4848000] [200.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 00:39:48,795 - Train: 20.04% [971600/4848000] [200.4/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 00:40:21,815 - Train: 20.04% [971700/4848000] [200.4/1000.0] [batch_t 0.335 (0.330)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-07 00:40:54,745 - Train: 20.05% [971800/4848000] [200.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 00:41:27,795 - Train: 20.05% [971900/4848000] [200.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 00:42:00,779 - Train: 20.05% [972000/4848000] [200.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 00:42:33,804 - Train: 20.05% [972100/4848000] [200.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 00:43:06,795 - Train: 20.05% [972200/4848000] [200.5/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 00:43:39,882 - Train: 20.06% [972300/4848000] [200.6/1000.0] [batch_t 0.337 (0.331)] [data_t 0.003] [optim_t 0.335] [lr 0.005000] 2024-04-07 00:44:13,003 - Train: 20.06% [972400/4848000] [200.6/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 00:44:46,038 - Train: 20.06% [972500/4848000] [200.6/1000.0] [batch_t 0.323 (0.330)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 00:45:19,060 - Train: 20.06% [972600/4848000] [200.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 00:45:52,212 - Train: 20.06% [972700/4848000] [200.6/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 00:46:25,240 - Train: 20.07% [972800/4848000] [200.7/1000.0] [batch_t 0.335 (0.330)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-07 00:46:58,239 - Train: 20.07% [972900/4848000] [200.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 00:47:31,203 - Train: 20.07% [973000/4848000] [200.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 00:48:04,200 - Train: 20.07% [973100/4848000] [200.7/1000.0] [batch_t 0.338 (0.330)] [data_t 0.003] [optim_t 0.335] [lr 0.005000] 2024-04-07 00:48:37,182 - Train: 20.07% [973200/4848000] [200.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 00:49:10,219 - Train: 20.08% [973300/4848000] [200.8/1000.0] [batch_t 0.336 (0.330)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-07 00:49:43,302 - Train: 20.08% [973400/4848000] [200.8/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 00:50:16,291 - Train: 20.08% [973500/4848000] [200.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 00:50:49,304 - Train: 20.08% [973600/4848000] [200.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 00:51:22,346 - Train: 20.08% [973700/4848000] [200.8/1000.0] [batch_t 0.335 (0.330)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-07 00:51:55,419 - Train: 20.09% [973800/4848000] [200.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 00:52:28,414 - Train: 20.09% [973900/4848000] [200.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 00:53:01,372 - Train: 20.09% [974000/4848000] [200.9/1000.0] [batch_t 0.335 (0.329)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-07 00:53:34,510 - Train: 20.09% [974100/4848000] [200.9/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 00:54:07,451 - Train: 20.09% [974200/4848000] [200.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 00:54:40,444 - Train: 20.10% [974300/4848000] [201.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 00:55:13,486 - Train: 20.10% [974400/4848000] [201.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 00:55:29,335 - ==> Total time: 4 days, 1:29:27 Eta: 16 days, 3:32:20 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 00:55:48,633 - Train: 20.10% [974500/4848000] [201.0/1000.0] [batch_t 0.333 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 00:56:21,595 - Train: 20.10% [974600/4848000] [201.0/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 00:56:54,551 - Train: 20.11% [974700/4848000] [201.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 00:57:27,579 - Train: 20.11% [974800/4848000] [201.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 00:58:00,521 - Train: 20.11% [974900/4848000] [201.1/1000.0] [batch_t 0.324 (0.329)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-07 00:58:33,487 - Train: 20.11% [975000/4848000] [201.1/1000.0] [batch_t 0.334 (0.330)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 00:59:06,572 - Train: 20.11% [975100/4848000] [201.1/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 00:59:39,575 - Train: 20.12% [975200/4848000] [201.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 01:00:12,498 - Train: 20.12% [975300/4848000] [201.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 01:00:45,445 - Train: 20.12% [975400/4848000] [201.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 01:01:18,599 - Train: 20.12% [975500/4848000] [201.2/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 01:01:51,564 - Train: 20.12% [975600/4848000] [201.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 01:02:24,622 - Train: 20.13% [975700/4848000] [201.3/1000.0] [batch_t 0.335 (0.330)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-07 01:02:57,719 - Train: 20.13% [975800/4848000] [201.3/1000.0] [batch_t 0.337 (0.331)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-07 01:03:30,609 - Train: 20.13% [975900/4848000] [201.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 01:04:03,645 - Train: 20.13% [976000/4848000] [201.3/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 01:04:36,644 - Train: 20.13% [976100/4848000] [201.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 01:05:09,650 - Train: 20.14% [976200/4848000] [201.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 01:05:42,570 - Train: 20.14% [976300/4848000] [201.4/1000.0] [batch_t 0.325 (0.329)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-07 01:06:15,533 - Train: 20.14% [976400/4848000] [201.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 01:06:48,442 - Train: 20.14% [976500/4848000] [201.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 01:07:21,363 - Train: 20.14% [976600/4848000] [201.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 01:07:54,321 - Train: 20.15% [976700/4848000] [201.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 01:08:27,413 - Train: 20.15% [976800/4848000] [201.5/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 01:09:00,465 - Train: 20.15% [976900/4848000] [201.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 01:09:33,463 - Train: 20.15% [977000/4848000] [201.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 01:10:06,370 - Train: 20.15% [977100/4848000] [201.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 01:10:39,313 - Train: 20.16% [977200/4848000] [201.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 01:11:12,275 - Train: 20.16% [977300/4848000] [201.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 01:11:45,162 - Train: 20.16% [977400/4848000] [201.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 01:12:18,127 - Train: 20.16% [977500/4848000] [201.6/1000.0] [batch_t 0.334 (0.330)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 01:12:51,063 - Train: 20.17% [977600/4848000] [201.7/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 01:13:24,056 - Train: 20.17% [977700/4848000] [201.7/1000.0] [batch_t 0.341 (0.330)] [data_t 0.003] [optim_t 0.338] [lr 0.005000] 2024-04-07 01:13:57,054 - Train: 20.17% [977800/4848000] [201.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 01:14:29,995 - Train: 20.17% [977900/4848000] [201.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 01:15:02,963 - Train: 20.17% [978000/4848000] [201.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 01:15:35,891 - Train: 20.18% [978100/4848000] [201.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 01:16:08,851 - Train: 20.18% [978200/4848000] [201.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 01:16:41,936 - Train: 20.18% [978300/4848000] [201.8/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 01:17:14,866 - Train: 20.18% [978400/4848000] [201.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 01:17:47,861 - Train: 20.18% [978500/4848000] [201.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 01:18:20,828 - Train: 20.19% [978600/4848000] [201.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 01:18:53,799 - Train: 20.19% [978700/4848000] [201.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 01:19:26,715 - Train: 20.19% [978800/4848000] [201.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 01:19:59,709 - Train: 20.19% [978900/4848000] [201.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 01:20:32,624 - Train: 20.19% [979000/4848000] [201.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 01:21:05,660 - Train: 20.20% [979100/4848000] [202.0/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 01:21:38,654 - Train: 20.20% [979200/4848000] [202.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 01:22:10,342 - ==> Total time: 4 days, 1:56:08 Eta: 16 days, 2:53:41 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 01:22:13,537 - Train: 20.20% [979300/4848000] [202.0/1000.0] [batch_t 0.326 (0.332)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 01:22:46,478 - Train: 20.20% [979400/4848000] [202.0/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 01:23:19,397 - Train: 20.20% [979500/4848000] [202.0/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 01:23:52,502 - Train: 20.21% [979600/4848000] [202.1/1000.0] [batch_t 0.335 (0.331)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-07 01:24:25,452 - Train: 20.21% [979700/4848000] [202.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 01:24:58,414 - Train: 20.21% [979800/4848000] [202.1/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 01:25:31,346 - Train: 20.21% [979900/4848000] [202.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 01:26:04,305 - Train: 20.21% [980000/4848000] [202.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 01:26:37,255 - Train: 20.22% [980100/4848000] [202.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 01:27:10,263 - Train: 20.22% [980200/4848000] [202.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 01:27:43,195 - Train: 20.22% [980300/4848000] [202.2/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 01:28:16,172 - Train: 20.22% [980400/4848000] [202.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 01:28:49,227 - Train: 20.22% [980500/4848000] [202.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 01:29:22,270 - Train: 20.23% [980600/4848000] [202.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 01:29:55,223 - Train: 20.23% [980700/4848000] [202.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 01:30:28,237 - Train: 20.23% [980800/4848000] [202.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 01:31:01,223 - Train: 20.23% [980900/4848000] [202.3/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 01:31:34,273 - Train: 20.24% [981000/4848000] [202.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 01:32:07,587 - Train: 20.24% [981100/4848000] [202.4/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 01:32:40,518 - Train: 20.24% [981200/4848000] [202.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 01:33:13,389 - Train: 20.24% [981300/4848000] [202.4/1000.0] [batch_t 0.335 (0.329)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-07 01:33:46,316 - Train: 20.24% [981400/4848000] [202.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 01:34:19,239 - Train: 20.25% [981500/4848000] [202.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 01:34:52,166 - Train: 20.25% [981600/4848000] [202.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 01:35:25,101 - Train: 20.25% [981700/4848000] [202.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 01:35:58,054 - Train: 20.25% [981800/4848000] [202.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 01:36:31,012 - Train: 20.25% [981900/4848000] [202.5/1000.0] [batch_t 0.335 (0.329)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-07 01:37:04,019 - Train: 20.26% [982000/4848000] [202.6/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 01:37:36,991 - Train: 20.26% [982100/4848000] [202.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 01:38:09,939 - Train: 20.26% [982200/4848000] [202.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 01:38:42,862 - Train: 20.26% [982300/4848000] [202.6/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 01:39:15,794 - Train: 20.26% [982400/4848000] [202.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 01:39:48,804 - Train: 20.27% [982500/4848000] [202.7/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 01:40:21,738 - Train: 20.27% [982600/4848000] [202.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 01:40:54,648 - Train: 20.27% [982700/4848000] [202.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 01:41:27,647 - Train: 20.27% [982800/4848000] [202.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 01:42:00,541 - Train: 20.27% [982900/4848000] [202.7/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 01:42:33,487 - Train: 20.28% [983000/4848000] [202.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 01:43:06,372 - Train: 20.28% [983100/4848000] [202.8/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 01:43:39,272 - Train: 20.28% [983200/4848000] [202.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 01:44:12,183 - Train: 20.28% [983300/4848000] [202.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 01:44:45,135 - Train: 20.28% [983400/4848000] [202.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 01:45:18,085 - Train: 20.29% [983500/4848000] [202.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 01:45:51,022 - Train: 20.29% [983600/4848000] [202.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 01:46:23,964 - Train: 20.29% [983700/4848000] [202.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 01:46:56,889 - Train: 20.29% [983800/4848000] [202.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 01:47:29,959 - Train: 20.29% [983900/4848000] [202.9/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 01:48:02,898 - Train: 20.30% [984000/4848000] [203.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 01:48:35,953 - Train: 20.30% [984100/4848000] [203.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 01:48:50,442 - ==> Total time: 4 days, 2:22:48 Eta: 16 days, 2:15:05 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 01:49:10,885 - Train: 20.30% [984200/4848000] [203.0/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 01:49:43,837 - Train: 20.30% [984300/4848000] [203.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 01:50:16,761 - Train: 20.31% [984400/4848000] [203.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 01:50:49,717 - Train: 20.31% [984500/4848000] [203.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 01:51:22,632 - Train: 20.31% [984600/4848000] [203.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 01:51:55,610 - Train: 20.31% [984700/4848000] [203.1/1000.0] [batch_t 0.326 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 01:52:28,668 - Train: 20.31% [984800/4848000] [203.1/1000.0] [batch_t 0.336 (0.330)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-07 01:53:01,555 - Train: 20.32% [984900/4848000] [203.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 01:53:34,485 - Train: 20.32% [985000/4848000] [203.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 01:54:07,420 - Train: 20.32% [985100/4848000] [203.2/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 01:54:40,504 - Train: 20.32% [985200/4848000] [203.2/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 01:55:13,487 - Train: 20.32% [985300/4848000] [203.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 01:55:46,427 - Train: 20.33% [985400/4848000] [203.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 01:56:19,331 - Train: 20.33% [985500/4848000] [203.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 01:56:52,278 - Train: 20.33% [985600/4848000] [203.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 01:57:25,196 - Train: 20.33% [985700/4848000] [203.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 01:57:58,123 - Train: 20.33% [985800/4848000] [203.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 01:58:31,059 - Train: 20.34% [985900/4848000] [203.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 01:59:04,086 - Train: 20.34% [986000/4848000] [203.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 01:59:37,121 - Train: 20.34% [986100/4848000] [203.4/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 02:00:10,216 - Train: 20.34% [986200/4848000] [203.4/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-07 02:00:43,224 - Train: 20.34% [986300/4848000] [203.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 02:01:16,261 - Train: 20.35% [986400/4848000] [203.5/1000.0] [batch_t 0.336 (0.330)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-07 02:01:49,296 - Train: 20.35% [986500/4848000] [203.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 02:02:22,256 - Train: 20.35% [986600/4848000] [203.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 02:02:55,405 - Train: 20.35% [986700/4848000] [203.5/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 02:03:28,375 - Train: 20.35% [986800/4848000] [203.5/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 02:04:01,352 - Train: 20.36% [986900/4848000] [203.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 02:04:34,375 - Train: 20.36% [987000/4848000] [203.6/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 02:05:07,380 - Train: 20.36% [987100/4848000] [203.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 02:05:40,379 - Train: 20.36% [987200/4848000] [203.6/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 02:06:13,401 - Train: 20.37% [987300/4848000] [203.7/1000.0] [batch_t 0.337 (0.330)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-07 02:06:46,483 - Train: 20.37% [987400/4848000] [203.7/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 02:07:19,480 - Train: 20.37% [987500/4848000] [203.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 02:07:52,460 - Train: 20.37% [987600/4848000] [203.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 02:08:25,419 - Train: 20.37% [987700/4848000] [203.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 02:08:58,475 - Train: 20.38% [987800/4848000] [203.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 02:09:31,465 - Train: 20.38% [987900/4848000] [203.8/1000.0] [batch_t 0.337 (0.330)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-07 02:10:04,426 - Train: 20.38% [988000/4848000] [203.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 02:10:37,587 - Train: 20.38% [988100/4848000] [203.8/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 02:11:10,624 - Train: 20.38% [988200/4848000] [203.8/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 02:11:43,683 - Train: 20.39% [988300/4848000] [203.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 02:12:16,744 - Train: 20.39% [988400/4848000] [203.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 02:12:49,735 - Train: 20.39% [988500/4848000] [203.9/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 02:13:22,723 - Train: 20.39% [988600/4848000] [203.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 02:13:55,749 - Train: 20.39% [988700/4848000] [203.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 02:14:28,736 - Train: 20.40% [988800/4848000] [204.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 02:15:01,729 - Train: 20.40% [988900/4848000] [204.0/1000.0] [batch_t 0.334 (0.330)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-07 02:15:32,108 - ==> Total time: 4 days, 2:49:30 Eta: 16 days, 1:36:42 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 02:15:36,751 - Train: 20.40% [989000/4848000] [204.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 02:16:09,717 - Train: 20.40% [989100/4848000] [204.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 02:16:42,730 - Train: 20.40% [989200/4848000] [204.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 02:17:15,680 - Train: 20.41% [989300/4848000] [204.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 02:17:48,776 - Train: 20.41% [989400/4848000] [204.1/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 02:18:21,872 - Train: 20.41% [989500/4848000] [204.1/1000.0] [batch_t 0.336 (0.331)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-07 02:18:54,874 - Train: 20.41% [989600/4848000] [204.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 02:19:27,820 - Train: 20.41% [989700/4848000] [204.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 02:20:00,922 - Train: 20.42% [989800/4848000] [204.2/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 02:20:33,978 - Train: 20.42% [989900/4848000] [204.2/1000.0] [batch_t 0.336 (0.330)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-07 02:21:06,989 - Train: 20.42% [990000/4848000] [204.2/1000.0] [batch_t 0.335 (0.330)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-07 02:21:39,928 - Train: 20.42% [990100/4848000] [204.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 02:22:12,881 - Train: 20.42% [990200/4848000] [204.2/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 02:22:45,921 - Train: 20.43% [990300/4848000] [204.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 02:23:18,826 - Train: 20.43% [990400/4848000] [204.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 02:23:51,812 - Train: 20.43% [990500/4848000] [204.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 02:24:24,780 - Train: 20.43% [990600/4848000] [204.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 02:24:57,681 - Train: 20.44% [990700/4848000] [204.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 02:25:30,720 - Train: 20.44% [990800/4848000] [204.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 02:26:03,664 - Train: 20.44% [990900/4848000] [204.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 02:26:36,647 - Train: 20.44% [991000/4848000] [204.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 02:27:09,644 - Train: 20.44% [991100/4848000] [204.4/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 02:27:42,665 - Train: 20.45% [991200/4848000] [204.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 02:28:15,670 - Train: 20.45% [991300/4848000] [204.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 02:28:48,629 - Train: 20.45% [991400/4848000] [204.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 02:29:21,597 - Train: 20.45% [991500/4848000] [204.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 02:29:54,612 - Train: 20.45% [991600/4848000] [204.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 02:30:27,587 - Train: 20.46% [991700/4848000] [204.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 02:31:00,586 - Train: 20.46% [991800/4848000] [204.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 02:31:33,542 - Train: 20.46% [991900/4848000] [204.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 02:32:06,512 - Train: 20.46% [992000/4848000] [204.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 02:32:39,550 - Train: 20.46% [992100/4848000] [204.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 02:33:12,492 - Train: 20.47% [992200/4848000] [204.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 02:33:45,511 - Train: 20.47% [992300/4848000] [204.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 02:34:18,500 - Train: 20.47% [992400/4848000] [204.7/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 02:34:51,557 - Train: 20.47% [992500/4848000] [204.7/1000.0] [batch_t 0.336 (0.330)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-07 02:35:24,516 - Train: 20.47% [992600/4848000] [204.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 02:35:57,558 - Train: 20.48% [992700/4848000] [204.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 02:36:30,579 - Train: 20.48% [992800/4848000] [204.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 02:37:03,514 - Train: 20.48% [992900/4848000] [204.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 02:37:36,420 - Train: 20.48% [993000/4848000] [204.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 02:38:09,450 - Train: 20.48% [993100/4848000] [204.8/1000.0] [batch_t 0.335 (0.330)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-07 02:38:42,440 - Train: 20.49% [993200/4848000] [204.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 02:39:15,420 - Train: 20.49% [993300/4848000] [204.9/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 02:39:48,334 - Train: 20.49% [993400/4848000] [204.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 02:40:21,318 - Train: 20.49% [993500/4848000] [204.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 02:40:54,292 - Train: 20.50% [993600/4848000] [205.0/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 02:41:27,411 - Train: 20.50% [993700/4848000] [205.0/1000.0] [batch_t 0.333 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 02:42:00,403 - Train: 20.50% [993800/4848000] [205.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 02:42:13,590 - ==> Total time: 4 days, 3:16:12 Eta: 16 days, 0:58:26 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 02:42:35,414 - Train: 20.50% [993900/4848000] [205.0/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 02:43:08,413 - Train: 20.50% [994000/4848000] [205.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 02:43:41,412 - Train: 20.51% [994100/4848000] [205.1/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 02:44:14,429 - Train: 20.51% [994200/4848000] [205.1/1000.0] [batch_t 0.333 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 02:44:47,380 - Train: 20.51% [994300/4848000] [205.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 02:45:20,353 - Train: 20.51% [994400/4848000] [205.1/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 02:45:53,359 - Train: 20.51% [994500/4848000] [205.1/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 02:46:26,367 - Train: 20.52% [994600/4848000] [205.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 02:46:59,335 - Train: 20.52% [994700/4848000] [205.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 02:47:32,299 - Train: 20.52% [994800/4848000] [205.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 02:48:05,301 - Train: 20.52% [994900/4848000] [205.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 02:48:38,409 - Train: 20.52% [995000/4848000] [205.2/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 02:49:11,377 - Train: 20.53% [995100/4848000] [205.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 02:49:44,350 - Train: 20.53% [995200/4848000] [205.3/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 02:50:17,419 - Train: 20.53% [995300/4848000] [205.3/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 02:50:50,412 - Train: 20.53% [995400/4848000] [205.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 02:51:23,401 - Train: 20.53% [995500/4848000] [205.3/1000.0] [batch_t 0.345 (0.330)] [data_t 0.003] [optim_t 0.342] [lr 0.005000] 2024-04-07 02:51:56,403 - Train: 20.54% [995600/4848000] [205.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 02:52:29,342 - Train: 20.54% [995700/4848000] [205.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 02:53:02,284 - Train: 20.54% [995800/4848000] [205.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 02:53:35,236 - Train: 20.54% [995900/4848000] [205.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 02:54:08,207 - Train: 20.54% [996000/4848000] [205.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 02:54:41,176 - Train: 20.55% [996100/4848000] [205.5/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 02:55:14,158 - Train: 20.55% [996200/4848000] [205.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 02:55:47,129 - Train: 20.55% [996300/4848000] [205.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 02:56:20,281 - Train: 20.55% [996400/4848000] [205.5/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 02:56:53,219 - Train: 20.55% [996500/4848000] [205.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 02:57:26,157 - Train: 20.56% [996600/4848000] [205.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 02:57:59,175 - Train: 20.56% [996700/4848000] [205.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 02:58:32,101 - Train: 20.56% [996800/4848000] [205.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 02:59:05,063 - Train: 20.56% [996900/4848000] [205.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 02:59:38,022 - Train: 20.57% [997000/4848000] [205.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 03:00:10,986 - Train: 20.57% [997100/4848000] [205.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 03:00:44,015 - Train: 20.57% [997200/4848000] [205.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 03:01:16,978 - Train: 20.57% [997300/4848000] [205.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 03:01:50,048 - Train: 20.57% [997400/4848000] [205.7/1000.0] [batch_t 0.342 (0.331)] [data_t 0.003] [optim_t 0.339] [lr 0.005000] 2024-04-07 03:02:23,022 - Train: 20.58% [997500/4848000] [205.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 03:02:55,986 - Train: 20.58% [997600/4848000] [205.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 03:03:29,017 - Train: 20.58% [997700/4848000] [205.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 03:04:02,068 - Train: 20.58% [997800/4848000] [205.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 03:04:35,093 - Train: 20.58% [997900/4848000] [205.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 03:05:08,078 - Train: 20.59% [998000/4848000] [205.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 03:05:41,106 - Train: 20.59% [998100/4848000] [205.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 03:06:14,109 - Train: 20.59% [998200/4848000] [205.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 03:06:47,057 - Train: 20.59% [998300/4848000] [205.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 03:07:20,000 - Train: 20.59% [998400/4848000] [205.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 03:07:52,957 - Train: 20.60% [998500/4848000] [206.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 03:08:25,923 - Train: 20.60% [998600/4848000] [206.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 03:08:54,899 - ==> Total time: 4 days, 3:42:53 Eta: 16 days, 0:20:15 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 03:09:01,013 - Train: 20.60% [998700/4848000] [206.0/1000.0] [batch_t 0.330 (0.328)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 03:09:33,945 - Train: 20.60% [998800/4848000] [206.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 03:10:06,920 - Train: 20.60% [998900/4848000] [206.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 03:10:39,874 - Train: 20.61% [999000/4848000] [206.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 03:11:12,842 - Train: 20.61% [999100/4848000] [206.1/1000.0] [batch_t 0.333 (0.330)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 03:11:46,008 - Train: 20.61% [999200/4848000] [206.1/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 03:12:19,004 - Train: 20.61% [999300/4848000] [206.1/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 03:12:51,988 - Train: 20.61% [999400/4848000] [206.1/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 03:13:24,911 - Train: 20.62% [999500/4848000] [206.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 03:13:57,910 - Train: 20.62% [999600/4848000] [206.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 03:14:30,834 - Train: 20.62% [999700/4848000] [206.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 03:15:03,833 - Train: 20.62% [999800/4848000] [206.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 03:15:36,771 - Train: 20.62% [999900/4848000] [206.2/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 03:16:09,778 - Train: 20.63% [1000000/4848000] [206.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 03:16:42,739 - Train: 20.63% [1000100/4848000] [206.3/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 03:17:15,664 - Train: 20.63% [1000200/4848000] [206.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 03:17:48,633 - Train: 20.63% [1000300/4848000] [206.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 03:18:21,604 - Train: 20.64% [1000400/4848000] [206.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 03:18:54,554 - Train: 20.64% [1000500/4848000] [206.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 03:19:27,733 - Train: 20.64% [1000600/4848000] [206.4/1000.0] [batch_t 0.332 (0.332)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 03:20:00,686 - Train: 20.64% [1000700/4848000] [206.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 03:20:33,598 - Train: 20.64% [1000800/4848000] [206.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 03:21:06,621 - Train: 20.65% [1000900/4848000] [206.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 03:21:39,535 - Train: 20.65% [1001000/4848000] [206.5/1000.0] [batch_t 0.333 (0.329)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 03:22:12,514 - Train: 20.65% [1001100/4848000] [206.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 03:22:45,450 - Train: 20.65% [1001200/4848000] [206.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 03:23:18,371 - Train: 20.65% [1001300/4848000] [206.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 03:23:51,316 - Train: 20.66% [1001400/4848000] [206.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 03:24:24,282 - Train: 20.66% [1001500/4848000] [206.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 03:24:57,232 - Train: 20.66% [1001600/4848000] [206.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 03:25:30,197 - Train: 20.66% [1001700/4848000] [206.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 03:26:03,143 - Train: 20.66% [1001800/4848000] [206.6/1000.0] [batch_t 0.338 (0.329)] [data_t 0.003] [optim_t 0.335] [lr 0.005000] 2024-04-07 03:26:36,129 - Train: 20.67% [1001900/4848000] [206.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 03:27:09,321 - Train: 20.67% [1002000/4848000] [206.7/1000.0] [batch_t 0.330 (0.332)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 03:27:42,299 - Train: 20.67% [1002100/4848000] [206.7/1000.0] [batch_t 0.333 (0.330)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 03:28:15,245 - Train: 20.67% [1002200/4848000] [206.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 03:28:48,207 - Train: 20.67% [1002300/4848000] [206.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 03:29:21,140 - Train: 20.68% [1002400/4848000] [206.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 03:29:54,145 - Train: 20.68% [1002500/4848000] [206.8/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 03:30:27,174 - Train: 20.68% [1002600/4848000] [206.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 03:31:00,203 - Train: 20.68% [1002700/4848000] [206.8/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 03:31:33,186 - Train: 20.68% [1002800/4848000] [206.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 03:32:06,184 - Train: 20.69% [1002900/4848000] [206.9/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 03:32:39,172 - Train: 20.69% [1003000/4848000] [206.9/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 03:33:12,164 - Train: 20.69% [1003100/4848000] [206.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 03:33:45,141 - Train: 20.69% [1003200/4848000] [206.9/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 03:34:18,138 - Train: 20.70% [1003300/4848000] [207.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 03:34:51,364 - Train: 20.70% [1003400/4848000] [207.0/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 03:35:24,397 - Train: 20.70% [1003500/4848000] [207.0/1000.0] [batch_t 0.334 (0.330)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-07 03:35:36,284 - ==> Total time: 4 days, 4:09:34 Eta: 15 days, 23:42:12 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 03:35:59,477 - Train: 20.70% [1003600/4848000] [207.0/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 03:36:32,473 - Train: 20.70% [1003700/4848000] [207.0/1000.0] [batch_t 0.333 (0.330)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 03:37:05,480 - Train: 20.71% [1003800/4848000] [207.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 03:37:38,452 - Train: 20.71% [1003900/4848000] [207.1/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 03:38:11,404 - Train: 20.71% [1004000/4848000] [207.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-07 03:38:44,369 - Train: 20.71% [1004100/4848000] [207.1/1000.0] [batch_t 0.333 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 03:39:17,416 - Train: 20.71% [1004200/4848000] [207.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 03:39:50,335 - Train: 20.72% [1004300/4848000] [207.2/1000.0] [batch_t 0.324 (0.329)] [data_t 0.003] [optim_t 0.321] [lr 0.005000] 2024-04-07 03:40:23,352 - Train: 20.72% [1004400/4848000] [207.2/1000.0] [batch_t 0.335 (0.330)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-07 03:40:56,391 - Train: 20.72% [1004500/4848000] [207.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 03:41:29,323 - Train: 20.72% [1004600/4848000] [207.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 03:42:02,344 - Train: 20.72% [1004700/4848000] [207.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 03:42:35,553 - Train: 20.73% [1004800/4848000] [207.3/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 03:43:08,487 - Train: 20.73% [1004900/4848000] [207.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 03:43:41,425 - Train: 20.73% [1005000/4848000] [207.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 03:44:14,383 - Train: 20.73% [1005100/4848000] [207.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 03:44:47,341 - Train: 20.73% [1005200/4848000] [207.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 03:45:20,385 - Train: 20.74% [1005300/4848000] [207.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 03:45:53,355 - Train: 20.74% [1005400/4848000] [207.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 03:46:26,352 - Train: 20.74% [1005500/4848000] [207.4/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 03:46:59,281 - Train: 20.74% [1005600/4848000] [207.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 03:47:32,217 - Train: 20.74% [1005700/4848000] [207.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 03:48:05,151 - Train: 20.75% [1005800/4848000] [207.5/1000.0] [batch_t 0.333 (0.329)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 03:48:38,157 - Train: 20.75% [1005900/4848000] [207.5/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 03:49:11,104 - Train: 20.75% [1006000/4848000] [207.5/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 03:49:44,009 - Train: 20.75% [1006100/4848000] [207.5/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 03:50:17,067 - Train: 20.75% [1006200/4848000] [207.5/1000.0] [batch_t 0.337 (0.330)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-07 03:50:50,089 - Train: 20.76% [1006300/4848000] [207.6/1000.0] [batch_t 0.336 (0.330)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-07 03:51:23,108 - Train: 20.76% [1006400/4848000] [207.6/1000.0] [batch_t 0.333 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 03:51:56,032 - Train: 20.76% [1006500/4848000] [207.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 03:52:28,950 - Train: 20.76% [1006600/4848000] [207.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 03:53:01,917 - Train: 20.77% [1006700/4848000] [207.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 03:53:34,892 - Train: 20.77% [1006800/4848000] [207.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 03:54:07,842 - Train: 20.77% [1006900/4848000] [207.7/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 03:54:40,786 - Train: 20.77% [1007000/4848000] [207.7/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 03:55:13,786 - Train: 20.77% [1007100/4848000] [207.7/1000.0] [batch_t 0.337 (0.330)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-07 03:55:46,753 - Train: 20.78% [1007200/4848000] [207.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 03:56:19,710 - Train: 20.78% [1007300/4848000] [207.8/1000.0] [batch_t 0.334 (0.329)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 03:56:52,647 - Train: 20.78% [1007400/4848000] [207.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 03:57:25,623 - Train: 20.78% [1007500/4848000] [207.8/1000.0] [batch_t 0.333 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 03:57:58,690 - Train: 20.78% [1007600/4848000] [207.8/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 03:58:31,656 - Train: 20.79% [1007700/4848000] [207.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 03:59:04,575 - Train: 20.79% [1007800/4848000] [207.9/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 03:59:37,578 - Train: 20.79% [1007900/4848000] [207.9/1000.0] [batch_t 0.336 (0.330)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-07 04:00:10,560 - Train: 20.79% [1008000/4848000] [207.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 04:00:43,532 - Train: 20.79% [1008100/4848000] [207.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 04:01:16,478 - Train: 20.80% [1008200/4848000] [208.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 04:01:49,392 - Train: 20.80% [1008300/4848000] [208.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 04:02:17,088 - ==> Total time: 4 days, 4:36:15 Eta: 15 days, 23:04:13 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 04:02:24,443 - Train: 20.80% [1008400/4848000] [208.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 04:02:57,454 - Train: 20.80% [1008500/4848000] [208.0/1000.0] [batch_t 0.333 (0.330)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 04:03:30,535 - Train: 20.80% [1008600/4848000] [208.0/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 04:04:03,498 - Train: 20.81% [1008700/4848000] [208.1/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 04:04:36,409 - Train: 20.81% [1008800/4848000] [208.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 04:05:09,483 - Train: 20.81% [1008900/4848000] [208.1/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 04:05:42,490 - Train: 20.81% [1009000/4848000] [208.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 04:06:15,472 - Train: 20.81% [1009100/4848000] [208.1/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 04:06:48,389 - Train: 20.82% [1009200/4848000] [208.2/1000.0] [batch_t 0.335 (0.329)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-07 04:07:21,299 - Train: 20.82% [1009300/4848000] [208.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 04:07:54,271 - Train: 20.82% [1009400/4848000] [208.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 04:08:27,241 - Train: 20.82% [1009500/4848000] [208.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 04:09:00,213 - Train: 20.83% [1009600/4848000] [208.3/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 04:09:33,217 - Train: 20.83% [1009700/4848000] [208.3/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 04:10:06,189 - Train: 20.83% [1009800/4848000] [208.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 04:10:39,189 - Train: 20.83% [1009900/4848000] [208.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 04:11:12,160 - Train: 20.83% [1010000/4848000] [208.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 04:11:45,137 - Train: 20.84% [1010100/4848000] [208.4/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 04:12:18,152 - Train: 20.84% [1010200/4848000] [208.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 04:12:51,122 - Train: 20.84% [1010300/4848000] [208.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 04:13:24,259 - Train: 20.84% [1010400/4848000] [208.4/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 04:13:57,230 - Train: 20.84% [1010500/4848000] [208.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 04:14:30,179 - Train: 20.85% [1010600/4848000] [208.5/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 04:15:03,100 - Train: 20.85% [1010700/4848000] [208.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 04:15:36,050 - Train: 20.85% [1010800/4848000] [208.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 04:16:09,043 - Train: 20.85% [1010900/4848000] [208.5/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 04:16:42,014 - Train: 20.85% [1011000/4848000] [208.5/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 04:17:14,946 - Train: 20.86% [1011100/4848000] [208.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 04:17:47,910 - Train: 20.86% [1011200/4848000] [208.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 04:18:20,871 - Train: 20.86% [1011300/4848000] [208.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 04:18:53,844 - Train: 20.86% [1011400/4848000] [208.6/1000.0] [batch_t 0.325 (0.330)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-07 04:19:26,791 - Train: 20.86% [1011500/4848000] [208.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 04:19:59,768 - Train: 20.87% [1011600/4848000] [208.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 04:20:32,783 - Train: 20.87% [1011700/4848000] [208.7/1000.0] [batch_t 0.325 (0.330)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-07 04:21:05,874 - Train: 20.87% [1011800/4848000] [208.7/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 04:21:38,874 - Train: 20.87% [1011900/4848000] [208.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 04:22:11,930 - Train: 20.87% [1012000/4848000] [208.7/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 04:22:44,945 - Train: 20.88% [1012100/4848000] [208.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 04:23:17,900 - Train: 20.88% [1012200/4848000] [208.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 04:23:50,883 - Train: 20.88% [1012300/4848000] [208.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 04:24:23,886 - Train: 20.88% [1012400/4848000] [208.8/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 04:24:56,848 - Train: 20.88% [1012500/4848000] [208.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 04:25:29,863 - Train: 20.89% [1012600/4848000] [208.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 04:26:02,811 - Train: 20.89% [1012700/4848000] [208.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 04:26:35,835 - Train: 20.89% [1012800/4848000] [208.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 04:27:08,891 - Train: 20.89% [1012900/4848000] [208.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 04:27:41,867 - Train: 20.90% [1013000/4848000] [209.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 04:28:14,871 - Train: 20.90% [1013100/4848000] [209.0/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 04:28:47,952 - Train: 20.90% [1013200/4848000] [209.0/1000.0] [batch_t 0.324 (0.331)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-07 04:28:58,530 - ==> Total time: 4 days, 5:02:57 Eta: 15 days, 22:26:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 04:29:23,002 - Train: 20.90% [1013300/4848000] [209.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 04:29:55,988 - Train: 20.90% [1013400/4848000] [209.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 04:30:29,040 - Train: 20.91% [1013500/4848000] [209.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 04:31:02,051 - Train: 20.91% [1013600/4848000] [209.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 04:31:35,012 - Train: 20.91% [1013700/4848000] [209.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 04:32:07,940 - Train: 20.91% [1013800/4848000] [209.1/1000.0] [batch_t 0.326 (0.329)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-07 04:32:40,917 - Train: 20.91% [1013900/4848000] [209.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 04:33:13,876 - Train: 20.92% [1014000/4848000] [209.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 04:33:46,857 - Train: 20.92% [1014100/4848000] [209.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 04:34:19,865 - Train: 20.92% [1014200/4848000] [209.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 04:34:52,926 - Train: 20.92% [1014300/4848000] [209.2/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 04:35:25,921 - Train: 20.92% [1014400/4848000] [209.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 04:35:59,010 - Train: 20.93% [1014500/4848000] [209.3/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 04:36:32,027 - Train: 20.93% [1014600/4848000] [209.3/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 04:37:05,009 - Train: 20.93% [1014700/4848000] [209.3/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 04:37:37,949 - Train: 20.93% [1014800/4848000] [209.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 04:38:10,921 - Train: 20.93% [1014900/4848000] [209.3/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 04:38:43,894 - Train: 20.94% [1015000/4848000] [209.4/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 04:39:16,902 - Train: 20.94% [1015100/4848000] [209.4/1000.0] [batch_t 0.326 (0.330)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-07 04:39:49,870 - Train: 20.94% [1015200/4848000] [209.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 04:40:22,821 - Train: 20.94% [1015300/4848000] [209.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 04:40:55,770 - Train: 20.94% [1015400/4848000] [209.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 04:41:28,714 - Train: 20.95% [1015500/4848000] [209.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 04:42:01,661 - Train: 20.95% [1015600/4848000] [209.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 04:42:34,605 - Train: 20.95% [1015700/4848000] [209.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 04:43:07,581 - Train: 20.95% [1015800/4848000] [209.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 04:43:40,594 - Train: 20.96% [1015900/4848000] [209.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 04:44:13,665 - Train: 20.96% [1016000/4848000] [209.6/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 04:44:46,674 - Train: 20.96% [1016100/4848000] [209.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 04:45:19,696 - Train: 20.96% [1016200/4848000] [209.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 04:45:52,669 - Train: 20.96% [1016300/4848000] [209.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 04:46:25,657 - Train: 20.97% [1016400/4848000] [209.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 04:46:58,621 - Train: 20.97% [1016500/4848000] [209.7/1000.0] [batch_t 0.326 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 04:47:31,628 - Train: 20.97% [1016600/4848000] [209.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 04:48:04,630 - Train: 20.97% [1016700/4848000] [209.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 04:48:37,636 - Train: 20.97% [1016800/4848000] [209.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 04:49:10,616 - Train: 20.98% [1016900/4848000] [209.8/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 04:49:43,516 - Train: 20.98% [1017000/4848000] [209.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 04:50:16,419 - Train: 20.98% [1017100/4848000] [209.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 04:50:49,319 - Train: 20.98% [1017200/4848000] [209.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 04:51:22,251 - Train: 20.98% [1017300/4848000] [209.8/1000.0] [batch_t 0.336 (0.329)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-07 04:51:55,295 - Train: 20.99% [1017400/4848000] [209.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 04:52:28,222 - Train: 20.99% [1017500/4848000] [209.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 04:53:01,146 - Train: 20.99% [1017600/4848000] [209.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 04:53:34,088 - Train: 20.99% [1017700/4848000] [209.9/1000.0] [batch_t 0.337 (0.329)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-07 04:54:07,081 - Train: 20.99% [1017800/4848000] [209.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 04:54:40,038 - Train: 21.00% [1017900/4848000] [210.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 04:55:13,020 - Train: 21.00% [1018000/4848000] [210.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 04:55:39,439 - ==> Total time: 4 days, 5:29:37 Eta: 15 days, 21:48:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 04:55:48,101 - Train: 21.00% [1018100/4848000] [210.0/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 04:56:21,068 - Train: 21.00% [1018200/4848000] [210.0/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 04:56:54,132 - Train: 21.00% [1018300/4848000] [210.0/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 04:57:27,087 - Train: 21.01% [1018400/4848000] [210.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 04:58:00,038 - Train: 21.01% [1018500/4848000] [210.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 04:58:32,991 - Train: 21.01% [1018600/4848000] [210.1/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 04:59:06,187 - Train: 21.01% [1018700/4848000] [210.1/1000.0] [batch_t 0.326 (0.332)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-07 04:59:39,165 - Train: 21.01% [1018800/4848000] [210.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 05:00:12,177 - Train: 21.02% [1018900/4848000] [210.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 05:00:45,186 - Train: 21.02% [1019000/4848000] [210.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 05:01:18,082 - Train: 21.02% [1019100/4848000] [210.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 05:01:51,066 - Train: 21.02% [1019200/4848000] [210.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 05:02:24,031 - Train: 21.03% [1019300/4848000] [210.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 05:02:56,996 - Train: 21.03% [1019400/4848000] [210.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 05:03:29,974 - Train: 21.03% [1019500/4848000] [210.3/1000.0] [batch_t 0.334 (0.330)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 05:04:02,920 - Train: 21.03% [1019600/4848000] [210.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 05:04:35,866 - Train: 21.03% [1019700/4848000] [210.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 05:05:08,779 - Train: 21.04% [1019800/4848000] [210.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-07 05:05:41,793 - Train: 21.04% [1019900/4848000] [210.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 05:06:14,731 - Train: 21.04% [1020000/4848000] [210.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 05:06:47,808 - Train: 21.04% [1020100/4848000] [210.4/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 05:07:20,756 - Train: 21.04% [1020200/4848000] [210.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-07 05:07:53,740 - Train: 21.05% [1020300/4848000] [210.5/1000.0] [batch_t 0.326 (0.330)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-07 05:08:26,718 - Train: 21.05% [1020400/4848000] [210.5/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 05:08:59,688 - Train: 21.05% [1020500/4848000] [210.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 05:09:32,617 - Train: 21.05% [1020600/4848000] [210.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 05:10:05,590 - Train: 21.05% [1020700/4848000] [210.5/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 05:10:38,518 - Train: 21.06% [1020800/4848000] [210.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 05:11:11,510 - Train: 21.06% [1020900/4848000] [210.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 05:11:44,446 - Train: 21.06% [1021000/4848000] [210.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 05:12:17,380 - Train: 21.06% [1021100/4848000] [210.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 05:12:50,357 - Train: 21.06% [1021200/4848000] [210.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 05:13:23,287 - Train: 21.07% [1021300/4848000] [210.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 05:13:56,233 - Train: 21.07% [1021400/4848000] [210.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 05:14:29,227 - Train: 21.07% [1021500/4848000] [210.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 05:15:02,284 - Train: 21.07% [1021600/4848000] [210.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 05:15:35,233 - Train: 21.07% [1021700/4848000] [210.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 05:16:08,217 - Train: 21.08% [1021800/4848000] [210.8/1000.0] [batch_t 0.333 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 05:16:41,329 - Train: 21.08% [1021900/4848000] [210.8/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 05:17:14,329 - Train: 21.08% [1022000/4848000] [210.8/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 05:17:47,306 - Train: 21.08% [1022100/4848000] [210.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 05:18:20,281 - Train: 21.08% [1022200/4848000] [210.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 05:18:53,280 - Train: 21.09% [1022300/4848000] [210.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 05:19:26,250 - Train: 21.09% [1022400/4848000] [210.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 05:19:59,200 - Train: 21.09% [1022500/4848000] [210.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 05:20:32,131 - Train: 21.09% [1022600/4848000] [210.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 05:21:05,111 - Train: 21.10% [1022700/4848000] [211.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 05:21:38,058 - Train: 21.10% [1022800/4848000] [211.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 05:22:11,056 - Train: 21.10% [1022900/4848000] [211.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 05:22:20,267 - ==> Total time: 4 days, 5:56:18 Eta: 15 days, 21:10:57 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 05:22:46,367 - Train: 21.10% [1023000/4848000] [211.0/1000.0] [batch_t 0.326 (0.330)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-07 05:23:19,265 - Train: 21.10% [1023100/4848000] [211.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 05:23:52,256 - Train: 21.11% [1023200/4848000] [211.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 05:24:25,218 - Train: 21.11% [1023300/4848000] [211.1/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 05:24:58,192 - Train: 21.11% [1023400/4848000] [211.1/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 05:25:31,252 - Train: 21.11% [1023500/4848000] [211.1/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 05:26:04,227 - Train: 21.11% [1023600/4848000] [211.1/1000.0] [batch_t 0.337 (0.330)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-07 05:26:37,241 - Train: 21.12% [1023700/4848000] [211.2/1000.0] [batch_t 0.334 (0.330)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 05:27:10,214 - Train: 21.12% [1023800/4848000] [211.2/1000.0] [batch_t 0.326 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 05:27:43,167 - Train: 21.12% [1023900/4848000] [211.2/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 05:28:16,127 - Train: 21.12% [1024000/4848000] [211.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 05:28:49,109 - Train: 21.12% [1024100/4848000] [211.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 05:29:22,053 - Train: 21.13% [1024200/4848000] [211.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 05:29:54,936 - Train: 21.13% [1024300/4848000] [211.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 05:30:28,109 - Train: 21.13% [1024400/4848000] [211.3/1000.0] [batch_t 0.333 (0.332)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 05:31:01,097 - Train: 21.13% [1024500/4848000] [211.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 05:31:34,147 - Train: 21.13% [1024600/4848000] [211.3/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 05:32:07,081 - Train: 21.14% [1024700/4848000] [211.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 05:32:40,063 - Train: 21.14% [1024800/4848000] [211.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 05:33:13,027 - Train: 21.14% [1024900/4848000] [211.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 05:33:45,961 - Train: 21.14% [1025000/4848000] [211.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 05:34:18,914 - Train: 21.14% [1025100/4848000] [211.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 05:34:51,815 - Train: 21.15% [1025200/4848000] [211.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 05:35:24,740 - Train: 21.15% [1025300/4848000] [211.5/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 05:35:57,652 - Train: 21.15% [1025400/4848000] [211.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 05:36:30,606 - Train: 21.15% [1025500/4848000] [211.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 05:37:03,602 - Train: 21.16% [1025600/4848000] [211.6/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 05:37:36,511 - Train: 21.16% [1025700/4848000] [211.6/1000.0] [batch_t 0.325 (0.329)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-07 05:38:09,661 - Train: 21.16% [1025800/4848000] [211.6/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 05:38:42,604 - Train: 21.16% [1025900/4848000] [211.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 05:39:15,588 - Train: 21.16% [1026000/4848000] [211.6/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 05:39:48,582 - Train: 21.17% [1026100/4848000] [211.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 05:40:21,614 - Train: 21.17% [1026200/4848000] [211.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 05:40:54,655 - Train: 21.17% [1026300/4848000] [211.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 05:41:27,646 - Train: 21.17% [1026400/4848000] [211.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 05:42:00,645 - Train: 21.17% [1026500/4848000] [211.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 05:42:33,682 - Train: 21.18% [1026600/4848000] [211.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 05:43:06,602 - Train: 21.18% [1026700/4848000] [211.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 05:43:39,609 - Train: 21.18% [1026800/4848000] [211.8/1000.0] [batch_t 0.325 (0.330)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-07 05:44:12,630 - Train: 21.18% [1026900/4848000] [211.8/1000.0] [batch_t 0.335 (0.330)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-07 05:44:45,692 - Train: 21.18% [1027000/4848000] [211.8/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 05:45:18,781 - Train: 21.19% [1027100/4848000] [211.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 05:45:51,887 - Train: 21.19% [1027200/4848000] [211.9/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 05:46:24,875 - Train: 21.19% [1027300/4848000] [211.9/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 05:46:57,879 - Train: 21.19% [1027400/4848000] [211.9/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 05:47:30,820 - Train: 21.19% [1027500/4848000] [211.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 05:48:03,836 - Train: 21.20% [1027600/4848000] [212.0/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 05:48:36,804 - Train: 21.20% [1027700/4848000] [212.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 05:49:01,868 - ==> Total time: 4 days, 6:23:00 Eta: 15 days, 20:33:26 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 05:49:12,900 - Train: 21.20% [1027800/4848000] [212.0/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 05:49:45,866 - Train: 21.20% [1027900/4848000] [212.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 05:50:18,870 - Train: 21.20% [1028000/4848000] [212.0/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 05:50:51,803 - Train: 21.21% [1028100/4848000] [212.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 05:51:24,771 - Train: 21.21% [1028200/4848000] [212.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 05:51:57,745 - Train: 21.21% [1028300/4848000] [212.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 05:52:30,750 - Train: 21.21% [1028400/4848000] [212.1/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 05:53:03,733 - Train: 21.21% [1028500/4848000] [212.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 05:53:36,868 - Train: 21.22% [1028600/4848000] [212.2/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 05:54:09,896 - Train: 21.22% [1028700/4848000] [212.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 05:54:42,899 - Train: 21.22% [1028800/4848000] [212.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 05:55:15,816 - Train: 21.22% [1028900/4848000] [212.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 05:55:48,815 - Train: 21.23% [1029000/4848000] [212.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 05:56:21,795 - Train: 21.23% [1029100/4848000] [212.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 05:56:54,730 - Train: 21.23% [1029200/4848000] [212.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 05:57:27,714 - Train: 21.23% [1029300/4848000] [212.3/1000.0] [batch_t 0.335 (0.330)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-07 05:58:00,696 - Train: 21.23% [1029400/4848000] [212.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 05:58:33,657 - Train: 21.24% [1029500/4848000] [212.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 05:59:06,753 - Train: 21.24% [1029600/4848000] [212.4/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 05:59:39,759 - Train: 21.24% [1029700/4848000] [212.4/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 06:00:12,776 - Train: 21.24% [1029800/4848000] [212.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 06:00:45,734 - Train: 21.24% [1029900/4848000] [212.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 06:01:18,934 - Train: 21.25% [1030000/4848000] [212.5/1000.0] [batch_t 0.332 (0.332)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 06:01:51,881 - Train: 21.25% [1030100/4848000] [212.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 06:02:24,872 - Train: 21.25% [1030200/4848000] [212.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 06:02:57,860 - Train: 21.25% [1030300/4848000] [212.5/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 06:03:30,805 - Train: 21.25% [1030400/4848000] [212.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 06:04:03,734 - Train: 21.26% [1030500/4848000] [212.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 06:04:36,685 - Train: 21.26% [1030600/4848000] [212.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 06:05:09,683 - Train: 21.26% [1030700/4848000] [212.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 06:05:42,617 - Train: 21.26% [1030800/4848000] [212.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 06:06:15,591 - Train: 21.26% [1030900/4848000] [212.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 06:06:48,516 - Train: 21.27% [1031000/4848000] [212.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 06:07:21,446 - Train: 21.27% [1031100/4848000] [212.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 06:07:54,424 - Train: 21.27% [1031200/4848000] [212.7/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 06:08:27,357 - Train: 21.27% [1031300/4848000] [212.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 06:09:00,432 - Train: 21.27% [1031400/4848000] [212.7/1000.0] [batch_t 0.327 (0.331)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 06:09:33,401 - Train: 21.28% [1031500/4848000] [212.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 06:10:06,383 - Train: 21.28% [1031600/4848000] [212.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 06:10:39,355 - Train: 21.28% [1031700/4848000] [212.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 06:11:12,327 - Train: 21.28% [1031800/4848000] [212.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 06:11:45,405 - Train: 21.29% [1031900/4848000] [212.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 06:12:18,372 - Train: 21.29% [1032000/4848000] [212.9/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 06:12:51,284 - Train: 21.29% [1032100/4848000] [212.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 06:13:24,204 - Train: 21.29% [1032200/4848000] [212.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 06:13:57,163 - Train: 21.29% [1032300/4848000] [212.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 06:14:30,160 - Train: 21.30% [1032400/4848000] [213.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 06:15:03,154 - Train: 21.30% [1032500/4848000] [213.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 06:15:36,134 - Train: 21.30% [1032600/4848000] [213.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 06:15:44,046 - ==> Total time: 4 days, 6:49:42 Eta: 15 days, 19:56:03 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 06:16:11,172 - Train: 21.30% [1032700/4848000] [213.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 06:16:44,313 - Train: 21.30% [1032800/4848000] [213.0/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 06:17:17,329 - Train: 21.31% [1032900/4848000] [213.1/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 06:17:50,296 - Train: 21.31% [1033000/4848000] [213.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 06:18:23,235 - Train: 21.31% [1033100/4848000] [213.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 06:18:56,198 - Train: 21.31% [1033200/4848000] [213.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 06:19:29,160 - Train: 21.31% [1033300/4848000] [213.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 06:20:02,077 - Train: 21.32% [1033400/4848000] [213.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 06:20:35,031 - Train: 21.32% [1033500/4848000] [213.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 06:21:08,007 - Train: 21.32% [1033600/4848000] [213.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 06:21:40,964 - Train: 21.32% [1033700/4848000] [213.2/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 06:22:13,942 - Train: 21.32% [1033800/4848000] [213.2/1000.0] [batch_t 0.334 (0.330)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 06:22:46,951 - Train: 21.33% [1033900/4848000] [213.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 06:23:19,995 - Train: 21.33% [1034000/4848000] [213.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 06:23:52,974 - Train: 21.33% [1034100/4848000] [213.3/1000.0] [batch_t 0.323 (0.330)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 06:24:26,040 - Train: 21.33% [1034200/4848000] [213.3/1000.0] [batch_t 0.323 (0.331)] [data_t 0.003] [optim_t 0.320] [lr 0.005000] 2024-04-07 06:24:58,971 - Train: 21.33% [1034300/4848000] [213.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 06:25:31,936 - Train: 21.34% [1034400/4848000] [213.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 06:26:04,950 - Train: 21.34% [1034500/4848000] [213.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 06:26:37,949 - Train: 21.34% [1034600/4848000] [213.4/1000.0] [batch_t 0.326 (0.330)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-07 06:27:10,913 - Train: 21.34% [1034700/4848000] [213.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 06:27:43,898 - Train: 21.34% [1034800/4848000] [213.4/1000.0] [batch_t 0.326 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 06:28:16,813 - Train: 21.35% [1034900/4848000] [213.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 06:28:49,710 - Train: 21.35% [1035000/4848000] [213.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 06:29:22,654 - Train: 21.35% [1035100/4848000] [213.5/1000.0] [batch_t 0.334 (0.329)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 06:29:55,592 - Train: 21.35% [1035200/4848000] [213.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 06:30:28,600 - Train: 21.36% [1035300/4848000] [213.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 06:31:01,574 - Train: 21.36% [1035400/4848000] [213.6/1000.0] [batch_t 0.336 (0.330)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-07 06:31:34,597 - Train: 21.36% [1035500/4848000] [213.6/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 06:32:07,712 - Train: 21.36% [1035600/4848000] [213.6/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 06:32:40,643 - Train: 21.36% [1035700/4848000] [213.6/1000.0] [batch_t 0.334 (0.329)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 06:33:13,674 - Train: 21.37% [1035800/4848000] [213.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 06:33:46,620 - Train: 21.37% [1035900/4848000] [213.7/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 06:34:19,627 - Train: 21.37% [1036000/4848000] [213.7/1000.0] [batch_t 0.335 (0.330)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-07 06:34:52,598 - Train: 21.37% [1036100/4848000] [213.7/1000.0] [batch_t 0.337 (0.330)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-07 06:35:25,545 - Train: 21.37% [1036200/4848000] [213.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 06:35:58,545 - Train: 21.38% [1036300/4848000] [213.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 06:36:31,457 - Train: 21.38% [1036400/4848000] [213.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 06:37:04,436 - Train: 21.38% [1036500/4848000] [213.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 06:37:37,382 - Train: 21.38% [1036600/4848000] [213.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 06:38:10,345 - Train: 21.38% [1036700/4848000] [213.8/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 06:38:43,305 - Train: 21.39% [1036800/4848000] [213.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 06:39:16,237 - Train: 21.39% [1036900/4848000] [213.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 06:39:49,188 - Train: 21.39% [1037000/4848000] [213.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 06:40:22,235 - Train: 21.39% [1037100/4848000] [213.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 06:40:55,212 - Train: 21.39% [1037200/4848000] [213.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 06:41:28,196 - Train: 21.40% [1037300/4848000] [214.0/1000.0] [batch_t 0.334 (0.330)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 06:42:01,182 - Train: 21.40% [1037400/4848000] [214.0/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 06:42:24,964 - ==> Total time: 4 days, 7:16:23 Eta: 15 days, 19:18:42 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 06:42:36,280 - Train: 21.40% [1037500/4848000] [214.0/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 06:43:09,222 - Train: 21.40% [1037600/4848000] [214.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 06:43:42,286 - Train: 21.40% [1037700/4848000] [214.0/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 06:44:15,264 - Train: 21.41% [1037800/4848000] [214.1/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 06:44:48,242 - Train: 21.41% [1037900/4848000] [214.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 06:45:21,185 - Train: 21.41% [1038000/4848000] [214.1/1000.0] [batch_t 0.334 (0.329)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 06:45:54,144 - Train: 21.41% [1038100/4848000] [214.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 06:46:27,117 - Train: 21.42% [1038200/4848000] [214.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 06:47:00,211 - Train: 21.42% [1038300/4848000] [214.2/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 06:47:33,149 - Train: 21.42% [1038400/4848000] [214.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 06:48:06,118 - Train: 21.42% [1038500/4848000] [214.2/1000.0] [batch_t 0.337 (0.330)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-07 06:48:39,073 - Train: 21.42% [1038600/4848000] [214.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 06:49:12,026 - Train: 21.43% [1038700/4848000] [214.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 06:49:44,977 - Train: 21.43% [1038800/4848000] [214.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 06:50:17,936 - Train: 21.43% [1038900/4848000] [214.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 06:50:50,864 - Train: 21.43% [1039000/4848000] [214.3/1000.0] [batch_t 0.324 (0.329)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-07 06:51:23,800 - Train: 21.43% [1039100/4848000] [214.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 06:51:56,723 - Train: 21.44% [1039200/4848000] [214.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 06:52:29,682 - Train: 21.44% [1039300/4848000] [214.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 06:53:02,668 - Train: 21.44% [1039400/4848000] [214.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 06:53:35,651 - Train: 21.44% [1039500/4848000] [214.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 06:54:08,579 - Train: 21.44% [1039600/4848000] [214.4/1000.0] [batch_t 0.335 (0.329)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-07 06:54:41,522 - Train: 21.45% [1039700/4848000] [214.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 06:55:14,589 - Train: 21.45% [1039800/4848000] [214.5/1000.0] [batch_t 0.326 (0.331)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-07 06:55:47,581 - Train: 21.45% [1039900/4848000] [214.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 06:56:20,516 - Train: 21.45% [1040000/4848000] [214.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 06:56:53,508 - Train: 21.45% [1040100/4848000] [214.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 06:57:26,436 - Train: 21.46% [1040200/4848000] [214.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 06:57:59,367 - Train: 21.46% [1040300/4848000] [214.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 06:58:32,315 - Train: 21.46% [1040400/4848000] [214.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 06:59:05,276 - Train: 21.46% [1040500/4848000] [214.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 06:59:38,365 - Train: 21.46% [1040600/4848000] [214.6/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 07:00:11,342 - Train: 21.47% [1040700/4848000] [214.7/1000.0] [batch_t 0.337 (0.330)] [data_t 0.003] [optim_t 0.335] [lr 0.005000] 2024-04-07 07:00:44,358 - Train: 21.47% [1040800/4848000] [214.7/1000.0] [batch_t 0.335 (0.330)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-07 07:01:17,345 - Train: 21.47% [1040900/4848000] [214.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 07:01:50,264 - Train: 21.47% [1041000/4848000] [214.7/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 07:02:23,270 - Train: 21.47% [1041100/4848000] [214.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 07:02:56,474 - Train: 21.48% [1041200/4848000] [214.8/1000.0] [batch_t 0.327 (0.332)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 07:03:29,482 - Train: 21.48% [1041300/4848000] [214.8/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 07:04:02,482 - Train: 21.48% [1041400/4848000] [214.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 07:04:35,421 - Train: 21.48% [1041500/4848000] [214.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 07:05:08,368 - Train: 21.49% [1041600/4848000] [214.9/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 07:05:41,301 - Train: 21.49% [1041700/4848000] [214.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 07:06:14,299 - Train: 21.49% [1041800/4848000] [214.9/1000.0] [batch_t 0.334 (0.330)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-07 07:06:47,284 - Train: 21.49% [1041900/4848000] [214.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 07:07:20,309 - Train: 21.49% [1042000/4848000] [214.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 07:07:53,292 - Train: 21.50% [1042100/4848000] [215.0/1000.0] [batch_t 0.324 (0.330)] [data_t 0.003] [optim_t 0.321] [lr 0.005000] 2024-04-07 07:08:26,225 - Train: 21.50% [1042200/4848000] [215.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 07:08:59,199 - Train: 21.50% [1042300/4848000] [215.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 07:09:05,796 - ==> Total time: 4 days, 7:43:04 Eta: 15 days, 18:41:26 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 07:09:34,054 - Train: 21.50% [1042400/4848000] [215.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 07:10:07,131 - Train: 21.50% [1042500/4848000] [215.0/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 07:10:40,084 - Train: 21.51% [1042600/4848000] [215.1/1000.0] [batch_t 0.333 (0.329)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 07:11:13,020 - Train: 21.51% [1042700/4848000] [215.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 07:11:45,979 - Train: 21.51% [1042800/4848000] [215.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 07:12:18,984 - Train: 21.51% [1042900/4848000] [215.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 07:12:52,046 - Train: 21.51% [1043000/4848000] [215.1/1000.0] [batch_t 0.333 (0.331)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 07:13:25,094 - Train: 21.52% [1043100/4848000] [215.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 07:13:58,114 - Train: 21.52% [1043200/4848000] [215.2/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 07:14:31,117 - Train: 21.52% [1043300/4848000] [215.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 07:15:04,150 - Train: 21.52% [1043400/4848000] [215.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 07:15:37,086 - Train: 21.52% [1043500/4848000] [215.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 07:16:09,994 - Train: 21.53% [1043600/4848000] [215.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 07:16:42,942 - Train: 21.53% [1043700/4848000] [215.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 07:17:15,947 - Train: 21.53% [1043800/4848000] [215.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 07:17:49,082 - Train: 21.53% [1043900/4848000] [215.3/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 07:18:22,052 - Train: 21.53% [1044000/4848000] [215.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 07:18:55,064 - Train: 21.54% [1044100/4848000] [215.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 07:19:28,012 - Train: 21.54% [1044200/4848000] [215.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 07:20:00,948 - Train: 21.54% [1044300/4848000] [215.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 07:20:33,868 - Train: 21.54% [1044400/4848000] [215.4/1000.0] [batch_t 0.333 (0.329)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 07:21:06,843 - Train: 21.54% [1044500/4848000] [215.4/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 07:21:39,821 - Train: 21.55% [1044600/4848000] [215.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 07:22:12,763 - Train: 21.55% [1044700/4848000] [215.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 07:22:45,750 - Train: 21.55% [1044800/4848000] [215.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 07:23:18,728 - Train: 21.55% [1044900/4848000] [215.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 07:23:51,705 - Train: 21.56% [1045000/4848000] [215.6/1000.0] [batch_t 0.334 (0.330)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 07:24:24,696 - Train: 21.56% [1045100/4848000] [215.6/1000.0] [batch_t 0.334 (0.330)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 07:24:57,655 - Train: 21.56% [1045200/4848000] [215.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 07:25:30,629 - Train: 21.56% [1045300/4848000] [215.6/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 07:26:03,719 - Train: 21.56% [1045400/4848000] [215.6/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 07:26:36,679 - Train: 21.57% [1045500/4848000] [215.7/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 07:27:09,582 - Train: 21.57% [1045600/4848000] [215.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 07:27:42,499 - Train: 21.57% [1045700/4848000] [215.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 07:28:15,437 - Train: 21.57% [1045800/4848000] [215.7/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 07:28:48,394 - Train: 21.57% [1045900/4848000] [215.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 07:29:21,336 - Train: 21.58% [1046000/4848000] [215.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 07:29:54,286 - Train: 21.58% [1046100/4848000] [215.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 07:30:27,179 - Train: 21.58% [1046200/4848000] [215.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 07:31:00,091 - Train: 21.58% [1046300/4848000] [215.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 07:31:32,986 - Train: 21.58% [1046400/4848000] [215.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 07:32:05,897 - Train: 21.59% [1046500/4848000] [215.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 07:32:38,837 - Train: 21.59% [1046600/4848000] [215.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-07 07:33:11,731 - Train: 21.59% [1046700/4848000] [215.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 07:33:44,786 - Train: 21.59% [1046800/4848000] [215.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 07:34:17,690 - Train: 21.59% [1046900/4848000] [215.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 07:34:50,605 - Train: 21.60% [1047000/4848000] [216.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 07:35:23,583 - Train: 21.60% [1047100/4848000] [216.0/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 07:35:45,998 - ==> Total time: 4 days, 8:09:44 Eta: 15 days, 18:04:14 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 07:35:58,641 - Train: 21.60% [1047200/4848000] [216.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 07:36:31,585 - Train: 21.60% [1047300/4848000] [216.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 07:37:04,551 - Train: 21.60% [1047400/4848000] [216.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 07:37:37,451 - Train: 21.61% [1047500/4848000] [216.1/1000.0] [batch_t 0.335 (0.329)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-07 07:38:10,365 - Train: 21.61% [1047600/4848000] [216.1/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 07:38:43,436 - Train: 21.61% [1047700/4848000] [216.1/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 07:39:16,418 - Train: 21.61% [1047800/4848000] [216.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 07:39:49,346 - Train: 21.62% [1047900/4848000] [216.2/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 07:40:22,259 - Train: 21.62% [1048000/4848000] [216.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 07:40:55,338 - Train: 21.62% [1048100/4848000] [216.2/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 07:41:28,268 - Train: 21.62% [1048200/4848000] [216.2/1000.0] [batch_t 0.324 (0.329)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-07 07:42:01,212 - Train: 21.62% [1048300/4848000] [216.2/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 07:42:34,274 - Train: 21.63% [1048400/4848000] [216.3/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 07:43:07,230 - Train: 21.63% [1048500/4848000] [216.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 07:43:40,192 - Train: 21.63% [1048600/4848000] [216.3/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 07:44:13,132 - Train: 21.63% [1048700/4848000] [216.3/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 07:44:46,117 - Train: 21.63% [1048800/4848000] [216.3/1000.0] [batch_t 0.337 (0.330)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-07 07:45:19,126 - Train: 21.64% [1048900/4848000] [216.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 07:45:52,124 - Train: 21.64% [1049000/4848000] [216.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 07:46:25,070 - Train: 21.64% [1049100/4848000] [216.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 07:46:57,990 - Train: 21.64% [1049200/4848000] [216.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 07:47:30,898 - Train: 21.64% [1049300/4848000] [216.4/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 07:48:03,835 - Train: 21.65% [1049400/4848000] [216.5/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 07:48:36,876 - Train: 21.65% [1049500/4848000] [216.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 07:49:09,806 - Train: 21.65% [1049600/4848000] [216.5/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 07:49:42,726 - Train: 21.65% [1049700/4848000] [216.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 07:50:15,736 - Train: 21.65% [1049800/4848000] [216.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 07:50:48,713 - Train: 21.66% [1049900/4848000] [216.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 07:51:21,676 - Train: 21.66% [1050000/4848000] [216.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 07:51:54,687 - Train: 21.66% [1050100/4848000] [216.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 07:52:27,706 - Train: 21.66% [1050200/4848000] [216.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 07:53:00,734 - Train: 21.66% [1050300/4848000] [216.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 07:53:33,745 - Train: 21.67% [1050400/4848000] [216.7/1000.0] [batch_t 0.324 (0.330)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-07 07:54:06,785 - Train: 21.67% [1050500/4848000] [216.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 07:54:39,725 - Train: 21.67% [1050600/4848000] [216.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 07:55:12,743 - Train: 21.67% [1050700/4848000] [216.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 07:55:45,651 - Train: 21.67% [1050800/4848000] [216.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 07:56:18,775 - Train: 21.68% [1050900/4848000] [216.8/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 07:56:51,859 - Train: 21.68% [1051000/4848000] [216.8/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 07:57:24,854 - Train: 21.68% [1051100/4848000] [216.8/1000.0] [batch_t 0.333 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 07:57:58,034 - Train: 21.68% [1051200/4848000] [216.8/1000.0] [batch_t 0.331 (0.332)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 07:58:31,122 - Train: 21.69% [1051300/4848000] [216.9/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 07:59:04,169 - Train: 21.69% [1051400/4848000] [216.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 07:59:37,198 - Train: 21.69% [1051500/4848000] [216.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 08:00:10,217 - Train: 21.69% [1051600/4848000] [216.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 08:00:43,236 - Train: 21.69% [1051700/4848000] [216.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 08:01:16,287 - Train: 21.70% [1051800/4848000] [217.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 08:01:49,261 - Train: 21.70% [1051900/4848000] [217.0/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 08:02:22,230 - Train: 21.70% [1052000/4848000] [217.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 08:02:27,508 - ==> Total time: 4 days, 8:36:26 Eta: 15 days, 17:27:13 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 08:02:57,151 - Train: 21.70% [1052100/4848000] [217.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 08:03:30,140 - Train: 21.70% [1052200/4848000] [217.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 08:04:03,359 - Train: 21.71% [1052300/4848000] [217.1/1000.0] [batch_t 0.330 (0.332)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 08:04:36,321 - Train: 21.71% [1052400/4848000] [217.1/1000.0] [batch_t 0.324 (0.330)] [data_t 0.003] [optim_t 0.321] [lr 0.005000] 2024-04-07 08:05:09,302 - Train: 21.71% [1052500/4848000] [217.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 08:05:42,279 - Train: 21.71% [1052600/4848000] [217.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 08:06:15,175 - Train: 21.71% [1052700/4848000] [217.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 08:06:48,120 - Train: 21.72% [1052800/4848000] [217.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 08:07:21,028 - Train: 21.72% [1052900/4848000] [217.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 08:07:53,989 - Train: 21.72% [1053000/4848000] [217.2/1000.0] [batch_t 0.333 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 08:08:26,959 - Train: 21.72% [1053100/4848000] [217.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 08:08:59,878 - Train: 21.72% [1053200/4848000] [217.2/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 08:09:32,930 - Train: 21.73% [1053300/4848000] [217.3/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 08:10:05,855 - Train: 21.73% [1053400/4848000] [217.3/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 08:10:38,766 - Train: 21.73% [1053500/4848000] [217.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 08:11:11,686 - Train: 21.73% [1053600/4848000] [217.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 08:11:44,885 - Train: 21.73% [1053700/4848000] [217.3/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 08:12:17,948 - Train: 21.74% [1053800/4848000] [217.4/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 08:12:59,486 - Train: 21.74% [1053900/4848000] [217.4/1000.0] [batch_t 0.330 (0.415)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 08:13:32,419 - Train: 21.74% [1054000/4848000] [217.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 08:14:05,318 - Train: 21.74% [1054100/4848000] [217.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 08:14:38,254 - Train: 21.75% [1054200/4848000] [217.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 08:15:11,197 - Train: 21.75% [1054300/4848000] [217.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 08:15:44,258 - Train: 21.75% [1054400/4848000] [217.5/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 08:16:17,254 - Train: 21.75% [1054500/4848000] [217.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 08:16:50,221 - Train: 21.75% [1054600/4848000] [217.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 08:17:23,141 - Train: 21.76% [1054700/4848000] [217.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 08:17:56,137 - Train: 21.76% [1054800/4848000] [217.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 08:18:29,089 - Train: 21.76% [1054900/4848000] [217.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 08:19:02,085 - Train: 21.76% [1055000/4848000] [217.6/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 08:19:35,348 - Train: 21.76% [1055100/4848000] [217.6/1000.0] [batch_t 0.329 (0.333)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 08:20:08,266 - Train: 21.77% [1055200/4848000] [217.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 08:20:41,204 - Train: 21.77% [1055300/4848000] [217.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 08:21:14,127 - Train: 21.77% [1055400/4848000] [217.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 08:21:47,058 - Train: 21.77% [1055500/4848000] [217.7/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 08:22:19,992 - Train: 21.77% [1055600/4848000] [217.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 08:22:53,148 - Train: 21.78% [1055700/4848000] [217.8/1000.0] [batch_t 0.336 (0.331)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-07 08:23:26,073 - Train: 21.78% [1055800/4848000] [217.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 08:23:59,010 - Train: 21.78% [1055900/4848000] [217.8/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 08:24:32,025 - Train: 21.78% [1056000/4848000] [217.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 08:25:04,921 - Train: 21.78% [1056100/4848000] [217.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 08:25:37,881 - Train: 21.79% [1056200/4848000] [217.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 08:26:10,796 - Train: 21.79% [1056300/4848000] [217.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 08:26:43,712 - Train: 21.79% [1056400/4848000] [217.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 08:27:16,800 - Train: 21.79% [1056500/4848000] [217.9/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 08:27:49,731 - Train: 21.79% [1056600/4848000] [217.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 08:28:22,662 - Train: 21.80% [1056700/4848000] [218.0/1000.0] [batch_t 0.326 (0.329)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-07 08:28:55,590 - Train: 21.80% [1056800/4848000] [218.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 08:29:16,675 - ==> Total time: 4 days, 9:03:15 Eta: 15 days, 16:50:45 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 08:29:30,443 - Train: 21.80% [1056900/4848000] [218.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 08:30:03,372 - Train: 21.80% [1057000/4848000] [218.0/1000.0] [batch_t 0.338 (0.329)] [data_t 0.003] [optim_t 0.336] [lr 0.005000] 2024-04-07 08:30:36,385 - Train: 21.80% [1057100/4848000] [218.0/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 08:31:09,365 - Train: 21.81% [1057200/4848000] [218.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 08:31:42,365 - Train: 21.81% [1057300/4848000] [218.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 08:32:15,334 - Train: 21.81% [1057400/4848000] [218.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 08:32:48,268 - Train: 21.81% [1057500/4848000] [218.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 08:33:21,305 - Train: 21.82% [1057600/4848000] [218.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 08:33:54,332 - Train: 21.82% [1057700/4848000] [218.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 08:34:27,389 - Train: 21.82% [1057800/4848000] [218.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 08:35:00,595 - Train: 21.82% [1057900/4848000] [218.2/1000.0] [batch_t 0.331 (0.332)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 08:35:33,597 - Train: 21.82% [1058000/4848000] [218.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 08:36:06,618 - Train: 21.83% [1058100/4848000] [218.3/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 08:36:39,572 - Train: 21.83% [1058200/4848000] [218.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 08:37:12,574 - Train: 21.83% [1058300/4848000] [218.3/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 08:37:45,554 - Train: 21.83% [1058400/4848000] [218.3/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 08:38:18,550 - Train: 21.83% [1058500/4848000] [218.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 08:38:51,551 - Train: 21.84% [1058600/4848000] [218.4/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 08:39:24,532 - Train: 21.84% [1058700/4848000] [218.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 08:39:57,512 - Train: 21.84% [1058800/4848000] [218.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 08:40:30,485 - Train: 21.84% [1058900/4848000] [218.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 08:41:03,523 - Train: 21.84% [1059000/4848000] [218.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 08:41:36,507 - Train: 21.85% [1059100/4848000] [218.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 08:42:09,501 - Train: 21.85% [1059200/4848000] [218.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 08:42:42,555 - Train: 21.85% [1059300/4848000] [218.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 08:43:15,572 - Train: 21.85% [1059400/4848000] [218.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 08:43:48,593 - Train: 21.85% [1059500/4848000] [218.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 08:44:21,574 - Train: 21.86% [1059600/4848000] [218.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 08:44:54,635 - Train: 21.86% [1059700/4848000] [218.6/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 08:45:27,588 - Train: 21.86% [1059800/4848000] [218.6/1000.0] [batch_t 0.335 (0.329)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-07 08:46:00,571 - Train: 21.86% [1059900/4848000] [218.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 08:46:33,573 - Train: 21.86% [1060000/4848000] [218.6/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 08:47:06,582 - Train: 21.87% [1060100/4848000] [218.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 08:47:39,609 - Train: 21.87% [1060200/4848000] [218.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 08:48:12,587 - Train: 21.87% [1060300/4848000] [218.7/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 08:48:45,625 - Train: 21.87% [1060400/4848000] [218.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 08:49:18,682 - Train: 21.88% [1060500/4848000] [218.8/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 08:49:51,671 - Train: 21.88% [1060600/4848000] [218.8/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 08:50:24,717 - Train: 21.88% [1060700/4848000] [218.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 08:50:57,695 - Train: 21.88% [1060800/4848000] [218.8/1000.0] [batch_t 0.336 (0.330)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-07 08:51:30,713 - Train: 21.88% [1060900/4848000] [218.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 08:52:03,727 - Train: 21.89% [1061000/4848000] [218.9/1000.0] [batch_t 0.326 (0.330)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-07 08:52:36,754 - Train: 21.89% [1061100/4848000] [218.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 08:53:09,777 - Train: 21.89% [1061200/4848000] [218.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 08:53:42,759 - Train: 21.89% [1061300/4848000] [218.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 08:54:15,749 - Train: 21.89% [1061400/4848000] [218.9/1000.0] [batch_t 0.337 (0.330)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-07 08:54:48,742 - Train: 21.90% [1061500/4848000] [219.0/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 08:55:21,785 - Train: 21.90% [1061600/4848000] [219.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 08:55:54,829 - Train: 21.90% [1061700/4848000] [219.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 08:55:58,810 - ==> Total time: 4 days, 9:29:57 Eta: 15 days, 16:13:56 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 08:56:29,934 - Train: 21.90% [1061800/4848000] [219.0/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 08:57:02,902 - Train: 21.90% [1061900/4848000] [219.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 08:57:35,972 - Train: 21.91% [1062000/4848000] [219.1/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 08:58:08,904 - Train: 21.91% [1062100/4848000] [219.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 08:58:41,913 - Train: 21.91% [1062200/4848000] [219.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 08:59:14,872 - Train: 21.91% [1062300/4848000] [219.1/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 08:59:47,854 - Train: 21.91% [1062400/4848000] [219.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 09:00:20,846 - Train: 21.92% [1062500/4848000] [219.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 09:00:53,813 - Train: 21.92% [1062600/4848000] [219.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 09:01:26,770 - Train: 21.92% [1062700/4848000] [219.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 09:01:59,746 - Train: 21.92% [1062800/4848000] [219.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 09:02:32,678 - Train: 21.92% [1062900/4848000] [219.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 09:03:05,650 - Train: 21.93% [1063000/4848000] [219.3/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 09:03:38,642 - Train: 21.93% [1063100/4848000] [219.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 09:04:11,602 - Train: 21.93% [1063200/4848000] [219.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 09:04:44,608 - Train: 21.93% [1063300/4848000] [219.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 09:05:17,599 - Train: 21.93% [1063400/4848000] [219.3/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 09:05:50,679 - Train: 21.94% [1063500/4848000] [219.4/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 09:06:23,600 - Train: 21.94% [1063600/4848000] [219.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 09:06:56,614 - Train: 21.94% [1063700/4848000] [219.4/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 09:07:29,593 - Train: 21.94% [1063800/4848000] [219.4/1000.0] [batch_t 0.326 (0.330)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-07 09:08:02,619 - Train: 21.95% [1063900/4848000] [219.5/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 09:08:35,702 - Train: 21.95% [1064000/4848000] [219.5/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 09:09:08,695 - Train: 21.95% [1064100/4848000] [219.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 09:09:41,626 - Train: 21.95% [1064200/4848000] [219.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 09:10:14,577 - Train: 21.95% [1064300/4848000] [219.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 09:10:47,586 - Train: 21.96% [1064400/4848000] [219.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 09:11:20,558 - Train: 21.96% [1064500/4848000] [219.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 09:11:53,534 - Train: 21.96% [1064600/4848000] [219.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 09:12:26,518 - Train: 21.96% [1064700/4848000] [219.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 09:12:59,524 - Train: 21.96% [1064800/4848000] [219.6/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 09:13:32,574 - Train: 21.97% [1064900/4848000] [219.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 09:14:05,557 - Train: 21.97% [1065000/4848000] [219.7/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 09:14:38,558 - Train: 21.97% [1065100/4848000] [219.7/1000.0] [batch_t 0.324 (0.330)] [data_t 0.003] [optim_t 0.321] [lr 0.005000] 2024-04-07 09:15:11,575 - Train: 21.97% [1065200/4848000] [219.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 09:15:44,495 - Train: 21.97% [1065300/4848000] [219.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 09:16:17,444 - Train: 21.98% [1065400/4848000] [219.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 09:16:50,411 - Train: 21.98% [1065500/4848000] [219.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 09:17:23,346 - Train: 21.98% [1065600/4848000] [219.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 09:17:56,314 - Train: 21.98% [1065700/4848000] [219.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 09:18:29,318 - Train: 21.98% [1065800/4848000] [219.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 09:19:02,351 - Train: 21.99% [1065900/4848000] [219.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 09:19:35,346 - Train: 21.99% [1066000/4848000] [219.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 09:20:08,371 - Train: 21.99% [1066100/4848000] [219.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 09:20:41,437 - Train: 21.99% [1066200/4848000] [219.9/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 09:21:14,544 - Train: 21.99% [1066300/4848000] [219.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 09:21:47,580 - Train: 22.00% [1066400/4848000] [220.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 09:22:20,591 - Train: 22.00% [1066500/4848000] [220.0/1000.0] [batch_t 0.339 (0.330)] [data_t 0.003] [optim_t 0.337] [lr 0.005000] 2024-04-07 09:22:40,457 - ==> Total time: 4 days, 9:56:38 Eta: 15 days, 15:37:12 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 09:22:55,714 - Train: 22.00% [1066600/4848000] [220.0/1000.0] [batch_t 0.326 (0.330)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-07 09:23:28,759 - Train: 22.00% [1066700/4848000] [220.0/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 09:24:01,804 - Train: 22.00% [1066800/4848000] [220.0/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 09:24:34,803 - Train: 22.01% [1066900/4848000] [220.1/1000.0] [batch_t 0.336 (0.330)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-07 09:25:07,811 - Train: 22.01% [1067000/4848000] [220.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 09:25:40,760 - Train: 22.01% [1067100/4848000] [220.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 09:26:13,755 - Train: 22.01% [1067200/4848000] [220.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 09:26:46,734 - Train: 22.02% [1067300/4848000] [220.2/1000.0] [batch_t 0.335 (0.330)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-07 09:27:19,659 - Train: 22.02% [1067400/4848000] [220.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 09:27:52,602 - Train: 22.02% [1067500/4848000] [220.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 09:28:25,578 - Train: 22.02% [1067600/4848000] [220.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 09:28:58,696 - Train: 22.02% [1067700/4848000] [220.2/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 09:29:31,649 - Train: 22.03% [1067800/4848000] [220.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 09:30:04,634 - Train: 22.03% [1067900/4848000] [220.3/1000.0] [batch_t 0.336 (0.330)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-07 09:30:37,645 - Train: 22.03% [1068000/4848000] [220.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 09:31:10,607 - Train: 22.03% [1068100/4848000] [220.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 09:31:43,576 - Train: 22.03% [1068200/4848000] [220.3/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 09:32:16,570 - Train: 22.04% [1068300/4848000] [220.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 09:32:49,600 - Train: 22.04% [1068400/4848000] [220.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 09:33:22,593 - Train: 22.04% [1068500/4848000] [220.4/1000.0] [batch_t 0.325 (0.330)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-07 09:33:55,558 - Train: 22.04% [1068600/4848000] [220.4/1000.0] [batch_t 0.335 (0.330)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-07 09:34:28,529 - Train: 22.04% [1068700/4848000] [220.4/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 09:35:01,504 - Train: 22.05% [1068800/4848000] [220.5/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 09:35:34,484 - Train: 22.05% [1068900/4848000] [220.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 09:36:07,476 - Train: 22.05% [1069000/4848000] [220.5/1000.0] [batch_t 0.326 (0.330)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-07 09:36:40,603 - Train: 22.05% [1069100/4848000] [220.5/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 09:37:13,620 - Train: 22.05% [1069200/4848000] [220.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 09:37:46,570 - Train: 22.06% [1069300/4848000] [220.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 09:38:19,530 - Train: 22.06% [1069400/4848000] [220.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 09:38:52,501 - Train: 22.06% [1069500/4848000] [220.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 09:39:25,476 - Train: 22.06% [1069600/4848000] [220.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 09:39:58,446 - Train: 22.06% [1069700/4848000] [220.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 09:40:31,536 - Train: 22.07% [1069800/4848000] [220.7/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 09:41:04,511 - Train: 22.07% [1069900/4848000] [220.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 09:41:37,451 - Train: 22.07% [1070000/4848000] [220.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 09:42:10,389 - Train: 22.07% [1070100/4848000] [220.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 09:42:43,397 - Train: 22.08% [1070200/4848000] [220.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 09:43:16,417 - Train: 22.08% [1070300/4848000] [220.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 09:43:49,463 - Train: 22.08% [1070400/4848000] [220.8/1000.0] [batch_t 0.333 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 09:44:22,489 - Train: 22.08% [1070500/4848000] [220.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 09:44:55,419 - Train: 22.08% [1070600/4848000] [220.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 09:45:28,474 - Train: 22.09% [1070700/4848000] [220.9/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 09:46:01,502 - Train: 22.09% [1070800/4848000] [220.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 09:46:34,418 - Train: 22.09% [1070900/4848000] [220.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 09:47:07,365 - Train: 22.09% [1071000/4848000] [220.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 09:47:40,377 - Train: 22.09% [1071100/4848000] [220.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 09:48:13,592 - Train: 22.10% [1071200/4848000] [221.0/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 09:48:46,677 - Train: 22.10% [1071300/4848000] [221.0/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 09:49:19,635 - Train: 22.10% [1071400/4848000] [221.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 09:49:22,284 - ==> Total time: 4 days, 10:23:20 Eta: 15 days, 15:00:34 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 09:49:54,646 - Train: 22.10% [1071500/4848000] [221.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 09:50:27,597 - Train: 22.10% [1071600/4848000] [221.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 09:51:00,558 - Train: 22.11% [1071700/4848000] [221.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 09:51:33,611 - Train: 22.11% [1071800/4848000] [221.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 09:52:06,641 - Train: 22.11% [1071900/4848000] [221.1/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 09:52:39,617 - Train: 22.11% [1072000/4848000] [221.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 09:53:12,582 - Train: 22.11% [1072100/4848000] [221.1/1000.0] [batch_t 0.338 (0.330)] [data_t 0.003] [optim_t 0.335] [lr 0.005000] 2024-04-07 09:53:45,591 - Train: 22.12% [1072200/4848000] [221.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 09:54:18,625 - Train: 22.12% [1072300/4848000] [221.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 09:54:51,568 - Train: 22.12% [1072400/4848000] [221.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 09:55:24,528 - Train: 22.12% [1072500/4848000] [221.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 09:55:57,471 - Train: 22.12% [1072600/4848000] [221.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 09:56:30,417 - Train: 22.13% [1072700/4848000] [221.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 09:57:03,432 - Train: 22.13% [1072800/4848000] [221.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 09:57:36,441 - Train: 22.13% [1072900/4848000] [221.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 09:58:09,427 - Train: 22.13% [1073000/4848000] [221.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 09:58:42,400 - Train: 22.13% [1073100/4848000] [221.3/1000.0] [batch_t 0.333 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 09:59:15,331 - Train: 22.14% [1073200/4848000] [221.4/1000.0] [batch_t 0.334 (0.329)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-07 09:59:48,601 - Train: 22.14% [1073300/4848000] [221.4/1000.0] [batch_t 0.335 (0.333)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-07 10:00:21,634 - Train: 22.14% [1073400/4848000] [221.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 10:00:54,654 - Train: 22.14% [1073500/4848000] [221.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 10:01:27,619 - Train: 22.15% [1073600/4848000] [221.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 10:02:00,577 - Train: 22.15% [1073700/4848000] [221.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 10:02:33,549 - Train: 22.15% [1073800/4848000] [221.5/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 10:03:06,570 - Train: 22.15% [1073900/4848000] [221.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 10:03:39,517 - Train: 22.15% [1074000/4848000] [221.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 10:04:12,469 - Train: 22.16% [1074100/4848000] [221.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 10:04:45,452 - Train: 22.16% [1074200/4848000] [221.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 10:05:18,409 - Train: 22.16% [1074300/4848000] [221.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 10:05:51,392 - Train: 22.16% [1074400/4848000] [221.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 10:06:24,339 - Train: 22.16% [1074500/4848000] [221.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 10:06:57,301 - Train: 22.17% [1074600/4848000] [221.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 10:07:30,324 - Train: 22.17% [1074700/4848000] [221.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 10:08:03,279 - Train: 22.17% [1074800/4848000] [221.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 10:08:36,237 - Train: 22.17% [1074900/4848000] [221.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 10:09:09,185 - Train: 22.17% [1075000/4848000] [221.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 10:09:42,141 - Train: 22.18% [1075100/4848000] [221.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 10:10:15,066 - Train: 22.18% [1075200/4848000] [221.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 10:10:48,181 - Train: 22.18% [1075300/4848000] [221.8/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 10:11:21,149 - Train: 22.18% [1075400/4848000] [221.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 10:11:54,052 - Train: 22.18% [1075500/4848000] [221.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 10:12:27,037 - Train: 22.19% [1075600/4848000] [221.9/1000.0] [batch_t 0.336 (0.330)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-07 10:12:59,999 - Train: 22.19% [1075700/4848000] [221.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 10:13:32,928 - Train: 22.19% [1075800/4848000] [221.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 10:14:05,932 - Train: 22.19% [1075900/4848000] [221.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 10:14:38,957 - Train: 22.19% [1076000/4848000] [221.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 10:15:12,092 - Train: 22.20% [1076100/4848000] [222.0/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 10:15:45,144 - Train: 22.20% [1076200/4848000] [222.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 10:16:03,590 - ==> Total time: 4 days, 10:50:02 Eta: 15 days, 14:23:59 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 10:16:20,818 - Train: 22.20% [1076300/4848000] [222.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 10:16:53,748 - Train: 22.20% [1076400/4848000] [222.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 10:17:26,708 - Train: 22.21% [1076500/4848000] [222.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 10:17:59,641 - Train: 22.21% [1076600/4848000] [222.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 10:18:32,564 - Train: 22.21% [1076700/4848000] [222.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 10:19:05,559 - Train: 22.21% [1076800/4848000] [222.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 10:19:38,500 - Train: 22.21% [1076900/4848000] [222.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 10:20:11,424 - Train: 22.22% [1077000/4848000] [222.2/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 10:20:44,321 - Train: 22.22% [1077100/4848000] [222.2/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 10:21:17,282 - Train: 22.22% [1077200/4848000] [222.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 10:21:50,260 - Train: 22.22% [1077300/4848000] [222.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 10:22:23,229 - Train: 22.22% [1077400/4848000] [222.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 10:22:56,469 - Train: 22.23% [1077500/4848000] [222.3/1000.0] [batch_t 0.331 (0.332)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 10:23:32,180 - Train: 22.23% [1077600/4848000] [222.3/1000.0] [batch_t 0.331 (0.357)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 10:24:58,535 - Train: 22.23% [1077700/4848000] [222.3/1000.0] [batch_t 0.328 (0.863)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 10:29:40,961 - Train: 22.23% [1077800/4848000] [222.3/1000.0] [batch_t 1.356 (2.824)] [data_t 0.576] [optim_t 0.779] [lr 0.005000] 2024-04-07 10:45:20,539 - Train: 22.23% [1077900/4848000] [222.3/1000.0] [batch_t 0.781 (9.396)] [data_t 0.003] [optim_t 0.778] [lr 0.005000] 2024-04-07 10:54:48,107 - Train: 22.24% [1078000/4848000] [222.4/1000.0] [batch_t 0.782 (5.676)] [data_t 0.003] [optim_t 0.778] [lr 0.005000] 2024-04-07 10:56:10,119 - Train: 22.24% [1078100/4848000] [222.4/1000.0] [batch_t 0.782 (0.820)] [data_t 0.003] [optim_t 0.778] [lr 0.005000] 2024-04-07 10:57:27,861 - Train: 22.24% [1078200/4848000] [222.4/1000.0] [batch_t 0.781 (0.777)] [data_t 0.004] [optim_t 0.777] [lr 0.005000] 2024-04-07 10:58:45,728 - Train: 22.24% [1078300/4848000] [222.4/1000.0] [batch_t 0.777 (0.779)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-07 11:00:03,664 - Train: 22.24% [1078400/4848000] [222.4/1000.0] [batch_t 0.787 (0.779)] [data_t 0.003] [optim_t 0.784] [lr 0.005000] 2024-04-07 11:01:21,555 - Train: 22.25% [1078500/4848000] [222.5/1000.0] [batch_t 0.784 (0.779)] [data_t 0.003] [optim_t 0.780] [lr 0.005000] 2024-04-07 11:02:39,430 - Train: 22.25% [1078600/4848000] [222.5/1000.0] [batch_t 0.782 (0.779)] [data_t 0.003] [optim_t 0.778] [lr 0.005000] 2024-04-07 11:03:57,201 - Train: 22.25% [1078700/4848000] [222.5/1000.0] [batch_t 0.777 (0.778)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-07 11:05:14,967 - Train: 22.25% [1078800/4848000] [222.5/1000.0] [batch_t 0.780 (0.778)] [data_t 0.003] [optim_t 0.777] [lr 0.005000] 2024-04-07 11:06:32,759 - Train: 22.25% [1078900/4848000] [222.5/1000.0] [batch_t 0.778 (0.778)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-07 11:07:50,488 - Train: 22.26% [1079000/4848000] [222.6/1000.0] [batch_t 0.781 (0.777)] [data_t 0.003] [optim_t 0.778] [lr 0.005000] 2024-04-07 11:09:08,250 - Train: 22.26% [1079100/4848000] [222.6/1000.0] [batch_t 0.782 (0.778)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-07 11:10:27,730 - Train: 22.26% [1079200/4848000] [222.6/1000.0] [batch_t 0.775 (0.795)] [data_t 0.003] [optim_t 0.772] [lr 0.005000] 2024-04-07 11:11:45,641 - Train: 22.26% [1079300/4848000] [222.6/1000.0] [batch_t 0.781 (0.779)] [data_t 0.003] [optim_t 0.778] [lr 0.005000] 2024-04-07 11:13:03,495 - Train: 22.26% [1079400/4848000] [222.6/1000.0] [batch_t 0.787 (0.778)] [data_t 0.003] [optim_t 0.783] [lr 0.005000] 2024-04-07 11:14:21,252 - Train: 22.27% [1079500/4848000] [222.7/1000.0] [batch_t 0.787 (0.777)] [data_t 0.003] [optim_t 0.784] [lr 0.005000] 2024-04-07 11:15:39,112 - Train: 22.27% [1079600/4848000] [222.7/1000.0] [batch_t 0.778 (0.779)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-07 11:16:57,006 - Train: 22.27% [1079700/4848000] [222.7/1000.0] [batch_t 0.772 (0.779)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-07 11:18:14,819 - Train: 22.27% [1079800/4848000] [222.7/1000.0] [batch_t 0.786 (0.778)] [data_t 0.003] [optim_t 0.783] [lr 0.005000] 2024-04-07 11:19:32,677 - Train: 22.28% [1079900/4848000] [222.8/1000.0] [batch_t 0.777 (0.778)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-07 11:20:50,567 - Train: 22.28% [1080000/4848000] [222.8/1000.0] [batch_t 0.776 (0.779)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-07 11:22:08,468 - Train: 22.28% [1080100/4848000] [222.8/1000.0] [batch_t 0.783 (0.779)] [data_t 0.002] [optim_t 0.780] [lr 0.005000] 2024-04-07 11:23:26,395 - Train: 22.28% [1080200/4848000] [222.8/1000.0] [batch_t 0.778 (0.779)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-07 11:24:44,277 - Train: 22.28% [1080300/4848000] [222.8/1000.0] [batch_t 0.782 (0.779)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-07 11:26:02,189 - Train: 22.29% [1080400/4848000] [222.9/1000.0] [batch_t 0.783 (0.779)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-07 11:27:20,126 - Train: 22.29% [1080500/4848000] [222.9/1000.0] [batch_t 0.778 (0.779)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-07 11:28:38,060 - Train: 22.29% [1080600/4848000] [222.9/1000.0] [batch_t 0.783 (0.779)] [data_t 0.003] [optim_t 0.780] [lr 0.005000] 2024-04-07 11:29:55,998 - Train: 22.29% [1080700/4848000] [222.9/1000.0] [batch_t 0.782 (0.779)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-07 11:31:13,921 - Train: 22.29% [1080800/4848000] [222.9/1000.0] [batch_t 0.786 (0.779)] [data_t 0.003] [optim_t 0.783] [lr 0.005000] 2024-04-07 11:32:31,781 - Train: 22.30% [1080900/4848000] [223.0/1000.0] [batch_t 0.782 (0.779)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-07 11:33:49,724 - Train: 22.30% [1081000/4848000] [223.0/1000.0] [batch_t 0.783 (0.779)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-07 11:35:07,614 - Train: 22.30% [1081100/4848000] [223.0/1000.0] [batch_t 0.778 (0.779)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-07 11:35:10,714 - ==> Total time: 4 days, 12:09:09 Eta: 15 days, 16:50:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 11:36:27,982 - Train: 22.30% [1081200/4848000] [223.0/1000.0] [batch_t 0.772 (0.781)] [data_t 0.004] [optim_t 0.768] [lr 0.005000] 2024-04-07 11:37:45,814 - Train: 22.30% [1081300/4848000] [223.0/1000.0] [batch_t 0.776 (0.778)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-07 11:39:04,826 - Train: 22.31% [1081400/4848000] [223.1/1000.0] [batch_t 0.777 (0.790)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-07 11:40:22,702 - Train: 22.31% [1081500/4848000] [223.1/1000.0] [batch_t 0.778 (0.779)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-07 11:46:18,479 - Train: 22.31% [1081600/4848000] [223.1/1000.0] [batch_t 43.633 (3.558)] [data_t 42.862] [optim_t 0.771] [lr 0.005000] 2024-04-07 11:50:52,528 - Train: 22.31% [1081700/4848000] [223.1/1000.0] [batch_t 0.777 (2.740)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-07 11:52:26,737 - Train: 22.31% [1081800/4848000] [223.1/1000.0] [batch_t 0.767 (0.942)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 11:55:38,890 - Train: 22.32% [1081900/4848000] [223.2/1000.0] [batch_t 0.776 (1.921)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-07 11:59:43,691 - Train: 22.32% [1082000/4848000] [223.2/1000.0] [batch_t 0.783 (2.448)] [data_t 0.003] [optim_t 0.780] [lr 0.005000] 2024-04-07 12:05:00,918 - Train: 22.32% [1082100/4848000] [223.2/1000.0] [batch_t 0.779 (3.172)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-07 12:11:43,828 - Train: 22.32% [1082200/4848000] [223.2/1000.0] [batch_t 0.787 (4.029)] [data_t 0.003] [optim_t 0.784] [lr 0.005000] 2024-04-07 12:16:32,411 - Train: 22.32% [1082300/4848000] [223.2/1000.0] [batch_t 0.782 (2.886)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-07 12:19:00,187 - Train: 22.33% [1082400/4848000] [223.3/1000.0] [batch_t 0.782 (1.478)] [data_t 0.003] [optim_t 0.778] [lr 0.005000] 2024-04-07 12:20:29,539 - Train: 22.33% [1082500/4848000] [223.3/1000.0] [batch_t 0.777 (0.893)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-07 12:21:47,406 - Train: 22.33% [1082600/4848000] [223.3/1000.0] [batch_t 0.772 (0.779)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-07 12:23:05,223 - Train: 22.33% [1082700/4848000] [223.3/1000.0] [batch_t 0.772 (0.778)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-07 12:24:23,045 - Train: 22.33% [1082800/4848000] [223.3/1000.0] [batch_t 0.788 (0.778)] [data_t 0.003] [optim_t 0.784] [lr 0.005000] 2024-04-07 12:25:40,871 - Train: 22.34% [1082900/4848000] [223.4/1000.0] [batch_t 0.776 (0.778)] [data_t 0.003] [optim_t 0.772] [lr 0.005000] 2024-04-07 12:26:58,790 - Train: 22.34% [1083000/4848000] [223.4/1000.0] [batch_t 0.782 (0.779)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-07 12:28:16,666 - Train: 22.34% [1083100/4848000] [223.4/1000.0] [batch_t 0.782 (0.779)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-07 12:29:34,595 - Train: 22.34% [1083200/4848000] [223.4/1000.0] [batch_t 0.774 (0.779)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-07 12:30:52,502 - Train: 22.35% [1083300/4848000] [223.5/1000.0] [batch_t 0.779 (0.779)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-07 12:32:10,357 - Train: 22.35% [1083400/4848000] [223.5/1000.0] [batch_t 0.773 (0.778)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-07 12:32:48,201 - Train: 22.35% [1083500/4848000] [223.5/1000.0] [batch_t 0.328 (0.378)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 12:33:21,394 - Train: 22.35% [1083600/4848000] [223.5/1000.0] [batch_t 0.325 (0.332)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 12:33:54,367 - Train: 22.35% [1083700/4848000] [223.5/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 12:34:27,697 - Train: 22.36% [1083800/4848000] [223.6/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 12:35:00,517 - Train: 22.36% [1083900/4848000] [223.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 12:35:33,418 - Train: 22.36% [1084000/4848000] [223.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 12:36:06,457 - Train: 22.36% [1084100/4848000] [223.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 12:36:39,392 - Train: 22.36% [1084200/4848000] [223.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 12:37:12,341 - Train: 22.37% [1084300/4848000] [223.7/1000.0] [batch_t 0.337 (0.329)] [data_t 0.003] [optim_t 0.335] [lr 0.005000] 2024-04-07 12:37:45,264 - Train: 22.37% [1084400/4848000] [223.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 12:38:18,254 - Train: 22.37% [1084500/4848000] [223.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 12:38:51,308 - Train: 22.37% [1084600/4848000] [223.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 12:39:24,305 - Train: 22.37% [1084700/4848000] [223.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 12:39:57,235 - Train: 22.38% [1084800/4848000] [223.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 12:40:30,651 - Train: 22.38% [1084900/4848000] [223.8/1000.0] [batch_t 0.330 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 12:41:03,642 - Train: 22.38% [1085000/4848000] [223.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 12:41:36,845 - Train: 22.38% [1085100/4848000] [223.8/1000.0] [batch_t 0.329 (0.332)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 12:42:09,866 - Train: 22.38% [1085200/4848000] [223.8/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 12:42:42,804 - Train: 22.39% [1085300/4848000] [223.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 12:43:15,794 - Train: 22.39% [1085400/4848000] [223.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 12:43:48,754 - Train: 22.39% [1085500/4848000] [223.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 12:44:23,108 - Train: 22.39% [1085600/4848000] [223.9/1000.0] [batch_t 0.332 (0.343)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 12:44:56,024 - Train: 22.39% [1085700/4848000] [223.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 12:45:29,344 - Train: 22.40% [1085800/4848000] [224.0/1000.0] [batch_t 0.324 (0.333)] [data_t 0.003] [optim_t 0.321] [lr 0.005000] 2024-04-07 12:46:02,305 - Train: 22.40% [1085900/4848000] [224.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 12:46:19,533 - ==> Total time: 4 days, 13:20:18 Eta: 15 days, 18:46:45 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 12:46:38,578 - Train: 22.40% [1086000/4848000] [224.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 12:47:11,573 - Train: 22.40% [1086100/4848000] [224.0/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 12:47:44,503 - Train: 22.41% [1086200/4848000] [224.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 12:48:18,230 - Train: 22.41% [1086300/4848000] [224.1/1000.0] [batch_t 0.331 (0.337)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 12:48:51,184 - Train: 22.41% [1086400/4848000] [224.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 12:49:24,123 - Train: 22.41% [1086500/4848000] [224.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 12:49:57,105 - Train: 22.41% [1086600/4848000] [224.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 12:50:30,503 - Train: 22.42% [1086700/4848000] [224.2/1000.0] [batch_t 0.330 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 12:51:03,452 - Train: 22.42% [1086800/4848000] [224.2/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 12:51:37,096 - Train: 22.42% [1086900/4848000] [224.2/1000.0] [batch_t 0.330 (0.336)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 12:52:10,010 - Train: 22.42% [1087000/4848000] [224.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 12:52:42,906 - Train: 22.42% [1087100/4848000] [224.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 12:53:15,813 - Train: 22.43% [1087200/4848000] [224.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 12:53:48,804 - Train: 22.43% [1087300/4848000] [224.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 12:54:21,895 - Train: 22.43% [1087400/4848000] [224.3/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 12:54:54,848 - Train: 22.43% [1087500/4848000] [224.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 12:55:27,840 - Train: 22.43% [1087600/4848000] [224.3/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 12:56:00,789 - Train: 22.44% [1087700/4848000] [224.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 12:56:35,604 - Train: 22.44% [1087800/4848000] [224.4/1000.0] [batch_t 0.327 (0.348)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 12:57:08,507 - Train: 22.44% [1087900/4848000] [224.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 12:57:41,462 - Train: 22.44% [1088000/4848000] [224.4/1000.0] [batch_t 0.333 (0.329)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 12:58:15,350 - Train: 22.44% [1088100/4848000] [224.4/1000.0] [batch_t 0.330 (0.339)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 12:58:48,296 - Train: 22.45% [1088200/4848000] [224.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 12:59:22,128 - Train: 22.45% [1088300/4848000] [224.5/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 12:59:55,087 - Train: 22.45% [1088400/4848000] [224.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 13:00:29,361 - Train: 22.45% [1088500/4848000] [224.5/1000.0] [batch_t 0.329 (0.343)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:01:02,349 - Train: 22.45% [1088600/4848000] [224.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 13:01:35,465 - Train: 22.46% [1088700/4848000] [224.6/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 13:02:08,449 - Train: 22.46% [1088800/4848000] [224.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 13:02:41,476 - Train: 22.46% [1088900/4848000] [224.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:03:14,457 - Train: 22.46% [1089000/4848000] [224.6/1000.0] [batch_t 0.335 (0.330)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-07 13:03:47,368 - Train: 22.46% [1089100/4848000] [224.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:04:20,316 - Train: 22.47% [1089200/4848000] [224.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:04:53,378 - Train: 22.47% [1089300/4848000] [224.7/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:05:26,275 - Train: 22.47% [1089400/4848000] [224.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:05:59,310 - Train: 22.47% [1089500/4848000] [224.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:06:32,223 - Train: 22.48% [1089600/4848000] [224.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:07:05,187 - Train: 22.48% [1089700/4848000] [224.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:07:38,155 - Train: 22.48% [1089800/4848000] [224.8/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 13:08:11,152 - Train: 22.48% [1089900/4848000] [224.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 13:08:44,113 - Train: 22.48% [1090000/4848000] [224.8/1000.0] [batch_t 0.335 (0.330)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-07 13:09:17,174 - Train: 22.49% [1090100/4848000] [224.9/1000.0] [batch_t 0.333 (0.331)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 13:09:50,198 - Train: 22.49% [1090200/4848000] [224.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 13:10:23,350 - Train: 22.49% [1090300/4848000] [224.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 13:10:56,296 - Train: 22.49% [1090400/4848000] [224.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 13:11:29,252 - Train: 22.49% [1090500/4848000] [224.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:12:02,143 - Train: 22.50% [1090600/4848000] [225.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 13:12:35,051 - Train: 22.50% [1090700/4848000] [225.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 13:13:08,015 - Train: 22.50% [1090800/4848000] [225.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:13:08,018 - ==> Total time: 4 days, 13:47:06 Eta: 15 days, 18:08:55 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 13:13:42,838 - Train: 22.50% [1090900/4848000] [225.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 13:14:15,783 - Train: 22.50% [1091000/4848000] [225.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:14:48,720 - Train: 22.51% [1091100/4848000] [225.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:15:21,726 - Train: 22.51% [1091200/4848000] [225.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 13:15:54,746 - Train: 22.51% [1091300/4848000] [225.1/1000.0] [batch_t 0.336 (0.330)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-07 13:16:27,665 - Train: 22.51% [1091400/4848000] [225.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 13:17:00,571 - Train: 22.51% [1091500/4848000] [225.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:17:33,639 - Train: 22.52% [1091600/4848000] [225.2/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:18:06,676 - Train: 22.52% [1091700/4848000] [225.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 13:18:39,574 - Train: 22.52% [1091800/4848000] [225.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 13:19:15,559 - Train: 22.52% [1091900/4848000] [225.2/1000.0] [batch_t 0.328 (0.360)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:19:48,511 - Train: 22.52% [1092000/4848000] [225.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:20:21,441 - Train: 22.53% [1092100/4848000] [225.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 13:20:54,413 - Train: 22.53% [1092200/4848000] [225.3/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 13:21:27,359 - Train: 22.53% [1092300/4848000] [225.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:22:00,297 - Train: 22.53% [1092400/4848000] [225.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 13:22:33,260 - Train: 22.54% [1092500/4848000] [225.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 13:23:06,195 - Train: 22.54% [1092600/4848000] [225.4/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 13:23:39,270 - Train: 22.54% [1092700/4848000] [225.4/1000.0] [batch_t 0.339 (0.331)] [data_t 0.003] [optim_t 0.336] [lr 0.005000] 2024-04-07 13:24:12,217 - Train: 22.54% [1092800/4848000] [225.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 13:24:45,153 - Train: 22.54% [1092900/4848000] [225.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 13:25:18,291 - Train: 22.55% [1093000/4848000] [225.5/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:25:51,294 - Train: 22.55% [1093100/4848000] [225.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:26:25,878 - Train: 22.55% [1093200/4848000] [225.5/1000.0] [batch_t 0.330 (0.346)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:26:58,860 - Train: 22.55% [1093300/4848000] [225.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:27:31,893 - Train: 22.55% [1093400/4848000] [225.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:28:04,822 - Train: 22.56% [1093500/4848000] [225.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:28:37,858 - Train: 22.56% [1093600/4848000] [225.6/1000.0] [batch_t 0.335 (0.330)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-07 13:29:10,784 - Train: 22.56% [1093700/4848000] [225.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 13:29:43,726 - Train: 22.56% [1093800/4848000] [225.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 13:30:16,685 - Train: 22.56% [1093900/4848000] [225.6/1000.0] [batch_t 0.325 (0.329)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-07 13:30:49,734 - Train: 22.57% [1094000/4848000] [225.7/1000.0] [batch_t 0.333 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 13:31:22,756 - Train: 22.57% [1094100/4848000] [225.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:31:55,755 - Train: 22.57% [1094200/4848000] [225.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:32:28,718 - Train: 22.57% [1094300/4848000] [225.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:33:01,692 - Train: 22.57% [1094400/4848000] [225.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 13:33:34,716 - Train: 22.58% [1094500/4848000] [225.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 13:34:07,647 - Train: 22.58% [1094600/4848000] [225.8/1000.0] [batch_t 0.334 (0.329)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 13:34:40,635 - Train: 22.58% [1094700/4848000] [225.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 13:35:13,687 - Train: 22.58% [1094800/4848000] [225.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:35:46,666 - Train: 22.58% [1094900/4848000] [225.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 13:36:19,664 - Train: 22.59% [1095000/4848000] [225.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:36:52,586 - Train: 22.59% [1095100/4848000] [225.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 13:37:25,540 - Train: 22.59% [1095200/4848000] [225.9/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 13:37:58,503 - Train: 22.59% [1095300/4848000] [225.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 13:38:31,514 - Train: 22.59% [1095400/4848000] [225.9/1000.0] [batch_t 0.337 (0.330)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-07 13:39:04,503 - Train: 22.60% [1095500/4848000] [226.0/1000.0] [batch_t 0.333 (0.330)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 13:39:37,446 - Train: 22.60% [1095600/4848000] [226.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:39:53,303 - ==> Total time: 4 days, 14:13:51 Eta: 15 days, 17:31:01 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 13:40:12,522 - Train: 22.60% [1095700/4848000] [226.0/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 13:40:45,562 - Train: 22.60% [1095800/4848000] [226.0/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 13:41:18,500 - Train: 22.61% [1095900/4848000] [226.1/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 13:41:51,412 - Train: 22.61% [1096000/4848000] [226.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:42:24,345 - Train: 22.61% [1096100/4848000] [226.1/1000.0] [batch_t 0.334 (0.329)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-07 13:42:57,312 - Train: 22.61% [1096200/4848000] [226.1/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 13:43:30,352 - Train: 22.61% [1096300/4848000] [226.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:44:03,358 - Train: 22.62% [1096400/4848000] [226.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:44:36,320 - Train: 22.62% [1096500/4848000] [226.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 13:45:09,310 - Train: 22.62% [1096600/4848000] [226.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 13:45:42,301 - Train: 22.62% [1096700/4848000] [226.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 13:46:15,243 - Train: 22.62% [1096800/4848000] [226.2/1000.0] [batch_t 0.334 (0.329)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 13:46:48,151 - Train: 22.63% [1096900/4848000] [226.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 13:47:21,111 - Train: 22.63% [1097000/4848000] [226.3/1000.0] [batch_t 0.334 (0.330)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 13:47:54,035 - Train: 22.63% [1097100/4848000] [226.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:48:27,092 - Train: 22.63% [1097200/4848000] [226.3/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 13:49:00,092 - Train: 22.63% [1097300/4848000] [226.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:49:33,071 - Train: 22.64% [1097400/4848000] [226.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:50:06,045 - Train: 22.64% [1097500/4848000] [226.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 13:50:38,982 - Train: 22.64% [1097600/4848000] [226.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 13:51:11,920 - Train: 22.64% [1097700/4848000] [226.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 13:51:44,879 - Train: 22.64% [1097800/4848000] [226.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 13:52:17,848 - Train: 22.65% [1097900/4848000] [226.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:52:50,835 - Train: 22.65% [1098000/4848000] [226.5/1000.0] [batch_t 0.334 (0.330)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 13:53:23,875 - Train: 22.65% [1098100/4848000] [226.5/1000.0] [batch_t 0.338 (0.330)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-07 13:53:56,967 - Train: 22.65% [1098200/4848000] [226.5/1000.0] [batch_t 0.335 (0.331)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-07 13:54:30,039 - Train: 22.65% [1098300/4848000] [226.5/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 13:55:03,015 - Train: 22.66% [1098400/4848000] [226.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 13:55:35,967 - Train: 22.66% [1098500/4848000] [226.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 13:56:08,911 - Train: 22.66% [1098600/4848000] [226.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 13:56:41,956 - Train: 22.66% [1098700/4848000] [226.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:57:14,947 - Train: 22.67% [1098800/4848000] [226.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 13:57:47,954 - Train: 22.67% [1098900/4848000] [226.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 13:58:20,906 - Train: 22.67% [1099000/4848000] [226.7/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 13:58:53,905 - Train: 22.67% [1099100/4848000] [226.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 13:59:26,815 - Train: 22.67% [1099200/4848000] [226.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 13:59:59,797 - Train: 22.68% [1099300/4848000] [226.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 14:00:32,773 - Train: 22.68% [1099400/4848000] [226.8/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 14:01:07,998 - Train: 22.68% [1099500/4848000] [226.8/1000.0] [batch_t 0.330 (0.352)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 14:01:40,967 - Train: 22.68% [1099600/4848000] [226.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 14:02:15,649 - Train: 22.68% [1099700/4848000] [226.8/1000.0] [batch_t 0.331 (0.347)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 14:02:48,648 - Train: 22.69% [1099800/4848000] [226.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 14:03:29,027 - Train: 22.69% [1099900/4848000] [226.9/1000.0] [batch_t 0.330 (0.404)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 14:04:12,046 - Train: 22.69% [1100000/4848000] [226.9/1000.0] [batch_t 0.337 (0.430)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-07 14:04:45,076 - Train: 22.69% [1100100/4848000] [226.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 14:05:18,030 - Train: 22.69% [1100200/4848000] [226.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 14:05:51,938 - Train: 22.70% [1100300/4848000] [227.0/1000.0] [batch_t 0.334 (0.339)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-07 14:07:06,512 - Train: 22.70% [1100400/4848000] [227.0/1000.0] [batch_t 0.329 (0.746)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 14:07:44,914 - ==> Total time: 4 days, 14:41:43 Eta: 15 days, 16:56:58 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 14:07:48,227 - Train: 22.70% [1100500/4848000] [227.0/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 14:08:21,175 - Train: 22.70% [1100600/4848000] [227.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 14:08:54,148 - Train: 22.70% [1100700/4848000] [227.0/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 14:09:27,117 - Train: 22.71% [1100800/4848000] [227.1/1000.0] [batch_t 0.325 (0.330)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-07 14:10:00,104 - Train: 22.71% [1100900/4848000] [227.1/1000.0] [batch_t 0.333 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 14:10:33,059 - Train: 22.71% [1101000/4848000] [227.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 14:17:38,795 - Train: 22.71% [1101100/4848000] [227.1/1000.0] [batch_t 21.184 (4.257)] [data_t 20.405] [optim_t 0.780] [lr 0.005000] 2024-04-07 14:27:32,460 - Train: 22.71% [1101200/4848000] [227.1/1000.0] [batch_t 0.778 (5.937)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-07 14:29:08,729 - Train: 22.72% [1101300/4848000] [227.2/1000.0] [batch_t 0.772 (0.963)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-07 14:30:26,710 - Train: 22.72% [1101400/4848000] [227.2/1000.0] [batch_t 0.782 (0.780)] [data_t 0.003] [optim_t 0.778] [lr 0.005000] 2024-04-07 14:31:44,704 - Train: 22.72% [1101500/4848000] [227.2/1000.0] [batch_t 0.787 (0.780)] [data_t 0.003] [optim_t 0.784] [lr 0.005000] 2024-04-07 14:33:07,407 - Train: 22.72% [1101600/4848000] [227.2/1000.0] [batch_t 0.778 (0.827)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-07 14:34:32,468 - Train: 22.72% [1101700/4848000] [227.2/1000.0] [batch_t 0.782 (0.851)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-07 14:35:54,660 - Train: 22.73% [1101800/4848000] [227.3/1000.0] [batch_t 0.788 (0.822)] [data_t 0.003] [optim_t 0.785] [lr 0.005000] 2024-04-07 14:37:12,494 - Train: 22.73% [1101900/4848000] [227.3/1000.0] [batch_t 0.777 (0.778)] [data_t 0.004] [optim_t 0.773] [lr 0.005000] 2024-04-07 14:38:30,338 - Train: 22.73% [1102000/4848000] [227.3/1000.0] [batch_t 0.782 (0.778)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-07 14:39:48,188 - Train: 22.73% [1102100/4848000] [227.3/1000.0] [batch_t 0.783 (0.778)] [data_t 0.003] [optim_t 0.780] [lr 0.005000] 2024-04-07 14:41:06,004 - Train: 22.74% [1102200/4848000] [227.4/1000.0] [batch_t 0.777 (0.778)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-07 14:42:23,893 - Train: 22.74% [1102300/4848000] [227.4/1000.0] [batch_t 0.782 (0.779)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-07 14:43:41,771 - Train: 22.74% [1102400/4848000] [227.4/1000.0] [batch_t 0.782 (0.779)] [data_t 0.003] [optim_t 0.778] [lr 0.005000] 2024-04-07 14:44:59,675 - Train: 22.74% [1102500/4848000] [227.4/1000.0] [batch_t 0.781 (0.779)] [data_t 0.003] [optim_t 0.778] [lr 0.005000] 2024-04-07 14:46:20,042 - Train: 22.74% [1102600/4848000] [227.4/1000.0] [batch_t 0.782 (0.804)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-07 14:47:37,851 - Train: 22.75% [1102700/4848000] [227.5/1000.0] [batch_t 0.776 (0.778)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-07 14:48:55,618 - Train: 22.75% [1102800/4848000] [227.5/1000.0] [batch_t 0.768 (0.778)] [data_t 0.003] [optim_t 0.765] [lr 0.005000] 2024-04-07 14:50:13,463 - Train: 22.75% [1102900/4848000] [227.5/1000.0] [batch_t 0.778 (0.778)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-07 14:51:31,326 - Train: 22.75% [1103000/4848000] [227.5/1000.0] [batch_t 0.780 (0.779)] [data_t 0.003] [optim_t 0.777] [lr 0.005000] 2024-04-07 14:52:49,217 - Train: 22.75% [1103100/4848000] [227.5/1000.0] [batch_t 0.787 (0.779)] [data_t 0.003] [optim_t 0.784] [lr 0.005000] 2024-04-07 14:54:07,156 - Train: 22.76% [1103200/4848000] [227.6/1000.0] [batch_t 0.778 (0.779)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-07 14:55:24,970 - Train: 22.76% [1103300/4848000] [227.6/1000.0] [batch_t 0.779 (0.778)] [data_t 0.003] [optim_t 0.776] [lr 0.005000] 2024-04-07 14:56:42,732 - Train: 22.76% [1103400/4848000] [227.6/1000.0] [batch_t 0.780 (0.778)] [data_t 0.003] [optim_t 0.777] [lr 0.005000] 2024-04-07 14:58:00,566 - Train: 22.76% [1103500/4848000] [227.6/1000.0] [batch_t 0.776 (0.778)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-07 14:59:18,437 - Train: 22.76% [1103600/4848000] [227.6/1000.0] [batch_t 0.783 (0.779)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-07 15:00:36,239 - Train: 22.77% [1103700/4848000] [227.7/1000.0] [batch_t 0.777 (0.778)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-07 15:01:54,138 - Train: 22.77% [1103800/4848000] [227.7/1000.0] [batch_t 0.773 (0.779)] [data_t 0.003] [optim_t 0.770] [lr 0.005000] 2024-04-07 15:03:12,007 - Train: 22.77% [1103900/4848000] [227.7/1000.0] [batch_t 0.787 (0.779)] [data_t 0.003] [optim_t 0.783] [lr 0.005000] 2024-04-07 15:04:29,835 - Train: 22.77% [1104000/4848000] [227.7/1000.0] [batch_t 0.771 (0.778)] [data_t 0.003] [optim_t 0.768] [lr 0.005000] 2024-04-07 15:05:47,677 - Train: 22.77% [1104100/4848000] [227.7/1000.0] [batch_t 0.774 (0.778)] [data_t 0.003] [optim_t 0.771] [lr 0.005000] 2024-04-07 15:07:05,568 - Train: 22.78% [1104200/4848000] [227.8/1000.0] [batch_t 0.788 (0.779)] [data_t 0.003] [optim_t 0.785] [lr 0.005000] 2024-04-07 15:08:23,419 - Train: 22.78% [1104300/4848000] [227.8/1000.0] [batch_t 0.783 (0.778)] [data_t 0.003] [optim_t 0.780] [lr 0.005000] 2024-04-07 15:09:41,225 - Train: 22.78% [1104400/4848000] [227.8/1000.0] [batch_t 0.767 (0.778)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 15:10:58,895 - Train: 22.78% [1104500/4848000] [227.8/1000.0] [batch_t 0.781 (0.777)] [data_t 0.003] [optim_t 0.778] [lr 0.005000] 2024-04-07 15:12:16,635 - Train: 22.78% [1104600/4848000] [227.8/1000.0] [batch_t 0.778 (0.777)] [data_t 0.003] [optim_t 0.775] [lr 0.005000] 2024-04-07 15:13:34,314 - Train: 22.79% [1104700/4848000] [227.9/1000.0] [batch_t 0.777 (0.777)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-07 15:14:52,137 - Train: 22.79% [1104800/4848000] [227.9/1000.0] [batch_t 0.772 (0.778)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-07 15:16:09,893 - Train: 22.79% [1104900/4848000] [227.9/1000.0] [batch_t 0.777 (0.777)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-07 15:17:27,651 - Train: 22.79% [1105000/4848000] [227.9/1000.0] [batch_t 0.782 (0.777)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-07 15:18:45,460 - Train: 22.79% [1105100/4848000] [227.9/1000.0] [batch_t 0.783 (0.778)] [data_t 0.003] [optim_t 0.779] [lr 0.005000] 2024-04-07 15:20:03,332 - Train: 22.80% [1105200/4848000] [228.0/1000.0] [batch_t 0.776 (0.779)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-07 15:21:21,236 - Train: 22.80% [1105300/4848000] [228.0/1000.0] [batch_t 0.786 (0.779)] [data_t 0.003] [optim_t 0.782] [lr 0.005000] 2024-04-07 15:21:55,480 - ==> Total time: 4 days, 15:55:53 Eta: 15 days, 18:59:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 15:22:41,142 - Train: 22.80% [1105400/4848000] [228.0/1000.0] [batch_t 0.777 (0.777)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-07 15:23:59,006 - Train: 22.80% [1105500/4848000] [228.0/1000.0] [batch_t 0.783 (0.779)] [data_t 0.003] [optim_t 0.780] [lr 0.005000] 2024-04-07 15:25:16,791 - Train: 22.81% [1105600/4848000] [228.1/1000.0] [batch_t 0.780 (0.778)] [data_t 0.003] [optim_t 0.777] [lr 0.005000] 2024-04-07 15:26:34,626 - Train: 22.81% [1105700/4848000] [228.1/1000.0] [batch_t 0.767 (0.778)] [data_t 0.003] [optim_t 0.764] [lr 0.005000] 2024-04-07 15:27:52,462 - Train: 22.81% [1105800/4848000] [228.1/1000.0] [batch_t 0.772 (0.778)] [data_t 0.003] [optim_t 0.769] [lr 0.005000] 2024-04-07 15:29:10,350 - Train: 22.81% [1105900/4848000] [228.1/1000.0] [batch_t 0.776 (0.779)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-07 15:30:28,203 - Train: 22.81% [1106000/4848000] [228.1/1000.0] [batch_t 0.777 (0.778)] [data_t 0.003] [optim_t 0.774] [lr 0.005000] 2024-04-07 15:31:46,170 - Train: 22.82% [1106100/4848000] [228.2/1000.0] [batch_t 0.776 (0.780)] [data_t 0.003] [optim_t 0.773] [lr 0.005000] 2024-04-07 15:32:40,377 - Train: 22.82% [1106200/4848000] [228.2/1000.0] [batch_t 0.329 (0.542)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 15:33:13,358 - Train: 22.82% [1106300/4848000] [228.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 15:33:46,347 - Train: 22.82% [1106400/4848000] [228.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 15:34:19,291 - Train: 22.82% [1106500/4848000] [228.2/1000.0] [batch_t 0.326 (0.329)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-07 15:34:52,214 - Train: 22.83% [1106600/4848000] [228.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 15:35:25,199 - Train: 22.83% [1106700/4848000] [228.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 15:35:58,159 - Train: 22.83% [1106800/4848000] [228.3/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 15:36:32,122 - Train: 22.83% [1106900/4848000] [228.3/1000.0] [batch_t 0.331 (0.340)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 15:37:10,784 - Train: 22.83% [1107000/4848000] [228.3/1000.0] [batch_t 0.329 (0.387)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 15:37:43,804 - Train: 22.84% [1107100/4848000] [228.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 15:38:26,982 - Train: 22.84% [1107200/4848000] [228.4/1000.0] [batch_t 0.329 (0.432)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 15:39:51,685 - Train: 22.84% [1107300/4848000] [228.4/1000.0] [batch_t 0.329 (0.847)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 15:41:30,489 - Train: 22.84% [1107400/4848000] [228.4/1000.0] [batch_t 0.331 (0.988)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 15:43:20,814 - Train: 22.84% [1107500/4848000] [228.4/1000.0] [batch_t 0.330 (1.103)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 15:45:22,897 - Train: 22.85% [1107600/4848000] [228.5/1000.0] [batch_t 2.216 (1.221)] [data_t 1.890] [optim_t 0.327] [lr 0.005000] 2024-04-07 15:47:15,830 - Train: 22.85% [1107700/4848000] [228.5/1000.0] [batch_t 4.222 (1.129)] [data_t 3.896] [optim_t 0.326] [lr 0.005000] 2024-04-07 15:48:45,641 - Train: 22.85% [1107800/4848000] [228.5/1000.0] [batch_t 0.330 (0.898)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 15:50:20,569 - Train: 22.85% [1107900/4848000] [228.5/1000.0] [batch_t 1.419 (0.949)] [data_t 1.092] [optim_t 0.327] [lr 0.005000] 2024-04-07 15:51:43,823 - Train: 22.85% [1108000/4848000] [228.5/1000.0] [batch_t 0.330 (0.832)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 15:52:23,133 - Train: 22.86% [1108100/4848000] [228.6/1000.0] [batch_t 0.330 (0.393)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 15:52:56,164 - Train: 22.86% [1108200/4848000] [228.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 15:53:29,430 - Train: 22.86% [1108300/4848000] [228.6/1000.0] [batch_t 0.329 (0.333)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 15:54:02,419 - Train: 22.86% [1108400/4848000] [228.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 15:54:35,571 - Train: 22.87% [1108500/4848000] [228.7/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 15:55:08,772 - Train: 22.87% [1108600/4848000] [228.7/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 15:55:41,876 - Train: 22.87% [1108700/4848000] [228.7/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 15:56:14,854 - Train: 22.87% [1108800/4848000] [228.7/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 15:56:47,871 - Train: 22.87% [1108900/4848000] [228.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 15:57:20,905 - Train: 22.88% [1109000/4848000] [228.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 15:57:53,898 - Train: 22.88% [1109100/4848000] [228.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 15:58:26,894 - Train: 22.88% [1109200/4848000] [228.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 15:58:59,918 - Train: 22.88% [1109300/4848000] [228.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 15:59:32,953 - Train: 22.88% [1109400/4848000] [228.8/1000.0] [batch_t 0.333 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 16:00:06,028 - Train: 22.89% [1109500/4848000] [228.9/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 16:00:39,016 - Train: 22.89% [1109600/4848000] [228.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 16:01:12,003 - Train: 22.89% [1109700/4848000] [228.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 16:01:44,966 - Train: 22.89% [1109800/4848000] [228.9/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 16:02:18,006 - Train: 22.89% [1109900/4848000] [228.9/1000.0] [batch_t 0.335 (0.330)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-07 16:02:51,138 - Train: 22.90% [1110000/4848000] [229.0/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 16:03:24,242 - Train: 22.90% [1110100/4848000] [229.0/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 16:03:54,567 - ==> Total time: 4 days, 16:37:53 Eta: 15 days, 19:12:31 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 16:03:59,787 - Train: 22.90% [1110200/4848000] [229.0/1000.0] [batch_t 0.337 (0.342)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-07 16:04:32,824 - Train: 22.90% [1110300/4848000] [229.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 16:05:05,818 - Train: 22.90% [1110400/4848000] [229.0/1000.0] [batch_t 0.335 (0.330)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-07 16:05:38,834 - Train: 22.91% [1110500/4848000] [229.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 16:06:12,110 - Train: 22.91% [1110600/4848000] [229.1/1000.0] [batch_t 0.332 (0.333)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 16:06:45,200 - Train: 22.91% [1110700/4848000] [229.1/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 16:07:18,312 - Train: 22.91% [1110800/4848000] [229.1/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 16:07:51,316 - Train: 22.91% [1110900/4848000] [229.1/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 16:08:24,353 - Train: 22.92% [1111000/4848000] [229.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 16:08:57,347 - Train: 22.92% [1111100/4848000] [229.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 16:09:30,417 - Train: 22.92% [1111200/4848000] [229.2/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 16:10:03,496 - Train: 22.92% [1111300/4848000] [229.2/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 16:10:36,633 - Train: 22.92% [1111400/4848000] [229.2/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 16:11:09,597 - Train: 22.93% [1111500/4848000] [229.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 16:11:42,593 - Train: 22.93% [1111600/4848000] [229.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 16:12:15,604 - Train: 22.93% [1111700/4848000] [229.3/1000.0] [batch_t 0.334 (0.330)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 16:12:48,606 - Train: 22.93% [1111800/4848000] [229.3/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 16:13:21,602 - Train: 22.94% [1111900/4848000] [229.4/1000.0] [batch_t 0.326 (0.330)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-07 16:13:54,720 - Train: 22.94% [1112000/4848000] [229.4/1000.0] [batch_t 0.338 (0.331)] [data_t 0.003] [optim_t 0.335] [lr 0.005000] 2024-04-07 16:14:27,739 - Train: 22.94% [1112100/4848000] [229.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 16:15:00,787 - Train: 22.94% [1112200/4848000] [229.4/1000.0] [batch_t 0.336 (0.330)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-07 16:15:33,844 - Train: 22.94% [1112300/4848000] [229.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 16:16:09,659 - Train: 22.95% [1112400/4848000] [229.5/1000.0] [batch_t 0.330 (0.358)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 16:16:42,655 - Train: 22.95% [1112500/4848000] [229.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 16:17:15,619 - Train: 22.95% [1112600/4848000] [229.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 16:17:48,617 - Train: 22.95% [1112700/4848000] [229.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 16:18:21,743 - Train: 22.95% [1112800/4848000] [229.5/1000.0] [batch_t 0.325 (0.331)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 16:18:54,821 - Train: 22.96% [1112900/4848000] [229.6/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 16:19:27,897 - Train: 22.96% [1113000/4848000] [229.6/1000.0] [batch_t 0.327 (0.331)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 16:20:00,886 - Train: 22.96% [1113100/4848000] [229.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 16:20:33,881 - Train: 22.96% [1113200/4848000] [229.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 16:21:06,903 - Train: 22.96% [1113300/4848000] [229.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 16:21:39,898 - Train: 22.97% [1113400/4848000] [229.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 16:22:12,903 - Train: 22.97% [1113500/4848000] [229.7/1000.0] [batch_t 0.333 (0.330)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 16:22:45,964 - Train: 22.97% [1113600/4848000] [229.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 16:23:18,997 - Train: 22.97% [1113700/4848000] [229.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 16:23:51,958 - Train: 22.97% [1113800/4848000] [229.7/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 16:24:25,071 - Train: 22.98% [1113900/4848000] [229.8/1000.0] [batch_t 0.335 (0.331)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-07 16:24:58,114 - Train: 22.98% [1114000/4848000] [229.8/1000.0] [batch_t 0.334 (0.330)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 16:25:31,172 - Train: 22.98% [1114100/4848000] [229.8/1000.0] [batch_t 0.326 (0.330)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-07 16:26:04,170 - Train: 22.98% [1114200/4848000] [229.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 16:26:37,329 - Train: 22.98% [1114300/4848000] [229.8/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 16:27:10,437 - Train: 22.99% [1114400/4848000] [229.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 16:27:43,452 - Train: 22.99% [1114500/4848000] [229.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 16:28:16,456 - Train: 22.99% [1114600/4848000] [229.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 16:28:49,464 - Train: 22.99% [1114700/4848000] [229.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 16:29:22,449 - Train: 23.00% [1114800/4848000] [230.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 16:29:55,439 - Train: 23.00% [1114900/4848000] [230.0/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 16:30:28,432 - Train: 23.00% [1115000/4848000] [230.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 16:30:41,648 - ==> Total time: 4 days, 17:04:40 Eta: 15 days, 18:33:53 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 16:31:03,578 - Train: 23.00% [1115100/4848000] [230.0/1000.0] [batch_t 0.335 (0.331)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-07 16:31:36,688 - Train: 23.00% [1115200/4848000] [230.0/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 16:32:09,765 - Train: 23.01% [1115300/4848000] [230.1/1000.0] [batch_t 0.335 (0.331)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-07 16:32:42,769 - Train: 23.01% [1115400/4848000] [230.1/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 16:33:15,817 - Train: 23.01% [1115500/4848000] [230.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 16:33:49,032 - Train: 23.01% [1115600/4848000] [230.1/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 16:34:23,126 - Train: 23.01% [1115700/4848000] [230.1/1000.0] [batch_t 0.333 (0.341)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 16:34:56,101 - Train: 23.02% [1115800/4848000] [230.2/1000.0] [batch_t 0.335 (0.330)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-07 16:35:29,063 - Train: 23.02% [1115900/4848000] [230.2/1000.0] [batch_t 0.335 (0.330)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-07 16:36:02,107 - Train: 23.02% [1116000/4848000] [230.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 16:36:35,162 - Train: 23.02% [1116100/4848000] [230.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 16:37:08,189 - Train: 23.02% [1116200/4848000] [230.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 16:37:41,153 - Train: 23.03% [1116300/4848000] [230.3/1000.0] [batch_t 0.334 (0.330)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 16:38:14,158 - Train: 23.03% [1116400/4848000] [230.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 16:38:47,208 - Train: 23.03% [1116500/4848000] [230.3/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 16:39:20,270 - Train: 23.03% [1116600/4848000] [230.3/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 16:39:53,295 - Train: 23.03% [1116700/4848000] [230.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 16:40:26,230 - Train: 23.04% [1116800/4848000] [230.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 16:40:59,177 - Train: 23.04% [1116900/4848000] [230.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 16:41:32,253 - Train: 23.04% [1117000/4848000] [230.4/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 16:42:05,228 - Train: 23.04% [1117100/4848000] [230.4/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 16:42:38,174 - Train: 23.04% [1117200/4848000] [230.4/1000.0] [batch_t 0.337 (0.329)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-07 16:43:11,256 - Train: 23.05% [1117300/4848000] [230.5/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 16:43:44,146 - Train: 23.05% [1117400/4848000] [230.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 16:44:17,141 - Train: 23.05% [1117500/4848000] [230.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 16:44:50,111 - Train: 23.05% [1117600/4848000] [230.5/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 16:45:23,129 - Train: 23.05% [1117700/4848000] [230.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 16:45:56,097 - Train: 23.06% [1117800/4848000] [230.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 16:46:29,913 - Train: 23.06% [1117900/4848000] [230.6/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 16:47:02,889 - Train: 23.06% [1118000/4848000] [230.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 16:47:36,857 - Train: 23.06% [1118100/4848000] [230.6/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 16:48:09,831 - Train: 23.07% [1118200/4848000] [230.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 16:48:42,765 - Train: 23.07% [1118300/4848000] [230.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 16:49:17,014 - Train: 23.07% [1118400/4848000] [230.7/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 16:49:49,958 - Train: 23.07% [1118500/4848000] [230.7/1000.0] [batch_t 0.334 (0.329)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-07 16:50:22,993 - Train: 23.07% [1118600/4848000] [230.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 16:50:55,997 - Train: 23.08% [1118700/4848000] [230.8/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 16:51:30,109 - Train: 23.08% [1118800/4848000] [230.8/1000.0] [batch_t 0.328 (0.341)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 16:52:03,076 - Train: 23.08% [1118900/4848000] [230.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 16:52:37,208 - Train: 23.08% [1119000/4848000] [230.8/1000.0] [batch_t 0.331 (0.341)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 16:53:10,974 - Train: 23.08% [1119100/4848000] [230.8/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 16:53:43,910 - Train: 23.09% [1119200/4848000] [230.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 16:54:16,865 - Train: 23.09% [1119300/4848000] [230.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 16:54:49,810 - Train: 23.09% [1119400/4848000] [230.9/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 16:55:24,363 - Train: 23.09% [1119500/4848000] [230.9/1000.0] [batch_t 0.326 (0.345)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 16:55:57,266 - Train: 23.09% [1119600/4848000] [230.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 16:56:31,649 - Train: 23.10% [1119700/4848000] [231.0/1000.0] [batch_t 0.328 (0.344)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 16:57:04,604 - Train: 23.10% [1119800/4848000] [231.0/1000.0] [batch_t 0.360 (0.329)] [data_t 0.035] [optim_t 0.324] [lr 0.005000] 2024-04-07 16:57:33,675 - ==> Total time: 4 days, 17:31:32 Eta: 15 days, 17:55:38 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 16:57:39,645 - Train: 23.10% [1119900/4848000] [231.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 16:58:13,177 - Train: 23.10% [1120000/4848000] [231.0/1000.0] [batch_t 0.331 (0.335)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 16:58:46,076 - Train: 23.10% [1120100/4848000] [231.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 16:59:19,260 - Train: 23.11% [1120200/4848000] [231.1/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 16:59:52,203 - Train: 23.11% [1120300/4848000] [231.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 17:00:25,429 - Train: 23.11% [1120400/4848000] [231.1/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 17:00:58,349 - Train: 23.11% [1120500/4848000] [231.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 17:01:32,251 - Train: 23.11% [1120600/4848000] [231.1/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 17:02:07,090 - Train: 23.12% [1120700/4848000] [231.2/1000.0] [batch_t 2.269 (0.348)] [data_t 1.942] [optim_t 0.328] [lr 0.005000] 2024-04-07 17:02:40,526 - Train: 23.12% [1120800/4848000] [231.2/1000.0] [batch_t 0.329 (0.334)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 17:03:13,487 - Train: 23.12% [1120900/4848000] [231.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 17:03:46,435 - Train: 23.12% [1121000/4848000] [231.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 17:04:19,410 - Train: 23.12% [1121100/4848000] [231.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 17:04:52,478 - Train: 23.13% [1121200/4848000] [231.3/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 17:05:26,823 - Train: 23.13% [1121300/4848000] [231.3/1000.0] [batch_t 0.330 (0.343)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 17:05:59,697 - Train: 23.13% [1121400/4848000] [231.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 17:06:32,578 - Train: 23.13% [1121500/4848000] [231.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 17:07:05,799 - Train: 23.14% [1121600/4848000] [231.4/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 17:07:38,789 - Train: 23.14% [1121700/4848000] [231.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 17:08:11,723 - Train: 23.14% [1121800/4848000] [231.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 17:08:44,682 - Train: 23.14% [1121900/4848000] [231.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 17:09:17,753 - Train: 23.14% [1122000/4848000] [231.4/1000.0] [batch_t 0.326 (0.331)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 17:09:50,629 - Train: 23.15% [1122100/4848000] [231.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 17:10:23,736 - Train: 23.15% [1122200/4848000] [231.5/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 17:10:56,564 - Train: 23.15% [1122300/4848000] [231.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 17:11:30,780 - Train: 23.15% [1122400/4848000] [231.5/1000.0] [batch_t 0.330 (0.342)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 17:12:03,603 - Train: 23.15% [1122500/4848000] [231.5/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 17:12:37,976 - Train: 23.16% [1122600/4848000] [231.6/1000.0] [batch_t 0.329 (0.344)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 17:13:10,872 - Train: 23.16% [1122700/4848000] [231.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 17:13:43,848 - Train: 23.16% [1122800/4848000] [231.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 17:14:16,805 - Train: 23.16% [1122900/4848000] [231.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 17:14:49,750 - Train: 23.16% [1123000/4848000] [231.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 17:15:23,744 - Train: 23.17% [1123100/4848000] [231.7/1000.0] [batch_t 0.329 (0.340)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 17:15:56,740 - Train: 23.17% [1123200/4848000] [231.7/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 17:16:31,330 - Train: 23.17% [1123300/4848000] [231.7/1000.0] [batch_t 0.330 (0.346)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 17:17:04,273 - Train: 23.17% [1123400/4848000] [231.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 17:17:39,863 - Train: 23.17% [1123500/4848000] [231.7/1000.0] [batch_t 0.331 (0.356)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 17:18:14,025 - Train: 23.18% [1123600/4848000] [231.8/1000.0] [batch_t 0.331 (0.342)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 17:18:47,150 - Train: 23.18% [1123700/4848000] [231.8/1000.0] [batch_t 0.327 (0.331)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 17:19:20,202 - Train: 23.18% [1123800/4848000] [231.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 17:19:53,252 - Train: 23.18% [1123900/4848000] [231.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 17:20:27,571 - Train: 23.18% [1124000/4848000] [231.8/1000.0] [batch_t 0.327 (0.343)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 17:21:00,699 - Train: 23.19% [1124100/4848000] [231.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 17:21:37,474 - Train: 23.19% [1124200/4848000] [231.9/1000.0] [batch_t 0.331 (0.368)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 17:22:11,831 - Train: 23.19% [1124300/4848000] [231.9/1000.0] [batch_t 0.331 (0.343)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 17:22:44,832 - Train: 23.19% [1124400/4848000] [231.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 17:23:18,713 - Train: 23.20% [1124500/4848000] [232.0/1000.0] [batch_t 0.327 (0.339)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 17:23:51,761 - Train: 23.20% [1124600/4848000] [232.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 17:24:26,254 - Train: 23.20% [1124700/4848000] [232.0/1000.0] [batch_t 0.331 (0.345)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 17:24:38,169 - ==> Total time: 4 days, 17:58:36 Eta: 15 days, 17:18:09 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 17:25:01,418 - Train: 23.20% [1124800/4848000] [232.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 17:25:37,055 - Train: 23.20% [1124900/4848000] [232.0/1000.0] [batch_t 0.332 (0.356)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 17:26:10,060 - Train: 23.21% [1125000/4848000] [232.1/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 17:26:43,035 - Train: 23.21% [1125100/4848000] [232.1/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 17:27:17,589 - Train: 23.21% [1125200/4848000] [232.1/1000.0] [batch_t 0.330 (0.345)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 17:27:50,570 - Train: 23.21% [1125300/4848000] [232.1/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 17:28:25,268 - Train: 23.21% [1125400/4848000] [232.1/1000.0] [batch_t 0.329 (0.347)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 17:28:58,323 - Train: 23.22% [1125500/4848000] [232.2/1000.0] [batch_t 0.325 (0.330)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-07 17:29:32,711 - Train: 23.22% [1125600/4848000] [232.2/1000.0] [batch_t 0.327 (0.344)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 17:30:08,216 - Train: 23.22% [1125700/4848000] [232.2/1000.0] [batch_t 0.329 (0.355)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 17:30:41,130 - Train: 23.22% [1125800/4848000] [232.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 17:31:17,282 - Train: 23.22% [1125900/4848000] [232.2/1000.0] [batch_t 0.328 (0.361)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 17:31:50,181 - Train: 23.23% [1126000/4848000] [232.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 17:32:25,085 - Train: 23.23% [1126100/4848000] [232.3/1000.0] [batch_t 0.328 (0.349)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 17:32:57,962 - Train: 23.23% [1126200/4848000] [232.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 17:33:32,984 - Train: 23.23% [1126300/4848000] [232.3/1000.0] [batch_t 0.330 (0.350)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 17:34:07,354 - Train: 23.23% [1126400/4848000] [232.3/1000.0] [batch_t 0.333 (0.344)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 17:34:40,283 - Train: 23.24% [1126500/4848000] [232.4/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 17:35:13,626 - Train: 23.24% [1126600/4848000] [232.4/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 17:35:46,509 - Train: 23.24% [1126700/4848000] [232.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 17:36:19,963 - Train: 23.24% [1126800/4848000] [232.4/1000.0] [batch_t 0.324 (0.334)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 17:36:52,823 - Train: 23.24% [1126900/4848000] [232.4/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 17:37:25,835 - Train: 23.25% [1127000/4848000] [232.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 17:37:58,685 - Train: 23.25% [1127100/4848000] [232.5/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 17:38:31,811 - Train: 23.25% [1127200/4848000] [232.5/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 17:39:04,710 - Train: 23.25% [1127300/4848000] [232.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 17:39:37,585 - Train: 23.25% [1127400/4848000] [232.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 17:40:10,575 - Train: 23.26% [1127500/4848000] [232.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 17:40:43,483 - Train: 23.26% [1127600/4848000] [232.6/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 17:41:16,361 - Train: 23.26% [1127700/4848000] [232.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 17:41:49,307 - Train: 23.26% [1127800/4848000] [232.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 17:42:22,237 - Train: 23.27% [1127900/4848000] [232.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 17:42:55,172 - Train: 23.27% [1128000/4848000] [232.7/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 17:43:28,046 - Train: 23.27% [1128100/4848000] [232.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 17:44:00,976 - Train: 23.27% [1128200/4848000] [232.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 17:44:33,959 - Train: 23.27% [1128300/4848000] [232.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 17:45:06,857 - Train: 23.28% [1128400/4848000] [232.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 17:45:39,769 - Train: 23.28% [1128500/4848000] [232.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 17:46:12,688 - Train: 23.28% [1128600/4848000] [232.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 17:46:45,572 - Train: 23.28% [1128700/4848000] [232.8/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 17:47:18,428 - Train: 23.28% [1128800/4848000] [232.8/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 17:47:51,329 - Train: 23.29% [1128900/4848000] [232.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 17:48:24,244 - Train: 23.29% [1129000/4848000] [232.9/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-07 17:48:57,116 - Train: 23.29% [1129100/4848000] [232.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 17:49:30,080 - Train: 23.29% [1129200/4848000] [232.9/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 17:50:03,000 - Train: 23.29% [1129300/4848000] [232.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 17:50:35,971 - Train: 23.30% [1129400/4848000] [233.0/1000.0] [batch_t 0.336 (0.330)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-07 17:51:08,881 - Train: 23.30% [1129500/4848000] [233.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 17:51:36,526 - ==> Total time: 4 days, 18:25:35 Eta: 15 days, 16:40:26 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 17:51:44,137 - Train: 23.30% [1129600/4848000] [233.0/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 17:52:16,983 - Train: 23.30% [1129700/4848000] [233.0/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 17:52:49,825 - Train: 23.30% [1129800/4848000] [233.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 17:53:22,738 - Train: 23.31% [1129900/4848000] [233.1/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 17:53:55,660 - Train: 23.31% [1130000/4848000] [233.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 17:54:28,594 - Train: 23.31% [1130100/4848000] [233.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 17:55:01,440 - Train: 23.31% [1130200/4848000] [233.1/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 17:55:34,298 - Train: 23.31% [1130300/4848000] [233.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 17:56:08,648 - Train: 23.32% [1130400/4848000] [233.2/1000.0] [batch_t 0.330 (0.343)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 17:56:41,563 - Train: 23.32% [1130500/4848000] [233.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 17:57:14,694 - Train: 23.32% [1130600/4848000] [233.2/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 17:57:47,618 - Train: 23.32% [1130700/4848000] [233.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 17:58:20,490 - Train: 23.33% [1130800/4848000] [233.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 17:58:53,444 - Train: 23.33% [1130900/4848000] [233.3/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 17:59:26,348 - Train: 23.33% [1131000/4848000] [233.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 17:59:59,405 - Train: 23.33% [1131100/4848000] [233.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 18:00:32,338 - Train: 23.33% [1131200/4848000] [233.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 18:01:05,202 - Train: 23.34% [1131300/4848000] [233.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 18:01:38,097 - Train: 23.34% [1131400/4848000] [233.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 18:02:11,003 - Train: 23.34% [1131500/4848000] [233.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 18:02:43,861 - Train: 23.34% [1131600/4848000] [233.4/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 18:03:16,743 - Train: 23.34% [1131700/4848000] [233.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 18:03:49,669 - Train: 23.35% [1131800/4848000] [233.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 18:04:22,675 - Train: 23.35% [1131900/4848000] [233.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 18:04:55,672 - Train: 23.35% [1132000/4848000] [233.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 18:05:28,665 - Train: 23.35% [1132100/4848000] [233.5/1000.0] [batch_t 0.326 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 18:06:01,670 - Train: 23.35% [1132200/4848000] [233.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 18:06:34,668 - Train: 23.36% [1132300/4848000] [233.6/1000.0] [batch_t 0.337 (0.330)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-07 18:07:07,705 - Train: 23.36% [1132400/4848000] [233.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 18:07:40,924 - Train: 23.36% [1132500/4848000] [233.6/1000.0] [batch_t 0.331 (0.332)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 18:08:14,051 - Train: 23.36% [1132600/4848000] [233.6/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 18:08:47,106 - Train: 23.36% [1132700/4848000] [233.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 18:09:20,205 - Train: 23.37% [1132800/4848000] [233.7/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 18:09:53,262 - Train: 23.37% [1132900/4848000] [233.7/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 18:10:26,251 - Train: 23.37% [1133000/4848000] [233.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 18:10:59,226 - Train: 23.37% [1133100/4848000] [233.7/1000.0] [batch_t 0.337 (0.330)] [data_t 0.003] [optim_t 0.335] [lr 0.005000] 2024-04-07 18:11:32,254 - Train: 23.37% [1133200/4848000] [233.7/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 18:12:05,287 - Train: 23.38% [1133300/4848000] [233.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 18:12:38,320 - Train: 23.38% [1133400/4848000] [233.8/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 18:13:11,292 - Train: 23.38% [1133500/4848000] [233.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 18:13:44,265 - Train: 23.38% [1133600/4848000] [233.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 18:14:17,331 - Train: 23.38% [1133700/4848000] [233.8/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 18:14:50,354 - Train: 23.39% [1133800/4848000] [233.9/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 18:15:23,483 - Train: 23.39% [1133900/4848000] [233.9/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 18:15:56,418 - Train: 23.39% [1134000/4848000] [233.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 18:16:29,382 - Train: 23.39% [1134100/4848000] [233.9/1000.0] [batch_t 0.336 (0.330)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-07 18:17:02,330 - Train: 23.40% [1134200/4848000] [234.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 18:17:35,259 - Train: 23.40% [1134300/4848000] [234.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 18:18:08,173 - Train: 23.40% [1134400/4848000] [234.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 18:18:18,783 - ==> Total time: 4 days, 18:52:17 Eta: 15 days, 16:01:56 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 18:18:43,188 - Train: 23.40% [1134500/4848000] [234.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 18:19:16,101 - Train: 23.40% [1134600/4848000] [234.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 18:19:49,147 - Train: 23.41% [1134700/4848000] [234.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 18:20:22,059 - Train: 23.41% [1134800/4848000] [234.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 18:20:55,017 - Train: 23.41% [1134900/4848000] [234.1/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 18:21:27,919 - Train: 23.41% [1135000/4848000] [234.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 18:22:00,828 - Train: 23.41% [1135100/4848000] [234.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 18:22:33,886 - Train: 23.42% [1135200/4848000] [234.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 18:23:06,820 - Train: 23.42% [1135300/4848000] [234.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 18:23:39,704 - Train: 23.42% [1135400/4848000] [234.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 18:24:12,666 - Train: 23.42% [1135500/4848000] [234.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 18:24:45,699 - Train: 23.42% [1135600/4848000] [234.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 18:25:18,726 - Train: 23.43% [1135700/4848000] [234.3/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 18:25:51,733 - Train: 23.43% [1135800/4848000] [234.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 18:26:24,722 - Train: 23.43% [1135900/4848000] [234.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 18:26:57,671 - Train: 23.43% [1136000/4848000] [234.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 18:27:30,620 - Train: 23.43% [1136100/4848000] [234.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 18:28:03,548 - Train: 23.44% [1136200/4848000] [234.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 18:28:36,560 - Train: 23.44% [1136300/4848000] [234.4/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 18:29:09,473 - Train: 23.44% [1136400/4848000] [234.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 18:29:42,437 - Train: 23.44% [1136500/4848000] [234.4/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 18:30:15,440 - Train: 23.44% [1136600/4848000] [234.4/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 18:30:48,364 - Train: 23.45% [1136700/4848000] [234.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 18:31:21,312 - Train: 23.45% [1136800/4848000] [234.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 18:31:54,221 - Train: 23.45% [1136900/4848000] [234.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 18:32:27,195 - Train: 23.45% [1137000/4848000] [234.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 18:32:59,979 - Train: 23.46% [1137100/4848000] [234.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 18:33:32,805 - Train: 23.46% [1137200/4848000] [234.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 18:34:05,627 - Train: 23.46% [1137300/4848000] [234.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 18:34:38,470 - Train: 23.46% [1137400/4848000] [234.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 18:35:11,322 - Train: 23.46% [1137500/4848000] [234.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 18:35:44,302 - Train: 23.47% [1137600/4848000] [234.7/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 18:36:17,237 - Train: 23.47% [1137700/4848000] [234.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 18:36:50,186 - Train: 23.47% [1137800/4848000] [234.7/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 18:37:23,128 - Train: 23.47% [1137900/4848000] [234.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 18:37:56,034 - Train: 23.47% [1138000/4848000] [234.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 18:38:29,339 - Train: 23.48% [1138100/4848000] [234.8/1000.0] [batch_t 0.332 (0.333)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 18:39:05,030 - Train: 23.48% [1138200/4848000] [234.8/1000.0] [batch_t 0.327 (0.357)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 18:39:37,965 - Train: 23.48% [1138300/4848000] [234.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 18:40:10,916 - Train: 23.48% [1138400/4848000] [234.8/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 18:40:43,940 - Train: 23.48% [1138500/4848000] [234.8/1000.0] [batch_t 0.333 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 18:41:16,889 - Train: 23.49% [1138600/4848000] [234.9/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 18:41:49,802 - Train: 23.49% [1138700/4848000] [234.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 18:42:22,737 - Train: 23.49% [1138800/4848000] [234.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 18:42:55,743 - Train: 23.49% [1138900/4848000] [234.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 18:43:28,727 - Train: 23.49% [1139000/4848000] [234.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 18:44:01,758 - Train: 23.50% [1139100/4848000] [235.0/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 18:44:34,798 - Train: 23.50% [1139200/4848000] [235.0/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 18:45:01,275 - ==> Total time: 4 days, 19:18:59 Eta: 15 days, 15:23:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 18:45:11,649 - Train: 23.50% [1139300/4848000] [235.0/1000.0] [batch_t 0.330 (0.345)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 18:45:44,776 - Train: 23.50% [1139400/4848000] [235.0/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 18:46:17,771 - Train: 23.50% [1139500/4848000] [235.0/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 18:46:50,785 - Train: 23.51% [1139600/4848000] [235.1/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 18:47:23,743 - Train: 23.51% [1139700/4848000] [235.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 18:47:56,741 - Train: 23.51% [1139800/4848000] [235.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 18:48:29,706 - Train: 23.51% [1139900/4848000] [235.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 18:49:02,672 - Train: 23.51% [1140000/4848000] [235.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 18:49:35,632 - Train: 23.52% [1140100/4848000] [235.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 18:50:08,676 - Train: 23.52% [1140200/4848000] [235.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 18:50:41,592 - Train: 23.52% [1140300/4848000] [235.2/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 18:51:14,560 - Train: 23.52% [1140400/4848000] [235.2/1000.0] [batch_t 0.324 (0.330)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-07 18:51:47,499 - Train: 23.53% [1140500/4848000] [235.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 18:52:20,576 - Train: 23.53% [1140600/4848000] [235.3/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 18:52:53,484 - Train: 23.53% [1140700/4848000] [235.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 18:53:26,651 - Train: 23.53% [1140800/4848000] [235.3/1000.0] [batch_t 0.331 (0.332)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 18:53:59,582 - Train: 23.53% [1140900/4848000] [235.3/1000.0] [batch_t 0.325 (0.329)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-07 18:54:32,538 - Train: 23.54% [1141000/4848000] [235.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 18:55:05,517 - Train: 23.54% [1141100/4848000] [235.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 18:55:38,523 - Train: 23.54% [1141200/4848000] [235.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 18:56:11,541 - Train: 23.54% [1141300/4848000] [235.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 18:56:44,452 - Train: 23.54% [1141400/4848000] [235.4/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 18:57:17,469 - Train: 23.55% [1141500/4848000] [235.5/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 18:57:50,416 - Train: 23.55% [1141600/4848000] [235.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 18:58:23,389 - Train: 23.55% [1141700/4848000] [235.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 18:58:56,437 - Train: 23.55% [1141800/4848000] [235.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-07 18:59:29,382 - Train: 23.55% [1141900/4848000] [235.5/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 19:00:02,399 - Train: 23.56% [1142000/4848000] [235.6/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 19:00:35,384 - Train: 23.56% [1142100/4848000] [235.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 19:01:08,415 - Train: 23.56% [1142200/4848000] [235.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:01:41,384 - Train: 23.56% [1142300/4848000] [235.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:02:14,318 - Train: 23.56% [1142400/4848000] [235.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 19:02:47,278 - Train: 23.57% [1142500/4848000] [235.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 19:03:20,261 - Train: 23.57% [1142600/4848000] [235.7/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 19:03:53,233 - Train: 23.57% [1142700/4848000] [235.7/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 19:04:26,223 - Train: 23.57% [1142800/4848000] [235.7/1000.0] [batch_t 0.335 (0.330)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-07 19:04:59,150 - Train: 23.57% [1142900/4848000] [235.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 19:05:32,077 - Train: 23.58% [1143000/4848000] [235.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:06:05,068 - Train: 23.58% [1143100/4848000] [235.8/1000.0] [batch_t 0.342 (0.330)] [data_t 0.003] [optim_t 0.339] [lr 0.005000] 2024-04-07 19:06:38,106 - Train: 23.58% [1143200/4848000] [235.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:07:11,094 - Train: 23.58% [1143300/4848000] [235.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 19:07:44,040 - Train: 23.58% [1143400/4848000] [235.8/1000.0] [batch_t 0.333 (0.329)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 19:08:17,015 - Train: 23.59% [1143500/4848000] [235.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:08:49,987 - Train: 23.59% [1143600/4848000] [235.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 19:09:23,090 - Train: 23.59% [1143700/4848000] [235.9/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:09:56,075 - Train: 23.59% [1143800/4848000] [235.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:10:28,984 - Train: 23.60% [1143900/4848000] [236.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:11:01,921 - Train: 23.60% [1144000/4848000] [236.0/1000.0] [batch_t 0.332 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 19:11:34,883 - Train: 23.60% [1144100/4848000] [236.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:11:44,102 - ==> Total time: 4 days, 19:45:42 Eta: 15 days, 14:45:15 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 19:12:09,712 - Train: 23.60% [1144200/4848000] [236.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:12:42,725 - Train: 23.60% [1144300/4848000] [236.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:13:15,644 - Train: 23.61% [1144400/4848000] [236.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-07 19:13:48,622 - Train: 23.61% [1144500/4848000] [236.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:14:21,568 - Train: 23.61% [1144600/4848000] [236.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 19:14:54,470 - Train: 23.61% [1144700/4848000] [236.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 19:15:27,409 - Train: 23.61% [1144800/4848000] [236.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 19:16:00,319 - Train: 23.62% [1144900/4848000] [236.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:16:33,446 - Train: 23.62% [1145000/4848000] [236.2/1000.0] [batch_t 0.325 (0.331)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 19:17:06,386 - Train: 23.62% [1145100/4848000] [236.2/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 19:17:39,370 - Train: 23.62% [1145200/4848000] [236.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 19:18:12,301 - Train: 23.62% [1145300/4848000] [236.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:18:45,256 - Train: 23.63% [1145400/4848000] [236.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:19:18,202 - Train: 23.63% [1145500/4848000] [236.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 19:19:51,168 - Train: 23.63% [1145600/4848000] [236.3/1000.0] [batch_t 0.337 (0.330)] [data_t 0.003] [optim_t 0.335] [lr 0.005000] 2024-04-07 19:20:24,232 - Train: 23.63% [1145700/4848000] [236.3/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:20:57,170 - Train: 23.63% [1145800/4848000] [236.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 19:21:30,174 - Train: 23.64% [1145900/4848000] [236.4/1000.0] [batch_t 0.333 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-07 19:22:03,245 - Train: 23.64% [1146000/4848000] [236.4/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-07 19:22:36,123 - Train: 23.64% [1146100/4848000] [236.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 19:23:09,590 - Train: 23.64% [1146200/4848000] [236.4/1000.0] [batch_t 0.328 (0.335)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:23:42,558 - Train: 23.64% [1146300/4848000] [236.4/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 19:24:16,124 - Train: 23.65% [1146400/4848000] [236.5/1000.0] [batch_t 0.334 (0.336)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 19:24:49,007 - Train: 23.65% [1146500/4848000] [236.5/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 19:25:21,967 - Train: 23.65% [1146600/4848000] [236.5/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 19:25:54,813 - Train: 23.65% [1146700/4848000] [236.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:26:27,698 - Train: 23.66% [1146800/4848000] [236.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 19:27:00,571 - Train: 23.66% [1146900/4848000] [236.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:27:33,591 - Train: 23.66% [1147000/4848000] [236.6/1000.0] [batch_t 0.322 (0.330)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-07 19:28:07,679 - Train: 23.66% [1147100/4848000] [236.6/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 19:28:40,569 - Train: 23.66% [1147200/4848000] [236.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:29:13,559 - Train: 23.67% [1147300/4848000] [236.7/1000.0] [batch_t 0.322 (0.330)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 19:29:46,412 - Train: 23.67% [1147400/4848000] [236.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:30:19,245 - Train: 23.67% [1147500/4848000] [236.7/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:30:52,100 - Train: 23.67% [1147600/4848000] [236.7/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:31:24,965 - Train: 23.67% [1147700/4848000] [236.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:31:57,836 - Train: 23.68% [1147800/4848000] [236.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:32:32,409 - Train: 23.68% [1147900/4848000] [236.8/1000.0] [batch_t 0.330 (0.346)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 19:33:06,712 - Train: 23.68% [1148000/4848000] [236.8/1000.0] [batch_t 0.331 (0.343)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 19:33:39,630 - Train: 23.68% [1148100/4848000] [236.8/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 19:34:13,314 - Train: 23.68% [1148200/4848000] [236.8/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:34:46,133 - Train: 23.69% [1148300/4848000] [236.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 19:35:19,048 - Train: 23.69% [1148400/4848000] [236.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:35:51,882 - Train: 23.69% [1148500/4848000] [236.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:36:26,523 - Train: 23.69% [1148600/4848000] [236.9/1000.0] [batch_t 0.329 (0.346)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:36:59,459 - Train: 23.69% [1148700/4848000] [236.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:37:33,525 - Train: 23.70% [1148800/4848000] [237.0/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:38:06,398 - Train: 23.70% [1148900/4848000] [237.0/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 19:38:31,347 - ==> Total time: 4 days, 20:12:29 Eta: 15 days, 14:07:19 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 19:38:41,188 - Train: 23.70% [1149000/4848000] [237.0/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 19:39:14,058 - Train: 23.70% [1149100/4848000] [237.0/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 19:39:47,053 - Train: 23.70% [1149200/4848000] [237.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:40:19,919 - Train: 23.71% [1149300/4848000] [237.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:40:52,788 - Train: 23.71% [1149400/4848000] [237.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 19:41:25,710 - Train: 23.71% [1149500/4848000] [237.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:41:58,553 - Train: 23.71% [1149600/4848000] [237.1/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 19:42:31,494 - Train: 23.71% [1149700/4848000] [237.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:43:04,314 - Train: 23.72% [1149800/4848000] [237.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 19:43:37,146 - Train: 23.72% [1149900/4848000] [237.2/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 19:44:10,096 - Train: 23.72% [1150000/4848000] [237.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 19:44:42,970 - Train: 23.72% [1150100/4848000] [237.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:45:15,828 - Train: 23.73% [1150200/4848000] [237.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:45:48,688 - Train: 23.73% [1150300/4848000] [237.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 19:46:21,567 - Train: 23.73% [1150400/4848000] [237.3/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 19:46:54,463 - Train: 23.73% [1150500/4848000] [237.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 19:47:27,363 - Train: 23.73% [1150600/4848000] [237.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:48:00,216 - Train: 23.74% [1150700/4848000] [237.4/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 19:48:33,028 - Train: 23.74% [1150800/4848000] [237.4/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 19:49:05,850 - Train: 23.74% [1150900/4848000] [237.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:49:38,667 - Train: 23.74% [1151000/4848000] [237.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:50:11,508 - Train: 23.74% [1151100/4848000] [237.4/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 19:50:44,381 - Train: 23.75% [1151200/4848000] [237.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 19:51:17,182 - Train: 23.75% [1151300/4848000] [237.5/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 19:51:50,004 - Train: 23.75% [1151400/4848000] [237.5/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:52:24,816 - Train: 23.75% [1151500/4848000] [237.5/1000.0] [batch_t 0.330 (0.348)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 19:52:57,680 - Train: 23.75% [1151600/4848000] [237.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 19:53:32,055 - Train: 23.76% [1151700/4848000] [237.6/1000.0] [batch_t 0.330 (0.344)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 19:54:04,882 - Train: 23.76% [1151800/4848000] [237.6/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 19:54:37,891 - Train: 23.76% [1151900/4848000] [237.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 19:55:11,448 - Train: 23.76% [1152000/4848000] [237.6/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 19:55:44,368 - Train: 23.76% [1152100/4848000] [237.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 19:56:19,551 - Train: 23.77% [1152200/4848000] [237.7/1000.0] [batch_t 0.326 (0.352)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-07 19:56:52,417 - Train: 23.77% [1152300/4848000] [237.7/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 19:57:26,319 - Train: 23.77% [1152400/4848000] [237.7/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 19:57:59,206 - Train: 23.77% [1152500/4848000] [237.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 19:58:35,293 - Train: 23.77% [1152600/4848000] [237.7/1000.0] [batch_t 0.329 (0.361)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 19:59:10,723 - Train: 23.78% [1152700/4848000] [237.8/1000.0] [batch_t 0.324 (0.354)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 19:59:43,589 - Train: 23.78% [1152800/4848000] [237.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 20:00:22,179 - Train: 23.78% [1152900/4848000] [237.8/1000.0] [batch_t 0.326 (0.386)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 20:00:55,058 - Train: 23.78% [1153000/4848000] [237.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 20:01:31,563 - Train: 23.79% [1153100/4848000] [237.9/1000.0] [batch_t 0.325 (0.365)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 20:02:04,440 - Train: 23.79% [1153200/4848000] [237.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 20:02:38,508 - Train: 23.79% [1153300/4848000] [237.9/1000.0] [batch_t 0.324 (0.340)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 20:03:13,836 - Train: 23.79% [1153400/4848000] [237.9/1000.0] [batch_t 0.334 (0.353)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 20:03:46,834 - Train: 23.79% [1153500/4848000] [237.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 20:04:24,677 - Train: 23.80% [1153600/4848000] [238.0/1000.0] [batch_t 0.327 (0.378)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 20:04:57,582 - Train: 23.80% [1153700/4848000] [238.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 20:05:30,572 - Train: 23.80% [1153800/4848000] [238.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 20:05:38,462 - ==> Total time: 4 days, 20:39:36 Eta: 15 days, 13:30:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 20:06:07,818 - Train: 23.80% [1153900/4848000] [238.0/1000.0] [batch_t 0.326 (0.359)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 20:06:40,712 - Train: 23.80% [1154000/4848000] [238.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:07:22,469 - Train: 23.81% [1154100/4848000] [238.1/1000.0] [batch_t 0.718 (0.417)] [data_t 0.390] [optim_t 0.329] [lr 0.005000] 2024-04-07 20:07:55,377 - Train: 23.81% [1154200/4848000] [238.1/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 20:08:51,473 - Train: 23.81% [1154300/4848000] [238.1/1000.0] [batch_t 0.329 (0.561)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:09:46,059 - Train: 23.81% [1154400/4848000] [238.1/1000.0] [batch_t 0.329 (0.546)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:10:21,788 - Train: 23.81% [1154500/4848000] [238.1/1000.0] [batch_t 0.323 (0.357)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 20:10:54,661 - Train: 23.82% [1154600/4848000] [238.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:11:27,716 - Train: 23.82% [1154700/4848000] [238.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:12:00,605 - Train: 23.82% [1154800/4848000] [238.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:12:33,643 - Train: 23.82% [1154900/4848000] [238.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:13:08,795 - Train: 23.82% [1155000/4848000] [238.2/1000.0] [batch_t 0.330 (0.351)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 20:13:41,677 - Train: 23.83% [1155100/4848000] [238.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 20:14:14,604 - Train: 23.83% [1155200/4848000] [238.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 20:14:47,439 - Train: 23.83% [1155300/4848000] [238.3/1000.0] [batch_t 0.321 (0.328)] [data_t 0.002] [optim_t 0.319] [lr 0.005000] 2024-04-07 20:15:22,229 - Train: 23.83% [1155400/4848000] [238.3/1000.0] [batch_t 0.328 (0.348)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:15:55,168 - Train: 23.83% [1155500/4848000] [238.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 20:16:29,718 - Train: 23.84% [1155600/4848000] [238.4/1000.0] [batch_t 0.329 (0.345)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 20:17:02,566 - Train: 23.84% [1155700/4848000] [238.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:17:37,420 - Train: 23.84% [1155800/4848000] [238.4/1000.0] [batch_t 0.329 (0.348)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:18:10,335 - Train: 23.84% [1155900/4848000] [238.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 20:18:43,204 - Train: 23.84% [1156000/4848000] [238.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:19:16,233 - Train: 23.85% [1156100/4848000] [238.5/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 20:19:49,097 - Train: 23.85% [1156200/4848000] [238.5/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 20:20:24,327 - Train: 23.85% [1156300/4848000] [238.5/1000.0] [batch_t 0.329 (0.352)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:20:57,240 - Train: 23.85% [1156400/4848000] [238.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:21:31,994 - Train: 23.86% [1156500/4848000] [238.6/1000.0] [batch_t 0.331 (0.347)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 20:22:05,695 - Train: 23.86% [1156600/4848000] [238.6/1000.0] [batch_t 1.101 (0.337)] [data_t 0.774] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:22:40,492 - Train: 23.86% [1156700/4848000] [238.6/1000.0] [batch_t 0.331 (0.348)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 20:23:16,245 - Train: 23.86% [1156800/4848000] [238.6/1000.0] [batch_t 0.328 (0.357)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:23:49,185 - Train: 23.86% [1156900/4848000] [238.6/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 20:24:22,401 - Train: 23.87% [1157000/4848000] [238.7/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 20:24:55,178 - Train: 23.87% [1157100/4848000] [238.7/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 20:25:31,041 - Train: 23.87% [1157200/4848000] [238.7/1000.0] [batch_t 0.330 (0.359)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 20:26:03,920 - Train: 23.87% [1157300/4848000] [238.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:26:40,647 - Train: 23.87% [1157400/4848000] [238.7/1000.0] [batch_t 0.329 (0.367)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:27:14,978 - Train: 23.88% [1157500/4848000] [238.8/1000.0] [batch_t 0.327 (0.343)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:27:47,812 - Train: 23.88% [1157600/4848000] [238.8/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 20:28:23,888 - Train: 23.88% [1157700/4848000] [238.8/1000.0] [batch_t 0.329 (0.361)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:28:56,771 - Train: 23.88% [1157800/4848000] [238.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 20:29:32,761 - Train: 23.88% [1157900/4848000] [238.8/1000.0] [batch_t 0.331 (0.360)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 20:30:07,316 - Train: 23.89% [1158000/4848000] [238.9/1000.0] [batch_t 0.329 (0.345)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:30:40,188 - Train: 23.89% [1158100/4848000] [238.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:31:14,031 - Train: 23.89% [1158200/4848000] [238.9/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 20:31:46,890 - Train: 23.89% [1158300/4848000] [238.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:32:21,132 - Train: 23.89% [1158400/4848000] [238.9/1000.0] [batch_t 0.328 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:32:57,339 - Train: 23.90% [1158500/4848000] [239.0/1000.0] [batch_t 0.330 (0.362)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 20:33:31,456 - Train: 23.90% [1158600/4848000] [239.0/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 20:33:55,111 - ==> Total time: 4 days, 21:07:53 Eta: 15 days, 12:57:31 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 20:34:07,550 - Train: 23.90% [1158700/4848000] [239.0/1000.0] [batch_t 0.329 (0.375)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 20:34:40,403 - Train: 23.90% [1158800/4848000] [239.0/1000.0] [batch_t 0.333 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 20:35:13,254 - Train: 23.90% [1158900/4848000] [239.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:35:46,195 - Train: 23.91% [1159000/4848000] [239.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:36:19,041 - Train: 23.91% [1159100/4848000] [239.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:36:51,861 - Train: 23.91% [1159200/4848000] [239.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 20:37:25,001 - Train: 23.91% [1159300/4848000] [239.1/1000.0] [batch_t 0.333 (0.331)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 20:37:57,798 - Train: 23.92% [1159400/4848000] [239.2/1000.0] [batch_t 0.324 (0.328)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 20:38:30,693 - Train: 23.92% [1159500/4848000] [239.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 20:39:03,533 - Train: 23.92% [1159600/4848000] [239.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 20:39:36,662 - Train: 23.92% [1159700/4848000] [239.2/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 20:40:09,492 - Train: 23.92% [1159800/4848000] [239.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:40:42,303 - Train: 23.93% [1159900/4848000] [239.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:41:16,284 - Train: 23.93% [1160000/4848000] [239.3/1000.0] [batch_t 0.325 (0.340)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 20:41:49,102 - Train: 23.93% [1160100/4848000] [239.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 20:42:22,296 - Train: 23.93% [1160200/4848000] [239.3/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 20:42:55,117 - Train: 23.93% [1160300/4848000] [239.3/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 20:43:27,968 - Train: 23.94% [1160400/4848000] [239.4/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 20:44:00,927 - Train: 23.94% [1160500/4848000] [239.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 20:44:35,492 - Train: 23.94% [1160600/4848000] [239.4/1000.0] [batch_t 0.328 (0.346)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:45:08,390 - Train: 23.94% [1160700/4848000] [239.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:45:41,420 - Train: 23.94% [1160800/4848000] [239.4/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 20:46:14,485 - Train: 23.95% [1160900/4848000] [239.5/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 20:46:47,353 - Train: 23.95% [1161000/4848000] [239.5/1000.0] [batch_t 0.325 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 20:47:21,264 - Train: 23.95% [1161100/4848000] [239.5/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 20:47:54,072 - Train: 23.95% [1161200/4848000] [239.5/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 20:48:28,098 - Train: 23.95% [1161300/4848000] [239.5/1000.0] [batch_t 0.333 (0.340)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 20:49:00,951 - Train: 23.96% [1161400/4848000] [239.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 20:49:33,909 - Train: 23.96% [1161500/4848000] [239.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 20:50:07,633 - Train: 23.96% [1161600/4848000] [239.6/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 20:50:40,469 - Train: 23.96% [1161700/4848000] [239.6/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 20:51:15,019 - Train: 23.96% [1161800/4848000] [239.6/1000.0] [batch_t 0.326 (0.345)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 20:51:47,880 - Train: 23.97% [1161900/4848000] [239.7/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:52:21,539 - Train: 23.97% [1162000/4848000] [239.7/1000.0] [batch_t 0.326 (0.337)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 20:52:54,394 - Train: 23.97% [1162100/4848000] [239.7/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 20:53:28,688 - Train: 23.97% [1162200/4848000] [239.7/1000.0] [batch_t 0.333 (0.343)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 20:54:01,507 - Train: 23.97% [1162300/4848000] [239.7/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 20:54:35,937 - Train: 23.98% [1162400/4848000] [239.8/1000.0] [batch_t 0.328 (0.344)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:55:10,798 - Train: 23.98% [1162500/4848000] [239.8/1000.0] [batch_t 0.329 (0.349)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:55:43,755 - Train: 23.98% [1162600/4848000] [239.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 20:56:19,507 - Train: 23.98% [1162700/4848000] [239.8/1000.0] [batch_t 0.329 (0.357)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 20:56:52,412 - Train: 23.99% [1162800/4848000] [239.9/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 20:57:27,107 - Train: 23.99% [1162900/4848000] [239.9/1000.0] [batch_t 0.327 (0.347)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 20:57:59,985 - Train: 23.99% [1163000/4848000] [239.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 20:58:35,082 - Train: 23.99% [1163100/4848000] [239.9/1000.0] [batch_t 0.329 (0.351)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 20:59:10,374 - Train: 23.99% [1163200/4848000] [239.9/1000.0] [batch_t 0.330 (0.353)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 20:59:43,344 - Train: 24.00% [1163300/4848000] [240.0/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-07 21:00:18,185 - Train: 24.00% [1163400/4848000] [240.0/1000.0] [batch_t 0.325 (0.348)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 21:00:51,197 - Train: 24.00% [1163500/4848000] [240.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 21:00:57,800 - ==> Total time: 4 days, 21:34:56 Eta: 15 days, 12:20:38 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 21:01:27,813 - Train: 24.00% [1163600/4848000] [240.0/1000.0] [batch_t 0.330 (0.348)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 21:02:00,828 - Train: 24.00% [1163700/4848000] [240.0/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 21:02:34,105 - Train: 24.01% [1163800/4848000] [240.1/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 21:03:08,177 - Train: 24.01% [1163900/4848000] [240.1/1000.0] [batch_t 0.334 (0.341)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 21:03:41,311 - Train: 24.01% [1164000/4848000] [240.1/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 21:04:16,061 - Train: 24.01% [1164100/4848000] [240.1/1000.0] [batch_t 0.336 (0.347)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-07 21:04:49,053 - Train: 24.01% [1164200/4848000] [240.1/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 21:05:23,459 - Train: 24.02% [1164300/4848000] [240.2/1000.0] [batch_t 0.335 (0.344)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-07 21:05:56,491 - Train: 24.02% [1164400/4848000] [240.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 21:06:31,304 - Train: 24.02% [1164500/4848000] [240.2/1000.0] [batch_t 0.330 (0.348)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 21:07:04,286 - Train: 24.02% [1164600/4848000] [240.2/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 21:07:38,667 - Train: 24.02% [1164700/4848000] [240.2/1000.0] [batch_t 0.327 (0.344)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 21:08:13,890 - Train: 24.03% [1164800/4848000] [240.3/1000.0] [batch_t 0.338 (0.352)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-07 21:08:46,878 - Train: 24.03% [1164900/4848000] [240.3/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-07 21:09:19,875 - Train: 24.03% [1165000/4848000] [240.3/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 21:09:52,826 - Train: 24.03% [1165100/4848000] [240.3/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 21:10:27,075 - Train: 24.03% [1165200/4848000] [240.3/1000.0] [batch_t 0.330 (0.342)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 21:11:00,108 - Train: 24.04% [1165300/4848000] [240.4/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-07 21:11:35,414 - Train: 24.04% [1165400/4848000] [240.4/1000.0] [batch_t 0.330 (0.353)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 21:12:09,134 - Train: 24.04% [1165500/4848000] [240.4/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 21:12:42,057 - Train: 24.04% [1165600/4848000] [240.4/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-07 21:13:16,024 - Train: 24.04% [1165700/4848000] [240.4/1000.0] [batch_t 0.331 (0.340)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 21:13:49,041 - Train: 24.05% [1165800/4848000] [240.5/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 21:14:22,044 - Train: 24.05% [1165900/4848000] [240.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 21:14:54,989 - Train: 24.05% [1166000/4848000] [240.5/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 21:15:29,855 - Train: 24.05% [1166100/4848000] [240.5/1000.0] [batch_t 0.330 (0.349)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 21:16:02,905 - Train: 24.06% [1166200/4848000] [240.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 21:16:37,120 - Train: 24.06% [1166300/4848000] [240.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 21:17:10,966 - Train: 24.06% [1166400/4848000] [240.6/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 21:17:43,910 - Train: 24.06% [1166500/4848000] [240.6/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 21:18:16,848 - Train: 24.06% [1166600/4848000] [240.6/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-07 21:18:49,856 - Train: 24.07% [1166700/4848000] [240.7/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 21:19:22,761 - Train: 24.07% [1166800/4848000] [240.7/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 21:19:55,741 - Train: 24.07% [1166900/4848000] [240.7/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 21:20:28,739 - Train: 24.07% [1167000/4848000] [240.7/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 21:21:01,794 - Train: 24.07% [1167100/4848000] [240.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 21:21:35,848 - Train: 24.08% [1167200/4848000] [240.8/1000.0] [batch_t 0.331 (0.340)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 21:22:09,847 - Train: 24.08% [1167300/4848000] [240.8/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 21:22:42,899 - Train: 24.08% [1167400/4848000] [240.8/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 21:23:16,100 - Train: 24.08% [1167500/4848000] [240.8/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 21:23:49,141 - Train: 24.08% [1167600/4848000] [240.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 21:24:21,963 - Train: 24.09% [1167700/4848000] [240.9/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 21:24:54,786 - Train: 24.09% [1167800/4848000] [240.9/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 21:25:28,431 - Train: 24.09% [1167900/4848000] [240.9/1000.0] [batch_t 0.325 (0.336)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 21:26:01,273 - Train: 24.09% [1168000/4848000] [240.9/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 21:26:34,105 - Train: 24.09% [1168100/4848000] [240.9/1000.0] [batch_t 0.326 (0.328)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 21:27:06,940 - Train: 24.10% [1168200/4848000] [241.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 21:27:39,805 - Train: 24.10% [1168300/4848000] [241.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 21:28:02,139 - ==> Total time: 4 days, 22:02:00 Eta: 15 days, 11:43:55 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 21:28:16,765 - Train: 24.10% [1168400/4848000] [241.0/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 21:28:49,561 - Train: 24.10% [1168500/4848000] [241.0/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 21:29:23,592 - Train: 24.10% [1168600/4848000] [241.0/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 21:29:56,425 - Train: 24.11% [1168700/4848000] [241.1/1000.0] [batch_t 0.332 (0.328)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 21:30:29,259 - Train: 24.11% [1168800/4848000] [241.1/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 21:31:02,278 - Train: 24.11% [1168900/4848000] [241.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 21:31:35,080 - Train: 24.11% [1169000/4848000] [241.1/1000.0] [batch_t 0.330 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 21:32:09,527 - Train: 24.12% [1169100/4848000] [241.2/1000.0] [batch_t 0.330 (0.344)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 21:32:42,414 - Train: 24.12% [1169200/4848000] [241.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 21:33:15,240 - Train: 24.12% [1169300/4848000] [241.2/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 21:33:48,037 - Train: 24.12% [1169400/4848000] [241.2/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 21:34:20,866 - Train: 24.12% [1169500/4848000] [241.2/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 21:34:53,679 - Train: 24.13% [1169600/4848000] [241.3/1000.0] [batch_t 0.331 (0.328)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 21:35:26,480 - Train: 24.13% [1169700/4848000] [241.3/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 21:35:59,289 - Train: 24.13% [1169800/4848000] [241.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 21:36:32,087 - Train: 24.13% [1169900/4848000] [241.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 21:37:04,900 - Train: 24.13% [1170000/4848000] [241.3/1000.0] [batch_t 0.329 (0.328)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 21:37:38,162 - Train: 24.14% [1170100/4848000] [241.4/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 21:38:10,991 - Train: 24.14% [1170200/4848000] [241.4/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 21:38:43,909 - Train: 24.14% [1170300/4848000] [241.4/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 21:39:17,724 - Train: 24.14% [1170400/4848000] [241.4/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 21:39:50,485 - Train: 24.14% [1170500/4848000] [241.4/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 21:40:24,169 - Train: 24.15% [1170600/4848000] [241.5/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 21:40:56,954 - Train: 24.15% [1170700/4848000] [241.5/1000.0] [batch_t 0.325 (0.328)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 21:41:29,844 - Train: 24.15% [1170800/4848000] [241.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 21:42:08,688 - Train: 24.15% [1170900/4848000] [241.5/1000.0] [batch_t 0.330 (0.388)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 21:42:41,596 - Train: 24.15% [1171000/4848000] [241.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 21:43:14,444 - Train: 24.16% [1171100/4848000] [241.6/1000.0] [batch_t 0.328 (0.328)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 21:43:49,873 - Train: 24.16% [1171200/4848000] [241.6/1000.0] [batch_t 0.329 (0.354)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 21:44:23,629 - Train: 24.16% [1171300/4848000] [241.6/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 21:44:56,552 - Train: 24.16% [1171400/4848000] [241.6/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 21:45:29,514 - Train: 24.16% [1171500/4848000] [241.6/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 21:46:02,437 - Train: 24.17% [1171600/4848000] [241.7/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-07 21:46:35,478 - Train: 24.17% [1171700/4848000] [241.7/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 21:47:08,382 - Train: 24.17% [1171800/4848000] [241.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 21:47:45,761 - Train: 24.17% [1171900/4848000] [241.7/1000.0] [batch_t 0.328 (0.374)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 21:48:18,628 - Train: 24.17% [1172000/4848000] [241.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 21:48:51,502 - Train: 24.18% [1172100/4848000] [241.8/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 21:49:24,464 - Train: 24.18% [1172200/4848000] [241.8/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 21:49:57,358 - Train: 24.18% [1172300/4848000] [241.8/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 21:50:30,372 - Train: 24.18% [1172400/4848000] [241.8/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 21:51:03,467 - Train: 24.19% [1172500/4848000] [241.9/1000.0] [batch_t 0.333 (0.331)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 21:51:36,501 - Train: 24.19% [1172600/4848000] [241.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 21:52:09,587 - Train: 24.19% [1172700/4848000] [241.9/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 21:52:42,673 - Train: 24.19% [1172800/4848000] [241.9/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 21:53:15,811 - Train: 24.19% [1172900/4848000] [241.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 21:53:48,865 - Train: 24.20% [1173000/4848000] [242.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 21:54:21,935 - Train: 24.20% [1173100/4848000] [242.0/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 21:54:55,215 - Train: 24.20% [1173200/4848000] [242.0/1000.0] [batch_t 0.338 (0.333)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-07 21:55:00,540 - ==> Total time: 4 days, 22:28:59 Eta: 15 days, 11:06:58 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 21:55:32,541 - Train: 24.20% [1173300/4848000] [242.0/1000.0] [batch_t 0.333 (0.338)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 21:56:05,553 - Train: 24.20% [1173400/4848000] [242.0/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 21:56:38,501 - Train: 24.21% [1173500/4848000] [242.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 21:57:11,495 - Train: 24.21% [1173600/4848000] [242.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 21:57:44,622 - Train: 24.21% [1173700/4848000] [242.1/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 21:58:17,601 - Train: 24.21% [1173800/4848000] [242.1/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 21:58:50,485 - Train: 24.21% [1173900/4848000] [242.1/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 21:59:23,452 - Train: 24.22% [1174000/4848000] [242.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 21:59:56,469 - Train: 24.22% [1174100/4848000] [242.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 22:00:29,462 - Train: 24.22% [1174200/4848000] [242.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 22:01:02,409 - Train: 24.22% [1174300/4848000] [242.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:01:35,430 - Train: 24.22% [1174400/4848000] [242.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 22:02:08,584 - Train: 24.23% [1174500/4848000] [242.3/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:02:41,688 - Train: 24.23% [1174600/4848000] [242.3/1000.0] [batch_t 0.333 (0.331)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 22:03:14,610 - Train: 24.23% [1174700/4848000] [242.3/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:03:47,562 - Train: 24.23% [1174800/4848000] [242.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 22:04:20,546 - Train: 24.23% [1174900/4848000] [242.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:04:54,228 - Train: 24.24% [1175000/4848000] [242.4/1000.0] [batch_t 0.332 (0.337)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 22:05:27,174 - Train: 24.24% [1175100/4848000] [242.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:06:00,145 - Train: 24.24% [1175200/4848000] [242.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:06:33,070 - Train: 24.24% [1175300/4848000] [242.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 22:07:06,065 - Train: 24.25% [1175400/4848000] [242.5/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 22:07:39,065 - Train: 24.25% [1175500/4848000] [242.5/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 22:08:12,151 - Train: 24.25% [1175600/4848000] [242.5/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:08:45,066 - Train: 24.25% [1175700/4848000] [242.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 22:09:18,063 - Train: 24.25% [1175800/4848000] [242.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 22:09:51,001 - Train: 24.26% [1175900/4848000] [242.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:10:24,068 - Train: 24.26% [1176000/4848000] [242.6/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:10:57,058 - Train: 24.26% [1176100/4848000] [242.6/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-07 22:11:29,984 - Train: 24.26% [1176200/4848000] [242.6/1000.0] [batch_t 0.331 (0.329)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-07 22:12:02,930 - Train: 24.26% [1176300/4848000] [242.6/1000.0] [batch_t 0.336 (0.329)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-07 22:12:35,870 - Train: 24.27% [1176400/4848000] [242.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 22:13:09,786 - Train: 24.27% [1176500/4848000] [242.7/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:13:42,747 - Train: 24.27% [1176600/4848000] [242.7/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:14:15,759 - Train: 24.27% [1176700/4848000] [242.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:14:48,747 - Train: 24.27% [1176800/4848000] [242.7/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 22:15:21,768 - Train: 24.28% [1176900/4848000] [242.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 22:15:54,713 - Train: 24.28% [1177000/4848000] [242.8/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 22:16:27,748 - Train: 24.28% [1177100/4848000] [242.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:17:00,822 - Train: 24.28% [1177200/4848000] [242.8/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:17:33,819 - Train: 24.28% [1177300/4848000] [242.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:18:06,975 - Train: 24.29% [1177400/4848000] [242.9/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:18:39,973 - Train: 24.29% [1177500/4848000] [242.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 22:19:12,957 - Train: 24.29% [1177600/4848000] [242.9/1000.0] [batch_t 0.341 (0.330)] [data_t 0.002] [optim_t 0.339] [lr 0.005000] 2024-04-07 22:19:45,931 - Train: 24.29% [1177700/4848000] [242.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:20:18,891 - Train: 24.29% [1177800/4848000] [242.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 22:20:51,907 - Train: 24.30% [1177900/4848000] [243.0/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 22:21:24,909 - Train: 24.30% [1178000/4848000] [243.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 22:21:46,039 - ==> Total time: 4 days, 22:55:44 Eta: 15 days, 10:29:27 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 22:22:00,206 - Train: 24.30% [1178100/4848000] [243.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:22:33,175 - Train: 24.30% [1178200/4848000] [243.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:23:06,194 - Train: 24.30% [1178300/4848000] [243.0/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-07 22:23:39,178 - Train: 24.31% [1178400/4848000] [243.1/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 22:24:12,105 - Train: 24.31% [1178500/4848000] [243.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:24:45,051 - Train: 24.31% [1178600/4848000] [243.1/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:25:18,193 - Train: 24.31% [1178700/4848000] [243.1/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 22:25:51,173 - Train: 24.32% [1178800/4848000] [243.2/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 22:26:24,109 - Train: 24.32% [1178900/4848000] [243.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:26:57,105 - Train: 24.32% [1179000/4848000] [243.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:27:30,078 - Train: 24.32% [1179100/4848000] [243.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 22:28:03,145 - Train: 24.32% [1179200/4848000] [243.2/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 22:28:36,232 - Train: 24.33% [1179300/4848000] [243.3/1000.0] [batch_t 0.325 (0.331)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 22:29:09,209 - Train: 24.33% [1179400/4848000] [243.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 22:29:42,193 - Train: 24.33% [1179500/4848000] [243.3/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 22:30:15,225 - Train: 24.33% [1179600/4848000] [243.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:30:48,235 - Train: 24.33% [1179700/4848000] [243.3/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-07 22:31:21,218 - Train: 24.34% [1179800/4848000] [243.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 22:31:54,257 - Train: 24.34% [1179900/4848000] [243.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:32:27,216 - Train: 24.34% [1180000/4848000] [243.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:33:00,335 - Train: 24.34% [1180100/4848000] [243.4/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:33:33,536 - Train: 24.34% [1180200/4848000] [243.4/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 22:34:06,459 - Train: 24.35% [1180300/4848000] [243.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:34:39,366 - Train: 24.35% [1180400/4848000] [243.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:35:12,321 - Train: 24.35% [1180500/4848000] [243.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 22:35:45,275 - Train: 24.35% [1180600/4848000] [243.5/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-07 22:36:18,348 - Train: 24.35% [1180700/4848000] [243.5/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 22:36:51,404 - Train: 24.36% [1180800/4848000] [243.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 22:37:24,402 - Train: 24.36% [1180900/4848000] [243.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 22:37:57,466 - Train: 24.36% [1181000/4848000] [243.6/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:38:30,511 - Train: 24.36% [1181100/4848000] [243.6/1000.0] [batch_t 0.348 (0.330)] [data_t 0.002] [optim_t 0.345] [lr 0.005000] 2024-04-07 22:39:03,553 - Train: 24.36% [1181200/4848000] [243.6/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 22:39:36,526 - Train: 24.37% [1181300/4848000] [243.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:40:09,557 - Train: 24.37% [1181400/4848000] [243.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 22:40:42,592 - Train: 24.37% [1181500/4848000] [243.7/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 22:41:15,808 - Train: 24.37% [1181600/4848000] [243.7/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 22:41:48,812 - Train: 24.38% [1181700/4848000] [243.8/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-07 22:42:21,817 - Train: 24.38% [1181800/4848000] [243.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:42:54,826 - Train: 24.38% [1181900/4848000] [243.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:43:27,898 - Train: 24.38% [1182000/4848000] [243.8/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 22:44:00,878 - Train: 24.38% [1182100/4848000] [243.8/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 22:44:33,837 - Train: 24.39% [1182200/4848000] [243.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:45:06,722 - Train: 24.39% [1182300/4848000] [243.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 22:45:39,715 - Train: 24.39% [1182400/4848000] [243.9/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 22:46:12,767 - Train: 24.39% [1182500/4848000] [243.9/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 22:46:45,662 - Train: 24.39% [1182600/4848000] [243.9/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 22:47:18,659 - Train: 24.40% [1182700/4848000] [244.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 22:47:51,618 - Train: 24.40% [1182800/4848000] [244.0/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 22:48:24,667 - Train: 24.40% [1182900/4848000] [244.0/1000.0] [batch_t 0.336 (0.330)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-07 22:48:28,636 - ==> Total time: 4 days, 23:22:27 Eta: 15 days, 9:51:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 22:49:00,025 - Train: 24.40% [1183000/4848000] [244.0/1000.0] [batch_t 0.334 (0.329)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 22:49:33,063 - Train: 24.40% [1183100/4848000] [244.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 22:50:06,199 - Train: 24.41% [1183200/4848000] [244.1/1000.0] [batch_t 0.333 (0.331)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 22:50:39,232 - Train: 24.41% [1183300/4848000] [244.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 22:51:12,373 - Train: 24.41% [1183400/4848000] [244.1/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 22:51:45,389 - Train: 24.41% [1183500/4848000] [244.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 22:52:18,466 - Train: 24.41% [1183600/4848000] [244.1/1000.0] [batch_t 0.325 (0.331)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 22:52:51,480 - Train: 24.42% [1183700/4848000] [244.2/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 22:53:24,540 - Train: 24.42% [1183800/4848000] [244.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 22:53:57,546 - Train: 24.42% [1183900/4848000] [244.2/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 22:54:30,505 - Train: 24.42% [1184000/4848000] [244.2/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 22:55:03,550 - Train: 24.42% [1184100/4848000] [244.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 22:55:36,498 - Train: 24.43% [1184200/4848000] [244.3/1000.0] [batch_t 0.324 (0.329)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 22:56:09,406 - Train: 24.43% [1184300/4848000] [244.3/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 22:56:42,489 - Train: 24.43% [1184400/4848000] [244.3/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 22:57:15,484 - Train: 24.43% [1184500/4848000] [244.3/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 22:57:48,376 - Train: 24.43% [1184600/4848000] [244.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 22:58:21,293 - Train: 24.44% [1184700/4848000] [244.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 22:58:54,243 - Train: 24.44% [1184800/4848000] [244.4/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-07 22:59:27,196 - Train: 24.44% [1184900/4848000] [244.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 23:00:00,169 - Train: 24.44% [1185000/4848000] [244.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 23:00:33,178 - Train: 24.45% [1185100/4848000] [244.5/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 23:01:06,237 - Train: 24.45% [1185200/4848000] [244.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 23:01:39,259 - Train: 24.45% [1185300/4848000] [244.5/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 23:02:12,265 - Train: 24.45% [1185400/4848000] [244.5/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 23:02:45,299 - Train: 24.45% [1185500/4848000] [244.5/1000.0] [batch_t 0.338 (0.330)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-07 23:03:18,346 - Train: 24.46% [1185600/4848000] [244.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 23:03:51,319 - Train: 24.46% [1185700/4848000] [244.6/1000.0] [batch_t 0.337 (0.330)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-07 23:04:24,371 - Train: 24.46% [1185800/4848000] [244.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:04:57,558 - Train: 24.46% [1185900/4848000] [244.6/1000.0] [batch_t 0.335 (0.332)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-07 23:05:30,612 - Train: 24.46% [1186000/4848000] [244.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 23:06:03,625 - Train: 24.47% [1186100/4848000] [244.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 23:06:36,681 - Train: 24.47% [1186200/4848000] [244.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 23:07:09,697 - Train: 24.47% [1186300/4848000] [244.7/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 23:07:42,694 - Train: 24.47% [1186400/4848000] [244.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 23:08:15,737 - Train: 24.47% [1186500/4848000] [244.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:08:48,719 - Train: 24.48% [1186600/4848000] [244.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 23:09:21,709 - Train: 24.48% [1186700/4848000] [244.8/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 23:09:54,742 - Train: 24.48% [1186800/4848000] [244.8/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 23:10:27,735 - Train: 24.48% [1186900/4848000] [244.8/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 23:11:00,746 - Train: 24.48% [1187000/4848000] [244.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 23:11:33,811 - Train: 24.49% [1187100/4848000] [244.9/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 23:12:06,844 - Train: 24.49% [1187200/4848000] [244.9/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 23:12:40,138 - Train: 24.49% [1187300/4848000] [244.9/1000.0] [batch_t 0.326 (0.333)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 23:13:13,157 - Train: 24.49% [1187400/4848000] [244.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 23:13:46,107 - Train: 24.49% [1187500/4848000] [244.9/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 23:14:19,097 - Train: 24.50% [1187600/4848000] [245.0/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 23:14:52,029 - Train: 24.50% [1187700/4848000] [245.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 23:15:11,903 - ==> Total time: 4 days, 23:49:10 Eta: 15 days, 9:14:23 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 23:15:27,254 - Train: 24.50% [1187800/4848000] [245.0/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 23:16:00,276 - Train: 24.50% [1187900/4848000] [245.0/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:16:33,929 - Train: 24.50% [1188000/4848000] [245.0/1000.0] [batch_t 0.331 (0.336)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 23:17:06,922 - Train: 24.51% [1188100/4848000] [245.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 23:17:39,883 - Train: 24.51% [1188200/4848000] [245.1/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:18:14,629 - Train: 24.51% [1188300/4848000] [245.1/1000.0] [batch_t 0.329 (0.347)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 23:18:47,644 - Train: 24.51% [1188400/4848000] [245.1/1000.0] [batch_t 0.337 (0.330)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-07 23:19:20,577 - Train: 24.52% [1188500/4848000] [245.2/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:19:53,798 - Train: 24.52% [1188600/4848000] [245.2/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 23:20:26,800 - Train: 24.52% [1188700/4848000] [245.2/1000.0] [batch_t 0.336 (0.330)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-07 23:20:59,756 - Train: 24.52% [1188800/4848000] [245.2/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 23:21:32,801 - Train: 24.52% [1188900/4848000] [245.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 23:22:05,771 - Train: 24.53% [1189000/4848000] [245.3/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 23:22:38,806 - Train: 24.53% [1189100/4848000] [245.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 23:23:11,795 - Train: 24.53% [1189200/4848000] [245.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 23:23:44,861 - Train: 24.53% [1189300/4848000] [245.3/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:24:17,874 - Train: 24.53% [1189400/4848000] [245.3/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 23:24:50,862 - Train: 24.54% [1189500/4848000] [245.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:25:23,959 - Train: 24.54% [1189600/4848000] [245.4/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 23:25:56,909 - Train: 24.54% [1189700/4848000] [245.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 23:26:29,860 - Train: 24.54% [1189800/4848000] [245.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 23:27:02,814 - Train: 24.54% [1189900/4848000] [245.4/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 23:27:37,118 - Train: 24.55% [1190000/4848000] [245.5/1000.0] [batch_t 0.328 (0.343)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:28:10,107 - Train: 24.55% [1190100/4848000] [245.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 23:28:43,115 - Train: 24.55% [1190200/4848000] [245.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 23:29:16,118 - Train: 24.55% [1190300/4848000] [245.5/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 23:29:49,089 - Train: 24.55% [1190400/4848000] [245.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 23:30:22,123 - Train: 24.56% [1190500/4848000] [245.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 23:30:55,092 - Train: 24.56% [1190600/4848000] [245.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:31:28,135 - Train: 24.56% [1190700/4848000] [245.6/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-07 23:32:01,123 - Train: 24.56% [1190800/4848000] [245.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:32:34,102 - Train: 24.56% [1190900/4848000] [245.6/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-07 23:33:07,080 - Train: 24.57% [1191000/4848000] [245.7/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-07 23:33:40,093 - Train: 24.57% [1191100/4848000] [245.7/1000.0] [batch_t 0.338 (0.330)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-07 23:34:13,088 - Train: 24.57% [1191200/4848000] [245.7/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-07 23:34:46,097 - Train: 24.57% [1191300/4848000] [245.7/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-07 23:35:19,154 - Train: 24.58% [1191400/4848000] [245.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 23:35:52,296 - Train: 24.58% [1191500/4848000] [245.8/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 23:36:25,344 - Train: 24.58% [1191600/4848000] [245.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 23:36:58,341 - Train: 24.58% [1191700/4848000] [245.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 23:37:31,333 - Train: 24.58% [1191800/4848000] [245.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 23:38:04,349 - Train: 24.59% [1191900/4848000] [245.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 23:38:37,270 - Train: 24.59% [1192000/4848000] [245.9/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 23:39:10,234 - Train: 24.59% [1192100/4848000] [245.9/1000.0] [batch_t 0.336 (0.330)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-07 23:39:43,153 - Train: 24.59% [1192200/4848000] [245.9/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 23:40:16,188 - Train: 24.59% [1192300/4848000] [245.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 23:40:49,142 - Train: 24.60% [1192400/4848000] [246.0/1000.0] [batch_t 0.333 (0.329)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 23:41:22,181 - Train: 24.60% [1192500/4848000] [246.0/1000.0] [batch_t 0.338 (0.330)] [data_t 0.002] [optim_t 0.337] [lr 0.005000] 2024-04-07 23:41:55,174 - Train: 24.60% [1192600/4848000] [246.0/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 23:41:57,819 - ==> Total time: 5 days, 0:15:56 Eta: 15 days, 8:37:08 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-07 23:42:30,523 - Train: 24.60% [1192700/4848000] [246.0/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 23:43:03,438 - Train: 24.60% [1192800/4848000] [246.0/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:43:37,432 - Train: 24.61% [1192900/4848000] [246.1/1000.0] [batch_t 0.326 (0.340)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 23:44:10,656 - Train: 24.61% [1193000/4848000] [246.1/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 23:44:43,738 - Train: 24.61% [1193100/4848000] [246.1/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 23:45:16,826 - Train: 24.61% [1193200/4848000] [246.1/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-07 23:45:49,834 - Train: 24.61% [1193300/4848000] [246.1/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-07 23:46:22,790 - Train: 24.62% [1193400/4848000] [246.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 23:46:55,760 - Train: 24.62% [1193500/4848000] [246.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 23:47:28,750 - Train: 24.62% [1193600/4848000] [246.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 23:48:01,737 - Train: 24.62% [1193700/4848000] [246.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 23:48:34,642 - Train: 24.62% [1193800/4848000] [246.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-07 23:49:07,516 - Train: 24.63% [1193900/4848000] [246.3/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 23:49:40,382 - Train: 24.63% [1194000/4848000] [246.3/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-07 23:50:14,222 - Train: 24.63% [1194100/4848000] [246.3/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 23:50:47,237 - Train: 24.63% [1194200/4848000] [246.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 23:51:20,185 - Train: 24.63% [1194300/4848000] [246.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 23:51:53,109 - Train: 24.64% [1194400/4848000] [246.4/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 23:52:26,017 - Train: 24.64% [1194500/4848000] [246.4/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 23:52:58,984 - Train: 24.64% [1194600/4848000] [246.4/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-07 23:53:31,927 - Train: 24.64% [1194700/4848000] [246.4/1000.0] [batch_t 0.338 (0.329)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-07 23:54:04,890 - Train: 24.65% [1194800/4848000] [246.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 23:54:37,802 - Train: 24.65% [1194900/4848000] [246.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 23:55:10,838 - Train: 24.65% [1195000/4848000] [246.5/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-07 23:55:43,791 - Train: 24.65% [1195100/4848000] [246.5/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 23:56:16,878 - Train: 24.65% [1195200/4848000] [246.5/1000.0] [batch_t 0.338 (0.331)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-07 23:56:49,951 - Train: 24.66% [1195300/4848000] [246.6/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-07 23:57:23,889 - Train: 24.66% [1195400/4848000] [246.6/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-07 23:57:57,140 - Train: 24.66% [1195500/4848000] [246.6/1000.0] [batch_t 0.339 (0.332)] [data_t 0.002] [optim_t 0.337] [lr 0.005000] 2024-04-07 23:58:30,895 - Train: 24.66% [1195600/4848000] [246.6/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-07 23:59:04,018 - Train: 24.66% [1195700/4848000] [246.6/1000.0] [batch_t 0.335 (0.331)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-07 23:59:37,152 - Train: 24.67% [1195800/4848000] [246.7/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 00:00:10,370 - Train: 24.67% [1195900/4848000] [246.7/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 00:00:43,621 - Train: 24.67% [1196000/4848000] [246.7/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 00:01:16,862 - Train: 24.67% [1196100/4848000] [246.7/1000.0] [batch_t 0.338 (0.332)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-08 00:01:50,190 - Train: 24.67% [1196200/4848000] [246.7/1000.0] [batch_t 0.341 (0.333)] [data_t 0.002] [optim_t 0.339] [lr 0.005000] 2024-04-08 00:02:23,472 - Train: 24.68% [1196300/4848000] [246.8/1000.0] [batch_t 0.339 (0.333)] [data_t 0.002] [optim_t 0.337] [lr 0.005000] 2024-04-08 00:02:56,692 - Train: 24.68% [1196400/4848000] [246.8/1000.0] [batch_t 0.337 (0.332)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-08 00:03:29,919 - Train: 24.68% [1196500/4848000] [246.8/1000.0] [batch_t 0.340 (0.332)] [data_t 0.002] [optim_t 0.338] [lr 0.005000] 2024-04-08 00:04:03,080 - Train: 24.68% [1196600/4848000] [246.8/1000.0] [batch_t 0.332 (0.332)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 00:04:38,833 - Train: 24.68% [1196700/4848000] [246.8/1000.0] [batch_t 0.334 (0.357)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 00:05:12,767 - Train: 24.69% [1196800/4848000] [246.9/1000.0] [batch_t 0.331 (0.339)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 00:05:45,773 - Train: 24.69% [1196900/4848000] [246.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 00:06:18,760 - Train: 24.69% [1197000/4848000] [246.9/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 00:06:51,784 - Train: 24.69% [1197100/4848000] [246.9/1000.0] [batch_t 0.336 (0.330)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-08 00:07:24,734 - Train: 24.69% [1197200/4848000] [246.9/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 00:07:57,729 - Train: 24.70% [1197300/4848000] [247.0/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 00:08:30,642 - Train: 24.70% [1197400/4848000] [247.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:08:49,052 - ==> Total time: 5 days, 0:42:47 Eta: 15 days, 8:00:15 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 00:09:05,700 - Train: 24.70% [1197500/4848000] [247.0/1000.0] [batch_t 0.335 (0.331)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 00:09:38,893 - Train: 24.70% [1197600/4848000] [247.0/1000.0] [batch_t 0.326 (0.332)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 00:10:12,177 - Train: 24.71% [1197700/4848000] [247.1/1000.0] [batch_t 0.341 (0.333)] [data_t 0.003] [optim_t 0.338] [lr 0.005000] 2024-04-08 00:10:45,379 - Train: 24.71% [1197800/4848000] [247.1/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 00:11:18,572 - Train: 24.71% [1197900/4848000] [247.1/1000.0] [batch_t 0.326 (0.332)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 00:11:51,829 - Train: 24.71% [1198000/4848000] [247.1/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 00:12:24,814 - Train: 24.71% [1198100/4848000] [247.1/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 00:12:58,038 - Train: 24.72% [1198200/4848000] [247.2/1000.0] [batch_t 0.337 (0.332)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-08 00:13:31,997 - Train: 24.72% [1198300/4848000] [247.2/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 00:14:05,442 - Train: 24.72% [1198400/4848000] [247.2/1000.0] [batch_t 0.643 (0.334)] [data_t 0.315] [optim_t 0.329] [lr 0.005000] 2024-04-08 00:14:38,408 - Train: 24.72% [1198500/4848000] [247.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:15:11,966 - Train: 24.72% [1198600/4848000] [247.2/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 00:15:44,841 - Train: 24.73% [1198700/4848000] [247.3/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 00:16:17,827 - Train: 24.73% [1198800/4848000] [247.3/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 00:16:50,826 - Train: 24.73% [1198900/4848000] [247.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 00:17:24,697 - Train: 24.73% [1199000/4848000] [247.3/1000.0] [batch_t 0.331 (0.339)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 00:17:57,755 - Train: 24.73% [1199100/4848000] [247.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 00:18:31,456 - Train: 24.74% [1199200/4848000] [247.4/1000.0] [batch_t 0.331 (0.337)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 00:19:04,477 - Train: 24.74% [1199300/4848000] [247.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 00:19:37,571 - Train: 24.74% [1199400/4848000] [247.4/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 00:20:10,679 - Train: 24.74% [1199500/4848000] [247.4/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 00:20:43,674 - Train: 24.74% [1199600/4848000] [247.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:21:16,756 - Train: 24.75% [1199700/4848000] [247.5/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 00:21:50,054 - Train: 24.75% [1199800/4848000] [247.5/1000.0] [batch_t 0.331 (0.333)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 00:22:23,130 - Train: 24.75% [1199900/4848000] [247.5/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 00:22:56,209 - Train: 24.75% [1200000/4848000] [247.5/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 00:23:30,252 - Train: 24.75% [1200100/4848000] [247.5/1000.0] [batch_t 0.333 (0.340)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 00:24:03,343 - Train: 24.76% [1200200/4848000] [247.6/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:24:37,221 - Train: 24.76% [1200300/4848000] [247.6/1000.0] [batch_t 0.331 (0.339)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 00:25:11,180 - Train: 24.76% [1200400/4848000] [247.6/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:25:44,188 - Train: 24.76% [1200500/4848000] [247.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:26:17,081 - Train: 24.76% [1200600/4848000] [247.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 00:26:50,110 - Train: 24.77% [1200700/4848000] [247.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:27:23,060 - Train: 24.77% [1200800/4848000] [247.7/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 00:27:56,074 - Train: 24.77% [1200900/4848000] [247.7/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 00:28:29,080 - Train: 24.77% [1201000/4848000] [247.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 00:29:02,095 - Train: 24.78% [1201100/4848000] [247.8/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 00:29:35,376 - Train: 24.78% [1201200/4848000] [247.8/1000.0] [batch_t 0.326 (0.333)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 00:30:09,406 - Train: 24.78% [1201300/4848000] [247.8/1000.0] [batch_t 0.331 (0.340)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 00:30:42,484 - Train: 24.78% [1201400/4848000] [247.8/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 00:31:17,335 - Train: 24.78% [1201500/4848000] [247.8/1000.0] [batch_t 0.335 (0.348)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 00:31:50,436 - Train: 24.79% [1201600/4848000] [247.9/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:32:24,473 - Train: 24.79% [1201700/4848000] [247.9/1000.0] [batch_t 0.330 (0.340)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 00:32:57,490 - Train: 24.79% [1201800/4848000] [247.9/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 00:33:31,233 - Train: 24.79% [1201900/4848000] [247.9/1000.0] [batch_t 0.325 (0.337)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 00:34:04,214 - Train: 24.79% [1202000/4848000] [247.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 00:34:38,052 - Train: 24.80% [1202100/4848000] [248.0/1000.0] [batch_t 0.332 (0.338)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 00:35:11,841 - Train: 24.80% [1202200/4848000] [248.0/1000.0] [batch_t 0.335 (0.338)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 00:35:44,848 - Train: 24.80% [1202300/4848000] [248.0/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 00:35:46,170 - ==> Total time: 5 days, 1:09:44 Eta: 15 days, 7:23:44 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 00:36:21,122 - Train: 24.80% [1202400/4848000] [248.0/1000.0] [batch_t 0.322 (0.342)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 00:36:54,078 - Train: 24.80% [1202500/4848000] [248.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 00:37:27,115 - Train: 24.81% [1202600/4848000] [248.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:38:00,060 - Train: 24.81% [1202700/4848000] [248.1/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 00:38:33,869 - Train: 24.81% [1202800/4848000] [248.1/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 00:39:07,958 - Train: 24.81% [1202900/4848000] [248.1/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 00:39:40,918 - Train: 24.81% [1203000/4848000] [248.1/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 00:40:14,741 - Train: 24.82% [1203100/4848000] [248.2/1000.0] [batch_t 0.332 (0.338)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 00:40:47,758 - Train: 24.82% [1203200/4848000] [248.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:41:21,801 - Train: 24.82% [1203300/4848000] [248.2/1000.0] [batch_t 0.327 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 00:41:54,717 - Train: 24.82% [1203400/4848000] [248.2/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:42:27,753 - Train: 24.82% [1203500/4848000] [248.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 00:43:00,751 - Train: 24.83% [1203600/4848000] [248.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:43:34,301 - Train: 24.83% [1203700/4848000] [248.3/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 00:44:08,213 - Train: 24.83% [1203800/4848000] [248.3/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 00:44:41,177 - Train: 24.83% [1203900/4848000] [248.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:45:14,265 - Train: 24.83% [1204000/4848000] [248.3/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 00:45:47,236 - Train: 24.84% [1204100/4848000] [248.4/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 00:46:21,337 - Train: 24.84% [1204200/4848000] [248.4/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 00:46:54,371 - Train: 24.84% [1204300/4848000] [248.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:47:27,716 - Train: 24.84% [1204400/4848000] [248.4/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:48:00,596 - Train: 24.85% [1204500/4848000] [248.5/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 00:48:33,468 - Train: 24.85% [1204600/4848000] [248.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 00:49:07,346 - Train: 24.85% [1204700/4848000] [248.5/1000.0] [batch_t 0.326 (0.339)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 00:49:40,264 - Train: 24.85% [1204800/4848000] [248.5/1000.0] [batch_t 0.332 (0.329)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 00:50:13,400 - Train: 24.85% [1204900/4848000] [248.5/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:50:46,296 - Train: 24.86% [1205000/4848000] [248.6/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:51:19,869 - Train: 24.86% [1205100/4848000] [248.6/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:51:52,968 - Train: 24.86% [1205200/4848000] [248.6/1000.0] [batch_t 0.335 (0.331)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 00:52:25,912 - Train: 24.86% [1205300/4848000] [248.6/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 00:52:59,018 - Train: 24.86% [1205400/4848000] [248.6/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 00:53:31,913 - Train: 24.87% [1205500/4848000] [248.7/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 00:54:05,752 - Train: 24.87% [1205600/4848000] [248.7/1000.0] [batch_t 1.239 (0.338)] [data_t 0.909] [optim_t 0.329] [lr 0.005000] 2024-04-08 00:54:38,688 - Train: 24.87% [1205700/4848000] [248.7/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:55:11,611 - Train: 24.87% [1205800/4848000] [248.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 00:55:44,524 - Train: 24.87% [1205900/4848000] [248.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 00:56:18,151 - Train: 24.88% [1206000/4848000] [248.8/1000.0] [batch_t 0.329 (0.336)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:56:51,149 - Train: 24.88% [1206100/4848000] [248.8/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 00:57:24,117 - Train: 24.88% [1206200/4848000] [248.8/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 00:57:56,992 - Train: 24.88% [1206300/4848000] [248.8/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 00:58:30,036 - Train: 24.88% [1206400/4848000] [248.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 00:59:03,007 - Train: 24.89% [1206500/4848000] [248.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 00:59:35,987 - Train: 24.89% [1206600/4848000] [248.9/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 01:00:09,010 - Train: 24.89% [1206700/4848000] [248.9/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 01:00:42,002 - Train: 24.89% [1206800/4848000] [248.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 01:01:15,109 - Train: 24.89% [1206900/4848000] [248.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 01:01:48,024 - Train: 24.90% [1207000/4848000] [249.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 01:02:20,967 - Train: 24.90% [1207100/4848000] [249.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 01:02:38,095 - ==> Total time: 5 days, 1:36:36 Eta: 15 days, 6:47:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 01:02:56,167 - Train: 24.90% [1207200/4848000] [249.0/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:03:29,104 - Train: 24.90% [1207300/4848000] [249.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 01:04:02,037 - Train: 24.91% [1207400/4848000] [249.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 01:04:35,041 - Train: 24.91% [1207500/4848000] [249.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 01:05:08,099 - Train: 24.91% [1207600/4848000] [249.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 01:05:41,054 - Train: 24.91% [1207700/4848000] [249.1/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 01:06:14,140 - Train: 24.91% [1207800/4848000] [249.1/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 01:06:47,132 - Train: 24.92% [1207900/4848000] [249.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 01:07:21,074 - Train: 24.92% [1208000/4848000] [249.2/1000.0] [batch_t 0.327 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:07:54,193 - Train: 24.92% [1208100/4848000] [249.2/1000.0] [batch_t 0.324 (0.331)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 01:08:27,350 - Train: 24.92% [1208200/4848000] [249.2/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:09:00,353 - Train: 24.92% [1208300/4848000] [249.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 01:09:33,411 - Train: 24.93% [1208400/4848000] [249.3/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 01:10:06,439 - Train: 24.93% [1208500/4848000] [249.3/1000.0] [batch_t 0.338 (0.330)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-08 01:10:39,407 - Train: 24.93% [1208600/4848000] [249.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 01:11:12,427 - Train: 24.93% [1208700/4848000] [249.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 01:11:45,400 - Train: 24.93% [1208800/4848000] [249.3/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 01:12:18,438 - Train: 24.94% [1208900/4848000] [249.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 01:12:51,380 - Train: 24.94% [1209000/4848000] [249.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:13:24,705 - Train: 24.94% [1209100/4848000] [249.4/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:13:57,775 - Train: 24.94% [1209200/4848000] [249.4/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:14:30,744 - Train: 24.94% [1209300/4848000] [249.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 01:15:03,763 - Train: 24.95% [1209400/4848000] [249.5/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 01:15:36,833 - Train: 24.95% [1209500/4848000] [249.5/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:16:09,911 - Train: 24.95% [1209600/4848000] [249.5/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 01:16:43,073 - Train: 24.95% [1209700/4848000] [249.5/1000.0] [batch_t 0.324 (0.332)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 01:17:16,055 - Train: 24.95% [1209800/4848000] [249.5/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:17:48,912 - Train: 24.96% [1209900/4848000] [249.6/1000.0] [batch_t 0.327 (0.328)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 01:18:21,805 - Train: 24.96% [1210000/4848000] [249.6/1000.0] [batch_t 0.337 (0.329)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-08 01:18:54,752 - Train: 24.96% [1210100/4848000] [249.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:19:27,679 - Train: 24.96% [1210200/4848000] [249.6/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 01:20:00,638 - Train: 24.96% [1210300/4848000] [249.6/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:20:33,622 - Train: 24.97% [1210400/4848000] [249.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 01:21:06,620 - Train: 24.97% [1210500/4848000] [249.7/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 01:21:39,684 - Train: 24.97% [1210600/4848000] [249.7/1000.0] [batch_t 0.340 (0.331)] [data_t 0.002] [optim_t 0.338] [lr 0.005000] 2024-04-08 01:22:12,705 - Train: 24.97% [1210700/4848000] [249.7/1000.0] [batch_t 0.338 (0.330)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-08 01:22:45,698 - Train: 24.98% [1210800/4848000] [249.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:23:18,712 - Train: 24.98% [1210900/4848000] [249.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 01:23:51,732 - Train: 24.98% [1211000/4848000] [249.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 01:24:25,880 - Train: 24.98% [1211100/4848000] [249.8/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:24:58,887 - Train: 24.98% [1211200/4848000] [249.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 01:25:33,073 - Train: 24.99% [1211300/4848000] [249.9/1000.0] [batch_t 0.327 (0.342)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 01:26:06,338 - Train: 24.99% [1211400/4848000] [249.9/1000.0] [batch_t 0.343 (0.333)] [data_t 0.002] [optim_t 0.341] [lr 0.005000] 2024-04-08 01:26:39,322 - Train: 24.99% [1211500/4848000] [249.9/1000.0] [batch_t 0.341 (0.330)] [data_t 0.002] [optim_t 0.339] [lr 0.005000] 2024-04-08 01:27:12,323 - Train: 24.99% [1211600/4848000] [249.9/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 01:27:45,284 - Train: 24.99% [1211700/4848000] [249.9/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 01:28:19,423 - Train: 25.00% [1211800/4848000] [250.0/1000.0] [batch_t 0.341 (0.341)] [data_t 0.002] [optim_t 0.339] [lr 0.005000] 2024-04-08 01:28:52,368 - Train: 25.00% [1211900/4848000] [250.0/1000.0] [batch_t 0.329 (0.329)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 01:29:25,412 - Train: 25.00% [1212000/4848000] [250.0/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 01:29:34,304 - Test: 16.13% [50/310] [batch_t 0.157 (0.161)] 2024-04-08 01:29:41,995 - Test: 32.26% [100/310] [batch_t 0.155 (0.157)] 2024-04-08 01:29:49,689 - Test: 48.39% [150/310] [batch_t 0.157 (0.156)] 2024-04-08 01:29:57,500 - Test: 64.52% [200/310] [batch_t 0.157 (0.156)] 2024-04-08 01:30:06,813 - Test: 80.65% [250/310] [batch_t 0.160 (0.162)] 2024-04-08 01:30:14,619 - Test: 96.77% [300/310] [batch_t 0.172 (0.161)] 2024-04-08 01:30:16,236 - Test: 100.00% [310/310] [batch_t 0.100 (0.161)] 2024-04-08 01:56:29,176 - ==> Metric Time for coco : 0.004 (mAUROC_sp_max) 0.001 (mAP_sp_max) 0.001 (mF1_max_sp_max) 379.342 (mAUROC_px) 286.800 (mAP_px) 34.951 (mF1_max_px) 794.924 (mAUPRO_px) 11.254 (mF1_px_0.2_0.8_0.1) 11.012 (mAcc_px_0.2_0.8_0.1) 10.941 (mIoU_px_0.2_0.8_0.1) 34.093 (mIoU_max_px) 2024-04-08 01:56:29,707 - | Name | mAUROC_sp_max | mAUROC_sp_max (Max) | mAP_sp_max | mAP_sp_max (Max) | mF1_max_sp_max | mF1_max_sp_max (Max) | mAUROC_px | mAUROC_px (Max) | mAP_px | mAP_px (Max) | mF1_max_px | mF1_max_px (Max) | mAUPRO_px | mAUPRO_px (Max) | mF1_px_0.2_0.8_0.1 | mF1_px_0.2_0.8_0.1 (Max) | mAcc_px_0.2_0.8_0.1 | mAcc_px_0.2_0.8_0.1 (Max) | mIoU_px_0.2_0.8_0.1 | mIoU_px_0.2_0.8_0.1 (Max) | mIoU_max_px | mIoU_max_px (Max) | |:------:|:---------------:|:---------------------:|:------------:|:------------------:|:----------------:|:----------------------:|:-----------:|:------------------:|:--------:|:-----------------:|:------------:|:------------------:|:-----------:|:------------------:|:--------------------:|:--------------------------:|:---------------------:|:---------------------------:|:---------------------:|:---------------------------:|:-------------:|:-------------------:| | coco | 55.736 | 58.404 (50 epoch) | 37.552 | 40.479 (50 epoch) | 51.758 | 51.858 (50 epoch) | 59.722 | 59.722 (250 epoch) | 8.696 | 8.714 (100 epoch) | 14.547 | 14.547 (250 epoch) | 29.178 | 34.736 (50 epoch) | 8.722 | 9.008 (50 epoch) | 38.883 | 38.999 (50 epoch) | 4.659 | 4.810 (50 epoch) | 7.844 | 7.844 (250 epoch) | 2024-04-08 01:56:30,341 - ==> Total time: 5 days, 2:30:28 Eta: 15 days, 7:31:26 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 01:57:49,493 - Train: 25.00% [1212100/4848000] [250.0/1000.0] [batch_t 0.759 (0.761)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-08 01:59:05,639 - Train: 25.00% [1212200/4848000] [250.0/1000.0] [batch_t 0.762 (0.761)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-08 02:00:21,792 - Train: 25.01% [1212300/4848000] [250.1/1000.0] [batch_t 0.759 (0.761)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-08 02:01:37,996 - Train: 25.01% [1212400/4848000] [250.1/1000.0] [batch_t 0.757 (0.762)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-08 02:02:54,074 - Train: 25.01% [1212500/4848000] [250.1/1000.0] [batch_t 0.750 (0.761)] [data_t 0.002] [optim_t 0.747] [lr 0.005000] 2024-04-08 02:04:10,168 - Train: 25.01% [1212600/4848000] [250.1/1000.0] [batch_t 0.758 (0.761)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-08 02:05:26,337 - Train: 25.01% [1212700/4848000] [250.1/1000.0] [batch_t 0.767 (0.762)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-08 02:06:42,548 - Train: 25.02% [1212800/4848000] [250.2/1000.0] [batch_t 0.762 (0.762)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-08 02:07:58,662 - Train: 25.02% [1212900/4848000] [250.2/1000.0] [batch_t 0.752 (0.761)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-08 02:09:14,757 - Train: 25.02% [1213000/4848000] [250.2/1000.0] [batch_t 0.760 (0.761)] [data_t 0.002] [optim_t 0.758] [lr 0.005000] 2024-04-08 02:10:31,032 - Train: 25.02% [1213100/4848000] [250.2/1000.0] [batch_t 0.763 (0.763)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-08 02:11:47,140 - Train: 25.02% [1213200/4848000] [250.2/1000.0] [batch_t 0.766 (0.761)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-08 02:13:03,269 - Train: 25.03% [1213300/4848000] [250.3/1000.0] [batch_t 0.763 (0.761)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-08 02:14:19,421 - Train: 25.03% [1213400/4848000] [250.3/1000.0] [batch_t 0.754 (0.761)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-08 02:15:35,519 - Train: 25.03% [1213500/4848000] [250.3/1000.0] [batch_t 0.760 (0.761)] [data_t 0.002] [optim_t 0.758] [lr 0.005000] 2024-04-08 02:16:51,623 - Train: 25.03% [1213600/4848000] [250.3/1000.0] [batch_t 0.754 (0.761)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-08 02:18:07,708 - Train: 25.04% [1213700/4848000] [250.4/1000.0] [batch_t 0.759 (0.761)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-08 02:19:23,880 - Train: 25.04% [1213800/4848000] [250.4/1000.0] [batch_t 0.764 (0.762)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-08 02:20:40,054 - Train: 25.04% [1213900/4848000] [250.4/1000.0] [batch_t 0.762 (0.762)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-08 02:21:56,158 - Train: 25.04% [1214000/4848000] [250.4/1000.0] [batch_t 0.762 (0.761)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-08 02:23:12,333 - Train: 25.04% [1214100/4848000] [250.4/1000.0] [batch_t 0.763 (0.762)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-08 02:24:28,431 - Train: 25.05% [1214200/4848000] [250.5/1000.0] [batch_t 0.755 (0.761)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-08 02:25:44,560 - Train: 25.05% [1214300/4848000] [250.5/1000.0] [batch_t 0.763 (0.761)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-08 02:27:00,639 - Train: 25.05% [1214400/4848000] [250.5/1000.0] [batch_t 0.763 (0.761)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-08 02:28:16,797 - Train: 25.05% [1214500/4848000] [250.5/1000.0] [batch_t 0.757 (0.761)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-08 02:29:32,988 - Train: 25.05% [1214600/4848000] [250.5/1000.0] [batch_t 0.754 (0.762)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-08 02:30:49,156 - Train: 25.06% [1214700/4848000] [250.6/1000.0] [batch_t 0.763 (0.762)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-08 02:32:05,296 - Train: 25.06% [1214800/4848000] [250.6/1000.0] [batch_t 0.752 (0.761)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-08 02:33:21,442 - Train: 25.06% [1214900/4848000] [250.6/1000.0] [batch_t 0.758 (0.761)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-08 02:34:37,570 - Train: 25.06% [1215000/4848000] [250.6/1000.0] [batch_t 0.759 (0.761)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-08 02:35:53,690 - Train: 25.06% [1215100/4848000] [250.6/1000.0] [batch_t 0.763 (0.761)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-08 02:37:09,793 - Train: 25.07% [1215200/4848000] [250.7/1000.0] [batch_t 0.759 (0.761)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-08 02:38:25,998 - Train: 25.07% [1215300/4848000] [250.7/1000.0] [batch_t 0.762 (0.762)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-08 02:39:42,085 - Train: 25.07% [1215400/4848000] [250.7/1000.0] [batch_t 0.767 (0.761)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-08 02:40:58,242 - Train: 25.07% [1215500/4848000] [250.7/1000.0] [batch_t 0.758 (0.761)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-08 02:42:14,387 - Train: 25.07% [1215600/4848000] [250.7/1000.0] [batch_t 0.760 (0.761)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-08 02:43:30,557 - Train: 25.08% [1215700/4848000] [250.8/1000.0] [batch_t 0.760 (0.762)] [data_t 0.002] [optim_t 0.758] [lr 0.005000] 2024-04-08 02:44:46,702 - Train: 25.08% [1215800/4848000] [250.8/1000.0] [batch_t 0.760 (0.761)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-08 02:46:02,863 - Train: 25.08% [1215900/4848000] [250.8/1000.0] [batch_t 0.759 (0.762)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-08 02:47:18,873 - Train: 25.08% [1216000/4848000] [250.8/1000.0] [batch_t 0.753 (0.760)] [data_t 0.002] [optim_t 0.750] [lr 0.005000] 2024-04-08 02:48:34,979 - Train: 25.08% [1216100/4848000] [250.8/1000.0] [batch_t 0.754 (0.761)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-08 02:49:51,099 - Train: 25.09% [1216200/4848000] [250.9/1000.0] [batch_t 0.758 (0.761)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-08 02:51:07,260 - Train: 25.09% [1216300/4848000] [250.9/1000.0] [batch_t 0.758 (0.762)] [data_t 0.002] [optim_t 0.755] [lr 0.005000] 2024-04-08 02:52:23,412 - Train: 25.09% [1216400/4848000] [250.9/1000.0] [batch_t 0.764 (0.761)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-08 02:53:39,594 - Train: 25.09% [1216500/4848000] [250.9/1000.0] [batch_t 0.763 (0.762)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-08 02:54:55,741 - Train: 25.09% [1216600/4848000] [250.9/1000.0] [batch_t 0.762 (0.761)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-08 02:56:11,966 - Train: 25.10% [1216700/4848000] [251.0/1000.0] [batch_t 0.753 (0.762)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-08 02:57:28,134 - Train: 25.10% [1216800/4848000] [251.0/1000.0] [batch_t 0.749 (0.762)] [data_t 0.002] [optim_t 0.747] [lr 0.005000] 2024-04-08 02:58:04,756 - ==> Total time: 5 days, 3:32:03 Eta: 15 days, 8:38:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 02:58:47,694 - Train: 25.10% [1216900/4848000] [251.0/1000.0] [batch_t 0.763 (0.762)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-08 03:00:03,846 - Train: 25.10% [1217000/4848000] [251.0/1000.0] [batch_t 0.759 (0.761)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-08 03:01:20,004 - Train: 25.11% [1217100/4848000] [251.1/1000.0] [batch_t 0.767 (0.761)] [data_t 0.002] [optim_t 0.765] [lr 0.005000] 2024-04-08 03:02:36,186 - Train: 25.11% [1217200/4848000] [251.1/1000.0] [batch_t 0.764 (0.762)] [data_t 0.003] [optim_t 0.761] [lr 0.005000] 2024-04-08 03:03:52,349 - Train: 25.11% [1217300/4848000] [251.1/1000.0] [batch_t 0.761 (0.762)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-08 03:05:08,495 - Train: 25.11% [1217400/4848000] [251.1/1000.0] [batch_t 0.767 (0.761)] [data_t 0.002] [optim_t 0.764] [lr 0.005000] 2024-04-08 03:06:24,729 - Train: 25.11% [1217500/4848000] [251.1/1000.0] [batch_t 0.759 (0.762)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-08 03:07:40,800 - Train: 25.12% [1217600/4848000] [251.2/1000.0] [batch_t 0.753 (0.761)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-08 03:08:56,965 - Train: 25.12% [1217700/4848000] [251.2/1000.0] [batch_t 0.754 (0.762)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-08 03:10:13,152 - Train: 25.12% [1217800/4848000] [251.2/1000.0] [batch_t 0.759 (0.762)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-08 03:11:29,274 - Train: 25.12% [1217900/4848000] [251.2/1000.0] [batch_t 0.759 (0.761)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-08 03:12:45,502 - Train: 25.12% [1218000/4848000] [251.2/1000.0] [batch_t 0.760 (0.762)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-08 03:14:01,624 - Train: 25.13% [1218100/4848000] [251.3/1000.0] [batch_t 0.753 (0.761)] [data_t 0.002] [optim_t 0.751] [lr 0.005000] 2024-04-08 03:15:17,770 - Train: 25.13% [1218200/4848000] [251.3/1000.0] [batch_t 0.759 (0.761)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-08 03:16:33,835 - Train: 25.13% [1218300/4848000] [251.3/1000.0] [batch_t 0.764 (0.761)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-08 03:17:49,945 - Train: 25.13% [1218400/4848000] [251.3/1000.0] [batch_t 0.763 (0.761)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-08 03:19:06,130 - Train: 25.13% [1218500/4848000] [251.3/1000.0] [batch_t 0.754 (0.762)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-08 03:20:22,316 - Train: 25.14% [1218600/4848000] [251.4/1000.0] [batch_t 0.754 (0.762)] [data_t 0.002] [optim_t 0.752] [lr 0.005000] 2024-04-08 03:21:38,524 - Train: 25.14% [1218700/4848000] [251.4/1000.0] [batch_t 0.759 (0.762)] [data_t 0.002] [optim_t 0.756] [lr 0.005000] 2024-04-08 03:22:54,640 - Train: 25.14% [1218800/4848000] [251.4/1000.0] [batch_t 0.759 (0.761)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-08 03:24:10,756 - Train: 25.14% [1218900/4848000] [251.4/1000.0] [batch_t 0.765 (0.761)] [data_t 0.002] [optim_t 0.762] [lr 0.005000] 2024-04-08 03:25:26,897 - Train: 25.14% [1219000/4848000] [251.4/1000.0] [batch_t 0.759 (0.761)] [data_t 0.002] [optim_t 0.757] [lr 0.005000] 2024-04-08 03:26:43,095 - Train: 25.15% [1219100/4848000] [251.5/1000.0] [batch_t 0.763 (0.762)] [data_t 0.002] [optim_t 0.760] [lr 0.005000] 2024-04-08 03:27:59,312 - Train: 25.15% [1219200/4848000] [251.5/1000.0] [batch_t 0.760 (0.762)] [data_t 0.003] [optim_t 0.757] [lr 0.005000] 2024-04-08 03:29:15,458 - Train: 25.15% [1219300/4848000] [251.5/1000.0] [batch_t 0.762 (0.761)] [data_t 0.002] [optim_t 0.759] [lr 0.005000] 2024-04-08 03:30:31,722 - Train: 25.15% [1219400/4848000] [251.5/1000.0] [batch_t 0.765 (0.763)] [data_t 0.002] [optim_t 0.763] [lr 0.005000] 2024-04-08 03:31:47,917 - Train: 25.15% [1219500/4848000] [251.5/1000.0] [batch_t 0.763 (0.762)] [data_t 0.002] [optim_t 0.761] [lr 0.005000] 2024-04-08 03:32:54,922 - Train: 25.16% [1219600/4848000] [251.6/1000.0] [batch_t 0.331 (0.670)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 03:33:28,011 - Train: 25.16% [1219700/4848000] [251.6/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 03:34:01,002 - Train: 25.16% [1219800/4848000] [251.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 03:34:33,999 - Train: 25.16% [1219900/4848000] [251.6/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 03:35:07,045 - Train: 25.17% [1220000/4848000] [251.7/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 03:35:40,093 - Train: 25.17% [1220100/4848000] [251.7/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 03:36:13,079 - Train: 25.17% [1220200/4848000] [251.7/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 03:36:46,105 - Train: 25.17% [1220300/4848000] [251.7/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 03:37:19,200 - Train: 25.17% [1220400/4848000] [251.7/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 03:37:52,227 - Train: 25.18% [1220500/4848000] [251.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 03:38:25,253 - Train: 25.18% [1220600/4848000] [251.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 03:38:58,383 - Train: 25.18% [1220700/4848000] [251.8/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 03:39:31,512 - Train: 25.18% [1220800/4848000] [251.8/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 03:40:04,696 - Train: 25.18% [1220900/4848000] [251.8/1000.0] [batch_t 0.332 (0.332)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 03:40:37,795 - Train: 25.19% [1221000/4848000] [251.9/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 03:41:10,922 - Train: 25.19% [1221100/4848000] [251.9/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 03:41:43,995 - Train: 25.19% [1221200/4848000] [251.9/1000.0] [batch_t 0.333 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 03:42:17,016 - Train: 25.19% [1221300/4848000] [251.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 03:42:50,040 - Train: 25.19% [1221400/4848000] [251.9/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 03:43:23,090 - Train: 25.20% [1221500/4848000] [252.0/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 03:43:56,188 - Train: 25.20% [1221600/4848000] [252.0/1000.0] [batch_t 0.335 (0.331)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 03:44:27,922 - ==> Total time: 5 days, 4:18:26 Eta: 15 days, 8:58:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 03:44:31,311 - Train: 25.20% [1221700/4848000] [252.0/1000.0] [batch_t 0.323 (0.332)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 03:45:04,301 - Train: 25.20% [1221800/4848000] [252.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 03:45:37,343 - Train: 25.20% [1221900/4848000] [252.0/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 03:46:10,482 - Train: 25.21% [1222000/4848000] [252.1/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 03:46:43,538 - Train: 25.21% [1222100/4848000] [252.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 03:47:16,742 - Train: 25.21% [1222200/4848000] [252.1/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 03:47:49,811 - Train: 25.21% [1222300/4848000] [252.1/1000.0] [batch_t 0.336 (0.331)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-08 03:48:23,670 - Train: 25.21% [1222400/4848000] [252.1/1000.0] [batch_t 0.338 (0.338)] [data_t 0.002] [optim_t 0.336] [lr 0.005000] 2024-04-08 03:48:56,697 - Train: 25.22% [1222500/4848000] [252.2/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 03:49:29,705 - Train: 25.22% [1222600/4848000] [252.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 03:50:02,713 - Train: 25.22% [1222700/4848000] [252.2/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 03:50:35,711 - Train: 25.22% [1222800/4848000] [252.2/1000.0] [batch_t 0.323 (0.330)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 03:51:09,674 - Train: 25.22% [1222900/4848000] [252.2/1000.0] [batch_t 0.332 (0.340)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 03:51:42,642 - Train: 25.23% [1223000/4848000] [252.3/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 03:52:16,403 - Train: 25.23% [1223100/4848000] [252.3/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 03:52:49,446 - Train: 25.23% [1223200/4848000] [252.3/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 03:53:22,494 - Train: 25.23% [1223300/4848000] [252.3/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 03:53:55,477 - Train: 25.24% [1223400/4848000] [252.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 03:54:28,503 - Train: 25.24% [1223500/4848000] [252.4/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 03:55:01,598 - Train: 25.24% [1223600/4848000] [252.4/1000.0] [batch_t 0.334 (0.331)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 03:55:34,635 - Train: 25.24% [1223700/4848000] [252.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 03:56:07,677 - Train: 25.24% [1223800/4848000] [252.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 03:56:40,678 - Train: 25.25% [1223900/4848000] [252.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 03:57:14,216 - Train: 25.25% [1224000/4848000] [252.5/1000.0] [batch_t 0.334 (0.335)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 03:57:47,286 - Train: 25.25% [1224100/4848000] [252.5/1000.0] [batch_t 0.347 (0.331)] [data_t 0.002] [optim_t 0.345] [lr 0.005000] 2024-04-08 03:58:21,021 - Train: 25.25% [1224200/4848000] [252.5/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 03:58:54,129 - Train: 25.25% [1224300/4848000] [252.5/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 03:59:27,944 - Train: 25.26% [1224400/4848000] [252.6/1000.0] [batch_t 0.336 (0.338)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-08 04:00:00,943 - Train: 25.26% [1224500/4848000] [252.6/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 04:00:34,050 - Train: 25.26% [1224600/4848000] [252.6/1000.0] [batch_t 0.336 (0.331)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-08 04:01:08,017 - Train: 25.26% [1224700/4848000] [252.6/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:01:41,139 - Train: 25.26% [1224800/4848000] [252.6/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 04:02:14,177 - Train: 25.27% [1224900/4848000] [252.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 04:02:47,439 - Train: 25.27% [1225000/4848000] [252.7/1000.0] [batch_t 0.331 (0.333)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 04:03:20,539 - Train: 25.27% [1225100/4848000] [252.7/1000.0] [batch_t 0.333 (0.331)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 04:03:53,585 - Train: 25.27% [1225200/4848000] [252.7/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 04:04:27,008 - Train: 25.27% [1225300/4848000] [252.7/1000.0] [batch_t 0.330 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 04:05:00,052 - Train: 25.28% [1225400/4848000] [252.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:05:33,132 - Train: 25.28% [1225500/4848000] [252.8/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:06:06,530 - Train: 25.28% [1225600/4848000] [252.8/1000.0] [batch_t 0.327 (0.334)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 04:06:39,669 - Train: 25.28% [1225700/4848000] [252.8/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 04:07:12,818 - Train: 25.28% [1225800/4848000] [252.8/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 04:07:45,902 - Train: 25.29% [1225900/4848000] [252.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 04:08:19,754 - Train: 25.29% [1226000/4848000] [252.9/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 04:08:52,868 - Train: 25.29% [1226100/4848000] [252.9/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 04:09:25,962 - Train: 25.29% [1226200/4848000] [252.9/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:09:58,986 - Train: 25.29% [1226300/4848000] [252.9/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 04:10:32,622 - Train: 25.30% [1226400/4848000] [253.0/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 04:11:06,612 - Train: 25.30% [1226500/4848000] [253.0/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:11:21,180 - ==> Total time: 5 days, 4:45:19 Eta: 15 days, 8:20:56 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 04:11:42,100 - Train: 25.30% [1226600/4848000] [253.0/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 04:12:15,602 - Train: 25.30% [1226700/4848000] [253.0/1000.0] [batch_t 0.323 (0.335)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 04:12:48,646 - Train: 25.31% [1226800/4848000] [253.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 04:13:23,582 - Train: 25.31% [1226900/4848000] [253.1/1000.0] [batch_t 0.330 (0.349)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 04:13:56,568 - Train: 25.31% [1227000/4848000] [253.1/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 04:14:29,591 - Train: 25.31% [1227100/4848000] [253.1/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 04:15:02,653 - Train: 25.31% [1227200/4848000] [253.1/1000.0] [batch_t 0.325 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 04:15:38,538 - Train: 25.32% [1227300/4848000] [253.2/1000.0] [batch_t 0.329 (0.359)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:16:13,056 - Train: 25.32% [1227400/4848000] [253.2/1000.0] [batch_t 0.330 (0.345)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 04:16:46,029 - Train: 25.32% [1227500/4848000] [253.2/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 04:17:19,088 - Train: 25.32% [1227600/4848000] [253.2/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 04:17:52,143 - Train: 25.32% [1227700/4848000] [253.2/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 04:18:25,879 - Train: 25.33% [1227800/4848000] [253.3/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 04:18:58,929 - Train: 25.33% [1227900/4848000] [253.3/1000.0] [batch_t 0.336 (0.330)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-08 04:19:31,955 - Train: 25.33% [1228000/4848000] [253.3/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 04:20:04,957 - Train: 25.33% [1228100/4848000] [253.3/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 04:20:38,135 - Train: 25.33% [1228200/4848000] [253.3/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:21:11,204 - Train: 25.34% [1228300/4848000] [253.4/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:21:44,202 - Train: 25.34% [1228400/4848000] [253.4/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 04:22:17,222 - Train: 25.34% [1228500/4848000] [253.4/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 04:22:50,243 - Train: 25.34% [1228600/4848000] [253.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:23:23,311 - Train: 25.34% [1228700/4848000] [253.4/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 04:23:56,383 - Train: 25.35% [1228800/4848000] [253.5/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:24:29,402 - Train: 25.35% [1228900/4848000] [253.5/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 04:25:02,428 - Train: 25.35% [1229000/4848000] [253.5/1000.0] [batch_t 0.337 (0.330)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-08 04:25:35,497 - Train: 25.35% [1229100/4848000] [253.5/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 04:26:08,670 - Train: 25.35% [1229200/4848000] [253.5/1000.0] [batch_t 0.333 (0.332)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 04:26:41,646 - Train: 25.36% [1229300/4848000] [253.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:27:14,670 - Train: 25.36% [1229400/4848000] [253.6/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 04:27:47,829 - Train: 25.36% [1229500/4848000] [253.6/1000.0] [batch_t 0.335 (0.331)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 04:28:20,942 - Train: 25.36% [1229600/4848000] [253.6/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:28:53,977 - Train: 25.37% [1229700/4848000] [253.7/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 04:29:27,063 - Train: 25.37% [1229800/4848000] [253.7/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:30:00,075 - Train: 25.37% [1229900/4848000] [253.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:30:33,115 - Train: 25.37% [1230000/4848000] [253.7/1000.0] [batch_t 0.323 (0.330)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 04:31:06,089 - Train: 25.37% [1230100/4848000] [253.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:31:39,075 - Train: 25.38% [1230200/4848000] [253.8/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 04:32:12,141 - Train: 25.38% [1230300/4848000] [253.8/1000.0] [batch_t 0.325 (0.331)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 04:32:45,172 - Train: 25.38% [1230400/4848000] [253.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 04:33:18,202 - Train: 25.38% [1230500/4848000] [253.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 04:33:51,376 - Train: 25.38% [1230600/4848000] [253.8/1000.0] [batch_t 0.332 (0.332)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 04:34:24,589 - Train: 25.39% [1230700/4848000] [253.9/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:34:57,618 - Train: 25.39% [1230800/4848000] [253.9/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:35:31,925 - Train: 25.39% [1230900/4848000] [253.9/1000.0] [batch_t 0.328 (0.343)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 04:36:04,935 - Train: 25.39% [1231000/4848000] [253.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 04:36:37,993 - Train: 25.39% [1231100/4848000] [253.9/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 04:37:11,022 - Train: 25.40% [1231200/4848000] [254.0/1000.0] [batch_t 0.323 (0.330)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 04:37:44,075 - Train: 25.40% [1231300/4848000] [254.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 04:38:14,867 - ==> Total time: 5 days, 5:12:13 Eta: 15 days, 7:43:27 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 04:38:19,652 - Train: 25.40% [1231400/4848000] [254.0/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 04:38:52,645 - Train: 25.40% [1231500/4848000] [254.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 04:39:25,611 - Train: 25.40% [1231600/4848000] [254.0/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 04:39:58,592 - Train: 25.41% [1231700/4848000] [254.1/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 04:40:32,052 - Train: 25.41% [1231800/4848000] [254.1/1000.0] [batch_t 0.331 (0.335)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 04:41:05,232 - Train: 25.41% [1231900/4848000] [254.1/1000.0] [batch_t 0.332 (0.332)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 04:41:38,364 - Train: 25.41% [1232000/4848000] [254.1/1000.0] [batch_t 0.333 (0.331)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 04:42:12,292 - Train: 25.41% [1232100/4848000] [254.1/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 04:42:45,380 - Train: 25.42% [1232200/4848000] [254.2/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 04:43:18,396 - Train: 25.42% [1232300/4848000] [254.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 04:43:51,411 - Train: 25.42% [1232400/4848000] [254.2/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 04:44:24,734 - Train: 25.42% [1232500/4848000] [254.2/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 04:44:57,765 - Train: 25.42% [1232600/4848000] [254.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:45:30,830 - Train: 25.43% [1232700/4848000] [254.3/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:46:03,833 - Train: 25.43% [1232800/4848000] [254.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:46:36,841 - Train: 25.43% [1232900/4848000] [254.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 04:47:09,892 - Train: 25.43% [1233000/4848000] [254.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:47:42,894 - Train: 25.44% [1233100/4848000] [254.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 04:48:15,827 - Train: 25.44% [1233200/4848000] [254.4/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 04:48:48,787 - Train: 25.44% [1233300/4848000] [254.4/1000.0] [batch_t 0.323 (0.329)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 04:49:21,881 - Train: 25.44% [1233400/4848000] [254.4/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 04:49:54,799 - Train: 25.44% [1233500/4848000] [254.4/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 04:50:28,212 - Train: 25.45% [1233600/4848000] [254.5/1000.0] [batch_t 0.328 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:51:01,287 - Train: 25.45% [1233700/4848000] [254.5/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 04:51:34,335 - Train: 25.45% [1233800/4848000] [254.5/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 04:52:08,074 - Train: 25.45% [1233900/4848000] [254.5/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:52:41,017 - Train: 25.45% [1234000/4848000] [254.5/1000.0] [batch_t 0.327 (0.329)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 04:53:14,000 - Train: 25.46% [1234100/4848000] [254.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:53:46,898 - Train: 25.46% [1234200/4848000] [254.6/1000.0] [batch_t 0.330 (0.329)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:54:20,596 - Train: 25.46% [1234300/4848000] [254.6/1000.0] [batch_t 0.336 (0.337)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-08 04:54:53,546 - Train: 25.46% [1234400/4848000] [254.6/1000.0] [batch_t 0.326 (0.329)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 04:55:27,349 - Train: 25.46% [1234500/4848000] [254.6/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:56:00,321 - Train: 25.47% [1234600/4848000] [254.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 04:56:34,060 - Train: 25.47% [1234700/4848000] [254.7/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:57:10,289 - Train: 25.47% [1234800/4848000] [254.7/1000.0] [batch_t 0.331 (0.362)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 04:57:43,354 - Train: 25.47% [1234900/4848000] [254.7/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:58:16,467 - Train: 25.47% [1235000/4848000] [254.7/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:58:49,621 - Train: 25.48% [1235100/4848000] [254.8/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 04:59:22,744 - Train: 25.48% [1235200/4848000] [254.8/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 04:59:55,849 - Train: 25.48% [1235300/4848000] [254.8/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 05:00:28,978 - Train: 25.48% [1235400/4848000] [254.8/1000.0] [batch_t 0.333 (0.331)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 05:01:02,023 - Train: 25.48% [1235500/4848000] [254.8/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 05:01:35,142 - Train: 25.49% [1235600/4848000] [254.9/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 05:02:08,979 - Train: 25.49% [1235700/4848000] [254.9/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 05:02:42,104 - Train: 25.49% [1235800/4848000] [254.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 05:03:15,220 - Train: 25.49% [1235900/4848000] [254.9/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 05:03:48,353 - Train: 25.50% [1236000/4848000] [255.0/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 05:04:21,437 - Train: 25.50% [1236100/4848000] [255.0/1000.0] [batch_t 0.340 (0.331)] [data_t 0.002] [optim_t 0.338] [lr 0.005000] 2024-04-08 05:04:54,793 - Train: 25.50% [1236200/4848000] [255.0/1000.0] [batch_t 0.326 (0.333)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 05:05:08,021 - ==> Total time: 5 days, 5:39:06 Eta: 15 days, 7:06:01 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 05:05:30,166 - Train: 25.50% [1236300/4848000] [255.0/1000.0] [batch_t 0.330 (0.332)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 05:06:03,196 - Train: 25.50% [1236400/4848000] [255.0/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 05:06:36,926 - Train: 25.51% [1236500/4848000] [255.1/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 05:07:11,040 - Train: 25.51% [1236600/4848000] [255.1/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 05:07:44,023 - Train: 25.51% [1236700/4848000] [255.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 05:08:17,089 - Train: 25.51% [1236800/4848000] [255.1/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 05:08:50,106 - Train: 25.51% [1236900/4848000] [255.1/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 05:09:25,310 - Train: 25.52% [1237000/4848000] [255.2/1000.0] [batch_t 0.327 (0.352)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 05:09:58,331 - Train: 25.52% [1237100/4848000] [255.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 05:10:32,394 - Train: 25.52% [1237200/4848000] [255.2/1000.0] [batch_t 0.332 (0.341)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 05:11:05,647 - Train: 25.52% [1237300/4848000] [255.2/1000.0] [batch_t 0.332 (0.332)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 05:11:38,644 - Train: 25.52% [1237400/4848000] [255.2/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 05:12:11,632 - Train: 25.53% [1237500/4848000] [255.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 05:12:44,800 - Train: 25.53% [1237600/4848000] [255.3/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 05:13:17,859 - Train: 25.53% [1237700/4848000] [255.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 05:13:50,837 - Train: 25.53% [1237800/4848000] [255.3/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 05:14:23,867 - Train: 25.53% [1237900/4848000] [255.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 05:14:56,906 - Train: 25.54% [1238000/4848000] [255.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 05:15:29,980 - Train: 25.54% [1238100/4848000] [255.4/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 05:16:03,030 - Train: 25.54% [1238200/4848000] [255.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 05:16:36,880 - Train: 25.54% [1238300/4848000] [255.4/1000.0] [batch_t 0.328 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 05:17:09,928 - Train: 25.54% [1238400/4848000] [255.4/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 05:17:42,978 - Train: 25.55% [1238500/4848000] [255.5/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 05:18:16,931 - Train: 25.55% [1238600/4848000] [255.5/1000.0] [batch_t 0.324 (0.339)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 05:18:49,957 - Train: 25.55% [1238700/4848000] [255.5/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 05:19:23,904 - Train: 25.55% [1238800/4848000] [255.5/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 05:19:57,050 - Train: 25.55% [1238900/4848000] [255.5/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 05:20:30,282 - Train: 25.56% [1239000/4848000] [255.6/1000.0] [batch_t 0.326 (0.332)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 05:21:03,307 - Train: 25.56% [1239100/4848000] [255.6/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 05:21:37,118 - Train: 25.56% [1239200/4848000] [255.6/1000.0] [batch_t 0.327 (0.338)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 05:22:10,179 - Train: 25.56% [1239300/4848000] [255.6/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 05:22:43,328 - Train: 25.57% [1239400/4848000] [255.7/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 05:23:16,754 - Train: 25.57% [1239500/4848000] [255.7/1000.0] [batch_t 0.329 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 05:23:49,848 - Train: 25.57% [1239600/4848000] [255.7/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 05:24:22,885 - Train: 25.57% [1239700/4848000] [255.7/1000.0] [batch_t 0.335 (0.330)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 05:24:55,907 - Train: 25.57% [1239800/4848000] [255.7/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 05:25:29,003 - Train: 25.58% [1239900/4848000] [255.8/1000.0] [batch_t 0.325 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 05:26:02,158 - Train: 25.58% [1240000/4848000] [255.8/1000.0] [batch_t 0.335 (0.331)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 05:26:35,724 - Train: 25.58% [1240100/4848000] [255.8/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 05:27:09,527 - Train: 25.58% [1240200/4848000] [255.8/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 05:27:42,586 - Train: 25.58% [1240300/4848000] [255.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 05:28:16,536 - Train: 25.59% [1240400/4848000] [255.9/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 05:28:49,539 - Train: 25.59% [1240500/4848000] [255.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 05:29:22,610 - Train: 25.59% [1240600/4848000] [255.9/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 05:29:55,634 - Train: 25.59% [1240700/4848000] [255.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 05:30:29,532 - Train: 25.59% [1240800/4848000] [255.9/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 05:31:02,574 - Train: 25.60% [1240900/4848000] [256.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 05:31:35,560 - Train: 25.60% [1241000/4848000] [256.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 05:32:04,583 - ==> Total time: 5 days, 6:06:03 Eta: 15 days, 6:28:50 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 05:32:11,563 - Train: 25.60% [1241100/4848000] [256.0/1000.0] [batch_t 0.335 (0.334)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 05:32:44,595 - Train: 25.60% [1241200/4848000] [256.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 05:33:17,634 - Train: 25.60% [1241300/4848000] [256.0/1000.0] [batch_t 0.340 (0.330)] [data_t 0.002] [optim_t 0.338] [lr 0.005000] 2024-04-08 05:33:50,652 - Train: 25.61% [1241400/4848000] [256.1/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 05:34:23,687 - Train: 25.61% [1241500/4848000] [256.1/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 05:34:56,695 - Train: 25.61% [1241600/4848000] [256.1/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 05:35:29,680 - Train: 25.61% [1241700/4848000] [256.1/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 05:36:02,814 - Train: 25.61% [1241800/4848000] [256.1/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 05:36:36,028 - Train: 25.62% [1241900/4848000] [256.2/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 05:37:09,726 - Train: 25.62% [1242000/4848000] [256.2/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 05:37:42,794 - Train: 25.62% [1242100/4848000] [256.2/1000.0] [batch_t 0.335 (0.331)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 05:38:16,440 - Train: 25.62% [1242200/4848000] [256.2/1000.0] [batch_t 0.326 (0.336)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 05:38:49,473 - Train: 25.62% [1242300/4848000] [256.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 05:39:23,326 - Train: 25.63% [1242400/4848000] [256.3/1000.0] [batch_t 0.326 (0.338)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 05:39:56,350 - Train: 25.63% [1242500/4848000] [256.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 05:40:29,694 - Train: 25.63% [1242600/4848000] [256.3/1000.0] [batch_t 0.332 (0.333)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 05:41:02,689 - Train: 25.63% [1242700/4848000] [256.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 05:41:36,870 - Train: 25.64% [1242800/4848000] [256.4/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 05:42:10,660 - Train: 25.64% [1242900/4848000] [256.4/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 05:42:43,805 - Train: 25.64% [1243000/4848000] [256.4/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 05:43:17,965 - Train: 25.64% [1243100/4848000] [256.4/1000.0] [batch_t 0.334 (0.342)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 05:43:51,004 - Train: 25.64% [1243200/4848000] [256.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 05:44:24,764 - Train: 25.65% [1243300/4848000] [256.5/1000.0] [batch_t 0.334 (0.338)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 05:44:57,822 - Train: 25.65% [1243400/4848000] [256.5/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 05:45:32,006 - Train: 25.65% [1243500/4848000] [256.5/1000.0] [batch_t 0.328 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 05:46:04,965 - Train: 25.65% [1243600/4848000] [256.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 05:46:38,104 - Train: 25.65% [1243700/4848000] [256.5/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 05:47:12,192 - Train: 25.66% [1243800/4848000] [256.6/1000.0] [batch_t 0.325 (0.341)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 05:47:45,293 - Train: 25.66% [1243900/4848000] [256.6/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 05:48:18,863 - Train: 25.66% [1244000/4848000] [256.6/1000.0] [batch_t 0.328 (0.336)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 05:48:51,889 - Train: 25.66% [1244100/4848000] [256.6/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 05:49:26,183 - Train: 25.66% [1244200/4848000] [256.6/1000.0] [batch_t 0.326 (0.343)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 05:49:59,212 - Train: 25.67% [1244300/4848000] [256.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 05:50:32,908 - Train: 25.67% [1244400/4848000] [256.7/1000.0] [batch_t 0.336 (0.337)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-08 05:51:06,960 - Train: 25.67% [1244500/4848000] [256.7/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 05:51:40,077 - Train: 25.67% [1244600/4848000] [256.7/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 05:52:13,059 - Train: 25.67% [1244700/4848000] [256.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 05:52:46,104 - Train: 25.68% [1244800/4848000] [256.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 05:53:19,102 - Train: 25.68% [1244900/4848000] [256.8/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 05:53:52,165 - Train: 25.68% [1245000/4848000] [256.8/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 05:54:26,562 - Train: 25.68% [1245100/4848000] [256.8/1000.0] [batch_t 0.330 (0.344)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 05:54:59,614 - Train: 25.68% [1245200/4848000] [256.8/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 05:55:33,522 - Train: 25.69% [1245300/4848000] [256.9/1000.0] [batch_t 0.325 (0.339)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 05:56:06,621 - Train: 25.69% [1245400/4848000] [256.9/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 05:56:39,710 - Train: 25.69% [1245500/4848000] [256.9/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 05:57:13,467 - Train: 25.69% [1245600/4848000] [256.9/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 05:57:46,470 - Train: 25.70% [1245700/4848000] [257.0/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 05:58:20,616 - Train: 25.70% [1245800/4848000] [257.0/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 05:58:53,676 - Train: 25.70% [1245900/4848000] [257.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 05:59:05,530 - ==> Total time: 5 days, 6:33:04 Eta: 15 days, 5:51:56 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 05:59:29,146 - Train: 25.70% [1246000/4848000] [257.0/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 06:00:02,108 - Train: 25.70% [1246100/4848000] [257.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 06:00:37,076 - Train: 25.71% [1246200/4848000] [257.1/1000.0] [batch_t 0.325 (0.350)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 06:01:10,150 - Train: 25.71% [1246300/4848000] [257.1/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 06:01:43,127 - Train: 25.71% [1246400/4848000] [257.1/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 06:02:17,271 - Train: 25.71% [1246500/4848000] [257.1/1000.0] [batch_t 0.333 (0.341)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 06:02:50,333 - Train: 25.71% [1246600/4848000] [257.1/1000.0] [batch_t 0.333 (0.331)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 06:03:24,226 - Train: 25.72% [1246700/4848000] [257.2/1000.0] [batch_t 0.332 (0.339)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 06:03:57,260 - Train: 25.72% [1246800/4848000] [257.2/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 06:04:31,596 - Train: 25.72% [1246900/4848000] [257.2/1000.0] [batch_t 0.328 (0.343)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 06:05:05,333 - Train: 25.72% [1247000/4848000] [257.2/1000.0] [batch_t 0.982 (0.337)] [data_t 0.654] [optim_t 0.328] [lr 0.005000] 2024-04-08 06:05:38,313 - Train: 25.72% [1247100/4848000] [257.2/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 06:06:12,513 - Train: 25.73% [1247200/4848000] [257.3/1000.0] [batch_t 0.339 (0.342)] [data_t 0.002] [optim_t 0.337] [lr 0.005000] 2024-04-08 06:06:45,461 - Train: 25.73% [1247300/4848000] [257.3/1000.0] [batch_t 0.330 (0.329)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 06:07:19,220 - Train: 25.73% [1247400/4848000] [257.3/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 06:07:52,167 - Train: 25.73% [1247500/4848000] [257.3/1000.0] [batch_t 0.335 (0.329)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 06:08:25,546 - Train: 25.73% [1247600/4848000] [257.3/1000.0] [batch_t 0.322 (0.334)] [data_t 0.002] [optim_t 0.320] [lr 0.005000] 2024-04-08 06:08:58,513 - Train: 25.74% [1247700/4848000] [257.4/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 06:09:32,907 - Train: 25.74% [1247800/4848000] [257.4/1000.0] [batch_t 0.330 (0.344)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 06:10:07,079 - Train: 25.74% [1247900/4848000] [257.4/1000.0] [batch_t 0.325 (0.342)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 06:10:40,112 - Train: 25.74% [1248000/4848000] [257.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 06:11:13,950 - Train: 25.74% [1248100/4848000] [257.4/1000.0] [batch_t 0.332 (0.338)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 06:11:46,854 - Train: 25.75% [1248200/4848000] [257.5/1000.0] [batch_t 0.328 (0.329)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 06:12:20,985 - Train: 25.75% [1248300/4848000] [257.5/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 06:12:54,063 - Train: 25.75% [1248400/4848000] [257.5/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 06:13:27,095 - Train: 25.75% [1248500/4848000] [257.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 06:14:00,144 - Train: 25.75% [1248600/4848000] [257.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 06:14:34,330 - Train: 25.76% [1248700/4848000] [257.6/1000.0] [batch_t 0.335 (0.342)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 06:15:07,356 - Train: 25.76% [1248800/4848000] [257.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 06:15:40,444 - Train: 25.76% [1248900/4848000] [257.6/1000.0] [batch_t 0.337 (0.331)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-08 06:16:14,265 - Train: 25.76% [1249000/4848000] [257.6/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 06:16:47,284 - Train: 25.77% [1249100/4848000] [257.7/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 06:17:21,426 - Train: 25.77% [1249200/4848000] [257.7/1000.0] [batch_t 0.332 (0.341)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 06:17:54,369 - Train: 25.77% [1249300/4848000] [257.7/1000.0] [batch_t 0.331 (0.329)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 06:18:28,286 - Train: 25.77% [1249400/4848000] [257.7/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 06:19:01,282 - Train: 25.77% [1249500/4848000] [257.7/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:19:35,774 - Train: 25.78% [1249600/4848000] [257.8/1000.0] [batch_t 0.331 (0.345)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 06:20:09,036 - Train: 25.78% [1249700/4848000] [257.8/1000.0] [batch_t 0.333 (0.333)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 06:20:42,090 - Train: 25.78% [1249800/4848000] [257.8/1000.0] [batch_t 0.325 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 06:21:15,131 - Train: 25.78% [1249900/4848000] [257.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 06:21:48,208 - Train: 25.78% [1250000/4848000] [257.8/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 06:22:22,324 - Train: 25.79% [1250100/4848000] [257.9/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 06:22:55,365 - Train: 25.79% [1250200/4848000] [257.9/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 06:23:29,629 - Train: 25.79% [1250300/4848000] [257.9/1000.0] [batch_t 0.330 (0.343)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 06:24:02,621 - Train: 25.79% [1250400/4848000] [257.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 06:24:35,909 - Train: 25.79% [1250500/4848000] [257.9/1000.0] [batch_t 0.339 (0.333)] [data_t 0.002] [optim_t 0.337] [lr 0.005000] 2024-04-08 06:25:09,866 - Train: 25.80% [1250600/4848000] [258.0/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 06:25:42,921 - Train: 25.80% [1250700/4848000] [258.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 06:26:11,520 - ==> Total time: 5 days, 7:00:10 Eta: 15 days, 5:15:21 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 06:26:19,131 - Train: 25.80% [1250800/4848000] [258.0/1000.0] [batch_t 0.331 (0.344)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 06:26:52,286 - Train: 25.80% [1250900/4848000] [258.0/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 06:27:26,146 - Train: 25.80% [1251000/4848000] [258.0/1000.0] [batch_t 0.324 (0.339)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 06:27:59,329 - Train: 25.81% [1251100/4848000] [258.1/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:28:33,188 - Train: 25.81% [1251200/4848000] [258.1/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:29:07,207 - Train: 25.81% [1251300/4848000] [258.1/1000.0] [batch_t 0.329 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:29:40,419 - Train: 25.81% [1251400/4848000] [258.1/1000.0] [batch_t 0.335 (0.332)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 06:30:14,230 - Train: 25.81% [1251500/4848000] [258.1/1000.0] [batch_t 0.329 (0.338)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:30:47,452 - Train: 25.82% [1251600/4848000] [258.2/1000.0] [batch_t 0.331 (0.332)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 06:31:21,881 - Train: 25.82% [1251700/4848000] [258.2/1000.0] [batch_t 0.329 (0.344)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:31:55,021 - Train: 25.82% [1251800/4848000] [258.2/1000.0] [batch_t 0.335 (0.331)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 06:32:28,600 - Train: 25.82% [1251900/4848000] [258.2/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 06:33:01,668 - Train: 25.83% [1252000/4848000] [258.3/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:33:35,857 - Train: 25.83% [1252100/4848000] [258.3/1000.0] [batch_t 0.330 (0.342)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 06:34:09,773 - Train: 25.83% [1252200/4848000] [258.3/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:34:42,841 - Train: 25.83% [1252300/4848000] [258.3/1000.0] [batch_t 0.325 (0.331)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 06:35:16,661 - Train: 25.83% [1252400/4848000] [258.3/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:35:49,715 - Train: 25.84% [1252500/4848000] [258.4/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 06:36:23,062 - Train: 25.84% [1252600/4848000] [258.4/1000.0] [batch_t 0.335 (0.333)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 06:36:56,096 - Train: 25.84% [1252700/4848000] [258.4/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 06:37:30,227 - Train: 25.84% [1252800/4848000] [258.4/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 06:38:03,263 - Train: 25.84% [1252900/4848000] [258.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:38:37,687 - Train: 25.85% [1253000/4848000] [258.5/1000.0] [batch_t 0.330 (0.344)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 06:39:11,913 - Train: 25.85% [1253100/4848000] [258.5/1000.0] [batch_t 0.330 (0.342)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 06:39:44,996 - Train: 25.85% [1253200/4848000] [258.5/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 06:40:19,127 - Train: 25.85% [1253300/4848000] [258.5/1000.0] [batch_t 0.335 (0.341)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 06:40:52,186 - Train: 25.85% [1253400/4848000] [258.5/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 06:41:26,323 - Train: 25.86% [1253500/4848000] [258.6/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 06:41:59,506 - Train: 25.86% [1253600/4848000] [258.6/1000.0] [batch_t 0.333 (0.332)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 06:42:33,466 - Train: 25.86% [1253700/4848000] [258.6/1000.0] [batch_t 0.330 (0.339)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 06:43:06,589 - Train: 25.86% [1253800/4848000] [258.6/1000.0] [batch_t 0.336 (0.331)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 06:43:39,676 - Train: 25.86% [1253900/4848000] [258.6/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 06:44:14,113 - Train: 25.87% [1254000/4848000] [258.7/1000.0] [batch_t 0.329 (0.344)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:44:47,193 - Train: 25.87% [1254100/4848000] [258.7/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 06:45:20,315 - Train: 25.87% [1254200/4848000] [258.7/1000.0] [batch_t 0.325 (0.331)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 06:45:53,493 - Train: 25.87% [1254300/4848000] [258.7/1000.0] [batch_t 0.328 (0.332)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 06:46:26,634 - Train: 25.87% [1254400/4848000] [258.7/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 06:46:59,934 - Train: 25.88% [1254500/4848000] [258.8/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 06:47:33,090 - Train: 25.88% [1254600/4848000] [258.8/1000.0] [batch_t 0.339 (0.331)] [data_t 0.002] [optim_t 0.337] [lr 0.005000] 2024-04-08 06:48:06,886 - Train: 25.88% [1254700/4848000] [258.8/1000.0] [batch_t 0.332 (0.338)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 06:48:40,161 - Train: 25.88% [1254800/4848000] [258.8/1000.0] [batch_t 0.331 (0.333)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 06:49:13,926 - Train: 25.88% [1254900/4848000] [258.8/1000.0] [batch_t 0.331 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 06:49:47,018 - Train: 25.89% [1255000/4848000] [258.9/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 06:50:20,084 - Train: 25.89% [1255100/4848000] [258.9/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 06:50:53,239 - Train: 25.89% [1255200/4848000] [258.9/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:51:26,968 - Train: 25.89% [1255300/4848000] [258.9/1000.0] [batch_t 0.330 (0.337)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:52:00,059 - Train: 25.90% [1255400/4848000] [259.0/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 06:52:34,460 - Train: 25.90% [1255500/4848000] [259.0/1000.0] [batch_t 0.329 (0.344)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 06:53:07,964 - Train: 25.90% [1255600/4848000] [259.0/1000.0] [batch_t 0.327 (0.335)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 06:53:18,609 - ==> Total time: 5 days, 7:27:17 Eta: 15 days, 4:38:54 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 06:53:43,390 - Train: 25.90% [1255700/4848000] [259.0/1000.0] [batch_t 0.327 (0.333)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 06:54:16,632 - Train: 25.90% [1255800/4848000] [259.0/1000.0] [batch_t 0.332 (0.332)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 06:54:49,776 - Train: 25.91% [1255900/4848000] [259.1/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 06:55:22,945 - Train: 25.91% [1256000/4848000] [259.1/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 06:55:56,137 - Train: 25.91% [1256100/4848000] [259.1/1000.0] [batch_t 0.328 (0.332)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 06:56:29,264 - Train: 25.91% [1256200/4848000] [259.1/1000.0] [batch_t 0.327 (0.331)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 06:57:02,461 - Train: 25.91% [1256300/4848000] [259.1/1000.0] [batch_t 0.345 (0.332)] [data_t 0.003] [optim_t 0.342] [lr 0.005000] 2024-04-08 06:57:35,631 - Train: 25.92% [1256400/4848000] [259.2/1000.0] [batch_t 0.334 (0.332)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 06:58:08,814 - Train: 25.92% [1256500/4848000] [259.2/1000.0] [batch_t 0.332 (0.332)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 06:58:41,976 - Train: 25.92% [1256600/4848000] [259.2/1000.0] [batch_t 0.327 (0.331)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 06:59:15,635 - Train: 25.92% [1256700/4848000] [259.2/1000.0] [batch_t 0.328 (0.336)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 06:59:48,753 - Train: 25.92% [1256800/4848000] [259.2/1000.0] [batch_t 0.326 (0.331)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 07:00:21,828 - Train: 25.93% [1256900/4848000] [259.3/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 07:00:54,964 - Train: 25.93% [1257000/4848000] [259.3/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 07:01:28,360 - Train: 25.93% [1257100/4848000] [259.3/1000.0] [batch_t 0.331 (0.334)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 07:02:01,518 - Train: 25.93% [1257200/4848000] [259.3/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:02:34,771 - Train: 25.93% [1257300/4848000] [259.3/1000.0] [batch_t 0.334 (0.332)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 07:03:08,609 - Train: 25.94% [1257400/4848000] [259.4/1000.0] [batch_t 0.335 (0.338)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 07:03:41,763 - Train: 25.94% [1257500/4848000] [259.4/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 07:04:14,984 - Train: 25.94% [1257600/4848000] [259.4/1000.0] [batch_t 0.343 (0.332)] [data_t 0.003] [optim_t 0.340] [lr 0.005000] 2024-04-08 07:04:48,224 - Train: 25.94% [1257700/4848000] [259.4/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 07:05:22,033 - Train: 25.94% [1257800/4848000] [259.4/1000.0] [batch_t 0.334 (0.338)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 07:05:55,122 - Train: 25.95% [1257900/4848000] [259.5/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:06:28,170 - Train: 25.95% [1258000/4848000] [259.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 07:07:01,321 - Train: 25.95% [1258100/4848000] [259.5/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 07:07:34,813 - Train: 25.95% [1258200/4848000] [259.5/1000.0] [batch_t 0.332 (0.335)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 07:08:08,946 - Train: 25.96% [1258300/4848000] [259.6/1000.0] [batch_t 0.327 (0.341)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 07:08:42,091 - Train: 25.96% [1258400/4848000] [259.6/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 07:09:15,461 - Train: 25.96% [1258500/4848000] [259.6/1000.0] [batch_t 0.335 (0.334)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 07:09:48,625 - Train: 25.96% [1258600/4848000] [259.6/1000.0] [batch_t 0.330 (0.332)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:10:21,978 - Train: 25.96% [1258700/4848000] [259.6/1000.0] [batch_t 0.331 (0.333)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 07:10:55,112 - Train: 25.97% [1258800/4848000] [259.7/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:11:28,289 - Train: 25.97% [1258900/4848000] [259.7/1000.0] [batch_t 0.337 (0.332)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-08 07:12:01,468 - Train: 25.97% [1259000/4848000] [259.7/1000.0] [batch_t 0.330 (0.332)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 07:12:34,659 - Train: 25.97% [1259100/4848000] [259.7/1000.0] [batch_t 0.334 (0.332)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 07:13:08,543 - Train: 25.97% [1259200/4848000] [259.7/1000.0] [batch_t 0.330 (0.339)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 07:13:41,658 - Train: 25.98% [1259300/4848000] [259.8/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 07:14:14,825 - Train: 25.98% [1259400/4848000] [259.8/1000.0] [batch_t 0.330 (0.332)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:14:47,939 - Train: 25.98% [1259500/4848000] [259.8/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 07:15:21,033 - Train: 25.98% [1259600/4848000] [259.8/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 07:15:54,166 - Train: 25.98% [1259700/4848000] [259.8/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:16:27,273 - Train: 25.99% [1259800/4848000] [259.9/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 07:17:00,465 - Train: 25.99% [1259900/4848000] [259.9/1000.0] [batch_t 0.332 (0.332)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 07:17:33,535 - Train: 25.99% [1260000/4848000] [259.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 07:18:06,870 - Train: 25.99% [1260100/4848000] [259.9/1000.0] [batch_t 0.329 (0.333)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 07:18:40,030 - Train: 25.99% [1260200/4848000] [259.9/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 07:19:13,753 - Train: 26.00% [1260300/4848000] [260.0/1000.0] [batch_t 0.332 (0.337)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 07:19:47,031 - Train: 26.00% [1260400/4848000] [260.0/1000.0] [batch_t 0.334 (0.333)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 07:20:14,205 - ==> Total time: 5 days, 7:54:12 Eta: 15 days, 4:01:59 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 07:20:23,077 - Train: 26.00% [1260500/4848000] [260.0/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 07:20:56,244 - Train: 26.00% [1260600/4848000] [260.0/1000.0] [batch_t 0.332 (0.332)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 07:21:29,377 - Train: 26.00% [1260700/4848000] [260.0/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 07:22:02,490 - Train: 26.01% [1260800/4848000] [260.1/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 07:22:35,636 - Train: 26.01% [1260900/4848000] [260.1/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 07:23:08,797 - Train: 26.01% [1261000/4848000] [260.1/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 07:23:41,864 - Train: 26.01% [1261100/4848000] [260.1/1000.0] [batch_t 0.333 (0.331)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 07:24:15,547 - Train: 26.01% [1261200/4848000] [260.1/1000.0] [batch_t 0.331 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 07:24:48,712 - Train: 26.02% [1261300/4848000] [260.2/1000.0] [batch_t 0.330 (0.332)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:25:22,725 - Train: 26.02% [1261400/4848000] [260.2/1000.0] [batch_t 0.332 (0.340)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 07:25:55,922 - Train: 26.02% [1261500/4848000] [260.2/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 07:26:29,044 - Train: 26.02% [1261600/4848000] [260.2/1000.0] [batch_t 0.333 (0.331)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 07:27:02,219 - Train: 26.03% [1261700/4848000] [260.3/1000.0] [batch_t 0.335 (0.332)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 07:27:35,420 - Train: 26.03% [1261800/4848000] [260.3/1000.0] [batch_t 0.332 (0.332)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 07:28:08,573 - Train: 26.03% [1261900/4848000] [260.3/1000.0] [batch_t 0.327 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 07:28:41,738 - Train: 26.03% [1262000/4848000] [260.3/1000.0] [batch_t 0.332 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 07:29:14,904 - Train: 26.03% [1262100/4848000] [260.3/1000.0] [batch_t 0.332 (0.332)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 07:29:48,032 - Train: 26.04% [1262200/4848000] [260.4/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 07:30:21,167 - Train: 26.04% [1262300/4848000] [260.4/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 07:30:54,239 - Train: 26.04% [1262400/4848000] [260.4/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 07:31:27,366 - Train: 26.04% [1262500/4848000] [260.4/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 07:32:00,461 - Train: 26.04% [1262600/4848000] [260.4/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:32:34,655 - Train: 26.05% [1262700/4848000] [260.5/1000.0] [batch_t 0.332 (0.342)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 07:33:07,779 - Train: 26.05% [1262800/4848000] [260.5/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:33:40,990 - Train: 26.05% [1262900/4848000] [260.5/1000.0] [batch_t 0.329 (0.332)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 07:34:14,070 - Train: 26.05% [1263000/4848000] [260.5/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 07:34:47,325 - Train: 26.05% [1263100/4848000] [260.5/1000.0] [batch_t 0.330 (0.332)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 07:35:22,462 - Train: 26.06% [1263200/4848000] [260.6/1000.0] [batch_t 0.329 (0.351)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 07:35:55,600 - Train: 26.06% [1263300/4848000] [260.6/1000.0] [batch_t 0.333 (0.331)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 07:36:30,466 - Train: 26.06% [1263400/4848000] [260.6/1000.0] [batch_t 0.330 (0.349)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:37:03,573 - Train: 26.06% [1263500/4848000] [260.6/1000.0] [batch_t 0.327 (0.331)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 07:37:36,732 - Train: 26.06% [1263600/4848000] [260.6/1000.0] [batch_t 0.327 (0.331)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 07:38:09,888 - Train: 26.07% [1263700/4848000] [260.7/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 07:38:43,071 - Train: 26.07% [1263800/4848000] [260.7/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 07:39:17,567 - Train: 26.07% [1263900/4848000] [260.7/1000.0] [batch_t 0.331 (0.345)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 07:39:50,771 - Train: 26.07% [1264000/4848000] [260.7/1000.0] [batch_t 0.325 (0.332)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 07:40:23,929 - Train: 26.07% [1264100/4848000] [260.7/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:40:57,066 - Train: 26.08% [1264200/4848000] [260.8/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 07:41:30,970 - Train: 26.08% [1264300/4848000] [260.8/1000.0] [batch_t 0.328 (0.339)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 07:42:04,352 - Train: 26.08% [1264400/4848000] [260.8/1000.0] [batch_t 0.334 (0.334)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 07:42:37,464 - Train: 26.08% [1264500/4848000] [260.8/1000.0] [batch_t 0.337 (0.331)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-08 07:43:10,618 - Train: 26.08% [1264600/4848000] [260.8/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:43:43,820 - Train: 26.09% [1264700/4848000] [260.9/1000.0] [batch_t 0.333 (0.332)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 07:44:17,090 - Train: 26.09% [1264800/4848000] [260.9/1000.0] [batch_t 0.327 (0.333)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 07:44:50,204 - Train: 26.09% [1264900/4848000] [260.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:45:23,325 - Train: 26.09% [1265000/4848000] [260.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:45:56,528 - Train: 26.10% [1265100/4848000] [261.0/1000.0] [batch_t 0.331 (0.332)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 07:46:29,837 - Train: 26.10% [1265200/4848000] [261.0/1000.0] [batch_t 0.334 (0.333)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 07:47:03,023 - Train: 26.10% [1265300/4848000] [261.0/1000.0] [batch_t 0.332 (0.332)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 07:47:13,460 - ==> Total time: 5 days, 8:21:11 Eta: 15 days, 3:25:18 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 07:47:39,712 - Train: 26.10% [1265400/4848000] [261.0/1000.0] [batch_t 0.327 (0.335)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 07:48:13,813 - Train: 26.10% [1265500/4848000] [261.0/1000.0] [batch_t 0.326 (0.341)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 07:48:46,975 - Train: 26.11% [1265600/4848000] [261.1/1000.0] [batch_t 0.329 (0.332)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:49:20,303 - Train: 26.11% [1265700/4848000] [261.1/1000.0] [batch_t 0.327 (0.333)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 07:49:53,427 - Train: 26.11% [1265800/4848000] [261.1/1000.0] [batch_t 0.327 (0.331)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 07:50:26,909 - Train: 26.11% [1265900/4848000] [261.1/1000.0] [batch_t 0.330 (0.335)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:51:00,077 - Train: 26.11% [1266000/4848000] [261.1/1000.0] [batch_t 0.331 (0.332)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 07:51:33,205 - Train: 26.12% [1266100/4848000] [261.2/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 07:52:06,890 - Train: 26.12% [1266200/4848000] [261.2/1000.0] [batch_t 0.329 (0.337)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 07:52:40,033 - Train: 26.12% [1266300/4848000] [261.2/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 07:53:14,092 - Train: 26.12% [1266400/4848000] [261.2/1000.0] [batch_t 0.331 (0.340)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 07:53:47,244 - Train: 26.12% [1266500/4848000] [261.2/1000.0] [batch_t 0.327 (0.331)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 07:54:20,396 - Train: 26.13% [1266600/4848000] [261.3/1000.0] [batch_t 0.338 (0.331)] [data_t 0.003] [optim_t 0.335] [lr 0.005000] 2024-04-08 07:54:53,514 - Train: 26.13% [1266700/4848000] [261.3/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 07:55:27,070 - Train: 26.13% [1266800/4848000] [261.3/1000.0] [batch_t 0.331 (0.335)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 07:56:00,144 - Train: 26.13% [1266900/4848000] [261.3/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 07:56:34,087 - Train: 26.13% [1267000/4848000] [261.3/1000.0] [batch_t 0.335 (0.339)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 07:57:07,580 - Train: 26.14% [1267100/4848000] [261.4/1000.0] [batch_t 0.330 (0.335)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 07:57:40,692 - Train: 26.14% [1267200/4848000] [261.4/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 07:58:13,887 - Train: 26.14% [1267300/4848000] [261.4/1000.0] [batch_t 0.330 (0.332)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 07:58:47,053 - Train: 26.14% [1267400/4848000] [261.4/1000.0] [batch_t 0.326 (0.332)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 07:59:21,088 - Train: 26.14% [1267500/4848000] [261.4/1000.0] [batch_t 0.331 (0.340)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 07:59:54,244 - Train: 26.15% [1267600/4848000] [261.5/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 08:00:27,463 - Train: 26.15% [1267700/4848000] [261.5/1000.0] [batch_t 0.332 (0.332)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 08:01:00,647 - Train: 26.15% [1267800/4848000] [261.5/1000.0] [batch_t 0.338 (0.332)] [data_t 0.003] [optim_t 0.336] [lr 0.005000] 2024-04-08 08:01:33,773 - Train: 26.15% [1267900/4848000] [261.5/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 08:02:06,929 - Train: 26.16% [1268000/4848000] [261.6/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:02:40,101 - Train: 26.16% [1268100/4848000] [261.6/1000.0] [batch_t 0.329 (0.332)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:03:14,281 - Train: 26.16% [1268200/4848000] [261.6/1000.0] [batch_t 0.329 (0.342)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:03:47,507 - Train: 26.16% [1268300/4848000] [261.6/1000.0] [batch_t 0.328 (0.332)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:04:21,157 - Train: 26.16% [1268400/4848000] [261.6/1000.0] [batch_t 0.324 (0.336)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-08 08:04:54,375 - Train: 26.17% [1268500/4848000] [261.7/1000.0] [batch_t 0.326 (0.332)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 08:05:28,241 - Train: 26.17% [1268600/4848000] [261.7/1000.0] [batch_t 0.336 (0.339)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 08:06:01,329 - Train: 26.17% [1268700/4848000] [261.7/1000.0] [batch_t 0.335 (0.331)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 08:06:35,192 - Train: 26.17% [1268800/4848000] [261.7/1000.0] [batch_t 0.331 (0.339)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 08:07:08,348 - Train: 26.17% [1268900/4848000] [261.7/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:07:41,537 - Train: 26.18% [1269000/4848000] [261.8/1000.0] [batch_t 0.330 (0.332)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:08:14,699 - Train: 26.18% [1269100/4848000] [261.8/1000.0] [batch_t 0.333 (0.332)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 08:08:47,833 - Train: 26.18% [1269200/4848000] [261.8/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:09:21,640 - Train: 26.18% [1269300/4848000] [261.8/1000.0] [batch_t 0.332 (0.338)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 08:09:54,850 - Train: 26.18% [1269400/4848000] [261.8/1000.0] [batch_t 0.329 (0.332)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:10:28,056 - Train: 26.19% [1269500/4848000] [261.9/1000.0] [batch_t 0.329 (0.332)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:11:01,223 - Train: 26.19% [1269600/4848000] [261.9/1000.0] [batch_t 0.336 (0.332)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 08:11:35,065 - Train: 26.19% [1269700/4848000] [261.9/1000.0] [batch_t 0.336 (0.338)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 08:12:08,984 - Train: 26.19% [1269800/4848000] [261.9/1000.0] [batch_t 0.329 (0.339)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:12:42,220 - Train: 26.19% [1269900/4848000] [261.9/1000.0] [batch_t 0.336 (0.332)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 08:13:16,190 - Train: 26.20% [1270000/4848000] [262.0/1000.0] [batch_t 0.322 (0.340)] [data_t 0.003] [optim_t 0.319] [lr 0.005000] 2024-04-08 08:13:49,305 - Train: 26.20% [1270100/4848000] [262.0/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:14:15,196 - ==> Total time: 5 days, 8:48:13 Eta: 15 days, 2:48:49 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 08:14:25,247 - Train: 26.20% [1270200/4848000] [262.0/1000.0] [batch_t 0.331 (0.332)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 08:14:58,360 - Train: 26.20% [1270300/4848000] [262.0/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 08:15:31,542 - Train: 26.20% [1270400/4848000] [262.0/1000.0] [batch_t 0.333 (0.332)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 08:16:05,977 - Train: 26.21% [1270500/4848000] [262.1/1000.0] [batch_t 1.614 (0.344)] [data_t 1.278] [optim_t 0.336] [lr 0.005000] 2024-04-08 08:16:39,114 - Train: 26.21% [1270600/4848000] [262.1/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 08:17:12,283 - Train: 26.21% [1270700/4848000] [262.1/1000.0] [batch_t 0.333 (0.332)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 08:17:45,428 - Train: 26.21% [1270800/4848000] [262.1/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:18:18,569 - Train: 26.21% [1270900/4848000] [262.1/1000.0] [batch_t 0.340 (0.331)] [data_t 0.003] [optim_t 0.337] [lr 0.005000] 2024-04-08 08:18:51,774 - Train: 26.22% [1271000/4848000] [262.2/1000.0] [batch_t 0.336 (0.332)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 08:19:26,915 - Train: 26.22% [1271100/4848000] [262.2/1000.0] [batch_t 0.332 (0.351)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 08:20:00,187 - Train: 26.22% [1271200/4848000] [262.2/1000.0] [batch_t 0.329 (0.333)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:20:33,298 - Train: 26.22% [1271300/4848000] [262.2/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 08:21:06,399 - Train: 26.23% [1271400/4848000] [262.3/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 08:21:39,568 - Train: 26.23% [1271500/4848000] [262.3/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:22:12,726 - Train: 26.23% [1271600/4848000] [262.3/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 08:22:45,893 - Train: 26.23% [1271700/4848000] [262.3/1000.0] [batch_t 0.339 (0.332)] [data_t 0.003] [optim_t 0.336] [lr 0.005000] 2024-04-08 08:23:19,104 - Train: 26.23% [1271800/4848000] [262.3/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:23:52,280 - Train: 26.24% [1271900/4848000] [262.4/1000.0] [batch_t 0.337 (0.332)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-08 08:24:25,473 - Train: 26.24% [1272000/4848000] [262.4/1000.0] [batch_t 0.341 (0.332)] [data_t 0.003] [optim_t 0.338] [lr 0.005000] 2024-04-08 08:24:58,736 - Train: 26.24% [1272100/4848000] [262.4/1000.0] [batch_t 0.330 (0.333)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:25:32,849 - Train: 26.24% [1272200/4848000] [262.4/1000.0] [batch_t 0.327 (0.341)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 08:26:07,061 - Train: 26.24% [1272300/4848000] [262.4/1000.0] [batch_t 0.334 (0.342)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 08:26:40,243 - Train: 26.25% [1272400/4848000] [262.5/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:27:13,676 - Train: 26.25% [1272500/4848000] [262.5/1000.0] [batch_t 0.324 (0.334)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 08:27:46,824 - Train: 26.25% [1272600/4848000] [262.5/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 08:28:20,125 - Train: 26.25% [1272700/4848000] [262.5/1000.0] [batch_t 0.329 (0.333)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:28:53,263 - Train: 26.25% [1272800/4848000] [262.5/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:29:26,385 - Train: 26.26% [1272900/4848000] [262.6/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 08:29:59,547 - Train: 26.26% [1273000/4848000] [262.6/1000.0] [batch_t 0.328 (0.332)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 08:30:32,683 - Train: 26.26% [1273100/4848000] [262.6/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 08:31:06,495 - Train: 26.26% [1273200/4848000] [262.6/1000.0] [batch_t 0.327 (0.338)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 08:31:39,648 - Train: 26.26% [1273300/4848000] [262.6/1000.0] [batch_t 0.338 (0.331)] [data_t 0.003] [optim_t 0.335] [lr 0.005000] 2024-04-08 08:32:13,182 - Train: 26.27% [1273400/4848000] [262.7/1000.0] [batch_t 0.333 (0.335)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 08:32:46,245 - Train: 26.27% [1273500/4848000] [262.7/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:33:19,479 - Train: 26.27% [1273600/4848000] [262.7/1000.0] [batch_t 0.330 (0.332)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 08:33:52,624 - Train: 26.27% [1273700/4848000] [262.7/1000.0] [batch_t 0.336 (0.331)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 08:34:26,890 - Train: 26.27% [1273800/4848000] [262.7/1000.0] [batch_t 0.330 (0.343)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 08:34:59,955 - Train: 26.28% [1273900/4848000] [262.8/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:35:33,062 - Train: 26.28% [1274000/4848000] [262.8/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:36:06,330 - Train: 26.28% [1274100/4848000] [262.8/1000.0] [batch_t 0.337 (0.333)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-08 08:36:39,541 - Train: 26.28% [1274200/4848000] [262.8/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:37:12,612 - Train: 26.29% [1274300/4848000] [262.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:37:45,826 - Train: 26.29% [1274400/4848000] [262.9/1000.0] [batch_t 0.345 (0.332)] [data_t 0.003] [optim_t 0.342] [lr 0.005000] 2024-04-08 08:38:18,983 - Train: 26.29% [1274500/4848000] [262.9/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 08:38:52,119 - Train: 26.29% [1274600/4848000] [262.9/1000.0] [batch_t 0.325 (0.331)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 08:39:25,616 - Train: 26.29% [1274700/4848000] [262.9/1000.0] [batch_t 0.329 (0.335)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:39:58,744 - Train: 26.30% [1274800/4848000] [263.0/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 08:40:31,821 - Train: 26.30% [1274900/4848000] [263.0/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:41:05,876 - Train: 26.30% [1275000/4848000] [263.0/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:41:13,807 - ==> Total time: 5 days, 9:15:12 Eta: 15 days, 2:12:16 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 08:41:41,043 - Train: 26.30% [1275100/4848000] [263.0/1000.0] [batch_t 0.326 (0.332)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 08:42:14,155 - Train: 26.30% [1275200/4848000] [263.0/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 08:42:47,295 - Train: 26.31% [1275300/4848000] [263.1/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:43:20,596 - Train: 26.31% [1275400/4848000] [263.1/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:43:53,684 - Train: 26.31% [1275500/4848000] [263.1/1000.0] [batch_t 0.336 (0.331)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 08:44:28,030 - Train: 26.31% [1275600/4848000] [263.1/1000.0] [batch_t 0.329 (0.343)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:45:01,129 - Train: 26.31% [1275700/4848000] [263.1/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 08:45:34,240 - Train: 26.32% [1275800/4848000] [263.2/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:46:07,722 - Train: 26.32% [1275900/4848000] [263.2/1000.0] [batch_t 0.326 (0.335)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 08:46:40,865 - Train: 26.32% [1276000/4848000] [263.2/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:47:13,928 - Train: 26.32% [1276100/4848000] [263.2/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:47:47,015 - Train: 26.32% [1276200/4848000] [263.2/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:48:20,138 - Train: 26.33% [1276300/4848000] [263.3/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 08:48:53,233 - Train: 26.33% [1276400/4848000] [263.3/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:49:27,610 - Train: 26.33% [1276500/4848000] [263.3/1000.0] [batch_t 0.330 (0.344)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:50:00,847 - Train: 26.33% [1276600/4848000] [263.3/1000.0] [batch_t 0.332 (0.332)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 08:50:34,756 - Train: 26.33% [1276700/4848000] [263.3/1000.0] [batch_t 0.331 (0.339)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 08:51:08,476 - Train: 26.34% [1276800/4848000] [263.4/1000.0] [batch_t 0.332 (0.337)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 08:51:41,894 - Train: 26.34% [1276900/4848000] [263.4/1000.0] [batch_t 0.330 (0.334)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:52:15,764 - Train: 26.34% [1277000/4848000] [263.4/1000.0] [batch_t 0.333 (0.339)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 08:52:49,081 - Train: 26.34% [1277100/4848000] [263.4/1000.0] [batch_t 0.335 (0.333)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 08:53:22,489 - Train: 26.34% [1277200/4848000] [263.4/1000.0] [batch_t 0.332 (0.334)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 08:53:55,675 - Train: 26.35% [1277300/4848000] [263.5/1000.0] [batch_t 0.331 (0.332)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 08:54:29,970 - Train: 26.35% [1277400/4848000] [263.5/1000.0] [batch_t 0.332 (0.343)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 08:55:03,266 - Train: 26.35% [1277500/4848000] [263.5/1000.0] [batch_t 0.331 (0.333)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 08:55:36,575 - Train: 26.35% [1277600/4848000] [263.5/1000.0] [batch_t 0.334 (0.333)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 08:56:10,820 - Train: 26.36% [1277700/4848000] [263.6/1000.0] [batch_t 0.330 (0.342)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 08:56:44,035 - Train: 26.36% [1277800/4848000] [263.6/1000.0] [batch_t 0.335 (0.332)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 08:57:17,923 - Train: 26.36% [1277900/4848000] [263.6/1000.0] [batch_t 0.331 (0.339)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 08:57:51,109 - Train: 26.36% [1278000/4848000] [263.6/1000.0] [batch_t 0.329 (0.332)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:58:24,288 - Train: 26.36% [1278100/4848000] [263.6/1000.0] [batch_t 0.326 (0.332)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 08:58:57,408 - Train: 26.37% [1278200/4848000] [263.7/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 08:59:31,745 - Train: 26.37% [1278300/4848000] [263.7/1000.0] [batch_t 0.329 (0.343)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:00:04,915 - Train: 26.37% [1278400/4848000] [263.7/1000.0] [batch_t 0.333 (0.332)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 09:00:38,466 - Train: 26.37% [1278500/4848000] [263.7/1000.0] [batch_t 0.328 (0.335)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 09:01:11,676 - Train: 26.37% [1278600/4848000] [263.7/1000.0] [batch_t 0.329 (0.332)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 09:01:44,886 - Train: 26.38% [1278700/4848000] [263.8/1000.0] [batch_t 0.341 (0.332)] [data_t 0.003] [optim_t 0.338] [lr 0.005000] 2024-04-08 09:02:18,566 - Train: 26.38% [1278800/4848000] [263.8/1000.0] [batch_t 0.331 (0.337)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 09:02:51,740 - Train: 26.38% [1278900/4848000] [263.8/1000.0] [batch_t 0.324 (0.332)] [data_t 0.003] [optim_t 0.321] [lr 0.005000] 2024-04-08 09:03:24,985 - Train: 26.38% [1279000/4848000] [263.8/1000.0] [batch_t 0.325 (0.332)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-08 09:03:58,150 - Train: 26.38% [1279100/4848000] [263.8/1000.0] [batch_t 0.330 (0.332)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:04:32,827 - Train: 26.39% [1279200/4848000] [263.9/1000.0] [batch_t 0.331 (0.347)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 09:05:05,992 - Train: 26.39% [1279300/4848000] [263.9/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:05:39,144 - Train: 26.39% [1279400/4848000] [263.9/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 09:06:14,365 - Train: 26.39% [1279500/4848000] [263.9/1000.0] [batch_t 0.329 (0.352)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 09:06:47,533 - Train: 26.39% [1279600/4848000] [263.9/1000.0] [batch_t 0.327 (0.332)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 09:07:20,907 - Train: 26.40% [1279700/4848000] [264.0/1000.0] [batch_t 0.331 (0.334)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 09:07:54,031 - Train: 26.40% [1279800/4848000] [264.0/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 09:08:19,283 - ==> Total time: 5 days, 9:42:17 Eta: 15 days, 1:36:05 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 09:08:30,762 - Train: 26.40% [1279900/4848000] [264.0/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 09:09:03,880 - Train: 26.40% [1280000/4848000] [264.0/1000.0] [batch_t 0.327 (0.331)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 09:09:36,952 - Train: 26.40% [1280100/4848000] [264.0/1000.0] [batch_t 0.339 (0.331)] [data_t 0.003] [optim_t 0.336] [lr 0.005000] 2024-04-08 09:10:10,934 - Train: 26.41% [1280200/4848000] [264.1/1000.0] [batch_t 0.330 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:10:44,057 - Train: 26.41% [1280300/4848000] [264.1/1000.0] [batch_t 0.333 (0.331)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 09:11:17,206 - Train: 26.41% [1280400/4848000] [264.1/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 09:11:50,405 - Train: 26.41% [1280500/4848000] [264.1/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:12:23,579 - Train: 26.42% [1280600/4848000] [264.2/1000.0] [batch_t 0.328 (0.332)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 09:12:56,692 - Train: 26.42% [1280700/4848000] [264.2/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:13:30,360 - Train: 26.42% [1280800/4848000] [264.2/1000.0] [batch_t 0.330 (0.337)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 09:14:03,519 - Train: 26.42% [1280900/4848000] [264.2/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 09:14:37,650 - Train: 26.42% [1281000/4848000] [264.2/1000.0] [batch_t 0.332 (0.341)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 09:15:10,780 - Train: 26.43% [1281100/4848000] [264.3/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 09:15:43,863 - Train: 26.43% [1281200/4848000] [264.3/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:16:16,928 - Train: 26.43% [1281300/4848000] [264.3/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 09:16:50,087 - Train: 26.43% [1281400/4848000] [264.3/1000.0] [batch_t 0.326 (0.331)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 09:17:23,207 - Train: 26.43% [1281500/4848000] [264.3/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 09:17:56,439 - Train: 26.44% [1281600/4848000] [264.4/1000.0] [batch_t 0.332 (0.332)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 09:18:29,611 - Train: 26.44% [1281700/4848000] [264.4/1000.0] [batch_t 0.329 (0.332)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:19:02,757 - Train: 26.44% [1281800/4848000] [264.4/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 09:19:35,861 - Train: 26.44% [1281900/4848000] [264.4/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 09:20:08,910 - Train: 26.44% [1282000/4848000] [264.4/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 09:20:41,971 - Train: 26.45% [1282100/4848000] [264.5/1000.0] [batch_t 0.326 (0.331)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 09:21:15,088 - Train: 26.45% [1282200/4848000] [264.5/1000.0] [batch_t 0.325 (0.331)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 09:21:48,248 - Train: 26.45% [1282300/4848000] [264.5/1000.0] [batch_t 0.338 (0.332)] [data_t 0.003] [optim_t 0.335] [lr 0.005000] 2024-04-08 09:22:21,391 - Train: 26.45% [1282400/4848000] [264.5/1000.0] [batch_t 0.327 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 09:22:54,614 - Train: 26.45% [1282500/4848000] [264.5/1000.0] [batch_t 0.330 (0.332)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:23:27,713 - Train: 26.46% [1282600/4848000] [264.6/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 09:24:00,914 - Train: 26.46% [1282700/4848000] [264.6/1000.0] [batch_t 0.328 (0.332)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 09:24:34,033 - Train: 26.46% [1282800/4848000] [264.6/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 09:25:07,257 - Train: 26.46% [1282900/4848000] [264.6/1000.0] [batch_t 0.325 (0.332)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-08 09:25:40,373 - Train: 26.46% [1283000/4848000] [264.6/1000.0] [batch_t 0.334 (0.331)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 09:26:13,559 - Train: 26.47% [1283100/4848000] [264.7/1000.0] [batch_t 0.336 (0.332)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 09:26:46,648 - Train: 26.47% [1283200/4848000] [264.7/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:27:19,770 - Train: 26.47% [1283300/4848000] [264.7/1000.0] [batch_t 0.327 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 09:27:52,916 - Train: 26.47% [1283400/4848000] [264.7/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 09:28:25,999 - Train: 26.47% [1283500/4848000] [264.7/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 09:28:59,199 - Train: 26.48% [1283600/4848000] [264.8/1000.0] [batch_t 0.329 (0.332)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 09:29:32,421 - Train: 26.48% [1283700/4848000] [264.8/1000.0] [batch_t 0.324 (0.332)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-08 09:30:05,477 - Train: 26.48% [1283800/4848000] [264.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:30:38,702 - Train: 26.48% [1283900/4848000] [264.8/1000.0] [batch_t 0.323 (0.332)] [data_t 0.003] [optim_t 0.321] [lr 0.005000] 2024-04-08 09:31:11,808 - Train: 26.49% [1284000/4848000] [264.9/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 09:31:44,945 - Train: 26.49% [1284100/4848000] [264.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:32:18,711 - Train: 26.49% [1284200/4848000] [264.9/1000.0] [batch_t 0.334 (0.338)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 09:32:51,897 - Train: 26.49% [1284300/4848000] [264.9/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 09:33:26,276 - Train: 26.49% [1284400/4848000] [264.9/1000.0] [batch_t 0.329 (0.344)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:34:00,830 - Train: 26.50% [1284500/4848000] [265.0/1000.0] [batch_t 0.328 (0.345)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 09:34:33,868 - Train: 26.50% [1284600/4848000] [265.0/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:35:10,777 - Train: 26.50% [1284700/4848000] [265.0/1000.0] [batch_t 0.331 (0.369)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 09:35:19,549 - ==> Total time: 5 days, 10:09:18 Eta: 15 days, 0:59:45 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 09:35:49,536 - Train: 26.50% [1284800/4848000] [265.0/1000.0] [batch_t 0.331 (0.348)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 09:36:25,212 - Train: 26.50% [1284900/4848000] [265.0/1000.0] [batch_t 0.331 (0.357)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 09:36:58,343 - Train: 26.51% [1285000/4848000] [265.1/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:37:32,413 - Train: 26.51% [1285100/4848000] [265.1/1000.0] [batch_t 0.323 (0.341)] [data_t 0.002] [optim_t 0.321] [lr 0.005000] 2024-04-08 09:38:06,120 - Train: 26.51% [1285200/4848000] [265.1/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 09:38:39,259 - Train: 26.51% [1285300/4848000] [265.1/1000.0] [batch_t 0.340 (0.331)] [data_t 0.002] [optim_t 0.338] [lr 0.005000] 2024-04-08 09:39:12,450 - Train: 26.51% [1285400/4848000] [265.1/1000.0] [batch_t 0.326 (0.332)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 09:39:45,549 - Train: 26.52% [1285500/4848000] [265.2/1000.0] [batch_t 0.338 (0.331)] [data_t 0.003] [optim_t 0.335] [lr 0.005000] 2024-04-08 09:40:18,650 - Train: 26.52% [1285600/4848000] [265.2/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 09:40:51,806 - Train: 26.52% [1285700/4848000] [265.2/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:41:26,218 - Train: 26.52% [1285800/4848000] [265.2/1000.0] [batch_t 0.333 (0.344)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 09:41:59,304 - Train: 26.52% [1285900/4848000] [265.2/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 09:42:36,346 - Train: 26.53% [1286000/4848000] [265.3/1000.0] [batch_t 0.330 (0.370)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 09:43:09,485 - Train: 26.53% [1286100/4848000] [265.3/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 09:43:42,553 - Train: 26.53% [1286200/4848000] [265.3/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 09:44:15,696 - Train: 26.53% [1286300/4848000] [265.3/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:44:48,821 - Train: 26.53% [1286400/4848000] [265.3/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 09:45:21,864 - Train: 26.54% [1286500/4848000] [265.4/1000.0] [batch_t 0.334 (0.330)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 09:45:59,751 - Train: 26.54% [1286600/4848000] [265.4/1000.0] [batch_t 0.328 (0.379)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 09:46:33,975 - Train: 26.54% [1286700/4848000] [265.4/1000.0] [batch_t 0.338 (0.342)] [data_t 0.003] [optim_t 0.335] [lr 0.005000] 2024-04-08 09:47:07,952 - Train: 26.54% [1286800/4848000] [265.4/1000.0] [batch_t 0.332 (0.340)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 09:47:40,972 - Train: 26.54% [1286900/4848000] [265.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 09:48:14,123 - Train: 26.55% [1287000/4848000] [265.5/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 09:48:47,247 - Train: 26.55% [1287100/4848000] [265.5/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:49:20,337 - Train: 26.55% [1287200/4848000] [265.5/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:49:53,396 - Train: 26.55% [1287300/4848000] [265.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:50:27,720 - Train: 26.56% [1287400/4848000] [265.6/1000.0] [batch_t 0.331 (0.343)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 09:51:00,815 - Train: 26.56% [1287500/4848000] [265.6/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 09:51:35,166 - Train: 26.56% [1287600/4848000] [265.6/1000.0] [batch_t 0.335 (0.343)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 09:52:08,706 - Train: 26.56% [1287700/4848000] [265.6/1000.0] [batch_t 0.332 (0.335)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 09:52:41,772 - Train: 26.56% [1287800/4848000] [265.6/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 09:53:15,461 - Train: 26.57% [1287900/4848000] [265.7/1000.0] [batch_t 0.333 (0.337)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 09:53:48,732 - Train: 26.57% [1288000/4848000] [265.7/1000.0] [batch_t 0.328 (0.333)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 09:54:23,147 - Train: 26.57% [1288100/4848000] [265.7/1000.0] [batch_t 0.330 (0.344)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 09:54:58,626 - Train: 26.57% [1288200/4848000] [265.7/1000.0] [batch_t 0.431 (0.355)] [data_t 0.102] [optim_t 0.329] [lr 0.005000] 2024-04-08 09:55:33,213 - Train: 26.57% [1288300/4848000] [265.7/1000.0] [batch_t 0.330 (0.346)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:56:06,752 - Train: 26.58% [1288400/4848000] [265.8/1000.0] [batch_t 0.330 (0.335)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 09:56:40,454 - Train: 26.58% [1288500/4848000] [265.8/1000.0] [batch_t 0.331 (0.337)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 09:57:13,541 - Train: 26.58% [1288600/4848000] [265.8/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 09:57:48,061 - Train: 26.58% [1288700/4848000] [265.8/1000.0] [batch_t 0.330 (0.345)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:58:22,207 - Train: 26.58% [1288800/4848000] [265.8/1000.0] [batch_t 0.330 (0.341)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 09:58:55,417 - Train: 26.59% [1288900/4848000] [265.9/1000.0] [batch_t 0.334 (0.332)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 09:59:29,647 - Train: 26.59% [1289000/4848000] [265.9/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 10:00:02,881 - Train: 26.59% [1289100/4848000] [265.9/1000.0] [batch_t 0.328 (0.332)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:00:36,138 - Train: 26.59% [1289200/4848000] [265.9/1000.0] [batch_t 0.327 (0.332)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:01:09,260 - Train: 26.59% [1289300/4848000] [265.9/1000.0] [batch_t 0.337 (0.331)] [data_t 0.003] [optim_t 0.335] [lr 0.005000] 2024-04-08 10:01:42,346 - Train: 26.60% [1289400/4848000] [266.0/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 10:02:16,680 - Train: 26.60% [1289500/4848000] [266.0/1000.0] [batch_t 0.331 (0.343)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 10:02:39,323 - ==> Total time: 5 days, 10:36:37 Eta: 15 days, 0:24:23 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 10:02:52,136 - Train: 26.60% [1289600/4848000] [266.0/1000.0] [batch_t 0.330 (0.335)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 10:03:25,845 - Train: 26.60% [1289700/4848000] [266.0/1000.0] [batch_t 0.326 (0.337)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 10:03:58,965 - Train: 26.60% [1289800/4848000] [266.0/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 10:04:32,167 - Train: 26.61% [1289900/4848000] [266.1/1000.0] [batch_t 0.328 (0.332)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 10:05:05,299 - Train: 26.61% [1290000/4848000] [266.1/1000.0] [batch_t 0.406 (0.331)] [data_t 0.077] [optim_t 0.329] [lr 0.005000] 2024-04-08 10:05:38,811 - Train: 26.61% [1290100/4848000] [266.1/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:06:11,936 - Train: 26.61% [1290200/4848000] [266.1/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 10:06:45,149 - Train: 26.62% [1290300/4848000] [266.2/1000.0] [batch_t 0.331 (0.332)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 10:07:20,237 - Train: 26.62% [1290400/4848000] [266.2/1000.0] [batch_t 0.333 (0.351)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 10:07:53,534 - Train: 26.62% [1290500/4848000] [266.2/1000.0] [batch_t 0.332 (0.333)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 10:08:26,719 - Train: 26.62% [1290600/4848000] [266.2/1000.0] [batch_t 0.329 (0.332)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 10:08:59,917 - Train: 26.62% [1290700/4848000] [266.2/1000.0] [batch_t 0.336 (0.332)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 10:09:33,149 - Train: 26.63% [1290800/4848000] [266.3/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 10:10:07,764 - Train: 26.63% [1290900/4848000] [266.3/1000.0] [batch_t 0.332 (0.346)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 10:10:40,991 - Train: 26.63% [1291000/4848000] [266.3/1000.0] [batch_t 0.326 (0.332)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 10:11:15,615 - Train: 26.63% [1291100/4848000] [266.3/1000.0] [batch_t 0.330 (0.346)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 10:11:48,714 - Train: 26.63% [1291200/4848000] [266.3/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 10:12:21,950 - Train: 26.64% [1291300/4848000] [266.4/1000.0] [batch_t 0.330 (0.332)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 10:12:55,032 - Train: 26.64% [1291400/4848000] [266.4/1000.0] [batch_t 0.327 (0.331)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 10:13:29,841 - Train: 26.64% [1291500/4848000] [266.4/1000.0] [batch_t 0.344 (0.348)] [data_t 0.003] [optim_t 0.341] [lr 0.005000] 2024-04-08 10:14:02,984 - Train: 26.64% [1291600/4848000] [266.4/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 10:14:44,335 - Train: 26.64% [1291700/4848000] [266.4/1000.0] [batch_t 0.329 (0.413)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 10:15:19,279 - Train: 26.65% [1291800/4848000] [266.5/1000.0] [batch_t 0.334 (0.349)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 10:15:52,374 - Train: 26.65% [1291900/4848000] [266.5/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 10:16:27,308 - Train: 26.65% [1292000/4848000] [266.5/1000.0] [batch_t 0.333 (0.349)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 10:17:00,444 - Train: 26.65% [1292100/4848000] [266.5/1000.0] [batch_t 0.333 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 10:17:35,125 - Train: 26.65% [1292200/4848000] [266.5/1000.0] [batch_t 0.328 (0.347)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 10:18:09,433 - Train: 26.66% [1292300/4848000] [266.6/1000.0] [batch_t 0.330 (0.343)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 10:18:42,456 - Train: 26.66% [1292400/4848000] [266.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 10:19:16,810 - Train: 26.66% [1292500/4848000] [266.6/1000.0] [batch_t 0.329 (0.343)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 10:19:49,921 - Train: 26.66% [1292600/4848000] [266.6/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:20:26,091 - Train: 26.66% [1292700/4848000] [266.6/1000.0] [batch_t 0.332 (0.362)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 10:20:59,133 - Train: 26.67% [1292800/4848000] [266.7/1000.0] [batch_t 0.334 (0.330)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 10:21:33,889 - Train: 26.67% [1292900/4848000] [266.7/1000.0] [batch_t 0.333 (0.347)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 10:22:08,129 - Train: 26.67% [1293000/4848000] [266.7/1000.0] [batch_t 0.331 (0.342)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 10:22:41,331 - Train: 26.67% [1293100/4848000] [266.7/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 10:23:16,493 - Train: 26.67% [1293200/4848000] [266.7/1000.0] [batch_t 0.327 (0.352)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 10:23:49,694 - Train: 26.68% [1293300/4848000] [266.8/1000.0] [batch_t 0.327 (0.332)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 10:24:24,494 - Train: 26.68% [1293400/4848000] [266.8/1000.0] [batch_t 0.330 (0.348)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 10:24:57,691 - Train: 26.68% [1293500/4848000] [266.8/1000.0] [batch_t 0.332 (0.332)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 10:25:31,665 - Train: 26.68% [1293600/4848000] [266.8/1000.0] [batch_t 0.333 (0.340)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 10:26:06,829 - Train: 26.69% [1293700/4848000] [266.9/1000.0] [batch_t 0.329 (0.352)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 10:26:40,268 - Train: 26.69% [1293800/4848000] [266.9/1000.0] [batch_t 0.340 (0.334)] [data_t 0.003] [optim_t 0.337] [lr 0.005000] 2024-04-08 10:27:15,183 - Train: 26.69% [1293900/4848000] [266.9/1000.0] [batch_t 0.331 (0.349)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 10:27:48,289 - Train: 26.69% [1294000/4848000] [266.9/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 10:28:23,939 - Train: 26.69% [1294100/4848000] [266.9/1000.0] [batch_t 0.327 (0.356)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:28:57,140 - Train: 26.70% [1294200/4848000] [267.0/1000.0] [batch_t 0.330 (0.332)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 10:29:31,322 - Train: 26.70% [1294300/4848000] [267.0/1000.0] [batch_t 0.332 (0.342)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 10:30:06,739 - Train: 26.70% [1294400/4848000] [267.0/1000.0] [batch_t 2.615 (0.354)] [data_t 2.282] [optim_t 0.333] [lr 0.005000] 2024-04-08 10:30:13,216 - ==> Total time: 5 days, 11:04:11 Eta: 14 days, 23:49:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 10:30:43,417 - Train: 26.70% [1294500/4848000] [267.0/1000.0] [batch_t 0.334 (0.334)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 10:31:19,562 - Train: 26.70% [1294600/4848000] [267.0/1000.0] [batch_t 0.335 (0.361)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 10:31:52,705 - Train: 26.71% [1294700/4848000] [267.1/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:32:28,174 - Train: 26.71% [1294800/4848000] [267.1/1000.0] [batch_t 0.334 (0.355)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 10:33:12,386 - Train: 26.71% [1294900/4848000] [267.1/1000.0] [batch_t 0.740 (0.442)] [data_t 0.412] [optim_t 0.327] [lr 0.005000] 2024-04-08 10:33:50,689 - Train: 26.71% [1295000/4848000] [267.1/1000.0] [batch_t 0.334 (0.383)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 10:34:25,699 - Train: 26.71% [1295100/4848000] [267.1/1000.0] [batch_t 0.329 (0.350)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 10:34:58,792 - Train: 26.72% [1295200/4848000] [267.2/1000.0] [batch_t 0.337 (0.331)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-08 10:35:36,846 - Train: 26.72% [1295300/4848000] [267.2/1000.0] [batch_t 0.332 (0.380)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 10:36:12,875 - Train: 26.72% [1295400/4848000] [267.2/1000.0] [batch_t 0.334 (0.360)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 10:36:46,262 - Train: 26.72% [1295500/4848000] [267.2/1000.0] [batch_t 0.332 (0.334)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 10:37:24,180 - Train: 26.72% [1295600/4848000] [267.2/1000.0] [batch_t 0.335 (0.379)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 10:37:57,379 - Train: 26.73% [1295700/4848000] [267.3/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 10:38:32,901 - Train: 26.73% [1295800/4848000] [267.3/1000.0] [batch_t 0.331 (0.355)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 10:39:08,137 - Train: 26.73% [1295900/4848000] [267.3/1000.0] [batch_t 0.330 (0.352)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 10:39:41,940 - Train: 26.73% [1296000/4848000] [267.3/1000.0] [batch_t 0.332 (0.338)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 10:40:15,641 - Train: 26.73% [1296100/4848000] [267.3/1000.0] [batch_t 0.334 (0.337)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 10:40:48,807 - Train: 26.74% [1296200/4848000] [267.4/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 10:41:21,998 - Train: 26.74% [1296300/4848000] [267.4/1000.0] [batch_t 0.334 (0.332)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 10:41:55,146 - Train: 26.74% [1296400/4848000] [267.4/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 10:42:29,661 - Train: 26.74% [1296500/4848000] [267.4/1000.0] [batch_t 0.326 (0.345)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 10:43:02,931 - Train: 26.75% [1296600/4848000] [267.5/1000.0] [batch_t 0.329 (0.333)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 10:43:38,669 - Train: 26.75% [1296700/4848000] [267.5/1000.0] [batch_t 0.327 (0.357)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 10:44:12,880 - Train: 26.75% [1296800/4848000] [267.5/1000.0] [batch_t 0.328 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 10:44:45,955 - Train: 26.75% [1296900/4848000] [267.5/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:45:21,716 - Train: 26.75% [1297000/4848000] [267.5/1000.0] [batch_t 0.334 (0.356)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 10:45:54,828 - Train: 26.76% [1297100/4848000] [267.6/1000.0] [batch_t 0.327 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:46:29,457 - Train: 26.76% [1297200/4848000] [267.6/1000.0] [batch_t 0.325 (0.346)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 10:47:02,504 - Train: 26.76% [1297300/4848000] [267.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 10:47:36,782 - Train: 26.76% [1297400/4848000] [267.6/1000.0] [batch_t 0.329 (0.343)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 10:48:11,439 - Train: 26.76% [1297500/4848000] [267.6/1000.0] [batch_t 0.327 (0.346)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:48:44,549 - Train: 26.77% [1297600/4848000] [267.7/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 10:49:22,011 - Train: 26.77% [1297700/4848000] [267.7/1000.0] [batch_t 0.329 (0.375)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 10:49:55,199 - Train: 26.77% [1297800/4848000] [267.7/1000.0] [batch_t 0.342 (0.332)] [data_t 0.003] [optim_t 0.340] [lr 0.005000] 2024-04-08 10:50:31,991 - Train: 26.77% [1297900/4848000] [267.7/1000.0] [batch_t 0.330 (0.368)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 10:51:06,986 - Train: 26.77% [1298000/4848000] [267.7/1000.0] [batch_t 0.327 (0.350)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 10:51:40,031 - Train: 26.78% [1298100/4848000] [267.8/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 10:52:15,417 - Train: 26.78% [1298200/4848000] [267.8/1000.0] [batch_t 0.326 (0.354)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 10:52:48,582 - Train: 26.78% [1298300/4848000] [267.8/1000.0] [batch_t 0.333 (0.332)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 10:53:23,298 - Train: 26.78% [1298400/4848000] [267.8/1000.0] [batch_t 0.327 (0.347)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:53:56,488 - Train: 26.78% [1298500/4848000] [267.8/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:54:31,058 - Train: 26.79% [1298600/4848000] [267.9/1000.0] [batch_t 0.332 (0.346)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 10:55:04,202 - Train: 26.79% [1298700/4848000] [267.9/1000.0] [batch_t 0.323 (0.331)] [data_t 0.003] [optim_t 0.321] [lr 0.005000] 2024-04-08 10:55:39,650 - Train: 26.79% [1298800/4848000] [267.9/1000.0] [batch_t 0.331 (0.354)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 10:56:14,212 - Train: 26.79% [1298900/4848000] [267.9/1000.0] [batch_t 0.329 (0.346)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 10:56:47,420 - Train: 26.79% [1299000/4848000] [267.9/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 10:57:21,787 - Train: 26.80% [1299100/4848000] [268.0/1000.0] [batch_t 0.331 (0.344)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 10:57:55,026 - Train: 26.80% [1299200/4848000] [268.0/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 10:58:17,893 - ==> Total time: 5 days, 11:32:16 Eta: 14 days, 23:16:30 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 10:58:32,442 - Train: 26.80% [1299300/4848000] [268.0/1000.0] [batch_t 0.334 (0.343)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 10:59:08,095 - Train: 26.80% [1299400/4848000] [268.0/1000.0] [batch_t 0.330 (0.356)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 10:59:41,227 - Train: 26.80% [1299500/4848000] [268.0/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 11:00:17,117 - Train: 26.81% [1299600/4848000] [268.1/1000.0] [batch_t 0.331 (0.359)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 11:00:50,257 - Train: 26.81% [1299700/4848000] [268.1/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 11:01:25,017 - Train: 26.81% [1299800/4848000] [268.1/1000.0] [batch_t 0.328 (0.347)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 11:01:58,083 - Train: 26.81% [1299900/4848000] [268.1/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 11:02:34,993 - Train: 26.82% [1300000/4848000] [268.2/1000.0] [batch_t 0.331 (0.369)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 11:03:10,187 - Train: 26.82% [1300100/4848000] [268.2/1000.0] [batch_t 0.325 (0.352)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-08 11:03:43,287 - Train: 26.82% [1300200/4848000] [268.2/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 11:04:18,258 - Train: 26.82% [1300300/4848000] [268.2/1000.0] [batch_t 0.327 (0.350)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 11:04:59,912 - Train: 26.82% [1300400/4848000] [268.2/1000.0] [batch_t 0.332 (0.416)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 11:05:39,380 - Train: 26.83% [1300500/4848000] [268.3/1000.0] [batch_t 0.330 (0.395)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 11:06:14,036 - Train: 26.83% [1300600/4848000] [268.3/1000.0] [batch_t 0.330 (0.346)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 11:06:47,180 - Train: 26.83% [1300700/4848000] [268.3/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 11:07:23,463 - Train: 26.83% [1300800/4848000] [268.3/1000.0] [batch_t 0.334 (0.363)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 11:07:56,563 - Train: 26.83% [1300900/4848000] [268.3/1000.0] [batch_t 0.325 (0.331)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 11:08:32,356 - Train: 26.84% [1301000/4848000] [268.4/1000.0] [batch_t 0.335 (0.358)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 11:09:08,513 - Train: 26.84% [1301100/4848000] [268.4/1000.0] [batch_t 0.322 (0.361)] [data_t 0.003] [optim_t 0.320] [lr 0.005000] 2024-04-08 11:09:41,594 - Train: 26.84% [1301200/4848000] [268.4/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 11:10:16,318 - Train: 26.84% [1301300/4848000] [268.4/1000.0] [batch_t 0.326 (0.347)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 11:10:49,492 - Train: 26.84% [1301400/4848000] [268.4/1000.0] [batch_t 0.337 (0.332)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-08 11:11:24,604 - Train: 26.85% [1301500/4848000] [268.5/1000.0] [batch_t 0.327 (0.351)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 11:11:57,712 - Train: 26.85% [1301600/4848000] [268.5/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 11:12:33,706 - Train: 26.85% [1301700/4848000] [268.5/1000.0] [batch_t 0.330 (0.360)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 11:13:09,424 - Train: 26.85% [1301800/4848000] [268.5/1000.0] [batch_t 0.329 (0.357)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 11:13:42,553 - Train: 26.85% [1301900/4848000] [268.5/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 11:14:18,285 - Train: 26.86% [1302000/4848000] [268.6/1000.0] [batch_t 0.330 (0.357)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 11:14:51,532 - Train: 26.86% [1302100/4848000] [268.6/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 11:15:27,340 - Train: 26.86% [1302200/4848000] [268.6/1000.0] [batch_t 0.332 (0.358)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 11:16:00,631 - Train: 26.86% [1302300/4848000] [268.6/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 11:16:34,459 - Train: 26.86% [1302400/4848000] [268.6/1000.0] [batch_t 0.332 (0.338)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 11:17:08,078 - Train: 26.87% [1302500/4848000] [268.7/1000.0] [batch_t 0.331 (0.336)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 11:17:41,262 - Train: 26.87% [1302600/4848000] [268.7/1000.0] [batch_t 0.331 (0.332)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 11:18:16,571 - Train: 26.87% [1302700/4848000] [268.7/1000.0] [batch_t 0.331 (0.353)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 11:18:49,707 - Train: 26.87% [1302800/4848000] [268.7/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 11:19:24,343 - Train: 26.88% [1302900/4848000] [268.8/1000.0] [batch_t 0.338 (0.346)] [data_t 0.003] [optim_t 0.335] [lr 0.005000] 2024-04-08 11:19:57,469 - Train: 26.88% [1303000/4848000] [268.8/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 11:20:31,932 - Train: 26.88% [1303100/4848000] [268.8/1000.0] [batch_t 0.337 (0.345)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-08 11:21:07,012 - Train: 26.88% [1303200/4848000] [268.8/1000.0] [batch_t 2.169 (0.351)] [data_t 1.839] [optim_t 0.330] [lr 0.005000] 2024-04-08 11:21:40,081 - Train: 26.88% [1303300/4848000] [268.8/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 11:22:14,320 - Train: 26.89% [1303400/4848000] [268.9/1000.0] [batch_t 0.337 (0.342)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-08 11:22:47,500 - Train: 26.89% [1303500/4848000] [268.9/1000.0] [batch_t 0.330 (0.332)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 11:23:21,732 - Train: 26.89% [1303600/4848000] [268.9/1000.0] [batch_t 0.331 (0.342)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 11:23:55,053 - Train: 26.89% [1303700/4848000] [268.9/1000.0] [batch_t 0.329 (0.333)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 11:24:34,435 - Train: 26.89% [1303800/4848000] [268.9/1000.0] [batch_t 0.338 (0.394)] [data_t 0.003] [optim_t 0.335] [lr 0.005000] 2024-04-08 11:25:08,846 - Train: 26.90% [1303900/4848000] [269.0/1000.0] [batch_t 0.339 (0.344)] [data_t 0.003] [optim_t 0.336] [lr 0.005000] 2024-04-08 11:25:42,147 - Train: 26.90% [1304000/4848000] [269.0/1000.0] [batch_t 0.329 (0.333)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 11:26:17,682 - Train: 26.90% [1304100/4848000] [269.0/1000.0] [batch_t 0.330 (0.355)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 11:26:21,654 - ==> Total time: 5 days, 12:00:20 Eta: 14 days, 22:43:17 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 11:26:53,164 - Train: 26.90% [1304200/4848000] [269.0/1000.0] [batch_t 0.335 (0.334)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 11:27:28,576 - Train: 26.90% [1304300/4848000] [269.0/1000.0] [batch_t 0.328 (0.354)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 11:28:01,739 - Train: 26.91% [1304400/4848000] [269.1/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 11:28:39,032 - Train: 26.91% [1304500/4848000] [269.1/1000.0] [batch_t 0.336 (0.373)] [data_t 0.006] [optim_t 0.330] [lr 0.005000] 2024-04-08 11:29:23,981 - Train: 26.91% [1304600/4848000] [269.1/1000.0] [batch_t 0.331 (0.449)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 11:29:57,104 - Train: 26.91% [1304700/4848000] [269.1/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 11:30:30,798 - Train: 26.91% [1304800/4848000] [269.1/1000.0] [batch_t 0.325 (0.337)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 11:31:04,057 - Train: 26.92% [1304900/4848000] [269.2/1000.0] [batch_t 0.328 (0.332)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 11:31:41,839 - Train: 26.92% [1305000/4848000] [269.2/1000.0] [batch_t 0.327 (0.378)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 11:32:16,730 - Train: 26.92% [1305100/4848000] [269.2/1000.0] [batch_t 0.335 (0.349)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 11:32:49,835 - Train: 26.92% [1305200/4848000] [269.2/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 11:33:24,168 - Train: 26.92% [1305300/4848000] [269.2/1000.0] [batch_t 0.330 (0.343)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 11:33:57,357 - Train: 26.93% [1305400/4848000] [269.3/1000.0] [batch_t 0.335 (0.332)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 11:34:31,944 - Train: 26.93% [1305500/4848000] [269.3/1000.0] [batch_t 0.326 (0.346)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 11:35:06,395 - Train: 26.93% [1305600/4848000] [269.3/1000.0] [batch_t 1.619 (0.344)] [data_t 1.286] [optim_t 0.333] [lr 0.005000] 2024-04-08 11:35:39,509 - Train: 26.93% [1305700/4848000] [269.3/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 11:36:14,264 - Train: 26.93% [1305800/4848000] [269.3/1000.0] [batch_t 0.332 (0.347)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 11:36:47,330 - Train: 26.94% [1305900/4848000] [269.4/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 11:37:21,760 - Train: 26.94% [1306000/4848000] [269.4/1000.0] [batch_t 0.327 (0.344)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 11:37:54,822 - Train: 26.94% [1306100/4848000] [269.4/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 11:38:30,077 - Train: 26.94% [1306200/4848000] [269.4/1000.0] [batch_t 0.335 (0.352)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 11:39:03,208 - Train: 26.95% [1306300/4848000] [269.5/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 11:39:37,035 - Train: 26.95% [1306400/4848000] [269.5/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 11:40:11,578 - Train: 26.95% [1306500/4848000] [269.5/1000.0] [batch_t 0.328 (0.345)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 11:40:44,660 - Train: 26.95% [1306600/4848000] [269.5/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 11:41:18,962 - Train: 26.95% [1306700/4848000] [269.5/1000.0] [batch_t 0.331 (0.343)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 11:41:52,091 - Train: 26.96% [1306800/4848000] [269.6/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 11:42:27,278 - Train: 26.96% [1306900/4848000] [269.6/1000.0] [batch_t 0.332 (0.352)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 11:43:00,400 - Train: 26.96% [1307000/4848000] [269.6/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 11:43:35,614 - Train: 26.96% [1307100/4848000] [269.6/1000.0] [batch_t 0.338 (0.352)] [data_t 0.003] [optim_t 0.335] [lr 0.005000] 2024-04-08 11:44:10,920 - Train: 26.96% [1307200/4848000] [269.6/1000.0] [batch_t 0.330 (0.353)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 11:44:44,055 - Train: 26.97% [1307300/4848000] [269.7/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 11:45:19,302 - Train: 26.97% [1307400/4848000] [269.7/1000.0] [batch_t 0.330 (0.352)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 11:45:52,393 - Train: 26.97% [1307500/4848000] [269.7/1000.0] [batch_t 0.333 (0.331)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 11:46:27,247 - Train: 26.97% [1307600/4848000] [269.7/1000.0] [batch_t 0.329 (0.348)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 11:47:00,463 - Train: 26.97% [1307700/4848000] [269.7/1000.0] [batch_t 0.331 (0.332)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 11:47:34,975 - Train: 26.98% [1307800/4848000] [269.8/1000.0] [batch_t 0.333 (0.345)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 11:48:10,503 - Train: 26.98% [1307900/4848000] [269.8/1000.0] [batch_t 0.327 (0.355)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 11:48:43,667 - Train: 26.98% [1308000/4848000] [269.8/1000.0] [batch_t 0.334 (0.332)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 11:49:18,414 - Train: 26.98% [1308100/4848000] [269.8/1000.0] [batch_t 0.326 (0.347)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 11:49:51,588 - Train: 26.98% [1308200/4848000] [269.8/1000.0] [batch_t 0.330 (0.332)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 11:50:25,943 - Train: 26.99% [1308300/4848000] [269.9/1000.0] [batch_t 0.337 (0.343)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-08 11:50:59,167 - Train: 26.99% [1308400/4848000] [269.9/1000.0] [batch_t 0.330 (0.332)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 11:51:32,544 - Train: 26.99% [1308500/4848000] [269.9/1000.0] [batch_t 0.336 (0.334)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-08 11:52:06,823 - Train: 26.99% [1308600/4848000] [269.9/1000.0] [batch_t 0.330 (0.343)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 11:52:39,993 - Train: 26.99% [1308700/4848000] [269.9/1000.0] [batch_t 0.330 (0.332)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 11:53:15,334 - Train: 27.00% [1308800/4848000] [270.0/1000.0] [batch_t 0.329 (0.353)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 11:53:48,524 - Train: 27.00% [1308900/4848000] [270.0/1000.0] [batch_t 0.329 (0.332)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 11:54:08,424 - ==> Total time: 5 days, 12:28:06 Eta: 14 days, 22:09:20 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 11:54:24,130 - Train: 27.00% [1309000/4848000] [270.0/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 11:54:57,268 - Train: 27.00% [1309100/4848000] [270.0/1000.0] [batch_t 0.338 (0.331)] [data_t 0.003] [optim_t 0.335] [lr 0.005000] 2024-04-08 11:55:32,031 - Train: 27.00% [1309200/4848000] [270.0/1000.0] [batch_t 0.327 (0.348)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 11:56:06,039 - Train: 27.01% [1309300/4848000] [270.1/1000.0] [batch_t 1.237 (0.340)] [data_t 0.909] [optim_t 0.328] [lr 0.005000] 2024-04-08 11:56:39,398 - Train: 27.01% [1309400/4848000] [270.1/1000.0] [batch_t 0.331 (0.333)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 11:57:13,794 - Train: 27.01% [1309500/4848000] [270.1/1000.0] [batch_t 0.333 (0.344)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 11:57:47,020 - Train: 27.01% [1309600/4848000] [270.1/1000.0] [batch_t 0.328 (0.332)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 11:58:20,155 - Train: 27.02% [1309700/4848000] [270.2/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 11:58:53,810 - Train: 27.02% [1309800/4848000] [270.2/1000.0] [batch_t 0.332 (0.336)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 11:59:29,559 - Train: 27.02% [1309900/4848000] [270.2/1000.0] [batch_t 0.327 (0.357)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 12:00:04,290 - Train: 27.02% [1310000/4848000] [270.2/1000.0] [batch_t 0.328 (0.347)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 12:00:43,397 - Train: 27.02% [1310100/4848000] [270.2/1000.0] [batch_t 0.329 (0.391)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 12:01:17,584 - Train: 27.03% [1310200/4848000] [270.3/1000.0] [batch_t 0.332 (0.342)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 12:01:50,681 - Train: 27.03% [1310300/4848000] [270.3/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 12:02:24,790 - Train: 27.03% [1310400/4848000] [270.3/1000.0] [batch_t 0.331 (0.341)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 12:02:57,883 - Train: 27.03% [1310500/4848000] [270.3/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 12:03:34,220 - Train: 27.03% [1310600/4848000] [270.3/1000.0] [batch_t 0.521 (0.363)] [data_t 0.188] [optim_t 0.334] [lr 0.005000] 2024-04-08 12:04:17,236 - Train: 27.04% [1310700/4848000] [270.4/1000.0] [batch_t 0.325 (0.430)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-08 12:04:50,411 - Train: 27.04% [1310800/4848000] [270.4/1000.0] [batch_t 0.330 (0.332)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 12:05:24,734 - Train: 27.04% [1310900/4848000] [270.4/1000.0] [batch_t 0.330 (0.343)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 12:05:59,127 - Train: 27.04% [1311000/4848000] [270.4/1000.0] [batch_t 0.333 (0.344)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 12:06:37,417 - Train: 27.04% [1311100/4848000] [270.4/1000.0] [batch_t 0.330 (0.383)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 12:07:38,573 - Train: 27.05% [1311200/4848000] [270.5/1000.0] [batch_t 0.328 (0.611)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 12:08:44,021 - Train: 27.05% [1311300/4848000] [270.5/1000.0] [batch_t 0.329 (0.654)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 12:09:29,026 - Train: 27.05% [1311400/4848000] [270.5/1000.0] [batch_t 0.327 (0.450)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 12:10:02,769 - Train: 27.05% [1311500/4848000] [270.5/1000.0] [batch_t 0.328 (0.337)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 12:10:41,673 - Train: 27.05% [1311600/4848000] [270.5/1000.0] [batch_t 0.330 (0.389)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 12:11:14,798 - Train: 27.06% [1311700/4848000] [270.6/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 12:11:47,877 - Train: 27.06% [1311800/4848000] [270.6/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 12:12:21,925 - Train: 27.06% [1311900/4848000] [270.6/1000.0] [batch_t 0.330 (0.340)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 12:12:54,970 - Train: 27.06% [1312000/4848000] [270.6/1000.0] [batch_t 0.334 (0.330)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 12:13:29,731 - Train: 27.06% [1312100/4848000] [270.6/1000.0] [batch_t 0.334 (0.348)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 12:14:09,246 - Train: 27.07% [1312200/4848000] [270.7/1000.0] [batch_t 0.325 (0.395)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-08 12:14:42,370 - Train: 27.07% [1312300/4848000] [270.7/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 12:15:43,024 - Train: 27.07% [1312400/4848000] [270.7/1000.0] [batch_t 1.235 (0.606)] [data_t 0.911] [optim_t 0.323] [lr 0.005000] 2024-04-08 12:16:26,373 - Train: 27.07% [1312500/4848000] [270.7/1000.0] [batch_t 0.329 (0.433)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 12:16:59,500 - Train: 27.08% [1312600/4848000] [270.8/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 12:17:34,509 - Train: 27.08% [1312700/4848000] [270.8/1000.0] [batch_t 0.329 (0.350)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 12:18:07,560 - Train: 27.08% [1312800/4848000] [270.8/1000.0] [batch_t 0.334 (0.330)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 12:18:40,687 - Train: 27.08% [1312900/4848000] [270.8/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 12:19:15,823 - Train: 27.08% [1313000/4848000] [270.8/1000.0] [batch_t 0.334 (0.351)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 12:19:53,301 - Train: 27.09% [1313100/4848000] [270.9/1000.0] [batch_t 0.330 (0.375)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 12:20:26,945 - Train: 27.09% [1313200/4848000] [270.9/1000.0] [batch_t 0.328 (0.336)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 12:21:00,116 - Train: 27.09% [1313300/4848000] [270.9/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 12:21:35,880 - Train: 27.09% [1313400/4848000] [270.9/1000.0] [batch_t 0.329 (0.358)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 12:22:12,148 - Train: 27.09% [1313500/4848000] [270.9/1000.0] [batch_t 0.335 (0.363)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 12:22:45,474 - Train: 27.10% [1313600/4848000] [271.0/1000.0] [batch_t 0.334 (0.333)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 12:23:18,690 - Train: 27.10% [1313700/4848000] [271.0/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 12:23:51,798 - Train: 27.10% [1313800/4848000] [271.0/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 12:23:54,451 - ==> Total time: 5 days, 12:57:52 Eta: 14 days, 21:40:47 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 12:24:29,001 - Train: 27.10% [1313900/4848000] [271.0/1000.0] [batch_t 0.330 (0.343)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 12:25:02,131 - Train: 27.10% [1314000/4848000] [271.0/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 12:25:40,463 - Train: 27.11% [1314100/4848000] [271.1/1000.0] [batch_t 0.328 (0.383)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 12:26:26,399 - Train: 27.11% [1314200/4848000] [271.1/1000.0] [batch_t 0.333 (0.459)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 12:27:09,787 - Train: 27.11% [1314300/4848000] [271.1/1000.0] [batch_t 0.325 (0.434)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 12:28:00,842 - Train: 27.11% [1314400/4848000] [271.1/1000.0] [batch_t 0.330 (0.510)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 12:28:36,971 - Train: 27.11% [1314500/4848000] [271.1/1000.0] [batch_t 0.331 (0.361)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 12:29:10,113 - Train: 27.12% [1314600/4848000] [271.2/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 12:29:43,202 - Train: 27.12% [1314700/4848000] [271.2/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 12:30:16,323 - Train: 27.12% [1314800/4848000] [271.2/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 12:30:49,457 - Train: 27.12% [1314900/4848000] [271.2/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 12:31:22,954 - Train: 27.12% [1315000/4848000] [271.2/1000.0] [batch_t 0.331 (0.335)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 12:31:56,082 - Train: 27.13% [1315100/4848000] [271.3/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 12:32:30,000 - Train: 27.13% [1315200/4848000] [271.3/1000.0] [batch_t 0.329 (0.339)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 12:33:03,094 - Train: 27.13% [1315300/4848000] [271.3/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 12:33:36,127 - Train: 27.13% [1315400/4848000] [271.3/1000.0] [batch_t 0.337 (0.330)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-08 12:34:09,978 - Train: 27.13% [1315500/4848000] [271.3/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 12:34:43,057 - Train: 27.14% [1315600/4848000] [271.4/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 12:35:16,030 - Train: 27.14% [1315700/4848000] [271.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 12:35:49,049 - Train: 27.14% [1315800/4848000] [271.4/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 12:36:22,175 - Train: 27.14% [1315900/4848000] [271.4/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 12:36:55,190 - Train: 27.15% [1316000/4848000] [271.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 12:37:28,296 - Train: 27.15% [1316100/4848000] [271.5/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 12:38:01,427 - Train: 27.15% [1316200/4848000] [271.5/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 12:38:34,590 - Train: 27.15% [1316300/4848000] [271.5/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 12:39:07,700 - Train: 27.15% [1316400/4848000] [271.5/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 12:39:40,973 - Train: 27.16% [1316500/4848000] [271.6/1000.0] [batch_t 0.330 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 12:40:14,766 - Train: 27.16% [1316600/4848000] [271.6/1000.0] [batch_t 0.332 (0.338)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 12:40:47,838 - Train: 27.16% [1316700/4848000] [271.6/1000.0] [batch_t 0.327 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 12:41:21,239 - Train: 27.16% [1316800/4848000] [271.6/1000.0] [batch_t 0.329 (0.334)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 12:41:54,322 - Train: 27.16% [1316900/4848000] [271.6/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 12:42:28,447 - Train: 27.17% [1317000/4848000] [271.7/1000.0] [batch_t 0.331 (0.341)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 12:43:01,564 - Train: 27.17% [1317100/4848000] [271.7/1000.0] [batch_t 0.335 (0.331)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 12:43:35,414 - Train: 27.17% [1317200/4848000] [271.7/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 12:44:08,549 - Train: 27.17% [1317300/4848000] [271.7/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 12:44:41,675 - Train: 27.17% [1317400/4848000] [271.7/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 12:45:14,872 - Train: 27.18% [1317500/4848000] [271.8/1000.0] [batch_t 0.327 (0.332)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 12:45:48,078 - Train: 27.18% [1317600/4848000] [271.8/1000.0] [batch_t 0.331 (0.332)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 12:46:21,231 - Train: 27.18% [1317700/4848000] [271.8/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 12:46:54,513 - Train: 27.18% [1317800/4848000] [271.8/1000.0] [batch_t 0.331 (0.333)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 12:47:27,825 - Train: 27.18% [1317900/4848000] [271.8/1000.0] [batch_t 0.326 (0.333)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 12:48:00,980 - Train: 27.19% [1318000/4848000] [271.9/1000.0] [batch_t 0.326 (0.331)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 12:48:34,076 - Train: 27.19% [1318100/4848000] [271.9/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 12:49:07,239 - Train: 27.19% [1318200/4848000] [271.9/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 12:49:40,422 - Train: 27.19% [1318300/4848000] [271.9/1000.0] [batch_t 0.331 (0.332)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 12:50:13,626 - Train: 27.19% [1318400/4848000] [271.9/1000.0] [batch_t 0.329 (0.332)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 12:50:46,805 - Train: 27.20% [1318500/4848000] [272.0/1000.0] [batch_t 0.339 (0.332)] [data_t 0.003] [optim_t 0.336] [lr 0.005000] 2024-04-08 12:51:24,890 - Train: 27.20% [1318600/4848000] [272.0/1000.0] [batch_t 0.329 (0.381)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 12:51:44,446 - ==> Total time: 5 days, 13:25:42 Eta: 14 days, 21:07:03 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 12:52:02,722 - Train: 27.20% [1318700/4848000] [272.0/1000.0] [batch_t 0.329 (0.352)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 12:52:35,861 - Train: 27.20% [1318800/4848000] [272.0/1000.0] [batch_t 0.336 (0.331)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 12:53:10,582 - Train: 27.21% [1318900/4848000] [272.1/1000.0] [batch_t 0.330 (0.347)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 12:53:43,718 - Train: 27.21% [1319000/4848000] [272.1/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 12:54:18,318 - Train: 27.21% [1319100/4848000] [272.1/1000.0] [batch_t 0.325 (0.346)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 12:54:51,519 - Train: 27.21% [1319200/4848000] [272.1/1000.0] [batch_t 0.328 (0.332)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 12:55:24,665 - Train: 27.21% [1319300/4848000] [272.1/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 12:55:57,868 - Train: 27.22% [1319400/4848000] [272.2/1000.0] [batch_t 0.335 (0.332)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 12:56:32,317 - Train: 27.22% [1319500/4848000] [272.2/1000.0] [batch_t 0.339 (0.344)] [data_t 0.003] [optim_t 0.336] [lr 0.005000] 2024-04-08 12:57:10,203 - Train: 27.22% [1319600/4848000] [272.2/1000.0] [batch_t 0.332 (0.379)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 12:57:43,365 - Train: 27.22% [1319700/4848000] [272.2/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 12:58:18,626 - Train: 27.22% [1319800/4848000] [272.2/1000.0] [batch_t 0.327 (0.352)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 12:58:55,099 - Train: 27.23% [1319900/4848000] [272.3/1000.0] [batch_t 0.330 (0.365)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 12:59:30,562 - Train: 27.23% [1320000/4848000] [272.3/1000.0] [batch_t 0.327 (0.355)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 13:00:06,954 - Train: 27.23% [1320100/4848000] [272.3/1000.0] [batch_t 0.328 (0.364)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 13:00:40,131 - Train: 27.23% [1320200/4848000] [272.3/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 13:01:13,559 - Train: 27.23% [1320300/4848000] [272.3/1000.0] [batch_t 0.326 (0.334)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 13:01:46,700 - Train: 27.24% [1320400/4848000] [272.4/1000.0] [batch_t 0.333 (0.331)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 13:02:24,944 - Train: 27.24% [1320500/4848000] [272.4/1000.0] [batch_t 0.331 (0.382)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 13:03:00,789 - Train: 27.24% [1320600/4848000] [272.4/1000.0] [batch_t 0.595 (0.358)] [data_t 0.268] [optim_t 0.327] [lr 0.005000] 2024-04-08 13:03:43,161 - Train: 27.24% [1320700/4848000] [272.4/1000.0] [batch_t 0.331 (0.424)] [data_t 0.006] [optim_t 0.325] [lr 0.005000] 2024-04-08 13:04:17,368 - Train: 27.24% [1320800/4848000] [272.4/1000.0] [batch_t 0.329 (0.342)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 13:04:50,489 - Train: 27.25% [1320900/4848000] [272.5/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 13:05:28,516 - Train: 27.25% [1321000/4848000] [272.5/1000.0] [batch_t 0.329 (0.380)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 13:06:07,174 - Train: 27.25% [1321100/4848000] [272.5/1000.0] [batch_t 0.331 (0.386)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 13:06:40,321 - Train: 27.25% [1321200/4848000] [272.5/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 13:07:14,959 - Train: 27.25% [1321300/4848000] [272.5/1000.0] [batch_t 0.331 (0.346)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 13:07:48,019 - Train: 27.26% [1321400/4848000] [272.6/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 13:08:23,797 - Train: 27.26% [1321500/4848000] [272.6/1000.0] [batch_t 0.334 (0.358)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 13:08:56,950 - Train: 27.26% [1321600/4848000] [272.6/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 13:09:31,417 - Train: 27.26% [1321700/4848000] [272.6/1000.0] [batch_t 0.329 (0.345)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 13:10:04,551 - Train: 27.26% [1321800/4848000] [272.6/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 13:10:39,011 - Train: 27.27% [1321900/4848000] [272.7/1000.0] [batch_t 0.331 (0.344)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 13:11:13,315 - Train: 27.27% [1322000/4848000] [272.7/1000.0] [batch_t 0.341 (0.343)] [data_t 0.003] [optim_t 0.339] [lr 0.005000] 2024-04-08 13:11:46,629 - Train: 27.27% [1322100/4848000] [272.7/1000.0] [batch_t 0.326 (0.333)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 13:12:21,283 - Train: 27.27% [1322200/4848000] [272.7/1000.0] [batch_t 0.331 (0.346)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 13:12:54,368 - Train: 27.28% [1322300/4848000] [272.8/1000.0] [batch_t 0.327 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 13:13:27,622 - Train: 27.28% [1322400/4848000] [272.8/1000.0] [batch_t 0.335 (0.332)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 13:14:00,877 - Train: 27.28% [1322500/4848000] [272.8/1000.0] [batch_t 0.331 (0.332)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 13:14:38,885 - Train: 27.28% [1322600/4848000] [272.8/1000.0] [batch_t 0.331 (0.380)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 13:15:11,989 - Train: 27.28% [1322700/4848000] [272.8/1000.0] [batch_t 0.327 (0.331)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 13:15:45,165 - Train: 27.29% [1322800/4848000] [272.9/1000.0] [batch_t 0.338 (0.332)] [data_t 0.003] [optim_t 0.336] [lr 0.005000] 2024-04-08 13:16:18,293 - Train: 27.29% [1322900/4848000] [272.9/1000.0] [batch_t 0.326 (0.331)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 13:17:02,136 - Train: 27.29% [1323000/4848000] [272.9/1000.0] [batch_t 0.331 (0.438)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 13:17:39,135 - Train: 27.29% [1323100/4848000] [272.9/1000.0] [batch_t 0.390 (0.370)] [data_t 0.060] [optim_t 0.329] [lr 0.005000] 2024-04-08 13:18:13,050 - Train: 27.29% [1323200/4848000] [272.9/1000.0] [batch_t 0.337 (0.339)] [data_t 0.003] [optim_t 0.335] [lr 0.005000] 2024-04-08 13:18:51,117 - Train: 27.30% [1323300/4848000] [273.0/1000.0] [batch_t 0.498 (0.381)] [data_t 0.170] [optim_t 0.329] [lr 0.005000] 2024-04-08 13:19:28,714 - Train: 27.30% [1323400/4848000] [273.0/1000.0] [batch_t 0.332 (0.376)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 13:20:01,925 - Train: 27.30% [1323500/4848000] [273.0/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 13:20:03,255 - ==> Total time: 5 days, 13:54:01 Eta: 14 days, 20:34:38 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 13:20:37,696 - Train: 27.30% [1323600/4848000] [273.0/1000.0] [batch_t 0.331 (0.332)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 13:21:11,186 - Train: 27.30% [1323700/4848000] [273.0/1000.0] [batch_t 0.324 (0.335)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-08 13:21:44,229 - Train: 27.31% [1323800/4848000] [273.1/1000.0] [batch_t 0.326 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 13:22:42,741 - Train: 27.31% [1323900/4848000] [273.1/1000.0] [batch_t 2.444 (0.585)] [data_t 2.115] [optim_t 0.329] [lr 0.005000] 2024-04-08 13:23:42,730 - Train: 27.31% [1324000/4848000] [273.1/1000.0] [batch_t 0.329 (0.600)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 13:24:22,326 - Train: 27.31% [1324100/4848000] [273.1/1000.0] [batch_t 0.329 (0.396)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 13:24:55,584 - Train: 27.31% [1324200/4848000] [273.1/1000.0] [batch_t 0.326 (0.332)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 13:25:29,246 - Train: 27.32% [1324300/4848000] [273.2/1000.0] [batch_t 0.332 (0.337)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 13:26:02,445 - Train: 27.32% [1324400/4848000] [273.2/1000.0] [batch_t 0.336 (0.332)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 13:26:38,897 - Train: 27.32% [1324500/4848000] [273.2/1000.0] [batch_t 0.464 (0.364)] [data_t 0.136] [optim_t 0.328] [lr 0.005000] 2024-04-08 13:27:17,575 - Train: 27.32% [1324600/4848000] [273.2/1000.0] [batch_t 0.330 (0.387)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 13:27:51,669 - Train: 27.32% [1324700/4848000] [273.2/1000.0] [batch_t 0.332 (0.341)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 13:28:34,931 - Train: 27.33% [1324800/4848000] [273.3/1000.0] [batch_t 0.332 (0.433)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 13:29:17,618 - Train: 27.33% [1324900/4848000] [273.3/1000.0] [batch_t 0.751 (0.427)] [data_t 0.425] [optim_t 0.326] [lr 0.005000] 2024-04-08 13:30:03,327 - Train: 27.33% [1325000/4848000] [273.3/1000.0] [batch_t 0.329 (0.457)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 13:30:45,133 - Train: 27.33% [1325100/4848000] [273.3/1000.0] [batch_t 0.329 (0.418)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 13:31:30,494 - Train: 27.33% [1325200/4848000] [273.3/1000.0] [batch_t 0.781 (0.454)] [data_t 0.450] [optim_t 0.331] [lr 0.005000] 2024-04-08 13:32:16,590 - Train: 27.34% [1325300/4848000] [273.4/1000.0] [batch_t 0.329 (0.461)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 13:33:02,028 - Train: 27.34% [1325400/4848000] [273.4/1000.0] [batch_t 0.333 (0.454)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 13:33:43,577 - Train: 27.34% [1325500/4848000] [273.4/1000.0] [batch_t 0.329 (0.415)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 13:34:19,498 - Train: 27.34% [1325600/4848000] [273.4/1000.0] [batch_t 0.327 (0.359)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 13:34:52,547 - Train: 27.35% [1325700/4848000] [273.5/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 13:35:26,612 - Train: 27.35% [1325800/4848000] [273.5/1000.0] [batch_t 0.333 (0.341)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 13:36:00,324 - Train: 27.35% [1325900/4848000] [273.5/1000.0] [batch_t 0.336 (0.337)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 13:36:33,958 - Train: 27.35% [1326000/4848000] [273.5/1000.0] [batch_t 0.332 (0.336)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 13:37:09,329 - Train: 27.35% [1326100/4848000] [273.5/1000.0] [batch_t 0.335 (0.354)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 13:37:42,527 - Train: 27.36% [1326200/4848000] [273.6/1000.0] [batch_t 0.330 (0.332)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 13:38:16,207 - Train: 27.36% [1326300/4848000] [273.6/1000.0] [batch_t 0.333 (0.337)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 13:38:49,568 - Train: 27.36% [1326400/4848000] [273.6/1000.0] [batch_t 0.337 (0.333)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-08 13:39:22,727 - Train: 27.36% [1326500/4848000] [273.6/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 13:39:55,842 - Train: 27.36% [1326600/4848000] [273.6/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 13:40:28,996 - Train: 27.37% [1326700/4848000] [273.7/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 13:41:02,109 - Train: 27.37% [1326800/4848000] [273.7/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 13:41:35,848 - Train: 27.37% [1326900/4848000] [273.7/1000.0] [batch_t 0.332 (0.337)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 13:42:09,093 - Train: 27.37% [1327000/4848000] [273.7/1000.0] [batch_t 0.332 (0.332)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 13:42:42,225 - Train: 27.37% [1327100/4848000] [273.7/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 13:43:15,385 - Train: 27.38% [1327200/4848000] [273.8/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 13:43:48,604 - Train: 27.38% [1327300/4848000] [273.8/1000.0] [batch_t 0.332 (0.332)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 13:44:22,163 - Train: 27.38% [1327400/4848000] [273.8/1000.0] [batch_t 0.333 (0.335)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 13:44:56,991 - Train: 27.38% [1327500/4848000] [273.8/1000.0] [batch_t 0.333 (0.348)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 13:45:32,040 - Train: 27.38% [1327600/4848000] [273.8/1000.0] [batch_t 0.333 (0.350)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 13:46:05,117 - Train: 27.39% [1327700/4848000] [273.9/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 13:46:38,243 - Train: 27.39% [1327800/4848000] [273.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 13:47:11,710 - Train: 27.39% [1327900/4848000] [273.9/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 13:47:44,786 - Train: 27.39% [1328000/4848000] [273.9/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 13:48:19,541 - Train: 27.39% [1328100/4848000] [273.9/1000.0] [batch_t 0.329 (0.347)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 13:48:52,664 - Train: 27.40% [1328200/4848000] [274.0/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 13:49:26,846 - Train: 27.40% [1328300/4848000] [274.0/1000.0] [batch_t 0.331 (0.342)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 13:49:44,035 - ==> Total time: 5 days, 14:23:42 Eta: 14 days, 20:05:53 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 13:50:03,666 - Train: 27.40% [1328400/4848000] [274.0/1000.0] [batch_t 0.336 (0.342)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-08 13:50:36,974 - Train: 27.40% [1328500/4848000] [274.0/1000.0] [batch_t 0.331 (0.333)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 13:51:10,191 - Train: 27.41% [1328600/4848000] [274.1/1000.0] [batch_t 0.332 (0.332)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 13:51:43,206 - Train: 27.41% [1328700/4848000] [274.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 13:52:16,282 - Train: 27.41% [1328800/4848000] [274.1/1000.0] [batch_t 0.326 (0.331)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 13:52:49,324 - Train: 27.41% [1328900/4848000] [274.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 13:53:22,688 - Train: 27.41% [1329000/4848000] [274.1/1000.0] [batch_t 0.327 (0.334)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 13:53:55,741 - Train: 27.42% [1329100/4848000] [274.2/1000.0] [batch_t 0.334 (0.330)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 13:54:28,958 - Train: 27.42% [1329200/4848000] [274.2/1000.0] [batch_t 0.327 (0.332)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 13:55:02,054 - Train: 27.42% [1329300/4848000] [274.2/1000.0] [batch_t 0.333 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 13:55:36,036 - Train: 27.42% [1329400/4848000] [274.2/1000.0] [batch_t 0.329 (0.340)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 13:56:09,638 - Train: 27.42% [1329500/4848000] [274.2/1000.0] [batch_t 0.328 (0.336)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 13:56:42,706 - Train: 27.43% [1329600/4848000] [274.3/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 13:57:15,826 - Train: 27.43% [1329700/4848000] [274.3/1000.0] [batch_t 0.325 (0.331)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-08 13:57:49,498 - Train: 27.43% [1329800/4848000] [274.3/1000.0] [batch_t 0.329 (0.337)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 13:58:24,928 - Train: 27.43% [1329900/4848000] [274.3/1000.0] [batch_t 0.330 (0.354)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 13:58:58,033 - Train: 27.43% [1330000/4848000] [274.3/1000.0] [batch_t 0.327 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 13:59:31,998 - Train: 27.44% [1330100/4848000] [274.4/1000.0] [batch_t 0.331 (0.340)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 14:00:05,754 - Train: 27.44% [1330200/4848000] [274.4/1000.0] [batch_t 1.092 (0.337)] [data_t 0.764] [optim_t 0.328] [lr 0.005000] 2024-04-08 14:00:38,873 - Train: 27.44% [1330300/4848000] [274.4/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 14:01:11,923 - Train: 27.44% [1330400/4848000] [274.4/1000.0] [batch_t 0.326 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 14:01:45,022 - Train: 27.44% [1330500/4848000] [274.4/1000.0] [batch_t 0.333 (0.331)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 14:02:19,113 - Train: 27.45% [1330600/4848000] [274.5/1000.0] [batch_t 0.326 (0.341)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 14:02:52,310 - Train: 27.45% [1330700/4848000] [274.5/1000.0] [batch_t 0.330 (0.332)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 14:03:28,425 - Train: 27.45% [1330800/4848000] [274.5/1000.0] [batch_t 0.329 (0.361)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 14:04:01,578 - Train: 27.45% [1330900/4848000] [274.5/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 14:04:35,401 - Train: 27.45% [1331000/4848000] [274.5/1000.0] [batch_t 0.333 (0.338)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 14:05:08,432 - Train: 27.46% [1331100/4848000] [274.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 14:05:42,230 - Train: 27.46% [1331200/4848000] [274.6/1000.0] [batch_t 0.330 (0.338)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 14:06:15,633 - Train: 27.46% [1331300/4848000] [274.6/1000.0] [batch_t 0.329 (0.334)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 14:06:48,778 - Train: 27.46% [1331400/4848000] [274.6/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 14:07:22,214 - Train: 27.46% [1331500/4848000] [274.6/1000.0] [batch_t 0.331 (0.334)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 14:07:55,326 - Train: 27.47% [1331600/4848000] [274.7/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 14:08:28,669 - Train: 27.47% [1331700/4848000] [274.7/1000.0] [batch_t 0.331 (0.333)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 14:09:01,721 - Train: 27.47% [1331800/4848000] [274.7/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 14:09:34,772 - Train: 27.47% [1331900/4848000] [274.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 14:10:10,823 - Train: 27.48% [1332000/4848000] [274.8/1000.0] [batch_t 0.331 (0.360)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 14:10:48,188 - Train: 27.48% [1332100/4848000] [274.8/1000.0] [batch_t 0.330 (0.374)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 14:11:22,004 - Train: 27.48% [1332200/4848000] [274.8/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 14:11:55,068 - Train: 27.48% [1332300/4848000] [274.8/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 14:12:29,710 - Train: 27.48% [1332400/4848000] [274.8/1000.0] [batch_t 0.331 (0.346)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 14:13:02,726 - Train: 27.49% [1332500/4848000] [274.9/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 14:13:35,800 - Train: 27.49% [1332600/4848000] [274.9/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 14:14:09,168 - Train: 27.49% [1332700/4848000] [274.9/1000.0] [batch_t 0.330 (0.334)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 14:14:42,239 - Train: 27.49% [1332800/4848000] [274.9/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 14:15:16,118 - Train: 27.49% [1332900/4848000] [274.9/1000.0] [batch_t 0.336 (0.339)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 14:15:49,259 - Train: 27.50% [1333000/4848000] [275.0/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 14:16:24,424 - Train: 27.50% [1333100/4848000] [275.0/1000.0] [batch_t 0.329 (0.352)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 14:16:57,546 - Train: 27.50% [1333200/4848000] [275.0/1000.0] [batch_t 0.333 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 14:16:57,551 - ==> Total time: 5 days, 14:50:56 Eta: 14 days, 19:30:38 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 14:17:37,752 - Train: 27.50% [1333300/4848000] [275.0/1000.0] [batch_t 0.328 (0.379)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 14:18:10,953 - Train: 27.50% [1333400/4848000] [275.0/1000.0] [batch_t 0.328 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 14:18:44,048 - Train: 27.51% [1333500/4848000] [275.1/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 14:19:18,543 - Train: 27.51% [1333600/4848000] [275.1/1000.0] [batch_t 0.330 (0.345)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 14:19:51,576 - Train: 27.51% [1333700/4848000] [275.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 14:20:26,577 - Train: 27.51% [1333800/4848000] [275.1/1000.0] [batch_t 0.334 (0.350)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 14:20:59,660 - Train: 27.51% [1333900/4848000] [275.1/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 14:21:33,398 - Train: 27.52% [1334000/4848000] [275.2/1000.0] [batch_t 0.330 (0.337)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 14:22:07,461 - Train: 27.52% [1334100/4848000] [275.2/1000.0] [batch_t 0.333 (0.341)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 14:22:42,082 - Train: 27.52% [1334200/4848000] [275.2/1000.0] [batch_t 0.332 (0.346)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 14:23:43,547 - Train: 27.52% [1334300/4848000] [275.2/1000.0] [batch_t 0.332 (0.615)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 14:24:47,351 - Train: 27.52% [1334400/4848000] [275.2/1000.0] [batch_t 0.329 (0.638)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 14:25:21,621 - Train: 27.53% [1334500/4848000] [275.3/1000.0] [batch_t 0.337 (0.343)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-08 14:25:54,819 - Train: 27.53% [1334600/4848000] [275.3/1000.0] [batch_t 0.330 (0.332)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 14:26:39,179 - Train: 27.53% [1334700/4848000] [275.3/1000.0] [batch_t 0.329 (0.443)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 14:27:13,444 - Train: 27.53% [1334800/4848000] [275.3/1000.0] [batch_t 0.330 (0.343)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 14:27:48,540 - Train: 27.54% [1334900/4848000] [275.4/1000.0] [batch_t 0.327 (0.351)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 14:28:22,996 - Train: 27.54% [1335000/4848000] [275.4/1000.0] [batch_t 0.331 (0.344)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 14:29:11,590 - Train: 27.54% [1335100/4848000] [275.4/1000.0] [batch_t 0.329 (0.486)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 14:29:48,675 - Train: 27.54% [1335200/4848000] [275.4/1000.0] [batch_t 0.331 (0.371)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 14:30:23,444 - Train: 27.54% [1335300/4848000] [275.4/1000.0] [batch_t 0.325 (0.348)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 14:30:56,695 - Train: 27.55% [1335400/4848000] [275.5/1000.0] [batch_t 0.335 (0.332)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 14:31:35,802 - Train: 27.55% [1335500/4848000] [275.5/1000.0] [batch_t 0.332 (0.391)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 14:32:13,905 - Train: 27.55% [1335600/4848000] [275.5/1000.0] [batch_t 0.332 (0.381)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 14:32:47,081 - Train: 27.55% [1335700/4848000] [275.5/1000.0] [batch_t 0.331 (0.332)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 14:33:21,938 - Train: 27.55% [1335800/4848000] [275.5/1000.0] [batch_t 0.331 (0.348)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 14:33:55,184 - Train: 27.56% [1335900/4848000] [275.6/1000.0] [batch_t 0.327 (0.332)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 14:34:30,800 - Train: 27.56% [1336000/4848000] [275.6/1000.0] [batch_t 0.332 (0.356)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 14:35:04,140 - Train: 27.56% [1336100/4848000] [275.6/1000.0] [batch_t 0.327 (0.333)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 14:35:38,298 - Train: 27.56% [1336200/4848000] [275.6/1000.0] [batch_t 0.337 (0.341)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-08 14:36:16,126 - Train: 27.56% [1336300/4848000] [275.6/1000.0] [batch_t 0.330 (0.378)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 14:37:00,348 - Train: 27.57% [1336400/4848000] [275.7/1000.0] [batch_t 0.330 (0.442)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 14:37:46,547 - Train: 27.57% [1336500/4848000] [275.7/1000.0] [batch_t 0.331 (0.462)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 14:38:33,173 - Train: 27.57% [1336600/4848000] [275.7/1000.0] [batch_t 1.043 (0.466)] [data_t 0.717] [optim_t 0.326] [lr 0.005000] 2024-04-08 14:39:28,938 - Train: 27.57% [1336700/4848000] [275.7/1000.0] [batch_t 0.329 (0.558)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 14:40:13,938 - Train: 27.57% [1336800/4848000] [275.7/1000.0] [batch_t 0.603 (0.450)] [data_t 0.281] [optim_t 0.322] [lr 0.005000] 2024-04-08 14:41:07,707 - Train: 27.58% [1336900/4848000] [275.8/1000.0] [batch_t 0.330 (0.538)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 14:41:42,055 - Train: 27.58% [1337000/4848000] [275.8/1000.0] [batch_t 0.329 (0.343)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 14:42:17,995 - Train: 27.58% [1337100/4848000] [275.8/1000.0] [batch_t 0.329 (0.359)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 14:43:06,164 - Train: 27.58% [1337200/4848000] [275.8/1000.0] [batch_t 0.330 (0.482)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 14:43:39,232 - Train: 27.58% [1337300/4848000] [275.8/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 14:44:12,787 - Train: 27.59% [1337400/4848000] [275.9/1000.0] [batch_t 0.331 (0.335)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 14:44:45,923 - Train: 27.59% [1337500/4848000] [275.9/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 14:45:20,193 - Train: 27.59% [1337600/4848000] [275.9/1000.0] [batch_t 0.329 (0.343)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 14:45:54,082 - Train: 27.59% [1337700/4848000] [275.9/1000.0] [batch_t 0.331 (0.339)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 14:46:33,440 - Train: 27.59% [1337800/4848000] [275.9/1000.0] [batch_t 0.330 (0.393)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 14:47:07,160 - Train: 27.60% [1337900/4848000] [276.0/1000.0] [batch_t 0.331 (0.337)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 14:47:40,211 - Train: 27.60% [1338000/4848000] [276.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 14:47:56,136 - ==> Total time: 5 days, 15:21:54 Eta: 14 days, 19:05:18 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 14:48:16,976 - Train: 27.60% [1338100/4848000] [276.0/1000.0] [batch_t 0.333 (0.357)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 14:48:50,072 - Train: 27.60% [1338200/4848000] [276.0/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 14:49:23,947 - Train: 27.61% [1338300/4848000] [276.1/1000.0] [batch_t 0.328 (0.339)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 14:49:56,992 - Train: 27.61% [1338400/4848000] [276.1/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 14:50:31,441 - Train: 27.61% [1338500/4848000] [276.1/1000.0] [batch_t 0.331 (0.344)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 14:51:04,522 - Train: 27.61% [1338600/4848000] [276.1/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 14:51:39,102 - Train: 27.61% [1338700/4848000] [276.1/1000.0] [batch_t 0.332 (0.346)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 14:52:12,291 - Train: 27.62% [1338800/4848000] [276.2/1000.0] [batch_t 0.331 (0.332)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 14:52:45,351 - Train: 27.62% [1338900/4848000] [276.2/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 14:53:19,826 - Train: 27.62% [1339000/4848000] [276.2/1000.0] [batch_t 0.331 (0.345)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 14:53:52,911 - Train: 27.62% [1339100/4848000] [276.2/1000.0] [batch_t 0.327 (0.331)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 14:54:27,273 - Train: 27.62% [1339200/4848000] [276.2/1000.0] [batch_t 0.328 (0.344)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 14:55:00,391 - Train: 27.63% [1339300/4848000] [276.3/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 14:55:34,996 - Train: 27.63% [1339400/4848000] [276.3/1000.0] [batch_t 0.331 (0.346)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 14:56:09,924 - Train: 27.63% [1339500/4848000] [276.3/1000.0] [batch_t 0.331 (0.349)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 14:56:43,037 - Train: 27.63% [1339600/4848000] [276.3/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 14:57:17,431 - Train: 27.63% [1339700/4848000] [276.3/1000.0] [batch_t 0.328 (0.344)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 14:57:50,838 - Train: 27.64% [1339800/4848000] [276.4/1000.0] [batch_t 0.692 (0.334)] [data_t 0.364] [optim_t 0.328] [lr 0.005000] 2024-04-08 14:58:31,552 - Train: 27.64% [1339900/4848000] [276.4/1000.0] [batch_t 0.332 (0.407)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 14:59:17,574 - Train: 27.64% [1340000/4848000] [276.4/1000.0] [batch_t 1.386 (0.460)] [data_t 1.055] [optim_t 0.330] [lr 0.005000] 2024-04-08 15:00:06,456 - Train: 27.64% [1340100/4848000] [276.4/1000.0] [batch_t 0.334 (0.489)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 15:00:41,110 - Train: 27.64% [1340200/4848000] [276.4/1000.0] [batch_t 0.330 (0.346)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 15:01:33,032 - Train: 27.65% [1340300/4848000] [276.5/1000.0] [batch_t 1.792 (0.519)] [data_t 1.463] [optim_t 0.330] [lr 0.005000] 2024-04-08 15:02:38,966 - Train: 27.65% [1340400/4848000] [276.5/1000.0] [batch_t 0.331 (0.659)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 15:03:15,255 - Train: 27.65% [1340500/4848000] [276.5/1000.0] [batch_t 0.333 (0.363)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 15:03:48,559 - Train: 27.65% [1340600/4848000] [276.5/1000.0] [batch_t 0.334 (0.333)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 15:04:23,170 - Train: 27.65% [1340700/4848000] [276.5/1000.0] [batch_t 0.330 (0.346)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 15:04:57,381 - Train: 27.66% [1340800/4848000] [276.6/1000.0] [batch_t 0.330 (0.342)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 15:05:36,837 - Train: 27.66% [1340900/4848000] [276.6/1000.0] [batch_t 0.327 (0.394)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 15:06:13,746 - Train: 27.66% [1341000/4848000] [276.6/1000.0] [batch_t 0.332 (0.369)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 15:06:51,948 - Train: 27.66% [1341100/4848000] [276.6/1000.0] [batch_t 0.329 (0.382)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 15:07:25,709 - Train: 27.67% [1341200/4848000] [276.7/1000.0] [batch_t 0.331 (0.338)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 15:07:58,731 - Train: 27.67% [1341300/4848000] [276.7/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.323] [lr 0.005000] 2024-04-08 15:08:40,865 - Train: 27.67% [1341400/4848000] [276.7/1000.0] [batch_t 0.328 (0.421)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 15:09:14,546 - Train: 27.67% [1341500/4848000] [276.7/1000.0] [batch_t 0.332 (0.337)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 15:09:47,562 - Train: 27.67% [1341600/4848000] [276.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 15:10:23,617 - Train: 27.68% [1341700/4848000] [276.8/1000.0] [batch_t 0.332 (0.360)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 15:10:56,801 - Train: 27.68% [1341800/4848000] [276.8/1000.0] [batch_t 0.330 (0.332)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 15:11:31,962 - Train: 27.68% [1341900/4848000] [276.8/1000.0] [batch_t 0.331 (0.352)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 15:12:06,350 - Train: 27.68% [1342000/4848000] [276.8/1000.0] [batch_t 0.336 (0.344)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 15:12:39,777 - Train: 27.68% [1342100/4848000] [276.8/1000.0] [batch_t 0.325 (0.334)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-08 15:13:14,827 - Train: 27.69% [1342200/4848000] [276.9/1000.0] [batch_t 0.330 (0.350)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 15:13:53,315 - Train: 27.69% [1342300/4848000] [276.9/1000.0] [batch_t 0.339 (0.385)] [data_t 0.003] [optim_t 0.336] [lr 0.005000] 2024-04-08 15:14:36,421 - Train: 27.69% [1342400/4848000] [276.9/1000.0] [batch_t 0.695 (0.431)] [data_t 0.361] [optim_t 0.334] [lr 0.005000] 2024-04-08 15:15:11,898 - Train: 27.69% [1342500/4848000] [276.9/1000.0] [batch_t 0.328 (0.355)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 15:15:58,358 - Train: 27.69% [1342600/4848000] [276.9/1000.0] [batch_t 0.331 (0.464)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 15:16:36,088 - Train: 27.70% [1342700/4848000] [277.0/1000.0] [batch_t 0.328 (0.377)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 15:17:17,652 - Train: 27.70% [1342800/4848000] [277.0/1000.0] [batch_t 0.331 (0.416)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 15:17:50,541 - ==> Total time: 5 days, 15:51:49 Eta: 14 days, 18:37:07 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 15:17:54,620 - Train: 27.70% [1342900/4848000] [277.0/1000.0] [batch_t 0.330 (0.335)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 15:18:33,134 - Train: 27.70% [1343000/4848000] [277.0/1000.0] [batch_t 0.326 (0.385)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 15:19:20,203 - Train: 27.70% [1343100/4848000] [277.0/1000.0] [batch_t 0.330 (0.471)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 15:20:02,138 - Train: 27.71% [1343200/4848000] [277.1/1000.0] [batch_t 0.324 (0.419)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-08 15:21:16,248 - Train: 27.71% [1343300/4848000] [277.1/1000.0] [batch_t 0.328 (0.741)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 15:23:27,302 - Train: 27.71% [1343400/4848000] [277.1/1000.0] [batch_t 3.027 (1.310)] [data_t 2.700] [optim_t 0.327] [lr 0.005000] 2024-04-08 15:25:46,482 - Train: 27.71% [1343500/4848000] [277.1/1000.0] [batch_t 0.330 (1.392)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 15:28:20,431 - Train: 27.71% [1343600/4848000] [277.1/1000.0] [batch_t 0.334 (1.539)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 15:29:13,144 - Train: 27.72% [1343700/4848000] [277.2/1000.0] [batch_t 1.280 (0.527)] [data_t 0.951] [optim_t 0.329] [lr 0.005000] 2024-04-08 15:33:12,967 - Train: 27.72% [1343800/4848000] [277.2/1000.0] [batch_t 4.265 (2.398)] [data_t 3.937] [optim_t 0.328] [lr 0.005000] 2024-04-08 15:39:33,595 - Train: 27.72% [1343900/4848000] [277.2/1000.0] [batch_t 4.884 (3.806)] [data_t 4.559] [optim_t 0.325] [lr 0.005000] 2024-04-08 15:43:49,775 - Train: 27.72% [1344000/4848000] [277.2/1000.0] [batch_t 5.945 (2.562)] [data_t 5.617] [optim_t 0.327] [lr 0.005000] 2024-04-08 15:45:57,600 - Train: 27.72% [1344100/4848000] [277.2/1000.0] [batch_t 0.330 (1.278)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 15:46:40,984 - Train: 27.73% [1344200/4848000] [277.3/1000.0] [batch_t 0.328 (0.434)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 15:47:35,810 - Train: 27.73% [1344300/4848000] [277.3/1000.0] [batch_t 1.412 (0.548)] [data_t 1.084] [optim_t 0.328] [lr 0.005000] 2024-04-08 15:48:26,852 - Train: 27.73% [1344400/4848000] [277.3/1000.0] [batch_t 0.339 (0.507)] [data_t 0.003] [optim_t 0.336] [lr 0.005000] 2024-04-08 15:49:06,390 - Train: 27.73% [1344500/4848000] [277.3/1000.0] [batch_t 1.899 (0.395)] [data_t 1.571] [optim_t 0.328] [lr 0.005000] 2024-04-08 15:50:12,765 - Train: 27.74% [1344600/4848000] [277.4/1000.0] [batch_t 0.331 (0.664)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 15:51:15,331 - Train: 27.74% [1344700/4848000] [277.4/1000.0] [batch_t 0.332 (0.626)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 15:52:01,473 - Train: 27.74% [1344800/4848000] [277.4/1000.0] [batch_t 1.339 (0.461)] [data_t 1.012] [optim_t 0.326] [lr 0.005000] 2024-04-08 15:53:11,069 - Train: 27.74% [1344900/4848000] [277.4/1000.0] [batch_t 0.334 (0.696)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 15:54:12,572 - Train: 27.74% [1345000/4848000] [277.4/1000.0] [batch_t 0.329 (0.615)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 15:55:43,445 - Train: 27.75% [1345100/4848000] [277.5/1000.0] [batch_t 0.332 (0.909)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 15:56:50,768 - Train: 27.75% [1345200/4848000] [277.5/1000.0] [batch_t 0.332 (0.673)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 15:57:44,932 - Train: 27.75% [1345300/4848000] [277.5/1000.0] [batch_t 0.329 (0.542)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 15:58:35,298 - Train: 27.75% [1345400/4848000] [277.5/1000.0] [batch_t 0.331 (0.504)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 15:59:28,095 - Train: 27.75% [1345500/4848000] [277.5/1000.0] [batch_t 0.330 (0.528)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 16:00:13,225 - Train: 27.76% [1345600/4848000] [277.6/1000.0] [batch_t 0.331 (0.451)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 16:00:48,872 - Train: 27.76% [1345700/4848000] [277.6/1000.0] [batch_t 0.329 (0.356)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 16:01:23,724 - Train: 27.76% [1345800/4848000] [277.6/1000.0] [batch_t 0.329 (0.348)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 16:01:56,800 - Train: 27.76% [1345900/4848000] [277.6/1000.0] [batch_t 0.336 (0.331)] [data_t 0.002] [optim_t 0.334] [lr 0.005000] 2024-04-08 16:02:37,247 - Train: 27.76% [1346000/4848000] [277.6/1000.0] [batch_t 1.480 (0.404)] [data_t 1.157] [optim_t 0.323] [lr 0.005000] 2024-04-08 16:03:21,176 - Train: 27.77% [1346100/4848000] [277.7/1000.0] [batch_t 0.331 (0.439)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 16:03:57,971 - Train: 27.77% [1346200/4848000] [277.7/1000.0] [batch_t 0.329 (0.368)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 16:04:34,482 - Train: 27.77% [1346300/4848000] [277.7/1000.0] [batch_t 0.331 (0.365)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 16:05:10,095 - Train: 27.77% [1346400/4848000] [277.7/1000.0] [batch_t 0.331 (0.356)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 16:05:46,440 - Train: 27.77% [1346500/4848000] [277.7/1000.0] [batch_t 0.331 (0.363)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 16:06:21,742 - Train: 27.78% [1346600/4848000] [277.8/1000.0] [batch_t 0.332 (0.353)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 16:06:54,853 - Train: 27.78% [1346700/4848000] [277.8/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 16:07:31,430 - Train: 27.78% [1346800/4848000] [277.8/1000.0] [batch_t 0.331 (0.366)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 16:08:05,241 - Train: 27.78% [1346900/4848000] [277.8/1000.0] [batch_t 0.328 (0.338)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 16:08:38,300 - Train: 27.78% [1347000/4848000] [277.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 16:09:11,395 - Train: 27.79% [1347100/4848000] [277.9/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 16:09:47,084 - Train: 27.79% [1347200/4848000] [277.9/1000.0] [batch_t 0.330 (0.357)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 16:10:21,303 - Train: 27.79% [1347300/4848000] [277.9/1000.0] [batch_t 0.328 (0.342)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 16:10:54,437 - Train: 27.79% [1347400/4848000] [277.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 16:11:28,565 - Train: 27.79% [1347500/4848000] [277.9/1000.0] [batch_t 0.331 (0.341)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 16:12:01,809 - Train: 27.80% [1347600/4848000] [278.0/1000.0] [batch_t 0.333 (0.332)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 16:12:34,962 - Train: 27.80% [1347700/4848000] [278.0/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 16:12:49,530 - ==> Total time: 5 days, 16:46:48 Eta: 14 days, 19:14:03 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 16:13:10,895 - Train: 27.80% [1347800/4848000] [278.0/1000.0] [batch_t 0.332 (0.337)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 16:13:44,062 - Train: 27.80% [1347900/4848000] [278.0/1000.0] [batch_t 0.327 (0.332)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 16:14:18,685 - Train: 27.81% [1348000/4848000] [278.1/1000.0] [batch_t 0.332 (0.346)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 16:14:51,798 - Train: 27.81% [1348100/4848000] [278.1/1000.0] [batch_t 0.337 (0.331)] [data_t 0.002] [optim_t 0.335] [lr 0.005000] 2024-04-08 16:15:24,882 - Train: 27.81% [1348200/4848000] [278.1/1000.0] [batch_t 0.333 (0.331)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 16:15:57,959 - Train: 27.81% [1348300/4848000] [278.1/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 16:16:32,628 - Train: 27.81% [1348400/4848000] [278.1/1000.0] [batch_t 0.330 (0.347)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 16:17:07,693 - Train: 27.82% [1348500/4848000] [278.2/1000.0] [batch_t 0.328 (0.351)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 16:17:40,735 - Train: 27.82% [1348600/4848000] [278.2/1000.0] [batch_t 0.326 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 16:18:20,379 - Train: 27.82% [1348700/4848000] [278.2/1000.0] [batch_t 0.330 (0.396)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 16:18:53,432 - Train: 27.82% [1348800/4848000] [278.2/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 16:19:27,480 - Train: 27.82% [1348900/4848000] [278.2/1000.0] [batch_t 0.330 (0.340)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 16:20:00,579 - Train: 27.83% [1349000/4848000] [278.3/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 16:20:33,690 - Train: 27.83% [1349100/4848000] [278.3/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 16:21:07,601 - Train: 27.83% [1349200/4848000] [278.3/1000.0] [batch_t 0.331 (0.339)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 16:21:40,874 - Train: 27.83% [1349300/4848000] [278.3/1000.0] [batch_t 0.330 (0.333)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 16:22:15,611 - Train: 27.83% [1349400/4848000] [278.3/1000.0] [batch_t 0.337 (0.347)] [data_t 0.003] [optim_t 0.335] [lr 0.005000] 2024-04-08 16:22:48,739 - Train: 27.84% [1349500/4848000] [278.4/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 16:23:22,804 - Train: 27.84% [1349600/4848000] [278.4/1000.0] [batch_t 0.329 (0.341)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 16:23:55,910 - Train: 27.84% [1349700/4848000] [278.4/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 16:24:29,715 - Train: 27.84% [1349800/4848000] [278.4/1000.0] [batch_t 0.326 (0.338)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 16:25:02,808 - Train: 27.84% [1349900/4848000] [278.4/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 16:25:35,851 - Train: 27.85% [1350000/4848000] [278.5/1000.0] [batch_t 0.327 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 16:26:10,113 - Train: 27.85% [1350100/4848000] [278.5/1000.0] [batch_t 0.331 (0.343)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 16:26:44,863 - Train: 27.85% [1350200/4848000] [278.5/1000.0] [batch_t 0.341 (0.347)] [data_t 0.003] [optim_t 0.339] [lr 0.005000] 2024-04-08 16:27:18,440 - Train: 27.85% [1350300/4848000] [278.5/1000.0] [batch_t 0.330 (0.336)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 16:27:51,567 - Train: 27.85% [1350400/4848000] [278.5/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 16:28:28,314 - Train: 27.86% [1350500/4848000] [278.6/1000.0] [batch_t 0.333 (0.367)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 16:29:01,549 - Train: 27.86% [1350600/4848000] [278.6/1000.0] [batch_t 0.329 (0.332)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 16:29:37,797 - Train: 27.86% [1350700/4848000] [278.6/1000.0] [batch_t 0.328 (0.362)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 16:30:13,198 - Train: 27.86% [1350800/4848000] [278.6/1000.0] [batch_t 0.330 (0.354)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 16:30:46,283 - Train: 27.87% [1350900/4848000] [278.7/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 16:31:21,407 - Train: 27.87% [1351000/4848000] [278.7/1000.0] [batch_t 0.333 (0.351)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 16:31:54,506 - Train: 27.87% [1351100/4848000] [278.7/1000.0] [batch_t 0.327 (0.331)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 16:32:27,930 - Train: 27.87% [1351200/4848000] [278.7/1000.0] [batch_t 0.330 (0.334)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 16:33:06,984 - Train: 27.87% [1351300/4848000] [278.7/1000.0] [batch_t 0.330 (0.390)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 16:33:40,124 - Train: 27.88% [1351400/4848000] [278.8/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 16:34:13,410 - Train: 27.88% [1351500/4848000] [278.8/1000.0] [batch_t 0.331 (0.333)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 16:34:46,461 - Train: 27.88% [1351600/4848000] [278.8/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 16:35:19,767 - Train: 27.88% [1351700/4848000] [278.8/1000.0] [batch_t 0.342 (0.333)] [data_t 0.003] [optim_t 0.340] [lr 0.005000] 2024-04-08 16:35:52,969 - Train: 27.88% [1351800/4848000] [278.8/1000.0] [batch_t 0.334 (0.332)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 16:36:27,245 - Train: 27.89% [1351900/4848000] [278.9/1000.0] [batch_t 0.329 (0.343)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 16:37:00,408 - Train: 27.89% [1352000/4848000] [278.9/1000.0] [batch_t 0.329 (0.332)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 16:37:34,551 - Train: 27.89% [1352100/4848000] [278.9/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 16:38:08,732 - Train: 27.89% [1352200/4848000] [278.9/1000.0] [batch_t 0.333 (0.342)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 16:39:05,947 - Train: 27.89% [1352300/4848000] [278.9/1000.0] [batch_t 0.329 (0.572)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 16:40:06,404 - Train: 27.90% [1352400/4848000] [279.0/1000.0] [batch_t 0.333 (0.604)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 16:40:39,587 - Train: 27.90% [1352500/4848000] [279.0/1000.0] [batch_t 0.331 (0.332)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 16:41:13,259 - ==> Total time: 5 days, 17:15:11 Eta: 14 days, 18:41:38 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 16:41:18,084 - Train: 27.90% [1352600/4848000] [279.0/1000.0] [batch_t 0.324 (0.331)] [data_t 0.003] [optim_t 0.321] [lr 0.005000] 2024-04-08 16:41:51,165 - Train: 27.90% [1352700/4848000] [279.0/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 16:42:24,561 - Train: 27.90% [1352800/4848000] [279.0/1000.0] [batch_t 0.332 (0.334)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 16:42:57,597 - Train: 27.91% [1352900/4848000] [279.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 16:43:32,105 - Train: 27.91% [1353000/4848000] [279.1/1000.0] [batch_t 0.330 (0.345)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 16:44:05,895 - Train: 27.91% [1353100/4848000] [279.1/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 16:44:39,062 - Train: 27.91% [1353200/4848000] [279.1/1000.0] [batch_t 0.326 (0.332)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 16:45:13,613 - Train: 27.91% [1353300/4848000] [279.1/1000.0] [batch_t 0.327 (0.345)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 16:45:46,689 - Train: 27.92% [1353400/4848000] [279.2/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 16:46:21,667 - Train: 27.92% [1353500/4848000] [279.2/1000.0] [batch_t 0.327 (0.350)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 16:46:54,697 - Train: 27.92% [1353600/4848000] [279.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 16:47:28,633 - Train: 27.92% [1353700/4848000] [279.2/1000.0] [batch_t 0.341 (0.339)] [data_t 0.003] [optim_t 0.338] [lr 0.005000] 2024-04-08 16:48:01,683 - Train: 27.92% [1353800/4848000] [279.2/1000.0] [batch_t 0.334 (0.330)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 16:48:36,263 - Train: 27.93% [1353900/4848000] [279.3/1000.0] [batch_t 0.333 (0.346)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 16:49:13,616 - Train: 27.93% [1354000/4848000] [279.3/1000.0] [batch_t 0.330 (0.373)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 16:49:46,658 - Train: 27.93% [1354100/4848000] [279.3/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 16:50:21,902 - Train: 27.93% [1354200/4848000] [279.3/1000.0] [batch_t 0.330 (0.352)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 16:50:55,265 - Train: 27.94% [1354300/4848000] [279.4/1000.0] [batch_t 0.330 (0.334)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 16:51:29,601 - Train: 27.94% [1354400/4848000] [279.4/1000.0] [batch_t 0.326 (0.343)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 16:52:02,714 - Train: 27.94% [1354500/4848000] [279.4/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 16:52:36,646 - Train: 27.94% [1354600/4848000] [279.4/1000.0] [batch_t 0.329 (0.339)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 16:53:11,187 - Train: 27.94% [1354700/4848000] [279.4/1000.0] [batch_t 0.327 (0.345)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 16:53:47,216 - Train: 27.95% [1354800/4848000] [279.5/1000.0] [batch_t 0.329 (0.360)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 16:54:20,315 - Train: 27.95% [1354900/4848000] [279.5/1000.0] [batch_t 0.325 (0.331)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 16:54:53,539 - Train: 27.95% [1355000/4848000] [279.5/1000.0] [batch_t 0.325 (0.332)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-08 16:55:30,993 - Train: 27.95% [1355100/4848000] [279.5/1000.0] [batch_t 0.331 (0.374)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 16:56:04,074 - Train: 27.95% [1355200/4848000] [279.5/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 16:56:39,443 - Train: 27.96% [1355300/4848000] [279.6/1000.0] [batch_t 0.327 (0.354)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 16:57:14,425 - Train: 27.96% [1355400/4848000] [279.6/1000.0] [batch_t 0.328 (0.350)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 16:57:47,580 - Train: 27.96% [1355500/4848000] [279.6/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 16:58:22,470 - Train: 27.96% [1355600/4848000] [279.6/1000.0] [batch_t 0.332 (0.349)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 16:58:55,603 - Train: 27.96% [1355700/4848000] [279.6/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 16:59:30,870 - Train: 27.97% [1355800/4848000] [279.7/1000.0] [batch_t 0.329 (0.353)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 17:00:03,947 - Train: 27.97% [1355900/4848000] [279.7/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 17:00:41,739 - Train: 27.97% [1356000/4848000] [279.7/1000.0] [batch_t 0.328 (0.378)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 17:01:17,913 - Train: 27.97% [1356100/4848000] [279.7/1000.0] [batch_t 0.330 (0.362)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 17:01:51,056 - Train: 27.97% [1356200/4848000] [279.7/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 17:02:26,040 - Train: 27.98% [1356300/4848000] [279.8/1000.0] [batch_t 0.335 (0.350)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 17:03:00,247 - Train: 27.98% [1356400/4848000] [279.8/1000.0] [batch_t 0.331 (0.342)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 17:03:35,183 - Train: 27.98% [1356500/4848000] [279.8/1000.0] [batch_t 0.326 (0.349)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 17:04:09,561 - Train: 27.98% [1356600/4848000] [279.8/1000.0] [batch_t 0.340 (0.344)] [data_t 0.003] [optim_t 0.337] [lr 0.005000] 2024-04-08 17:04:42,568 - Train: 27.98% [1356700/4848000] [279.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 17:05:16,238 - Train: 27.99% [1356800/4848000] [279.9/1000.0] [batch_t 0.329 (0.337)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 17:05:49,206 - Train: 27.99% [1356900/4848000] [279.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 17:06:23,269 - Train: 27.99% [1357000/4848000] [279.9/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 17:06:56,353 - Train: 27.99% [1357100/4848000] [279.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 17:07:30,583 - Train: 28.00% [1357200/4848000] [280.0/1000.0] [batch_t 0.330 (0.342)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 17:08:03,636 - Train: 28.00% [1357300/4848000] [280.0/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 17:08:38,011 - Train: 28.00% [1357400/4848000] [280.0/1000.0] [batch_t 0.327 (0.344)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 17:08:51,234 - ==> Total time: 5 days, 17:42:49 Eta: 14 days, 18:07:16 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 17:09:16,279 - Train: 28.00% [1357500/4848000] [280.0/1000.0] [batch_t 0.333 (0.375)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 17:09:49,410 - Train: 28.00% [1357600/4848000] [280.0/1000.0] [batch_t 0.336 (0.331)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 17:10:25,060 - Train: 28.01% [1357700/4848000] [280.1/1000.0] [batch_t 0.331 (0.356)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 17:10:58,687 - Train: 28.01% [1357800/4848000] [280.1/1000.0] [batch_t 0.329 (0.336)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 17:11:32,733 - Train: 28.01% [1357900/4848000] [280.1/1000.0] [batch_t 0.327 (0.340)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 17:12:08,255 - Train: 28.01% [1358000/4848000] [280.1/1000.0] [batch_t 0.331 (0.355)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 17:12:41,363 - Train: 28.01% [1358100/4848000] [280.1/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 17:13:15,725 - Train: 28.02% [1358200/4848000] [280.2/1000.0] [batch_t 0.335 (0.344)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 17:13:48,820 - Train: 28.02% [1358300/4848000] [280.2/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 17:14:23,118 - Train: 28.02% [1358400/4848000] [280.2/1000.0] [batch_t 0.330 (0.343)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 17:14:56,228 - Train: 28.02% [1358500/4848000] [280.2/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 17:15:30,989 - Train: 28.02% [1358600/4848000] [280.2/1000.0] [batch_t 0.332 (0.348)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 17:16:04,105 - Train: 28.03% [1358700/4848000] [280.3/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 17:16:38,383 - Train: 28.03% [1358800/4848000] [280.3/1000.0] [batch_t 0.336 (0.343)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 17:17:11,568 - Train: 28.03% [1358900/4848000] [280.3/1000.0] [batch_t 0.332 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 17:17:44,701 - Train: 28.03% [1359000/4848000] [280.3/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 17:18:17,789 - Train: 28.03% [1359100/4848000] [280.3/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 17:18:50,996 - Train: 28.04% [1359200/4848000] [280.4/1000.0] [batch_t 0.336 (0.332)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 17:19:24,081 - Train: 28.04% [1359300/4848000] [280.4/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 17:19:57,159 - Train: 28.04% [1359400/4848000] [280.4/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 17:20:30,210 - Train: 28.04% [1359500/4848000] [280.4/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 17:21:03,270 - Train: 28.04% [1359600/4848000] [280.4/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 17:21:36,431 - Train: 28.05% [1359700/4848000] [280.5/1000.0] [batch_t 0.329 (0.332)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 17:22:09,527 - Train: 28.05% [1359800/4848000] [280.5/1000.0] [batch_t 0.333 (0.331)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 17:22:42,624 - Train: 28.05% [1359900/4848000] [280.5/1000.0] [batch_t 0.335 (0.331)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 17:23:15,719 - Train: 28.05% [1360000/4848000] [280.5/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 17:23:48,730 - Train: 28.05% [1360100/4848000] [280.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 17:24:21,845 - Train: 28.06% [1360200/4848000] [280.6/1000.0] [batch_t 0.325 (0.331)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 17:24:54,949 - Train: 28.06% [1360300/4848000] [280.6/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 17:25:28,078 - Train: 28.06% [1360400/4848000] [280.6/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 17:26:01,158 - Train: 28.06% [1360500/4848000] [280.6/1000.0] [batch_t 0.326 (0.331)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 17:26:34,352 - Train: 28.07% [1360600/4848000] [280.7/1000.0] [batch_t 0.328 (0.332)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 17:27:07,465 - Train: 28.07% [1360700/4848000] [280.7/1000.0] [batch_t 0.333 (0.331)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 17:27:40,580 - Train: 28.07% [1360800/4848000] [280.7/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 17:28:13,707 - Train: 28.07% [1360900/4848000] [280.7/1000.0] [batch_t 0.336 (0.331)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 17:28:46,903 - Train: 28.07% [1361000/4848000] [280.7/1000.0] [batch_t 0.329 (0.332)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 17:29:19,999 - Train: 28.08% [1361100/4848000] [280.8/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 17:29:53,093 - Train: 28.08% [1361200/4848000] [280.8/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 17:30:26,250 - Train: 28.08% [1361300/4848000] [280.8/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 17:30:59,433 - Train: 28.08% [1361400/4848000] [280.8/1000.0] [batch_t 0.335 (0.332)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 17:31:33,224 - Train: 28.08% [1361500/4848000] [280.8/1000.0] [batch_t 0.329 (0.338)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 17:32:06,871 - Train: 28.09% [1361600/4848000] [280.9/1000.0] [batch_t 0.329 (0.336)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 17:32:39,948 - Train: 28.09% [1361700/4848000] [280.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 17:33:14,251 - Train: 28.09% [1361800/4848000] [280.9/1000.0] [batch_t 0.330 (0.343)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 17:33:47,338 - Train: 28.09% [1361900/4848000] [280.9/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 17:34:23,541 - Train: 28.09% [1362000/4848000] [280.9/1000.0] [batch_t 0.332 (0.362)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 17:34:57,028 - Train: 28.10% [1362100/4848000] [281.0/1000.0] [batch_t 0.330 (0.335)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 17:35:31,522 - Train: 28.10% [1362200/4848000] [281.0/1000.0] [batch_t 0.330 (0.345)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 17:36:00,592 - ==> Total time: 5 days, 18:09:59 Eta: 14 days, 17:31:44 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 17:36:09,854 - Train: 28.10% [1362300/4848000] [281.0/1000.0] [batch_t 0.336 (0.586)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 17:36:42,961 - Train: 28.10% [1362400/4848000] [281.0/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 17:37:17,693 - Train: 28.10% [1362500/4848000] [281.0/1000.0] [batch_t 0.331 (0.347)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 17:37:50,863 - Train: 28.11% [1362600/4848000] [281.1/1000.0] [batch_t 0.331 (0.332)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 17:38:25,835 - Train: 28.11% [1362700/4848000] [281.1/1000.0] [batch_t 0.332 (0.350)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 17:38:58,865 - Train: 28.11% [1362800/4848000] [281.1/1000.0] [batch_t 0.333 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 17:39:34,465 - Train: 28.11% [1362900/4848000] [281.1/1000.0] [batch_t 0.332 (0.356)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 17:40:08,537 - Train: 28.11% [1363000/4848000] [281.1/1000.0] [batch_t 0.333 (0.341)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 17:40:41,568 - Train: 28.12% [1363100/4848000] [281.2/1000.0] [batch_t 0.333 (0.330)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 17:41:16,819 - Train: 28.12% [1363200/4848000] [281.2/1000.0] [batch_t 0.326 (0.352)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 17:42:03,319 - Train: 28.12% [1363300/4848000] [281.2/1000.0] [batch_t 0.335 (0.465)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 17:42:37,238 - Train: 28.12% [1363400/4848000] [281.2/1000.0] [batch_t 0.331 (0.339)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 17:43:11,617 - Train: 28.12% [1363500/4848000] [281.2/1000.0] [batch_t 0.329 (0.344)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 17:43:44,682 - Train: 28.13% [1363600/4848000] [281.3/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 17:44:18,512 - Train: 28.13% [1363700/4848000] [281.3/1000.0] [batch_t 0.330 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 17:44:51,666 - Train: 28.13% [1363800/4848000] [281.3/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 17:45:25,739 - Train: 28.13% [1363900/4848000] [281.3/1000.0] [batch_t 0.335 (0.341)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 17:45:58,879 - Train: 28.14% [1364000/4848000] [281.4/1000.0] [batch_t 0.336 (0.331)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 17:46:54,097 - Train: 28.14% [1364100/4848000] [281.4/1000.0] [batch_t 0.328 (0.552)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 17:47:28,375 - Train: 28.14% [1364200/4848000] [281.4/1000.0] [batch_t 0.329 (0.343)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 17:48:25,899 - Train: 28.14% [1364300/4848000] [281.4/1000.0] [batch_t 0.582 (0.575)] [data_t 0.254] [optim_t 0.328] [lr 0.005000] 2024-04-08 17:49:27,959 - Train: 28.14% [1364400/4848000] [281.4/1000.0] [batch_t 0.775 (0.620)] [data_t 0.445] [optim_t 0.329] [lr 0.005000] 2024-04-08 17:50:07,374 - Train: 28.15% [1364500/4848000] [281.5/1000.0] [batch_t 0.329 (0.394)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 17:50:56,125 - Train: 28.15% [1364600/4848000] [281.5/1000.0] [batch_t 0.328 (0.487)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 17:51:29,241 - Train: 28.15% [1364700/4848000] [281.5/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 17:52:02,462 - Train: 28.15% [1364800/4848000] [281.5/1000.0] [batch_t 0.331 (0.332)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 17:52:37,466 - Train: 28.15% [1364900/4848000] [281.5/1000.0] [batch_t 0.336 (0.350)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 17:53:26,521 - Train: 28.16% [1365000/4848000] [281.6/1000.0] [batch_t 0.331 (0.490)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 17:53:59,617 - Train: 28.16% [1365100/4848000] [281.6/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 17:54:33,863 - Train: 28.16% [1365200/4848000] [281.6/1000.0] [batch_t 0.329 (0.342)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 17:55:13,055 - Train: 28.16% [1365300/4848000] [281.6/1000.0] [batch_t 0.334 (0.392)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 17:55:55,212 - Train: 28.16% [1365400/4848000] [281.6/1000.0] [batch_t 0.336 (0.421)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 17:56:29,279 - Train: 28.17% [1365500/4848000] [281.7/1000.0] [batch_t 0.331 (0.341)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 17:57:02,378 - Train: 28.17% [1365600/4848000] [281.7/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 17:57:36,943 - Train: 28.17% [1365700/4848000] [281.7/1000.0] [batch_t 0.331 (0.346)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 17:58:25,840 - Train: 28.17% [1365800/4848000] [281.7/1000.0] [batch_t 0.329 (0.489)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 17:59:18,665 - Train: 28.17% [1365900/4848000] [281.7/1000.0] [batch_t 0.328 (0.528)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 18:00:24,534 - Train: 28.18% [1366000/4848000] [281.8/1000.0] [batch_t 3.262 (0.658)] [data_t 2.925] [optim_t 0.337] [lr 0.005000] 2024-04-08 18:01:02,219 - Train: 28.18% [1366100/4848000] [281.8/1000.0] [batch_t 0.337 (0.377)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-08 18:01:37,579 - Train: 28.18% [1366200/4848000] [281.8/1000.0] [batch_t 0.331 (0.354)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 18:02:50,767 - Train: 28.18% [1366300/4848000] [281.8/1000.0] [batch_t 0.336 (0.732)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 18:03:56,139 - Train: 28.18% [1366400/4848000] [281.8/1000.0] [batch_t 0.331 (0.654)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 18:04:30,879 - Train: 28.19% [1366500/4848000] [281.9/1000.0] [batch_t 0.333 (0.347)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 18:05:03,931 - Train: 28.19% [1366600/4848000] [281.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 18:05:59,141 - Train: 28.19% [1366700/4848000] [281.9/1000.0] [batch_t 0.325 (0.552)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 18:07:25,868 - Train: 28.19% [1366800/4848000] [281.9/1000.0] [batch_t 0.329 (0.867)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 18:07:58,882 - Train: 28.20% [1366900/4848000] [282.0/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 18:08:43,755 - Train: 28.20% [1367000/4848000] [282.0/1000.0] [batch_t 0.331 (0.449)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 18:09:17,797 - Train: 28.20% [1367100/4848000] [282.0/1000.0] [batch_t 0.329 (0.340)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 18:09:29,721 - ==> Total time: 5 days, 18:43:28 Eta: 14 days, 17:12:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 18:10:11,629 - Train: 28.20% [1367200/4848000] [282.0/1000.0] [batch_t 0.331 (0.610)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 18:10:44,777 - Train: 28.20% [1367300/4848000] [282.0/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 18:11:28,599 - Train: 28.21% [1367400/4848000] [282.1/1000.0] [batch_t 0.687 (0.438)] [data_t 0.367] [optim_t 0.321] [lr 0.005000] 2024-04-08 18:12:02,016 - Train: 28.21% [1367500/4848000] [282.1/1000.0] [batch_t 0.325 (0.334)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 18:12:38,192 - Train: 28.21% [1367600/4848000] [282.1/1000.0] [batch_t 0.328 (0.362)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 18:13:25,285 - Train: 28.21% [1367700/4848000] [282.1/1000.0] [batch_t 0.329 (0.471)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 18:13:58,404 - Train: 28.21% [1367800/4848000] [282.1/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 18:14:33,172 - Train: 28.22% [1367900/4848000] [282.2/1000.0] [batch_t 0.331 (0.348)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 18:15:07,435 - Train: 28.22% [1368000/4848000] [282.2/1000.0] [batch_t 0.325 (0.343)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 18:15:40,540 - Train: 28.22% [1368100/4848000] [282.2/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 18:16:17,971 - Train: 28.22% [1368200/4848000] [282.2/1000.0] [batch_t 0.329 (0.374)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 18:16:51,077 - Train: 28.22% [1368300/4848000] [282.2/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 18:17:24,516 - Train: 28.23% [1368400/4848000] [282.3/1000.0] [batch_t 0.333 (0.334)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 18:18:01,670 - Train: 28.23% [1368500/4848000] [282.3/1000.0] [batch_t 0.331 (0.371)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 18:18:36,181 - Train: 28.23% [1368600/4848000] [282.3/1000.0] [batch_t 0.333 (0.345)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 18:19:09,709 - Train: 28.23% [1368700/4848000] [282.3/1000.0] [batch_t 0.326 (0.335)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 18:19:42,779 - Train: 28.23% [1368800/4848000] [282.3/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 18:20:17,284 - Train: 28.24% [1368900/4848000] [282.4/1000.0] [batch_t 0.328 (0.345)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 18:20:50,295 - Train: 28.24% [1369000/4848000] [282.4/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 18:21:28,677 - Train: 28.24% [1369100/4848000] [282.4/1000.0] [batch_t 0.331 (0.384)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 18:22:01,814 - Train: 28.24% [1369200/4848000] [282.4/1000.0] [batch_t 0.339 (0.331)] [data_t 0.003] [optim_t 0.336] [lr 0.005000] 2024-04-08 18:22:36,922 - Train: 28.24% [1369300/4848000] [282.4/1000.0] [batch_t 0.330 (0.351)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 18:23:14,784 - Train: 28.25% [1369400/4848000] [282.5/1000.0] [batch_t 0.329 (0.379)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 18:23:47,956 - Train: 28.25% [1369500/4848000] [282.5/1000.0] [batch_t 0.330 (0.332)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 18:24:23,429 - Train: 28.25% [1369600/4848000] [282.5/1000.0] [batch_t 0.331 (0.355)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 18:24:56,611 - Train: 28.25% [1369700/4848000] [282.5/1000.0] [batch_t 0.329 (0.332)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 18:25:30,813 - Train: 28.25% [1369800/4848000] [282.5/1000.0] [batch_t 0.333 (0.342)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 18:26:03,927 - Train: 28.26% [1369900/4848000] [282.6/1000.0] [batch_t 0.324 (0.331)] [data_t 0.003] [optim_t 0.321] [lr 0.005000] 2024-04-08 18:26:39,208 - Train: 28.26% [1370000/4848000] [282.6/1000.0] [batch_t 0.326 (0.353)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 18:27:12,383 - Train: 28.26% [1370100/4848000] [282.6/1000.0] [batch_t 0.328 (0.332)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 18:27:45,520 - Train: 28.26% [1370200/4848000] [282.6/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 18:28:18,582 - Train: 28.27% [1370300/4848000] [282.7/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 18:28:51,653 - Train: 28.27% [1370400/4848000] [282.7/1000.0] [batch_t 0.336 (0.331)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 18:29:24,798 - Train: 28.27% [1370500/4848000] [282.7/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 18:29:58,026 - Train: 28.27% [1370600/4848000] [282.7/1000.0] [batch_t 0.330 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 18:30:31,162 - Train: 28.27% [1370700/4848000] [282.7/1000.0] [batch_t 0.327 (0.331)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 18:31:04,275 - Train: 28.28% [1370800/4848000] [282.8/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 18:31:37,312 - Train: 28.28% [1370900/4848000] [282.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 18:32:10,457 - Train: 28.28% [1371000/4848000] [282.8/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 18:32:43,607 - Train: 28.28% [1371100/4848000] [282.8/1000.0] [batch_t 0.336 (0.331)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 18:33:16,669 - Train: 28.28% [1371200/4848000] [282.8/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 18:33:49,742 - Train: 28.29% [1371300/4848000] [282.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 18:34:23,408 - Train: 28.29% [1371400/4848000] [282.9/1000.0] [batch_t 0.328 (0.337)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 18:34:56,461 - Train: 28.29% [1371500/4848000] [282.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 18:35:30,796 - Train: 28.29% [1371600/4848000] [282.9/1000.0] [batch_t 0.330 (0.343)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 18:36:04,012 - Train: 28.29% [1371700/4848000] [282.9/1000.0] [batch_t 0.332 (0.332)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 18:36:38,502 - Train: 28.30% [1371800/4848000] [283.0/1000.0] [batch_t 0.331 (0.345)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 18:37:12,901 - Train: 28.30% [1371900/4848000] [283.0/1000.0] [batch_t 0.331 (0.344)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 18:37:40,817 - ==> Total time: 5 days, 19:11:39 Eta: 14 days, 16:39:29 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 18:37:48,174 - Train: 28.30% [1372000/4848000] [283.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 18:38:21,412 - Train: 28.30% [1372100/4848000] [283.0/1000.0] [batch_t 0.338 (0.332)] [data_t 0.003] [optim_t 0.336] [lr 0.005000] 2024-04-08 18:38:54,507 - Train: 28.30% [1372200/4848000] [283.0/1000.0] [batch_t 0.338 (0.331)] [data_t 0.003] [optim_t 0.335] [lr 0.005000] 2024-04-08 18:39:27,761 - Train: 28.31% [1372300/4848000] [283.1/1000.0] [batch_t 0.332 (0.332)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 18:40:00,880 - Train: 28.31% [1372400/4848000] [283.1/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 18:40:33,897 - Train: 28.31% [1372500/4848000] [283.1/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 18:41:07,199 - Train: 28.31% [1372600/4848000] [283.1/1000.0] [batch_t 0.334 (0.333)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 18:41:40,306 - Train: 28.31% [1372700/4848000] [283.1/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 18:42:13,357 - Train: 28.32% [1372800/4848000] [283.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 18:42:46,499 - Train: 28.32% [1372900/4848000] [283.2/1000.0] [batch_t 0.326 (0.331)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 18:43:19,598 - Train: 28.32% [1373000/4848000] [283.2/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 18:43:52,738 - Train: 28.32% [1373100/4848000] [283.2/1000.0] [batch_t 0.327 (0.331)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 18:44:26,713 - Train: 28.33% [1373200/4848000] [283.3/1000.0] [batch_t 0.328 (0.340)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 18:44:59,915 - Train: 28.33% [1373300/4848000] [283.3/1000.0] [batch_t 0.335 (0.332)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 18:45:33,925 - Train: 28.33% [1373400/4848000] [283.3/1000.0] [batch_t 0.328 (0.340)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 18:46:06,990 - Train: 28.33% [1373500/4848000] [283.3/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 18:46:40,021 - Train: 28.33% [1373600/4848000] [283.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 18:47:13,118 - Train: 28.34% [1373700/4848000] [283.4/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 18:47:46,256 - Train: 28.34% [1373800/4848000] [283.4/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 18:48:19,419 - Train: 28.34% [1373900/4848000] [283.4/1000.0] [batch_t 0.336 (0.332)] [data_t 0.002] [optim_t 0.333] [lr 0.005000] 2024-04-08 18:48:52,553 - Train: 28.34% [1374000/4848000] [283.4/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 18:49:25,686 - Train: 28.34% [1374100/4848000] [283.4/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 18:49:58,831 - Train: 28.35% [1374200/4848000] [283.5/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 18:50:31,989 - Train: 28.35% [1374300/4848000] [283.5/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 18:51:05,199 - Train: 28.35% [1374400/4848000] [283.5/1000.0] [batch_t 0.324 (0.332)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-08 18:51:38,338 - Train: 28.35% [1374500/4848000] [283.5/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 18:52:11,521 - Train: 28.35% [1374600/4848000] [283.5/1000.0] [batch_t 0.332 (0.332)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 18:52:44,656 - Train: 28.36% [1374700/4848000] [283.6/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 18:53:17,975 - Train: 28.36% [1374800/4848000] [283.6/1000.0] [batch_t 0.329 (0.333)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 18:53:51,157 - Train: 28.36% [1374900/4848000] [283.6/1000.0] [batch_t 0.333 (0.332)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 18:54:24,275 - Train: 28.36% [1375000/4848000] [283.6/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 18:54:57,538 - Train: 28.36% [1375100/4848000] [283.6/1000.0] [batch_t 0.330 (0.333)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 18:55:36,214 - Train: 28.37% [1375200/4848000] [283.7/1000.0] [batch_t 0.329 (0.387)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 18:56:22,005 - Train: 28.37% [1375300/4848000] [283.7/1000.0] [batch_t 0.329 (0.458)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 18:56:55,230 - Train: 28.37% [1375400/4848000] [283.7/1000.0] [batch_t 0.329 (0.332)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 18:57:29,315 - Train: 28.37% [1375500/4848000] [283.7/1000.0] [batch_t 0.330 (0.341)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 18:58:02,494 - Train: 28.37% [1375600/4848000] [283.7/1000.0] [batch_t 0.329 (0.332)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 18:58:36,862 - Train: 28.38% [1375700/4848000] [283.8/1000.0] [batch_t 0.334 (0.344)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 18:59:10,003 - Train: 28.38% [1375800/4848000] [283.8/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 18:59:43,082 - Train: 28.38% [1375900/4848000] [283.8/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 19:00:17,729 - Train: 28.38% [1376000/4848000] [283.8/1000.0] [batch_t 0.331 (0.346)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 19:00:50,888 - Train: 28.38% [1376100/4848000] [283.8/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 19:01:25,360 - Train: 28.39% [1376200/4848000] [283.9/1000.0] [batch_t 0.339 (0.345)] [data_t 0.003] [optim_t 0.336] [lr 0.005000] 2024-04-08 19:02:07,140 - Train: 28.39% [1376300/4848000] [283.9/1000.0] [batch_t 0.328 (0.418)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 19:02:40,784 - Train: 28.39% [1376400/4848000] [283.9/1000.0] [batch_t 0.328 (0.336)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 19:03:23,077 - Train: 28.39% [1376500/4848000] [283.9/1000.0] [batch_t 0.332 (0.423)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 19:03:56,217 - Train: 28.40% [1376600/4848000] [284.0/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 19:04:30,018 - Train: 28.40% [1376700/4848000] [284.0/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:05:03,215 - Train: 28.40% [1376800/4848000] [284.0/1000.0] [batch_t 0.333 (0.332)] [data_t 0.002] [optim_t 0.331] [lr 0.005000] 2024-04-08 19:05:14,706 - ==> Total time: 5 days, 19:39:13 Eta: 14 days, 16:05:04 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 19:05:39,711 - Train: 28.40% [1376900/4848000] [284.0/1000.0] [batch_t 0.327 (0.333)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 19:06:12,794 - Train: 28.40% [1377000/4848000] [284.0/1000.0] [batch_t 0.335 (0.331)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 19:06:45,854 - Train: 28.41% [1377100/4848000] [284.1/1000.0] [batch_t 0.327 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 19:07:18,913 - Train: 28.41% [1377200/4848000] [284.1/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 19:07:51,976 - Train: 28.41% [1377300/4848000] [284.1/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:08:25,055 - Train: 28.41% [1377400/4848000] [284.1/1000.0] [batch_t 0.325 (0.331)] [data_t 0.003] [optim_t 0.322] [lr 0.005000] 2024-04-08 19:09:00,554 - Train: 28.41% [1377500/4848000] [284.1/1000.0] [batch_t 0.329 (0.355)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:09:33,882 - Train: 28.42% [1377600/4848000] [284.2/1000.0] [batch_t 0.332 (0.333)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 19:10:06,978 - Train: 28.42% [1377700/4848000] [284.2/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 19:10:40,019 - Train: 28.42% [1377800/4848000] [284.2/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:11:13,128 - Train: 28.42% [1377900/4848000] [284.2/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:11:46,271 - Train: 28.42% [1378000/4848000] [284.2/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 19:12:19,362 - Train: 28.43% [1378100/4848000] [284.3/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 19:12:52,479 - Train: 28.43% [1378200/4848000] [284.3/1000.0] [batch_t 0.339 (0.331)] [data_t 0.003] [optim_t 0.336] [lr 0.005000] 2024-04-08 19:13:25,540 - Train: 28.43% [1378300/4848000] [284.3/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:13:58,622 - Train: 28.43% [1378400/4848000] [284.3/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:14:31,708 - Train: 28.43% [1378500/4848000] [284.3/1000.0] [batch_t 0.337 (0.331)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-08 19:15:04,822 - Train: 28.44% [1378600/4848000] [284.4/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:15:37,906 - Train: 28.44% [1378700/4848000] [284.4/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 19:16:10,933 - Train: 28.44% [1378800/4848000] [284.4/1000.0] [batch_t 0.326 (0.330)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 19:16:44,085 - Train: 28.44% [1378900/4848000] [284.4/1000.0] [batch_t 0.323 (0.331)] [data_t 0.003] [optim_t 0.320] [lr 0.005000] 2024-04-08 19:17:18,561 - Train: 28.44% [1379000/4848000] [284.4/1000.0] [batch_t 0.330 (0.345)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:17:51,667 - Train: 28.45% [1379100/4848000] [284.5/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 19:18:25,672 - Train: 28.45% [1379200/4848000] [284.5/1000.0] [batch_t 0.332 (0.340)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 19:18:58,809 - Train: 28.45% [1379300/4848000] [284.5/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 19:19:32,182 - Train: 28.45% [1379400/4848000] [284.5/1000.0] [batch_t 0.327 (0.334)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 19:20:05,197 - Train: 28.46% [1379500/4848000] [284.6/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 19:20:38,238 - Train: 28.46% [1379600/4848000] [284.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 19:21:11,319 - Train: 28.46% [1379700/4848000] [284.6/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:21:44,381 - Train: 28.46% [1379800/4848000] [284.6/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:22:18,627 - Train: 28.46% [1379900/4848000] [284.6/1000.0] [batch_t 0.329 (0.342)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 19:22:51,702 - Train: 28.47% [1380000/4848000] [284.7/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:23:25,114 - Train: 28.47% [1380100/4848000] [284.7/1000.0] [batch_t 0.330 (0.334)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:23:58,151 - Train: 28.47% [1380200/4848000] [284.7/1000.0] [batch_t 0.326 (0.330)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 19:24:31,578 - Train: 28.47% [1380300/4848000] [284.7/1000.0] [batch_t 0.330 (0.334)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:25:04,826 - Train: 28.47% [1380400/4848000] [284.7/1000.0] [batch_t 0.385 (0.332)] [data_t 0.049] [optim_t 0.336] [lr 0.005000] 2024-04-08 19:25:37,954 - Train: 28.48% [1380500/4848000] [284.8/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:26:11,155 - Train: 28.48% [1380600/4848000] [284.8/1000.0] [batch_t 0.335 (0.332)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 19:26:44,285 - Train: 28.48% [1380700/4848000] [284.8/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 19:27:19,620 - Train: 28.48% [1380800/4848000] [284.8/1000.0] [batch_t 0.329 (0.353)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:27:53,555 - Train: 28.48% [1380900/4848000] [284.8/1000.0] [batch_t 0.337 (0.339)] [data_t 0.003] [optim_t 0.335] [lr 0.005000] 2024-04-08 19:28:26,668 - Train: 28.49% [1381000/4848000] [284.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:28:59,738 - Train: 28.49% [1381100/4848000] [284.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:29:32,811 - Train: 28.49% [1381200/4848000] [284.9/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:30:05,843 - Train: 28.49% [1381300/4848000] [284.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:30:38,882 - Train: 28.49% [1381400/4848000] [284.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:31:12,037 - Train: 28.50% [1381500/4848000] [285.0/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:31:45,181 - Train: 28.50% [1381600/4848000] [285.0/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 19:32:11,815 - ==> Total time: 5 days, 20:06:10 Eta: 14 days, 15:29:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 19:32:21,061 - Train: 28.50% [1381700/4848000] [285.0/1000.0] [batch_t 0.329 (0.350)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 19:32:54,118 - Train: 28.50% [1381800/4848000] [285.0/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 19:33:27,185 - Train: 28.50% [1381900/4848000] [285.0/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 19:34:00,281 - Train: 28.51% [1382000/4848000] [285.1/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:34:34,131 - Train: 28.51% [1382100/4848000] [285.1/1000.0] [batch_t 0.333 (0.338)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 19:35:07,274 - Train: 28.51% [1382200/4848000] [285.1/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:35:40,339 - Train: 28.51% [1382300/4848000] [285.1/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:36:13,411 - Train: 28.51% [1382400/4848000] [285.1/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 19:36:46,474 - Train: 28.52% [1382500/4848000] [285.2/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 19:37:20,112 - Train: 28.52% [1382600/4848000] [285.2/1000.0] [batch_t 0.325 (0.336)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 19:37:53,139 - Train: 28.52% [1382700/4848000] [285.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 19:38:26,726 - Train: 28.52% [1382800/4848000] [285.2/1000.0] [batch_t 0.326 (0.336)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 19:38:59,826 - Train: 28.53% [1382900/4848000] [285.3/1000.0] [batch_t 0.330 (0.331)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:39:34,000 - Train: 28.53% [1383000/4848000] [285.3/1000.0] [batch_t 0.330 (0.342)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:40:07,702 - Train: 28.53% [1383100/4848000] [285.3/1000.0] [batch_t 0.327 (0.337)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 19:40:41,015 - Train: 28.53% [1383200/4848000] [285.3/1000.0] [batch_t 0.339 (0.333)] [data_t 0.003] [optim_t 0.337] [lr 0.005000] 2024-04-08 19:41:14,080 - Train: 28.53% [1383300/4848000] [285.3/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 19:41:47,144 - Train: 28.54% [1383400/4848000] [285.4/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:42:21,518 - Train: 28.54% [1383500/4848000] [285.4/1000.0] [batch_t 0.334 (0.344)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 19:42:54,664 - Train: 28.54% [1383600/4848000] [285.4/1000.0] [batch_t 0.335 (0.331)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 19:43:28,786 - Train: 28.54% [1383700/4848000] [285.4/1000.0] [batch_t 0.330 (0.341)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:44:01,940 - Train: 28.54% [1383800/4848000] [285.4/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:44:36,156 - Train: 28.55% [1383900/4848000] [285.5/1000.0] [batch_t 0.331 (0.342)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 19:45:10,159 - Train: 28.55% [1384000/4848000] [285.5/1000.0] [batch_t 0.328 (0.340)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 19:45:43,360 - Train: 28.55% [1384100/4848000] [285.5/1000.0] [batch_t 0.328 (0.332)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 19:46:16,453 - Train: 28.55% [1384200/4848000] [285.5/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:46:49,472 - Train: 28.55% [1384300/4848000] [285.5/1000.0] [batch_t 0.334 (0.330)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 19:47:22,617 - Train: 28.56% [1384400/4848000] [285.6/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 19:47:55,737 - Train: 28.56% [1384500/4848000] [285.6/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 19:48:28,972 - Train: 28.56% [1384600/4848000] [285.6/1000.0] [batch_t 0.329 (0.332)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 19:49:02,019 - Train: 28.56% [1384700/4848000] [285.6/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:49:35,452 - Train: 28.56% [1384800/4848000] [285.6/1000.0] [batch_t 0.330 (0.334)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:50:08,924 - Train: 28.57% [1384900/4848000] [285.7/1000.0] [batch_t 0.329 (0.335)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:50:42,087 - Train: 28.57% [1385000/4848000] [285.7/1000.0] [batch_t 0.330 (0.332)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:51:15,491 - Train: 28.57% [1385100/4848000] [285.7/1000.0] [batch_t 0.331 (0.334)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 19:51:48,617 - Train: 28.57% [1385200/4848000] [285.7/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 19:52:23,390 - Train: 28.57% [1385300/4848000] [285.7/1000.0] [batch_t 0.330 (0.348)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:52:56,450 - Train: 28.58% [1385400/4848000] [285.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 19:53:29,611 - Train: 28.58% [1385500/4848000] [285.8/1000.0] [batch_t 0.327 (0.332)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 19:54:02,659 - Train: 28.58% [1385600/4848000] [285.8/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:54:37,365 - Train: 28.58% [1385700/4848000] [285.8/1000.0] [batch_t 0.330 (0.347)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:55:11,104 - Train: 28.58% [1385800/4848000] [285.8/1000.0] [batch_t 0.331 (0.337)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:55:44,150 - Train: 28.59% [1385900/4848000] [285.9/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 19:56:18,332 - Train: 28.59% [1386000/4848000] [285.9/1000.0] [batch_t 0.331 (0.342)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:56:51,765 - Train: 28.59% [1386100/4848000] [285.9/1000.0] [batch_t 0.333 (0.332)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 19:57:24,856 - Train: 28.59% [1386200/4848000] [285.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:57:58,496 - Train: 28.60% [1386300/4848000] [286.0/1000.0] [batch_t 0.330 (0.336)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:58:33,426 - Train: 28.60% [1386400/4848000] [286.0/1000.0] [batch_t 0.330 (0.349)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 19:59:09,299 - Train: 28.60% [1386500/4848000] [286.0/1000.0] [batch_t 0.336 (0.359)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 19:59:18,560 - ==> Total time: 5 days, 20:33:17 Eta: 14 days, 14:53:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 19:59:44,680 - Train: 28.60% [1386600/4848000] [286.0/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:00:18,682 - Train: 28.60% [1386700/4848000] [286.0/1000.0] [batch_t 0.330 (0.340)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 20:00:51,723 - Train: 28.61% [1386800/4848000] [286.1/1000.0] [batch_t 0.326 (0.330)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 20:01:25,699 - Train: 28.61% [1386900/4848000] [286.1/1000.0] [batch_t 0.327 (0.340)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 20:01:58,801 - Train: 28.61% [1387000/4848000] [286.1/1000.0] [batch_t 0.334 (0.331)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 20:02:31,852 - Train: 28.61% [1387100/4848000] [286.1/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 20:03:04,896 - Train: 28.61% [1387200/4848000] [286.1/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 20:03:37,925 - Train: 28.62% [1387300/4848000] [286.2/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 20:04:12,007 - Train: 28.62% [1387400/4848000] [286.2/1000.0] [batch_t 0.329 (0.341)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 20:04:45,100 - Train: 28.62% [1387500/4848000] [286.2/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 20:05:19,011 - Train: 28.62% [1387600/4848000] [286.2/1000.0] [batch_t 0.327 (0.339)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 20:05:52,107 - Train: 28.62% [1387700/4848000] [286.2/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 20:06:25,123 - Train: 28.63% [1387800/4848000] [286.3/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 20:06:58,130 - Train: 28.63% [1387900/4848000] [286.3/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 20:07:31,930 - Train: 28.63% [1388000/4848000] [286.3/1000.0] [batch_t 0.332 (0.338)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 20:08:06,386 - Train: 28.63% [1388100/4848000] [286.3/1000.0] [batch_t 0.333 (0.344)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 20:08:39,510 - Train: 28.63% [1388200/4848000] [286.3/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:09:13,551 - Train: 28.64% [1388300/4848000] [286.4/1000.0] [batch_t 0.330 (0.340)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:09:46,649 - Train: 28.64% [1388400/4848000] [286.4/1000.0] [batch_t 0.327 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 20:10:20,411 - Train: 28.64% [1388500/4848000] [286.4/1000.0] [batch_t 0.329 (0.338)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 20:10:53,619 - Train: 28.64% [1388600/4848000] [286.4/1000.0] [batch_t 0.331 (0.332)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:11:29,585 - Train: 28.64% [1388700/4848000] [286.4/1000.0] [batch_t 0.326 (0.360)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 20:12:02,856 - Train: 28.65% [1388800/4848000] [286.5/1000.0] [batch_t 0.328 (0.333)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 20:12:37,396 - Train: 28.65% [1388900/4848000] [286.5/1000.0] [batch_t 0.331 (0.345)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:13:11,175 - Train: 28.65% [1389000/4848000] [286.5/1000.0] [batch_t 0.330 (0.338)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:13:44,221 - Train: 28.65% [1389100/4848000] [286.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:14:19,859 - Train: 28.66% [1389200/4848000] [286.6/1000.0] [batch_t 0.337 (0.356)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-08 20:14:53,013 - Train: 28.66% [1389300/4848000] [286.6/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 20:15:27,003 - Train: 28.66% [1389400/4848000] [286.6/1000.0] [batch_t 0.331 (0.340)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:15:59,966 - Train: 28.66% [1389500/4848000] [286.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 20:16:33,892 - Train: 28.66% [1389600/4848000] [286.6/1000.0] [batch_t 0.331 (0.339)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:17:08,671 - Train: 28.67% [1389700/4848000] [286.7/1000.0] [batch_t 0.333 (0.348)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 20:17:41,845 - Train: 28.67% [1389800/4848000] [286.7/1000.0] [batch_t 0.327 (0.332)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 20:18:15,988 - Train: 28.67% [1389900/4848000] [286.7/1000.0] [batch_t 0.330 (0.341)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:18:48,992 - Train: 28.67% [1390000/4848000] [286.7/1000.0] [batch_t 0.333 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 20:19:22,084 - Train: 28.67% [1390100/4848000] [286.7/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 20:19:55,186 - Train: 28.68% [1390200/4848000] [286.8/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 20:20:29,383 - Train: 28.68% [1390300/4848000] [286.8/1000.0] [batch_t 0.332 (0.342)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 20:21:02,584 - Train: 28.68% [1390400/4848000] [286.8/1000.0] [batch_t 0.329 (0.332)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 20:21:35,736 - Train: 28.68% [1390500/4848000] [286.8/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 20:22:11,762 - Train: 28.68% [1390600/4848000] [286.8/1000.0] [batch_t 0.326 (0.360)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 20:22:44,877 - Train: 28.69% [1390700/4848000] [286.9/1000.0] [batch_t 0.336 (0.331)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 20:23:21,095 - Train: 28.69% [1390800/4848000] [286.9/1000.0] [batch_t 0.332 (0.362)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 20:23:54,160 - Train: 28.69% [1390900/4848000] [286.9/1000.0] [batch_t 0.329 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 20:24:31,873 - Train: 28.69% [1391000/4848000] [286.9/1000.0] [batch_t 0.331 (0.377)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:25:07,138 - Train: 28.69% [1391100/4848000] [286.9/1000.0] [batch_t 0.327 (0.353)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 20:25:40,226 - Train: 28.70% [1391200/4848000] [287.0/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 20:26:16,605 - Train: 28.70% [1391300/4848000] [287.0/1000.0] [batch_t 0.331 (0.364)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:26:41,734 - ==> Total time: 5 days, 21:00:40 Eta: 14 days, 14:19:01 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 20:26:52,105 - Train: 28.70% [1391400/4848000] [287.0/1000.0] [batch_t 0.333 (0.336)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 20:27:26,571 - Train: 28.70% [1391500/4848000] [287.0/1000.0] [batch_t 0.336 (0.345)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 20:27:59,818 - Train: 28.70% [1391600/4848000] [287.0/1000.0] [batch_t 0.331 (0.332)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 20:28:34,624 - Train: 28.71% [1391700/4848000] [287.1/1000.0] [batch_t 0.331 (0.348)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 20:29:09,461 - Train: 28.71% [1391800/4848000] [287.1/1000.0] [batch_t 0.331 (0.348)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:29:42,606 - Train: 28.71% [1391900/4848000] [287.1/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 20:30:18,795 - Train: 28.71% [1392000/4848000] [287.1/1000.0] [batch_t 0.330 (0.362)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:30:51,832 - Train: 28.71% [1392100/4848000] [287.1/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 20:31:26,327 - Train: 28.72% [1392200/4848000] [287.2/1000.0] [batch_t 0.326 (0.345)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 20:31:59,364 - Train: 28.72% [1392300/4848000] [287.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 20:32:33,467 - Train: 28.72% [1392400/4848000] [287.2/1000.0] [batch_t 0.328 (0.341)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 20:33:07,579 - Train: 28.72% [1392500/4848000] [287.2/1000.0] [batch_t 0.332 (0.341)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 20:33:40,629 - Train: 28.73% [1392600/4848000] [287.3/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 20:34:15,342 - Train: 28.73% [1392700/4848000] [287.3/1000.0] [batch_t 0.327 (0.347)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 20:34:48,424 - Train: 28.73% [1392800/4848000] [287.3/1000.0] [batch_t 0.324 (0.331)] [data_t 0.003] [optim_t 0.321] [lr 0.005000] 2024-04-08 20:35:22,195 - Train: 28.73% [1392900/4848000] [287.3/1000.0] [batch_t 0.333 (0.338)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 20:35:55,363 - Train: 28.73% [1393000/4848000] [287.3/1000.0] [batch_t 0.333 (0.332)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 20:36:29,259 - Train: 28.74% [1393100/4848000] [287.4/1000.0] [batch_t 0.331 (0.339)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:37:02,892 - Train: 28.74% [1393200/4848000] [287.4/1000.0] [batch_t 0.335 (0.336)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 20:37:36,728 - Train: 28.74% [1393300/4848000] [287.4/1000.0] [batch_t 0.331 (0.338)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:38:10,402 - Train: 28.74% [1393400/4848000] [287.4/1000.0] [batch_t 0.327 (0.337)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 20:38:43,483 - Train: 28.74% [1393500/4848000] [287.4/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 20:39:17,283 - Train: 28.75% [1393600/4848000] [287.5/1000.0] [batch_t 0.332 (0.338)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 20:39:50,310 - Train: 28.75% [1393700/4848000] [287.5/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 20:40:24,283 - Train: 28.75% [1393800/4848000] [287.5/1000.0] [batch_t 0.330 (0.340)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 20:40:57,366 - Train: 28.75% [1393900/4848000] [287.5/1000.0] [batch_t 0.335 (0.331)] [data_t 0.002] [optim_t 0.332] [lr 0.005000] 2024-04-08 20:41:30,542 - Train: 28.75% [1394000/4848000] [287.5/1000.0] [batch_t 0.328 (0.332)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 20:42:03,659 - Train: 28.76% [1394100/4848000] [287.6/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 20:42:36,820 - Train: 28.76% [1394200/4848000] [287.6/1000.0] [batch_t 0.330 (0.332)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 20:43:09,920 - Train: 28.76% [1394300/4848000] [287.6/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 20:43:42,999 - Train: 28.76% [1394400/4848000] [287.6/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 20:44:16,226 - Train: 28.76% [1394500/4848000] [287.6/1000.0] [batch_t 0.330 (0.332)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 20:44:49,311 - Train: 28.77% [1394600/4848000] [287.7/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 20:45:22,367 - Train: 28.77% [1394700/4848000] [287.7/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 20:45:55,428 - Train: 28.77% [1394800/4848000] [287.7/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 20:46:28,548 - Train: 28.77% [1394900/4848000] [287.7/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 20:47:01,638 - Train: 28.77% [1395000/4848000] [287.7/1000.0] [batch_t 0.336 (0.331)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 20:47:34,745 - Train: 28.78% [1395100/4848000] [287.8/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:48:07,889 - Train: 28.78% [1395200/4848000] [287.8/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 20:48:40,969 - Train: 28.78% [1395300/4848000] [287.8/1000.0] [batch_t 0.337 (0.331)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-08 20:49:14,088 - Train: 28.78% [1395400/4848000] [287.8/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:49:47,137 - Train: 28.79% [1395500/4848000] [287.9/1000.0] [batch_t 0.334 (0.330)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 20:50:20,234 - Train: 28.79% [1395600/4848000] [287.9/1000.0] [batch_t 0.333 (0.331)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 20:50:53,303 - Train: 28.79% [1395700/4848000] [287.9/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 20:51:26,423 - Train: 28.79% [1395800/4848000] [287.9/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 20:51:59,648 - Train: 28.79% [1395900/4848000] [287.9/1000.0] [batch_t 0.332 (0.332)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 20:52:32,745 - Train: 28.80% [1396000/4848000] [288.0/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 20:53:05,875 - Train: 28.80% [1396100/4848000] [288.0/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 20:53:38,946 - Train: 28.80% [1396200/4848000] [288.0/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:53:46,909 - ==> Total time: 5 days, 21:27:45 Eta: 14 days, 13:43:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 20:54:14,401 - Train: 28.80% [1396300/4848000] [288.0/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 20:54:47,505 - Train: 28.80% [1396400/4848000] [288.0/1000.0] [batch_t 0.336 (0.331)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 20:55:20,611 - Train: 28.81% [1396500/4848000] [288.1/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:55:53,681 - Train: 28.81% [1396600/4848000] [288.1/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:56:26,814 - Train: 28.81% [1396700/4848000] [288.1/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:56:59,895 - Train: 28.81% [1396800/4848000] [288.1/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 20:57:32,904 - Train: 28.81% [1396900/4848000] [288.1/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 20:58:06,061 - Train: 28.82% [1397000/4848000] [288.2/1000.0] [batch_t 0.340 (0.331)] [data_t 0.003] [optim_t 0.337] [lr 0.005000] 2024-04-08 20:58:39,172 - Train: 28.82% [1397100/4848000] [288.2/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 20:59:12,231 - Train: 28.82% [1397200/4848000] [288.2/1000.0] [batch_t 0.324 (0.330)] [data_t 0.002] [optim_t 0.322] [lr 0.005000] 2024-04-08 20:59:45,401 - Train: 28.82% [1397300/4848000] [288.2/1000.0] [batch_t 0.332 (0.332)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 21:00:18,529 - Train: 28.82% [1397400/4848000] [288.2/1000.0] [batch_t 0.332 (0.331)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 21:00:51,626 - Train: 28.83% [1397500/4848000] [288.3/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 21:01:24,747 - Train: 28.83% [1397600/4848000] [288.3/1000.0] [batch_t 0.336 (0.331)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 21:01:57,835 - Train: 28.83% [1397700/4848000] [288.3/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 21:02:30,946 - Train: 28.83% [1397800/4848000] [288.3/1000.0] [batch_t 0.326 (0.331)] [data_t 0.002] [optim_t 0.324] [lr 0.005000] 2024-04-08 21:03:04,020 - Train: 28.83% [1397900/4848000] [288.3/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 21:03:37,128 - Train: 28.84% [1398000/4848000] [288.4/1000.0] [batch_t 0.335 (0.331)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 21:04:10,196 - Train: 28.84% [1398100/4848000] [288.4/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 21:04:43,267 - Train: 28.84% [1398200/4848000] [288.4/1000.0] [batch_t 0.333 (0.331)] [data_t 0.003] [optim_t 0.330] [lr 0.005000] 2024-04-08 21:05:16,380 - Train: 28.84% [1398300/4848000] [288.4/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 21:05:49,409 - Train: 28.84% [1398400/4848000] [288.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 21:06:22,492 - Train: 28.85% [1398500/4848000] [288.5/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 21:06:55,527 - Train: 28.85% [1398600/4848000] [288.5/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 21:07:28,681 - Train: 28.85% [1398700/4848000] [288.5/1000.0] [batch_t 0.336 (0.331)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 21:08:01,724 - Train: 28.85% [1398800/4848000] [288.5/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 21:08:34,803 - Train: 28.86% [1398900/4848000] [288.6/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 21:09:08,175 - Train: 28.86% [1399000/4848000] [288.6/1000.0] [batch_t 0.331 (0.334)] [data_t 0.002] [optim_t 0.329] [lr 0.005000] 2024-04-08 21:09:41,240 - Train: 28.86% [1399100/4848000] [288.6/1000.0] [batch_t 0.327 (0.331)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 21:10:14,295 - Train: 28.86% [1399200/4848000] [288.6/1000.0] [batch_t 0.327 (0.330)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 21:10:47,353 - Train: 28.86% [1399300/4848000] [288.6/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 21:11:20,409 - Train: 28.87% [1399400/4848000] [288.7/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 21:11:53,423 - Train: 28.87% [1399500/4848000] [288.7/1000.0] [batch_t 0.332 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 21:12:26,647 - Train: 28.87% [1399600/4848000] [288.7/1000.0] [batch_t 0.337 (0.332)] [data_t 0.003] [optim_t 0.335] [lr 0.005000] 2024-04-08 21:12:59,714 - Train: 28.87% [1399700/4848000] [288.7/1000.0] [batch_t 0.331 (0.331)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 21:13:32,785 - Train: 28.87% [1399800/4848000] [288.7/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.332] [lr 0.005000] 2024-04-08 21:14:05,872 - Train: 28.88% [1399900/4848000] [288.8/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 21:14:38,911 - Train: 28.88% [1400000/4848000] [288.8/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 21:15:12,121 - Train: 28.88% [1400100/4848000] [288.8/1000.0] [batch_t 0.328 (0.332)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 21:15:45,220 - Train: 28.88% [1400200/4848000] [288.8/1000.0] [batch_t 0.325 (0.331)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 21:16:18,305 - Train: 28.88% [1400300/4848000] [288.8/1000.0] [batch_t 0.328 (0.331)] [data_t 0.002] [optim_t 0.326] [lr 0.005000] 2024-04-08 21:16:51,326 - Train: 28.89% [1400400/4848000] [288.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 21:17:24,403 - Train: 28.89% [1400500/4848000] [288.9/1000.0] [batch_t 0.333 (0.331)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 21:17:57,444 - Train: 28.89% [1400600/4848000] [288.9/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 21:18:30,603 - Train: 28.89% [1400700/4848000] [288.9/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 21:19:03,686 - Train: 28.89% [1400800/4848000] [288.9/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 21:19:36,753 - Train: 28.90% [1400900/4848000] [289.0/1000.0] [batch_t 0.338 (0.331)] [data_t 0.003] [optim_t 0.335] [lr 0.005000] 2024-04-08 21:20:09,804 - Train: 28.90% [1401000/4848000] [289.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 21:20:33,618 - ==> Total time: 5 days, 21:54:32 Eta: 14 days, 13:07:31 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_coco3_nhcs57c1d4_20240402-232601' 2024-04-08 21:20:44,907 - Train: 28.90% [1401100/4848000] [289.0/1000.0] [batch_t 0.334 (0.332)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 21:21:18,001 - Train: 28.90% [1401200/4848000] [289.0/1000.0] [batch_t 0.327 (0.331)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 21:21:51,036 - Train: 28.90% [1401300/4848000] [289.0/1000.0] [batch_t 0.330 (0.330)] [data_t 0.002] [optim_t 0.328] [lr 0.005000] 2024-04-08 21:22:24,114 - Train: 28.91% [1401400/4848000] [289.1/1000.0] [batch_t 0.337 (0.331)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-08 21:22:57,298 - Train: 28.91% [1401500/4848000] [289.1/1000.0] [batch_t 0.331 (0.332)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 21:23:30,493 - Train: 28.91% [1401600/4848000] [289.1/1000.0] [batch_t 0.327 (0.332)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 21:24:03,643 - Train: 28.91% [1401700/4848000] [289.1/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 21:24:36,785 - Train: 28.92% [1401800/4848000] [289.2/1000.0] [batch_t 0.335 (0.331)] [data_t 0.003] [optim_t 0.333] [lr 0.005000] 2024-04-08 21:25:09,818 - Train: 28.92% [1401900/4848000] [289.2/1000.0] [batch_t 0.328 (0.330)] [data_t 0.002] [optim_t 0.325] [lr 0.005000] 2024-04-08 21:25:42,860 - Train: 28.92% [1402000/4848000] [289.2/1000.0] [batch_t 0.332 (0.330)] [data_t 0.002] [optim_t 0.330] [lr 0.005000] 2024-04-08 21:26:15,939 - Train: 28.92% [1402100/4848000] [289.2/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 21:26:49,030 - Train: 28.92% [1402200/4848000] [289.2/1000.0] [batch_t 0.331 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 21:27:22,151 - Train: 28.93% [1402300/4848000] [289.3/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 21:27:55,215 - Train: 28.93% [1402400/4848000] [289.3/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 21:28:28,309 - Train: 28.93% [1402500/4848000] [289.3/1000.0] [batch_t 0.328 (0.331)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 21:29:01,451 - Train: 28.93% [1402600/4848000] [289.3/1000.0] [batch_t 0.336 (0.331)] [data_t 0.003] [optim_t 0.334] [lr 0.005000] 2024-04-08 21:29:34,518 - Train: 28.93% [1402700/4848000] [289.3/1000.0] [batch_t 0.325 (0.331)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 21:30:07,636 - Train: 28.94% [1402800/4848000] [289.4/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 21:30:40,842 - Train: 28.94% [1402900/4848000] [289.4/1000.0] [batch_t 0.329 (0.332)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 21:31:13,985 - Train: 28.94% [1403000/4848000] [289.4/1000.0] [batch_t 0.334 (0.331)] [data_t 0.003] [optim_t 0.331] [lr 0.005000] 2024-04-08 21:31:47,073 - Train: 28.94% [1403100/4848000] [289.4/1000.0] [batch_t 0.332 (0.331)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 21:32:20,130 - Train: 28.94% [1403200/4848000] [289.4/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 21:32:53,155 - Train: 28.95% [1403300/4848000] [289.5/1000.0] [batch_t 0.331 (0.330)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 21:33:27,245 - Train: 28.95% [1403400/4848000] [289.5/1000.0] [batch_t 0.332 (0.341)] [data_t 0.003] [optim_t 0.329] [lr 0.005000] 2024-04-08 21:34:01,162 - Train: 28.95% [1403500/4848000] [289.5/1000.0] [batch_t 0.329 (0.339)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 21:34:34,202 - Train: 28.95% [1403600/4848000] [289.5/1000.0] [batch_t 0.326 (0.330)] [data_t 0.003] [optim_t 0.323] [lr 0.005000] 2024-04-08 21:35:07,263 - Train: 28.95% [1403700/4848000] [289.5/1000.0] [batch_t 0.327 (0.331)] [data_t 0.003] [optim_t 0.324] [lr 0.005000] 2024-04-08 21:35:40,272 - Train: 28.96% [1403800/4848000] [289.6/1000.0] [batch_t 0.329 (0.330)] [data_t 0.002] [optim_t 0.327] [lr 0.005000] 2024-04-08 21:36:13,314 - Train: 28.96% [1403900/4848000] [289.6/1000.0] [batch_t 0.328 (0.330)] [data_t 0.003] [optim_t 0.325] [lr 0.005000] 2024-04-08 21:36:46,475 - Train: 28.96% [1404000/4848000] [289.6/1000.0] [batch_t 0.329 (0.332)] [data_t 0.003] [optim_t 0.326] [lr 0.005000] 2024-04-08 21:37:19,599 - Train: 28.96% [1404100/4848000] [289.6/1000.0] [batch_t 0.330 (0.331)] [data_t 0.003] [optim_t 0.328] [lr 0.005000] 2024-04-08 21:37:52,664 - Train: 28.96% [1404200/4848000] [289.6/1000.0] [batch_t 0.329 (0.331)] [data_t 0.003] [optim_t 0.327] [lr 0.005000] 2024-04-08 21:38:25,875 - Train: 28.97% [1404300/4848000] [289.7/1000.0] [batch_t 0.329 (0.332)] [data_t 0.003] [optim_t 0.326] [lr 0.005000]