2024-04-02 01:00:05,660 - ==> Logging on master GPU: 0 2024-04-02 01:00:05,667 - ==> Running Trainer: MAMBAADTrainer 2024-04-02 01:00:05,667 - ==> Using GPU: [0] for Training 2024-04-02 01:00:05,668 - ==> Building model 2024-04-02 01:00:06,341 - Loaded from checkpoint 'model/pretrain/resnet34-43635321.pth' 2024-04-02 01:00:26,538 - ------------------------------------ MAMBAAD ------------------------------------ | module | #parameters or shape | #flops | |:------------------------------------------------------------|:-----------------------|:-------------| | model | 25.667M | 8.3G | | net_t | 8.17M | 3.955G | | net_t.conv1 | 9.408K | 0.154G | | net_t.conv1.weight | (64, 3, 7, 7) | | | net_t.bn1 | 0.128K | 2.097M | | net_t.bn1.weight | (64,) | | | net_t.bn1.bias | (64,) | | | net_t.layer1 | 0.222M | 0.909G | | net_t.layer1.0 | 73.984K | 0.303G | | net_t.layer1.0.conv1 | 36.864K | 0.151G | | net_t.layer1.0.conv1.weight | (64, 64, 3, 3) | | | net_t.layer1.0.bn1 | 0.128K | 0.524M | | net_t.layer1.0.bn1.weight | (64,) | | | net_t.layer1.0.bn1.bias | (64,) | | | net_t.layer1.0.conv2 | 36.864K | 0.151G | | net_t.layer1.0.conv2.weight | (64, 64, 3, 3) | | | net_t.layer1.0.bn2 | 0.128K | 0.524M | | net_t.layer1.0.bn2.weight | (64,) | | | net_t.layer1.0.bn2.bias | (64,) | | | net_t.layer1.1 | 73.984K | 0.303G | | net_t.layer1.1.conv1 | 36.864K | 0.151G | | net_t.layer1.1.conv1.weight | (64, 64, 3, 3) | | | net_t.layer1.1.bn1 | 0.128K | 0.524M | | net_t.layer1.1.bn1.weight | (64,) | | | net_t.layer1.1.bn1.bias | (64,) | | | net_t.layer1.1.conv2 | 36.864K | 0.151G | | net_t.layer1.1.conv2.weight | (64, 64, 3, 3) | | | net_t.layer1.1.bn2 | 0.128K | 0.524M | | net_t.layer1.1.bn2.weight | (64,) | | | net_t.layer1.1.bn2.bias | (64,) | | | net_t.layer1.2 | 73.984K | 0.303G | | net_t.layer1.2.conv1 | 36.864K | 0.151G | | net_t.layer1.2.conv1.weight | (64, 64, 3, 3) | | | net_t.layer1.2.bn1 | 0.128K | 0.524M | | net_t.layer1.2.bn1.weight | (64,) | | | net_t.layer1.2.bn1.bias | (64,) | | | net_t.layer1.2.conv2 | 36.864K | 0.151G | | net_t.layer1.2.conv2.weight | (64, 64, 3, 3) | | | net_t.layer1.2.bn2 | 0.128K | 0.524M | | net_t.layer1.2.bn2.weight | (64,) | | | net_t.layer1.2.bn2.bias | (64,) | | | net_t.layer2 | 1.116M | 1.143G | | net_t.layer2.0 | 0.23M | 0.236G | | net_t.layer2.0.conv1 | 73.728K | 75.497M | | net_t.layer2.0.conv1.weight | (128, 64, 3, 3) | | | net_t.layer2.0.bn1 | 0.256K | 0.262M | | net_t.layer2.0.bn1.weight | (128,) | | | net_t.layer2.0.bn1.bias | (128,) | | | net_t.layer2.0.conv2 | 0.147M | 0.151G | | net_t.layer2.0.conv2.weight | (128, 128, 3, 3) | | | net_t.layer2.0.bn2 | 0.256K | 0.262M | | net_t.layer2.0.bn2.weight | (128,) | | | net_t.layer2.0.bn2.bias | (128,) | | | net_t.layer2.0.downsample | 8.448K | 8.651M | | net_t.layer2.0.downsample.0 | 8.192K | 8.389M | | net_t.layer2.0.downsample.1 | 0.256K | 0.262M | | net_t.layer2.1 | 0.295M | 0.303G | | net_t.layer2.1.conv1 | 0.147M | 0.151G | | net_t.layer2.1.conv1.weight | (128, 128, 3, 3) | | | net_t.layer2.1.bn1 | 0.256K | 0.262M | | net_t.layer2.1.bn1.weight | (128,) | | | net_t.layer2.1.bn1.bias | (128,) | | | net_t.layer2.1.conv2 | 0.147M | 0.151G | | net_t.layer2.1.conv2.weight | (128, 128, 3, 3) | | | net_t.layer2.1.bn2 | 0.256K | 0.262M | | net_t.layer2.1.bn2.weight | (128,) | | | net_t.layer2.1.bn2.bias | (128,) | | | net_t.layer2.2 | 0.295M | 0.303G | | net_t.layer2.2.conv1 | 0.147M | 0.151G | | net_t.layer2.2.conv1.weight | (128, 128, 3, 3) | | | net_t.layer2.2.bn1 | 0.256K | 0.262M | | net_t.layer2.2.bn1.weight | (128,) | | | net_t.layer2.2.bn1.bias | (128,) | | | net_t.layer2.2.conv2 | 0.147M | 0.151G | | net_t.layer2.2.conv2.weight | (128, 128, 3, 3) | | | net_t.layer2.2.bn2 | 0.256K | 0.262M | | net_t.layer2.2.bn2.weight | (128,) | | | net_t.layer2.2.bn2.bias | (128,) | | | net_t.layer2.3 | 0.295M | 0.303G | | net_t.layer2.3.conv1 | 0.147M | 0.151G | | net_t.layer2.3.conv1.weight | (128, 128, 3, 3) | | | net_t.layer2.3.bn1 | 0.256K | 0.262M | | net_t.layer2.3.bn1.weight | (128,) | | | net_t.layer2.3.bn1.bias | (128,) | | | net_t.layer2.3.conv2 | 0.147M | 0.151G | | net_t.layer2.3.conv2.weight | (128, 128, 3, 3) | | | net_t.layer2.3.bn2 | 0.256K | 0.262M | | net_t.layer2.3.bn2.weight | (128,) | | | net_t.layer2.3.bn2.bias | (128,) | | | net_t.layer3 | 6.822M | 1.747G | | net_t.layer3.0 | 0.919M | 0.235G | | net_t.layer3.0.conv1 | 0.295M | 75.497M | | net_t.layer3.0.conv1.weight | (256, 128, 3, 3) | | | net_t.layer3.0.bn1 | 0.512K | 0.131M | | net_t.layer3.0.bn1.weight | (256,) | | | net_t.layer3.0.bn1.bias | (256,) | | | net_t.layer3.0.conv2 | 0.59M | 0.151G | | net_t.layer3.0.conv2.weight | (256, 256, 3, 3) | | | net_t.layer3.0.bn2 | 0.512K | 0.131M | | net_t.layer3.0.bn2.weight | (256,) | | | net_t.layer3.0.bn2.bias | (256,) | | | net_t.layer3.0.downsample | 33.28K | 8.52M | | net_t.layer3.0.downsample.0 | 32.768K | 8.389M | | net_t.layer3.0.downsample.1 | 0.512K | 0.131M | | net_t.layer3.1 | 1.181M | 0.302G | | net_t.layer3.1.conv1 | 0.59M | 0.151G | | net_t.layer3.1.conv1.weight | (256, 256, 3, 3) | | | net_t.layer3.1.bn1 | 0.512K | 0.131M | | net_t.layer3.1.bn1.weight | (256,) | | | net_t.layer3.1.bn1.bias | (256,) | | | net_t.layer3.1.conv2 | 0.59M | 0.151G | | net_t.layer3.1.conv2.weight | (256, 256, 3, 3) | | | net_t.layer3.1.bn2 | 0.512K | 0.131M | | net_t.layer3.1.bn2.weight | (256,) | | | net_t.layer3.1.bn2.bias | (256,) | | | net_t.layer3.2 | 1.181M | 0.302G | | net_t.layer3.2.conv1 | 0.59M | 0.151G | | net_t.layer3.2.conv1.weight | (256, 256, 3, 3) | | | net_t.layer3.2.bn1 | 0.512K | 0.131M | | net_t.layer3.2.bn1.weight | (256,) | | | net_t.layer3.2.bn1.bias | (256,) | | | net_t.layer3.2.conv2 | 0.59M | 0.151G | | net_t.layer3.2.conv2.weight | (256, 256, 3, 3) | | | net_t.layer3.2.bn2 | 0.512K | 0.131M | | net_t.layer3.2.bn2.weight | (256,) | | | net_t.layer3.2.bn2.bias | (256,) | | | net_t.layer3.3 | 1.181M | 0.302G | | net_t.layer3.3.conv1 | 0.59M | 0.151G | | net_t.layer3.3.conv1.weight | (256, 256, 3, 3) | | | net_t.layer3.3.bn1 | 0.512K | 0.131M | | net_t.layer3.3.bn1.weight | (256,) | | | net_t.layer3.3.bn1.bias | (256,) | | | net_t.layer3.3.conv2 | 0.59M | 0.151G | | net_t.layer3.3.conv2.weight | (256, 256, 3, 3) | | | net_t.layer3.3.bn2 | 0.512K | 0.131M | | net_t.layer3.3.bn2.weight | (256,) | | | net_t.layer3.3.bn2.bias | (256,) | | | net_t.layer3.4 | 1.181M | 0.302G | | net_t.layer3.4.conv1 | 0.59M | 0.151G | | net_t.layer3.4.conv1.weight | (256, 256, 3, 3) | | | net_t.layer3.4.bn1 | 0.512K | 0.131M | | net_t.layer3.4.bn1.weight | (256,) | | | net_t.layer3.4.bn1.bias | (256,) | | | net_t.layer3.4.conv2 | 0.59M | 0.151G | | net_t.layer3.4.conv2.weight | (256, 256, 3, 3) | | | net_t.layer3.4.bn2 | 0.512K | 0.131M | | net_t.layer3.4.bn2.weight | (256,) | | | net_t.layer3.4.bn2.bias | (256,) | | | net_t.layer3.5 | 1.181M | 0.302G | | net_t.layer3.5.conv1 | 0.59M | 0.151G | | net_t.layer3.5.conv1.weight | (256, 256, 3, 3) | | | net_t.layer3.5.bn1 | 0.512K | 0.131M | | net_t.layer3.5.bn1.weight | (256,) | | | net_t.layer3.5.bn1.bias | (256,) | | | net_t.layer3.5.conv2 | 0.59M | 0.151G | | net_t.layer3.5.conv2.weight | (256, 256, 3, 3) | | | net_t.layer3.5.bn2 | 0.512K | 0.131M | | net_t.layer3.5.bn2.weight | (256,) | | | net_t.layer3.5.bn2.bias | (256,) | | | mff_oce | 1.458M | 0.269G | | mff_oce.bn_layer | 0.94M | 66.47M | | mff_oce.bn_layer.0 | 0.379M | 30.622M | | mff_oce.bn_layer.0.conv1 | 32.768K | 8.389M | | mff_oce.bn_layer.0.conv1.weight | (128, 256, 1, 1) | | | mff_oce.bn_layer.0.bn1 | 0.256K | 65.536K | | mff_oce.bn_layer.0.bn1.weight | (128,) | | | mff_oce.bn_layer.0.bn1.bias | (128,) | | | mff_oce.bn_layer.0.conv2 | 0.147M | 9.437M | | mff_oce.bn_layer.0.conv2.weight | (128, 128, 3, 3) | | | mff_oce.bn_layer.0.bn2 | 0.256K | 16.384K | | mff_oce.bn_layer.0.bn2.weight | (128,) | | | mff_oce.bn_layer.0.bn2.bias | (128,) | | | mff_oce.bn_layer.0.conv3 | 65.536K | 4.194M | | mff_oce.bn_layer.0.conv3.weight | (512, 128, 1, 1) | | | mff_oce.bn_layer.0.bn3 | 1.024K | 65.536K | | mff_oce.bn_layer.0.bn3.weight | (512,) | | | mff_oce.bn_layer.0.bn3.bias | (512,) | | | mff_oce.bn_layer.0.downsample | 0.132M | 8.454M | | mff_oce.bn_layer.0.downsample.0 | 0.131M | 8.389M | | mff_oce.bn_layer.0.downsample.1 | 1.024K | 65.536K | | mff_oce.bn_layer.1 | 0.28M | 17.924M | | mff_oce.bn_layer.1.conv1 | 65.536K | 4.194M | | mff_oce.bn_layer.1.conv1.weight | (128, 512, 1, 1) | | | mff_oce.bn_layer.1.bn1 | 0.256K | 16.384K | | mff_oce.bn_layer.1.bn1.weight | (128,) | | | mff_oce.bn_layer.1.bn1.bias | (128,) | | | mff_oce.bn_layer.1.conv2 | 0.147M | 9.437M | | mff_oce.bn_layer.1.conv2.weight | (128, 128, 3, 3) | | | mff_oce.bn_layer.1.bn2 | 0.256K | 16.384K | | mff_oce.bn_layer.1.bn2.weight | (128,) | | | mff_oce.bn_layer.1.bn2.bias | (128,) | | | mff_oce.bn_layer.1.conv3 | 65.536K | 4.194M | | mff_oce.bn_layer.1.conv3.weight | (512, 128, 1, 1) | | | mff_oce.bn_layer.1.bn3 | 1.024K | 65.536K | | mff_oce.bn_layer.1.bn3.weight | (512,) | | | mff_oce.bn_layer.1.bn3.bias | (512,) | | | mff_oce.bn_layer.2 | 0.28M | 17.924M | | mff_oce.bn_layer.2.conv1 | 65.536K | 4.194M | | mff_oce.bn_layer.2.conv1.weight | (128, 512, 1, 1) | | | mff_oce.bn_layer.2.bn1 | 0.256K | 16.384K | | mff_oce.bn_layer.2.bn1.weight | (128,) | | | mff_oce.bn_layer.2.bn1.bias | (128,) | | | mff_oce.bn_layer.2.conv2 | 0.147M | 9.437M | | mff_oce.bn_layer.2.conv2.weight | (128, 128, 3, 3) | | | mff_oce.bn_layer.2.bn2 | 0.256K | 16.384K | | mff_oce.bn_layer.2.bn2.weight | (128,) | | | mff_oce.bn_layer.2.bn2.bias | (128,) | | | mff_oce.bn_layer.2.conv3 | 65.536K | 4.194M | | mff_oce.bn_layer.2.conv3.weight | (512, 128, 1, 1) | | | mff_oce.bn_layer.2.bn3 | 1.024K | 65.536K | | mff_oce.bn_layer.2.bn3.weight | (512,) | | | mff_oce.bn_layer.2.bn3.bias | (512,) | | | mff_oce.conv1 | 73.728K | 75.497M | | mff_oce.conv1.weight | (128, 64, 3, 3) | | | mff_oce.bn1 | 0.256K | 0.262M | | mff_oce.bn1.weight | (128,) | | | mff_oce.bn1.bias | (128,) | | | mff_oce.conv2 | 0.295M | 75.497M | | mff_oce.conv2.weight | (256, 128, 3, 3) | | | mff_oce.bn2 | 0.512K | 0.131M | | mff_oce.bn2.weight | (256,) | | | mff_oce.bn2.bias | (256,) | | | mff_oce.conv21 | 16.512K | 16.777M | | mff_oce.conv21.weight | (128, 128, 1, 1) | | | mff_oce.conv21.bias | (128,) | | | mff_oce.bn21 | 0.256K | 0.262M | | mff_oce.bn21.weight | (128,) | | | mff_oce.bn21.bias | (128,) | | | mff_oce.conv31 | 65.792K | 16.777M | | mff_oce.conv31.weight | (256, 256, 1, 1) | | | mff_oce.conv31.bias | (256,) | | | mff_oce.bn31 | 0.512K | 0.131M | | mff_oce.bn31.weight | (256,) | | | mff_oce.bn31.bias | (256,) | | | mff_oce.convf | 65.792K | 16.777M | | mff_oce.convf.weight | (256, 256, 1, 1) | | | mff_oce.convf.bias | (256,) | | | mff_oce.bnf | 0.512K | 0.131M | | mff_oce.bnf.weight | (256,) | | | mff_oce.bnf.bias | (256,) | | | net_s.layers_up | 16.038M | 4.076G | | net_s.layers_up.0.blocks.0 | 9.441M | 0.577G | | net_s.layers_up.0.blocks.0.smm_blocks | 7.561M | 0.456G | | net_s.layers_up.0.blocks.0.smm_blocks.0 | 2.52M | 0.152G | | net_s.layers_up.0.blocks.0.smm_blocks.0.ln_1 | 1.024K | 0.164M | | net_s.layers_up.0.blocks.0.smm_blocks.0.self_attention | 2.519M | 0.152G | | net_s.layers_up.0.blocks.0.smm_blocks.1 | 2.52M | 0.152G | | net_s.layers_up.0.blocks.0.smm_blocks.1.ln_1 | 1.024K | 0.164M | | net_s.layers_up.0.blocks.0.smm_blocks.1.self_attention | 2.519M | 0.152G | | net_s.layers_up.0.blocks.0.smm_blocks.2 | 2.52M | 0.152G | | net_s.layers_up.0.blocks.0.smm_blocks.2.ln_1 | 1.024K | 0.164M | | net_s.layers_up.0.blocks.0.smm_blocks.2.self_attention | 2.519M | 0.152G | | net_s.layers_up.0.blocks.0.conv1b3 | 0.263M | 16.908M | | net_s.layers_up.0.blocks.0.conv1b3.0 | 0.263M | 16.777M | | net_s.layers_up.0.blocks.0.conv1b3.0.weight | (512, 512, 1, 1) | | | net_s.layers_up.0.blocks.0.conv1b3.0.bias | (512,) | | | net_s.layers_up.0.blocks.0.conv1b3.1 | | 0.131M | | net_s.layers_up.0.blocks.0.conv1a3 | 0.263M | 16.908M | | net_s.layers_up.0.blocks.0.conv1a3.0 | 0.263M | 16.777M | | net_s.layers_up.0.blocks.0.conv1a3.0.weight | (512, 512, 1, 1) | | | net_s.layers_up.0.blocks.0.conv1a3.0.bias | (512,) | | | net_s.layers_up.0.blocks.0.conv1a3.1 | | 0.131M | | net_s.layers_up.0.blocks.0.conv1b5 | 0.263M | 16.908M | | net_s.layers_up.0.blocks.0.conv1b5.0 | 0.263M | 16.777M | | net_s.layers_up.0.blocks.0.conv1b5.0.weight | (512, 512, 1, 1) | | | net_s.layers_up.0.blocks.0.conv1b5.0.bias | (512,) | | | net_s.layers_up.0.blocks.0.conv1b5.1 | | 0.131M | | net_s.layers_up.0.blocks.0.conv1a5 | 0.263M | 16.908M | | net_s.layers_up.0.blocks.0.conv1a5.0 | 0.263M | 16.777M | | net_s.layers_up.0.blocks.0.conv1a5.0.weight | (512, 512, 1, 1) | | | net_s.layers_up.0.blocks.0.conv1a5.0.bias | (512,) | | | net_s.layers_up.0.blocks.0.conv1a5.1 | | 0.131M | | net_s.layers_up.0.blocks.0.conv33.0 | 4.608K | | | net_s.layers_up.0.blocks.0.conv33.0.weight | (512, 1, 3, 3) | | | net_s.layers_up.0.blocks.0.conv55 | 12.8K | 0.95M | | net_s.layers_up.0.blocks.0.conv55.0 | 12.8K | 0.819M | | net_s.layers_up.0.blocks.0.conv55.0.weight | (512, 1, 5, 5) | | | net_s.layers_up.0.blocks.0.conv55.1 | | 0.131M | | net_s.layers_up.0.blocks.0.conv77 | 25.088K | 1.737M | | net_s.layers_up.0.blocks.0.conv77.0 | 25.088K | 1.606M | | net_s.layers_up.0.blocks.0.conv77.0.weight | (512, 1, 7, 7) | | | net_s.layers_up.0.blocks.0.conv77.1 | | 0.131M | | net_s.layers_up.0.blocks.0.finalconv11 | 0.787M | 50.332M | | net_s.layers_up.0.blocks.0.finalconv11.weight | (512, 1536, 1, 1) | | | net_s.layers_up.0.blocks.0.finalconv11.bias | (512,) | | | net_s.layers_up.1 | 4.432M | 0.961G | | net_s.layers_up.1.blocks | 3.908M | 0.928G | | net_s.layers_up.1.blocks.0 | 1.954M | 0.464G | | net_s.layers_up.1.blocks.0.smm_blocks | 1.473M | 0.34G | | net_s.layers_up.1.blocks.0.conv1b3 | 65.792K | 17.039M | | net_s.layers_up.1.blocks.0.conv1a3 | 65.792K | 17.039M | | net_s.layers_up.1.blocks.0.conv1b5 | 65.792K | 17.039M | | net_s.layers_up.1.blocks.0.conv1a5 | 65.792K | 17.039M | | net_s.layers_up.1.blocks.0.conv33.0 | 2.304K | | | net_s.layers_up.1.blocks.0.conv55 | 6.4K | 1.901M | | net_s.layers_up.1.blocks.0.conv77 | 12.544K | 3.473M | | net_s.layers_up.1.blocks.0.finalconv11 | 0.197M | 50.332M | | net_s.layers_up.1.blocks.1 | 1.954M | 0.464G | | net_s.layers_up.1.blocks.1.smm_blocks | 1.473M | 0.34G | | net_s.layers_up.1.blocks.1.conv1b3 | 65.792K | 17.039M | | net_s.layers_up.1.blocks.1.conv1a3 | 65.792K | 17.039M | | net_s.layers_up.1.blocks.1.conv1b5 | 65.792K | 17.039M | | net_s.layers_up.1.blocks.1.conv1a5 | 65.792K | 17.039M | | net_s.layers_up.1.blocks.1.conv33.0 | 2.304K | | | net_s.layers_up.1.blocks.1.conv55 | 6.4K | 1.901M | | net_s.layers_up.1.blocks.1.conv77 | 12.544K | 3.473M | | net_s.layers_up.1.blocks.1.finalconv11 | 0.197M | 50.332M | | net_s.layers_up.1.upsample | 0.525M | 33.882M | | net_s.layers_up.1.upsample.expand | 0.524M | 33.554M | | net_s.layers_up.1.upsample.expand.weight | (1024, 512) | | | net_s.layers_up.1.upsample.norm | 0.512K | 0.328M | | net_s.layers_up.1.upsample.norm.weight | (256,) | | | net_s.layers_up.1.upsample.norm.bias | (256,) | | | net_s.layers_up.2 | 1.816M | 1.529G | | net_s.layers_up.2.blocks | 1.685M | 1.495G | | net_s.layers_up.2.blocks.0 | 0.842M | 0.747G | | net_s.layers_up.2.blocks.0.smm_blocks | 0.717M | 0.617G | | net_s.layers_up.2.blocks.0.conv1b3 | 16.512K | 17.302M | | net_s.layers_up.2.blocks.0.conv1a3 | 16.512K | 17.302M | | net_s.layers_up.2.blocks.0.conv1b5 | 16.512K | 17.302M | | net_s.layers_up.2.blocks.0.conv1a5 | 16.512K | 17.302M | | net_s.layers_up.2.blocks.0.conv33.0 | 1.152K | | | net_s.layers_up.2.blocks.0.conv55 | 3.2K | 3.801M | | net_s.layers_up.2.blocks.0.conv77 | 6.272K | 6.947M | | net_s.layers_up.2.blocks.0.finalconv11 | 49.28K | 50.332M | | net_s.layers_up.2.blocks.1 | 0.842M | 0.747G | | net_s.layers_up.2.blocks.1.smm_blocks | 0.717M | 0.617G | | net_s.layers_up.2.blocks.1.conv1b3 | 16.512K | 17.302M | | net_s.layers_up.2.blocks.1.conv1a3 | 16.512K | 17.302M | | net_s.layers_up.2.blocks.1.conv1b5 | 16.512K | 17.302M | | net_s.layers_up.2.blocks.1.conv1a5 | 16.512K | 17.302M | | net_s.layers_up.2.blocks.1.conv33.0 | 1.152K | | | net_s.layers_up.2.blocks.1.conv55 | 3.2K | 3.801M | | net_s.layers_up.2.blocks.1.conv77 | 6.272K | 6.947M | | net_s.layers_up.2.blocks.1.finalconv11 | 49.28K | 50.332M | | net_s.layers_up.2.upsample | 0.131M | 34.21M | | net_s.layers_up.2.upsample.expand | 0.131M | 33.554M | | net_s.layers_up.2.upsample.expand.weight | (512, 256) | | | net_s.layers_up.2.upsample.norm | 0.256K | 0.655M | | net_s.layers_up.2.upsample.norm.weight | (128,) | | | net_s.layers_up.2.upsample.norm.bias | (128,) | | | net_s.layers_up.3 | 0.349M | 1.009G | | net_s.layers_up.3.blocks.0 | 0.316M | 0.974G | | net_s.layers_up.3.blocks.0.smm_blocks | 0.281M | 0.831G | | net_s.layers_up.3.blocks.0.smm_blocks.0 | 93.824K | 0.277G | | net_s.layers_up.3.blocks.0.smm_blocks.1 | 93.824K | 0.277G | | net_s.layers_up.3.blocks.0.smm_blocks.2 | 93.824K | 0.277G | | net_s.layers_up.3.blocks.0.conv1b3 | 4.16K | 17.826M | | net_s.layers_up.3.blocks.0.conv1b3.0 | 4.16K | 16.777M | | net_s.layers_up.3.blocks.0.conv1b3.1 | | 1.049M | | net_s.layers_up.3.blocks.0.conv1a3 | 4.16K | 17.826M | | net_s.layers_up.3.blocks.0.conv1a3.0 | 4.16K | 16.777M | | net_s.layers_up.3.blocks.0.conv1a3.1 | | 1.049M | | net_s.layers_up.3.blocks.0.conv1b5 | 4.16K | 17.826M | | net_s.layers_up.3.blocks.0.conv1b5.0 | 4.16K | 16.777M | | net_s.layers_up.3.blocks.0.conv1b5.1 | | 1.049M | | net_s.layers_up.3.blocks.0.conv1a5 | 4.16K | 17.826M | | net_s.layers_up.3.blocks.0.conv1a5.0 | 4.16K | 16.777M | | net_s.layers_up.3.blocks.0.conv1a5.1 | | 1.049M | | net_s.layers_up.3.blocks.0.conv33.0 | 0.576K | | | net_s.layers_up.3.blocks.0.conv33.0.weight | (64, 1, 3, 3) | | | net_s.layers_up.3.blocks.0.conv55 | 1.6K | 7.602M | | net_s.layers_up.3.blocks.0.conv55.0 | 1.6K | 6.554M | | net_s.layers_up.3.blocks.0.conv55.1 | | 1.049M | | net_s.layers_up.3.blocks.0.conv77 | 3.136K | 13.894M | | net_s.layers_up.3.blocks.0.conv77.0 | 3.136K | 12.845M | | net_s.layers_up.3.blocks.0.conv77.1 | | 1.049M | | net_s.layers_up.3.blocks.0.finalconv11 | 12.352K | 50.332M | | net_s.layers_up.3.blocks.0.finalconv11.weight | (64, 192, 1, 1) | | | net_s.layers_up.3.blocks.0.finalconv11.bias | (64,) | | | net_s.layers_up.3.upsample | 32.896K | 34.865M | | net_s.layers_up.3.upsample.expand | 32.768K | 33.554M | | net_s.layers_up.3.upsample.expand.weight | (256, 128) | | | net_s.layers_up.3.upsample.norm | 0.128K | 1.311M | | net_s.layers_up.3.upsample.norm.weight | (64,) | | | net_s.layers_up.3.upsample.norm.bias | (64,) | | --------------------------------------------------------------------------------- 2024-04-02 01:00:26,542 - ==> Creating optimizer 2024-04-02 01:00:26,550 - ==> Loading dataset: DefaultAD 2024-04-02 01:00:26,613 - ==> ********** cfg ********** epoch_full : 1000 fvcore_b : 1 fvcore_c : 3 metrics : ['mAUROC_sp_max', 'mAP_sp_max', 'mF1_max_sp_max', 'mAUROC_px', 'mAP_px', 'mF1_max_px', 'mAUPRO_px', 'mF1_px_0.2_0.8_0.1', 'mAcc_px_0.2_0.8_0.1', 'mIoU_px_0.2_0.8_0.1', 'mIoU_max_px'] evaluator.kwargs : {'metrics': ['mAUROC_sp_max', 'mAP_sp_max', 'mF1_max_sp_max', 'mAUROC_px', 'mAP_px', 'mF1_max_px', 'mAUPRO_px', 'mF1_px_0.2_0.8_0.1', 'mAcc_px_0.2_0.8_0.1', 'mIoU_px_0.2_0.8_0.1', 'mIoU_max_px'], 'pooling_ks': None, 'max_step_aupro': 100} optim.lr : 0.005 optim.kwargs : {'name': 'adamw', 'betas': (0.9, 0.999), 'eps': 1e-08, 'weight_decay': 0.0001, 'amsgrad': False} trainer.name : MAMBAADTrainer trainer.checkpoint : runs trainer.logdir_sub : trainer.resume_dir : trainer.cuda_deterministic : False trainer.epoch_full : 1000 trainer.scheduler_kwargs : {'name': 'step', 'lr_noise': None, 'noise_pct': 0.67, 'noise_std': 1.0, 'noise_seed': 42, 'lr_min': 5e-05, 'warmup_lr': 5e-06, 'warmup_iters': -1, 'cooldown_iters': 0, 'warmup_epochs': 0, 'cooldown_epochs': 0, 'use_iters': True, 'patience_iters': 0, 'patience_epochs': 0, 'decay_iters': 0, 'decay_epochs': 800, 'cycle_decay': 0.1, 'decay_rate': 0.1} trainer.mixup_kwargs : {'mixup_alpha': 0.8, 'cutmix_alpha': 1.0, 'cutmix_minmax': None, 'prob': 0.0, 'switch_prob': 0.5, 'mode': 'batch', 'correct_lam': True, 'label_smoothing': 0.1} trainer.test_start_epoch : 1000 trainer.test_per_epoch : 50 trainer.find_unused_parameters : False trainer.sync_BN : apex trainer.dist_BN : trainer.scaler : none trainer.data.batch_size : 16 trainer.data.batch_size_per_gpu : 16 trainer.data.batch_size_test : 16 trainer.data.batch_size_per_gpu_test : 16 trainer.data.num_workers_per_gpu : 4 trainer.data.drop_last : True trainer.data.pin_memory : True trainer.data.persistent_workers : False trainer.data.num_workers : 4 trainer.iter : 0 trainer.epoch : 0 trainer.iter_full : 541000 trainer.metric_recorder : {'mAUROC_sp_max_candle': [], 'mAP_sp_max_candle': [], 'mF1_max_sp_max_candle': [], 'mAUROC_px_candle': [], 'mAP_px_candle': [], 'mF1_max_px_candle': [], 'mAUPRO_px_candle': [], 'mF1_px_0.2_0.8_0.1_candle': [], 'mAcc_px_0.2_0.8_0.1_candle': [], 'mIoU_px_0.2_0.8_0.1_candle': [], 'mIoU_max_px_candle': [], 'mAUROC_sp_max_capsules': [], 'mAP_sp_max_capsules': [], 'mF1_max_sp_max_capsules': [], 'mAUROC_px_capsules': [], 'mAP_px_capsules': [], 'mF1_max_px_capsules': [], 'mAUPRO_px_capsules': [], 'mF1_px_0.2_0.8_0.1_capsules': [], 'mAcc_px_0.2_0.8_0.1_capsules': [], 'mIoU_px_0.2_0.8_0.1_capsules': [], 'mIoU_max_px_capsules': [], 'mAUROC_sp_max_cashew': [], 'mAP_sp_max_cashew': [], 'mF1_max_sp_max_cashew': [], 'mAUROC_px_cashew': [], 'mAP_px_cashew': [], 'mF1_max_px_cashew': [], 'mAUPRO_px_cashew': [], 'mF1_px_0.2_0.8_0.1_cashew': [], 'mAcc_px_0.2_0.8_0.1_cashew': [], 'mIoU_px_0.2_0.8_0.1_cashew': [], 'mIoU_max_px_cashew': [], 'mAUROC_sp_max_chewinggum': [], 'mAP_sp_max_chewinggum': [], 'mF1_max_sp_max_chewinggum': [], 'mAUROC_px_chewinggum': [], 'mAP_px_chewinggum': [], 'mF1_max_px_chewinggum': [], 'mAUPRO_px_chewinggum': [], 'mF1_px_0.2_0.8_0.1_chewinggum': [], 'mAcc_px_0.2_0.8_0.1_chewinggum': [], 'mIoU_px_0.2_0.8_0.1_chewinggum': [], 'mIoU_max_px_chewinggum': [], 'mAUROC_sp_max_fryum': [], 'mAP_sp_max_fryum': [], 'mF1_max_sp_max_fryum': [], 'mAUROC_px_fryum': [], 'mAP_px_fryum': [], 'mF1_max_px_fryum': [], 'mAUPRO_px_fryum': [], 'mF1_px_0.2_0.8_0.1_fryum': [], 'mAcc_px_0.2_0.8_0.1_fryum': [], 'mIoU_px_0.2_0.8_0.1_fryum': [], 'mIoU_max_px_fryum': [], 'mAUROC_sp_max_macaroni1': [], 'mAP_sp_max_macaroni1': [], 'mF1_max_sp_max_macaroni1': [], 'mAUROC_px_macaroni1': [], 'mAP_px_macaroni1': [], 'mF1_max_px_macaroni1': [], 'mAUPRO_px_macaroni1': [], 'mF1_px_0.2_0.8_0.1_macaroni1': [], 'mAcc_px_0.2_0.8_0.1_macaroni1': [], 'mIoU_px_0.2_0.8_0.1_macaroni1': [], 'mIoU_max_px_macaroni1': [], 'mAUROC_sp_max_macaroni2': [], 'mAP_sp_max_macaroni2': [], 'mF1_max_sp_max_macaroni2': [], 'mAUROC_px_macaroni2': [], 'mAP_px_macaroni2': [], 'mF1_max_px_macaroni2': [], 'mAUPRO_px_macaroni2': [], 'mF1_px_0.2_0.8_0.1_macaroni2': [], 'mAcc_px_0.2_0.8_0.1_macaroni2': [], 'mIoU_px_0.2_0.8_0.1_macaroni2': [], 'mIoU_max_px_macaroni2': [], 'mAUROC_sp_max_pcb1': [], 'mAP_sp_max_pcb1': [], 'mF1_max_sp_max_pcb1': [], 'mAUROC_px_pcb1': [], 'mAP_px_pcb1': [], 'mF1_max_px_pcb1': [], 'mAUPRO_px_pcb1': [], 'mF1_px_0.2_0.8_0.1_pcb1': [], 'mAcc_px_0.2_0.8_0.1_pcb1': [], 'mIoU_px_0.2_0.8_0.1_pcb1': [], 'mIoU_max_px_pcb1': [], 'mAUROC_sp_max_pcb2': [], 'mAP_sp_max_pcb2': [], 'mF1_max_sp_max_pcb2': [], 'mAUROC_px_pcb2': [], 'mAP_px_pcb2': [], 'mF1_max_px_pcb2': [], 'mAUPRO_px_pcb2': [], 'mF1_px_0.2_0.8_0.1_pcb2': [], 'mAcc_px_0.2_0.8_0.1_pcb2': [], 'mIoU_px_0.2_0.8_0.1_pcb2': [], 'mIoU_max_px_pcb2': [], 'mAUROC_sp_max_pcb3': [], 'mAP_sp_max_pcb3': [], 'mF1_max_sp_max_pcb3': [], 'mAUROC_px_pcb3': [], 'mAP_px_pcb3': [], 'mF1_max_px_pcb3': [], 'mAUPRO_px_pcb3': [], 'mF1_px_0.2_0.8_0.1_pcb3': [], 'mAcc_px_0.2_0.8_0.1_pcb3': [], 'mIoU_px_0.2_0.8_0.1_pcb3': [], 'mIoU_max_px_pcb3': [], 'mAUROC_sp_max_pcb4': [], 'mAP_sp_max_pcb4': [], 'mF1_max_sp_max_pcb4': [], 'mAUROC_px_pcb4': [], 'mAP_px_pcb4': [], 'mF1_max_px_pcb4': [], 'mAUPRO_px_pcb4': [], 'mF1_px_0.2_0.8_0.1_pcb4': [], 'mAcc_px_0.2_0.8_0.1_pcb4': [], 'mIoU_px_0.2_0.8_0.1_pcb4': [], 'mIoU_max_px_pcb4': [], 'mAUROC_sp_max_pipe_fryum': [], 'mAUROC_sp_max_Avg': [], 'mAP_sp_max_pipe_fryum': [], 'mAP_sp_max_Avg': [], 'mF1_max_sp_max_pipe_fryum': [], 'mF1_max_sp_max_Avg': [], 'mAUROC_px_pipe_fryum': [], 'mAUROC_px_Avg': [], 'mAP_px_pipe_fryum': [], 'mAP_px_Avg': [], 'mF1_max_px_pipe_fryum': [], 'mF1_max_px_Avg': [], 'mAUPRO_px_pipe_fryum': [], 'mAUPRO_px_Avg': [], 'mF1_px_0.2_0.8_0.1_pipe_fryum': [], 'mF1_px_0.2_0.8_0.1_Avg': [], 'mAcc_px_0.2_0.8_0.1_pipe_fryum': [], 'mAcc_px_0.2_0.8_0.1_Avg': [], 'mIoU_px_0.2_0.8_0.1_pipe_fryum': [], 'mIoU_px_0.2_0.8_0.1_Avg': [], 'mIoU_max_px_pipe_fryum': [], 'mIoU_max_px_Avg': []} loss.loss_terms : [{'type': 'L2Loss', 'name': 'pixel', 'lam': 5.0}] loss.clip_grad : 5.0 loss.create_graph : False loss.retain_graph : False adv : False logging.log_terms_train : [{'name': 'batch_t', 'fmt': ':>5.3f', 'add_name': 'avg'}, {'name': 'data_t', 'fmt': ':>5.3f'}, {'name': 'optim_t', 'fmt': ':>5.3f'}, {'name': 'lr', 'fmt': ':>7.6f'}, {'name': 'cos', 'suffixes': [''], 'fmt': ':>5.3f', 'add_name': 'avg'}] logging.log_terms_test : [{'name': 'batch_t', 'fmt': ':>5.3f', 'add_name': 'avg'}, {'name': 'cos', 'suffixes': [''], 'fmt': ':>5.3f', 'add_name': 'avg'}] logging.train_reset_log_per : 100 logging.train_log_per : 100 logging.test_log_per : 50 data.sampler : naive data.loader_type : pil data.loader_type_target : pil_L data.type : DefaultAD data.root : data/visa data.meta : meta.json data.cls_names : [] data.train_transforms : [{'type': 'Resize', 'size': (256, 256), 'interpolation': }, {'type': 'CenterCrop', 'size': (256, 256)}, {'type': 'ToTensor'}, {'type': 'Normalize', 'mean': (0.485, 0.456, 0.406), 'std': (0.229, 0.224, 0.225), 'inplace': True}] data.test_transforms : [{'type': 'Resize', 'size': (256, 256), 'interpolation': }, {'type': 'CenterCrop', 'size': (256, 256)}, {'type': 'ToTensor'}, {'type': 'Normalize', 'mean': (0.485, 0.456, 0.406), 'std': (0.229, 0.224, 0.225), 'inplace': True}] data.target_transforms : [{'type': 'Resize', 'size': (256, 256), 'interpolation': }, {'type': 'CenterCrop', 'size': (256, 256)}, {'type': 'ToTensor'}] data.train_size : 541 data.test_size : 136 data.train_length : 8659 data.test_length : 2162 model_t.name : timm_resnet34 model_t.kwargs : {'pretrained': False, 'checkpoint_path': 'model/pretrain/resnet34-43635321.pth', 'strict': False, 'features_only': True, 'out_indices': [1, 2, 3]} model_s : {'depths_decoder': [3, 4, 6, 3], 'scan_type': 'hilbert', 'num_direction': 8} model.name : mambaadhcs57c1 model.kwargs : {'pretrained': False, 'checkpoint_path': '', 'strict': True, 'model_t': Namespace(name='timm_resnet34', kwargs={'pretrained': False, 'checkpoint_path': 'model/pretrain/resnet34-43635321.pth', 'strict': False, 'features_only': True, 'out_indices': [1, 2, 3]}), 'model_s': {'depths_decoder': [3, 4, 6, 3], 'scan_type': 'hilbert', 'num_direction': 8}} seed : 42 size : 256 warmup_epochs : 0 test_start_epoch : 1000 test_per_epoch : 50 batch_train : 16 batch_test_per : 16 lr : 0.005 weight_decay : 0.0001 cfg_path : configs.mambaad.mambaad_visa_nhcs57c1 mode : train sleep : -1 memory : -1 dist_url : env:// logger_rank : 0 opts : [] command : python3 -m torch.distributed.launch --nproc_per_node=$nproc_per_node --nnodes=$nnodes --node_rank=$node_rank --master_addr=$master_addr --master_port=$master_port --use_env run.py -c configs.mambaad.mambaad_visa_nhcs57c1 -m train --sleep -1 --memory -1 --dist_url env:// --logger_rank 0 task_start_time : 3592036.892600932 dist : False world_size : 1 rank : 0 local_rank : 0 ngpus_per_node : 1 nnodes : 1 master : True logdir : runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005 logger.filters : [] logger.name : root logger.level : 20 logger.parent : None logger.propagate : True logger.disabled : False logdir_train : runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005/show_train logdir_test : runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005/show_test 2024-04-02 01:00:26,613 - ==> Starting training with 1 nodes x 1 GPUs 2024-04-02 01:01:36,247 - Train: 0.02% [100/541000] [0.2/1000.0] [batch_t 0.611 (0.692)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 01:02:41,181 - Train: 0.04% [200/541000] [0.4/1000.0] [batch_t 0.610 (0.649)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 01:03:42,474 - Train: 0.06% [300/541000] [0.6/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 01:04:43,142 - Train: 0.07% [400/541000] [0.7/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 01:05:43,833 - Train: 0.09% [500/541000] [0.9/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 01:06:08,728 - ==> Total time: 0:06:03 Eta: 4 days, 4:46:50 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 01:06:47,365 - Train: 0.11% [600/541000] [1.1/1000.0] [batch_t 0.603 (0.621)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-02 01:07:49,667 - Train: 0.13% [700/541000] [1.3/1000.0] [batch_t 0.611 (0.623)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 01:08:50,357 - Train: 0.15% [800/541000] [1.5/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 01:09:52,431 - Train: 0.17% [900/541000] [1.7/1000.0] [batch_t 0.611 (0.618)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 01:11:06,647 - Train: 0.18% [1000/541000] [1.8/1000.0] [batch_t 0.910 (0.742)] [data_t 0.305] [optim_t 0.605] [lr 0.005000] 2024-04-02 01:12:24,929 - ==> Total time: 0:12:19 Eta: 4 days, 6:29:07 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 01:12:43,063 - Train: 0.20% [1100/541000] [2.0/1000.0] [batch_t 1.089 (0.813)] [data_t 0.485] [optim_t 0.605] [lr 0.005000] 2024-04-02 01:14:21,977 - Train: 0.22% [1200/541000] [2.2/1000.0] [batch_t 0.601 (0.989)] [data_t 0.002] [optim_t 0.600] [lr 0.005000] 2024-04-02 01:15:44,094 - Train: 0.24% [1300/541000] [2.4/1000.0] [batch_t 0.604 (0.817)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-02 01:16:45,563 - Train: 0.26% [1400/541000] [2.6/1000.0] [batch_t 0.611 (0.615)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 01:17:46,292 - Train: 0.28% [1500/541000] [2.8/1000.0] [batch_t 0.614 (0.607)] [data_t 0.002] [optim_t 0.612] [lr 0.005000] 2024-04-02 01:18:46,986 - Train: 0.30% [1600/541000] [3.0/1000.0] [batch_t 0.610 (0.607)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 01:19:00,940 - ==> Total time: 0:18:55 Eta: 4 days, 8:48:46 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 01:19:53,783 - Train: 0.31% [1700/541000] [3.1/1000.0] [batch_t 0.603 (0.667)] [data_t 0.002] [optim_t 0.601] [lr 0.005000] 2024-04-02 01:20:54,442 - Train: 0.33% [1800/541000] [3.3/1000.0] [batch_t 0.606 (0.606)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 01:21:55,141 - Train: 0.35% [1900/541000] [3.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 01:22:55,944 - Train: 0.37% [2000/541000] [3.7/1000.0] [batch_t 0.604 (0.608)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 01:23:56,680 - Train: 0.39% [2100/541000] [3.9/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 01:24:35,492 - ==> Total time: 0:24:29 Eta: 4 days, 5:40:14 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 01:24:59,459 - Train: 0.41% [2200/541000] [4.1/1000.0] [batch_t 0.602 (0.624)] [data_t 0.002] [optim_t 0.600] [lr 0.005000] 2024-04-02 01:26:00,313 - Train: 0.43% [2300/541000] [4.3/1000.0] [batch_t 0.611 (0.608)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 01:27:00,947 - Train: 0.44% [2400/541000] [4.4/1000.0] [batch_t 0.605 (0.606)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 01:28:01,588 - Train: 0.46% [2500/541000] [4.6/1000.0] [batch_t 0.599 (0.606)] [data_t 0.002] [optim_t 0.597] [lr 0.005000] 2024-04-02 01:29:02,241 - Train: 0.48% [2600/541000] [4.8/1000.0] [batch_t 0.605 (0.606)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 01:30:02,896 - Train: 0.50% [2700/541000] [5.0/1000.0] [batch_t 0.611 (0.606)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 01:30:05,926 - ==> Total time: 0:30:00 Eta: 4 days, 3:31:13 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 01:31:07,048 - Train: 0.52% [2800/541000] [5.2/1000.0] [batch_t 0.609 (0.621)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 01:32:07,832 - Train: 0.54% [2900/541000] [5.4/1000.0] [batch_t 0.611 (0.608)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 01:33:08,447 - Train: 0.55% [3000/541000] [5.5/1000.0] [batch_t 0.604 (0.606)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-02 01:34:09,116 - Train: 0.57% [3100/541000] [5.7/1000.0] [batch_t 0.604 (0.607)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-02 01:35:09,759 - Train: 0.59% [3200/541000] [5.9/1000.0] [batch_t 0.603 (0.606)] [data_t 0.002] [optim_t 0.601] [lr 0.005000] 2024-04-02 01:35:37,651 - ==> Total time: 0:35:32 Eta: 4 days, 2:06:57 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 01:36:12,719 - Train: 0.61% [3300/541000] [6.1/1000.0] [batch_t 0.611 (0.619)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 01:37:13,365 - Train: 0.63% [3400/541000] [6.3/1000.0] [batch_t 0.610 (0.606)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 01:38:14,108 - Train: 0.65% [3500/541000] [6.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 01:39:15,415 - Train: 0.67% [3600/541000] [6.7/1000.0] [batch_t 0.604 (0.613)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-02 01:40:16,055 - Train: 0.68% [3700/541000] [6.8/1000.0] [batch_t 0.602 (0.606)] [data_t 0.002] [optim_t 0.600] [lr 0.005000] 2024-04-02 01:41:08,816 - ==> Total time: 0:41:03 Eta: 4 days, 1:03:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 01:41:18,710 - Train: 0.70% [3800/541000] [7.0/1000.0] [batch_t 0.606 (0.651)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 01:42:19,416 - Train: 0.72% [3900/541000] [7.2/1000.0] [batch_t 0.611 (0.607)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 01:43:20,075 - Train: 0.74% [4000/541000] [7.4/1000.0] [batch_t 0.611 (0.606)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 01:44:20,877 - Train: 0.76% [4100/541000] [7.6/1000.0] [batch_t 0.610 (0.608)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 01:45:21,543 - Train: 0.78% [4200/541000] [7.8/1000.0] [batch_t 0.604 (0.607)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-02 01:46:22,210 - Train: 0.79% [4300/541000] [7.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 01:46:39,186 - ==> Total time: 0:46:33 Eta: 4 days, 0:13:30 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 01:47:24,722 - Train: 0.81% [4400/541000] [8.1/1000.0] [batch_t 0.611 (0.613)] [data_t 0.002] [optim_t 0.610] [lr 0.005000] 2024-04-02 01:48:25,376 - Train: 0.83% [4500/541000] [8.3/1000.0] [batch_t 0.602 (0.606)] [data_t 0.002] [optim_t 0.600] [lr 0.005000] 2024-04-02 01:49:26,172 - Train: 0.85% [4600/541000] [8.5/1000.0] [batch_t 0.603 (0.608)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-02 01:50:26,830 - Train: 0.87% [4700/541000] [8.7/1000.0] [batch_t 0.605 (0.606)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 01:51:27,494 - Train: 0.89% [4800/541000] [8.9/1000.0] [batch_t 0.610 (0.607)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 01:52:09,348 - ==> Total time: 0:52:03 Eta: 3 days, 23:32:44 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 01:52:31,051 - Train: 0.91% [4900/541000] [9.1/1000.0] [batch_t 0.609 (0.640)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 01:53:31,735 - Train: 0.92% [5000/541000] [9.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 01:54:32,360 - Train: 0.94% [5100/541000] [9.4/1000.0] [batch_t 0.612 (0.606)] [data_t 0.002] [optim_t 0.610] [lr 0.005000] 2024-04-02 01:55:32,990 - Train: 0.96% [5200/541000] [9.6/1000.0] [batch_t 0.606 (0.606)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 01:56:33,770 - Train: 0.98% [5300/541000] [9.8/1000.0] [batch_t 0.611 (0.608)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 01:57:34,408 - Train: 1.00% [5400/541000] [10.0/1000.0] [batch_t 0.607 (0.606)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 01:57:40,475 - ==> Total time: 0:57:34 Eta: 3 days, 23:00:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 01:58:38,227 - Train: 1.02% [5500/541000] [10.2/1000.0] [batch_t 0.605 (0.620)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 01:59:38,934 - Train: 1.04% [5600/541000] [10.4/1000.0] [batch_t 0.604 (0.607)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-02 02:00:39,643 - Train: 1.05% [5700/541000] [10.5/1000.0] [batch_t 0.603 (0.607)] [data_t 0.002] [optim_t 0.601] [lr 0.005000] 2024-04-02 02:01:40,338 - Train: 1.07% [5800/541000] [10.7/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 02:02:41,132 - Train: 1.09% [5900/541000] [10.9/1000.0] [batch_t 0.610 (0.608)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 02:03:12,102 - ==> Total time: 1:03:06 Eta: 3 days, 22:34:05 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 02:03:44,433 - Train: 1.11% [6000/541000] [11.1/1000.0] [batch_t 0.605 (0.628)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 02:04:45,127 - Train: 1.13% [6100/541000] [11.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 02:05:45,813 - Train: 1.15% [6200/541000] [11.5/1000.0] [batch_t 0.611 (0.607)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 02:06:46,513 - Train: 1.16% [6300/541000] [11.6/1000.0] [batch_t 0.604 (0.607)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-02 02:07:47,925 - Train: 1.18% [6400/541000] [11.8/1000.0] [batch_t 0.607 (0.614)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 02:08:43,798 - ==> Total time: 1:08:38 Eta: 3 days, 22:11:08 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 02:08:51,051 - Train: 1.20% [6500/541000] [12.0/1000.0] [batch_t 0.611 (0.715)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 02:09:51,776 - Train: 1.22% [6600/541000] [12.2/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 02:10:52,498 - Train: 1.24% [6700/541000] [12.4/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 02:11:53,214 - Train: 1.26% [6800/541000] [12.6/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 02:12:53,905 - Train: 1.28% [6900/541000] [12.8/1000.0] [batch_t 0.603 (0.607)] [data_t 0.002] [optim_t 0.601] [lr 0.005000] 2024-04-02 02:13:54,751 - Train: 1.29% [7000/541000] [12.9/1000.0] [batch_t 0.604 (0.608)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-02 02:14:14,781 - ==> Total time: 1:14:09 Eta: 3 days, 21:49:58 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 02:14:58,146 - Train: 1.31% [7100/541000] [13.1/1000.0] [batch_t 0.611 (0.618)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 02:15:58,862 - Train: 1.33% [7200/541000] [13.3/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 02:16:59,624 - Train: 1.35% [7300/541000] [13.5/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 02:18:00,340 - Train: 1.37% [7400/541000] [13.7/1000.0] [batch_t 0.610 (0.607)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 02:19:01,049 - Train: 1.39% [7500/541000] [13.9/1000.0] [batch_t 0.603 (0.607)] [data_t 0.002] [optim_t 0.601] [lr 0.005000] 2024-04-02 02:19:46,166 - ==> Total time: 1:19:40 Eta: 3 days, 21:31:31 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 02:20:03,984 - Train: 1.40% [7600/541000] [14.0/1000.0] [batch_t 0.606 (0.625)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 02:21:04,703 - Train: 1.42% [7700/541000] [14.2/1000.0] [batch_t 0.610 (0.607)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 02:22:05,411 - Train: 1.44% [7800/541000] [14.4/1000.0] [batch_t 0.604 (0.607)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-02 02:23:06,107 - Train: 1.46% [7900/541000] [14.6/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 02:24:06,864 - Train: 1.48% [8000/541000] [14.8/1000.0] [batch_t 0.615 (0.607)] [data_t 0.002] [optim_t 0.613] [lr 0.005000] 2024-04-02 02:25:07,569 - Train: 1.50% [8100/541000] [15.0/1000.0] [batch_t 0.603 (0.607)] [data_t 0.002] [optim_t 0.601] [lr 0.005000] 2024-04-02 02:25:16,674 - ==> Total time: 1:25:11 Eta: 3 days, 21:13:50 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 02:26:10,904 - Train: 1.52% [8200/541000] [15.2/1000.0] [batch_t 0.608 (0.619)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 02:27:11,607 - Train: 1.53% [8300/541000] [15.3/1000.0] [batch_t 0.611 (0.607)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 02:28:12,501 - Train: 1.55% [8400/541000] [15.5/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 02:29:13,132 - Train: 1.57% [8500/541000] [15.7/1000.0] [batch_t 0.607 (0.606)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 02:30:13,828 - Train: 1.59% [8600/541000] [15.9/1000.0] [batch_t 0.610 (0.607)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 02:30:47,932 - ==> Total time: 1:30:42 Eta: 3 days, 20:58:26 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 02:31:16,948 - Train: 1.61% [8700/541000] [16.1/1000.0] [batch_t 0.605 (0.624)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 02:32:17,639 - Train: 1.63% [8800/541000] [16.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 02:33:18,365 - Train: 1.65% [8900/541000] [16.5/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 02:34:19,662 - Train: 1.66% [9000/541000] [16.6/1000.0] [batch_t 0.604 (0.613)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-02 02:35:20,372 - Train: 1.68% [9100/541000] [16.8/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 02:36:19,516 - ==> Total time: 1:36:13 Eta: 3 days, 20:44:30 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 02:36:23,494 - Train: 1.70% [9200/541000] [17.0/1000.0] [batch_t 0.611 (0.791)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 02:37:24,221 - Train: 1.72% [9300/541000] [17.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 02:38:24,914 - Train: 1.74% [9400/541000] [17.4/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 02:39:25,633 - Train: 1.76% [9500/541000] [17.6/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 02:40:26,393 - Train: 1.77% [9600/541000] [17.7/1000.0] [batch_t 0.606 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-02 02:41:27,384 - Train: 1.79% [9700/541000] [17.9/1000.0] [batch_t 0.608 (0.610)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 02:41:50,498 - ==> Total time: 1:41:44 Eta: 3 days, 20:30:58 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 02:42:30,540 - Train: 1.81% [9800/541000] [18.1/1000.0] [batch_t 0.613 (0.618)] [data_t 0.002] [optim_t 0.611] [lr 0.005000] 2024-04-02 02:43:31,278 - Train: 1.83% [9900/541000] [18.3/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 02:44:31,985 - Train: 1.85% [10000/541000] [18.5/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 02:45:32,689 - Train: 1.87% [10100/541000] [18.7/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 02:46:33,509 - Train: 1.89% [10200/541000] [18.9/1000.0] [batch_t 0.604 (0.608)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-02 02:47:21,527 - ==> Total time: 1:47:15 Eta: 3 days, 20:18:19 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 02:47:36,581 - Train: 1.90% [10300/541000] [19.0/1000.0] [batch_t 0.607 (0.641)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 02:48:37,291 - Train: 1.92% [10400/541000] [19.2/1000.0] [batch_t 0.611 (0.607)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 02:49:38,011 - Train: 1.94% [10500/541000] [19.4/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 02:50:39,302 - Train: 1.96% [10600/541000] [19.6/1000.0] [batch_t 0.608 (0.613)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 02:51:40,180 - Train: 1.98% [10700/541000] [19.8/1000.0] [batch_t 0.611 (0.609)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 02:52:40,854 - Train: 2.00% [10800/541000] [20.0/1000.0] [batch_t 0.603 (0.607)] [data_t 0.002] [optim_t 0.601] [lr 0.005000] 2024-04-02 02:52:53,015 - ==> Total time: 1:52:47 Eta: 3 days, 20:06:45 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 02:53:44,203 - Train: 2.01% [10900/541000] [20.1/1000.0] [batch_t 0.607 (0.620)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 02:54:44,963 - Train: 2.03% [11000/541000] [20.3/1000.0] [batch_t 0.615 (0.607)] [data_t 0.002] [optim_t 0.613] [lr 0.005000] 2024-04-02 02:55:45,681 - Train: 2.05% [11100/541000] [20.5/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 02:56:46,406 - Train: 2.07% [11200/541000] [20.7/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 02:57:47,157 - Train: 2.09% [11300/541000] [20.9/1000.0] [batch_t 0.611 (0.607)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 02:58:24,311 - ==> Total time: 1:58:18 Eta: 3 days, 19:55:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 02:58:50,218 - Train: 2.11% [11400/541000] [21.1/1000.0] [batch_t 0.605 (0.627)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 02:59:50,918 - Train: 2.13% [11500/541000] [21.3/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 03:00:51,639 - Train: 2.14% [11600/541000] [21.4/1000.0] [batch_t 0.611 (0.607)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 03:01:52,319 - Train: 2.16% [11700/541000] [21.6/1000.0] [batch_t 0.597 (0.607)] [data_t 0.002] [optim_t 0.596] [lr 0.005000] 2024-04-02 03:02:53,061 - Train: 2.18% [11800/541000] [21.8/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 03:03:53,919 - Train: 2.20% [11900/541000] [22.0/1000.0] [batch_t 0.610 (0.608)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 03:03:55,135 - ==> Total time: 2:03:49 Eta: 3 days, 19:44:38 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 03:04:58,501 - Train: 2.22% [12000/541000] [22.2/1000.0] [batch_t 0.606 (0.622)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 03:05:59,206 - Train: 2.24% [12100/541000] [22.4/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 03:07:00,586 - Train: 2.26% [12200/541000] [22.6/1000.0] [batch_t 0.609 (0.614)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 03:08:01,281 - Train: 2.27% [12300/541000] [22.7/1000.0] [batch_t 0.598 (0.607)] [data_t 0.002] [optim_t 0.596] [lr 0.005000] 2024-04-02 03:09:02,010 - Train: 2.29% [12400/541000] [22.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 03:09:28,119 - ==> Total time: 2:09:22 Eta: 3 days, 19:35:40 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 03:10:05,067 - Train: 2.31% [12500/541000] [23.1/1000.0] [batch_t 0.610 (0.621)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 03:11:05,764 - Train: 2.33% [12600/541000] [23.3/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 03:12:06,460 - Train: 2.35% [12700/541000] [23.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 03:13:08,400 - Train: 2.37% [12800/541000] [23.7/1000.0] [batch_t 0.612 (0.619)] [data_t 0.002] [optim_t 0.610] [lr 0.005000] 2024-04-02 03:14:09,676 - Train: 2.38% [12900/541000] [23.8/1000.0] [batch_t 0.603 (0.613)] [data_t 0.002] [optim_t 0.601] [lr 0.005000] 2024-04-02 03:15:00,680 - ==> Total time: 2:14:55 Eta: 3 days, 19:26:41 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 03:15:16,592 - Train: 2.40% [13000/541000] [24.0/1000.0] [batch_t 0.600 (0.701)] [data_t 0.002] [optim_t 0.598] [lr 0.005000] 2024-04-02 03:16:17,333 - Train: 2.42% [13100/541000] [24.2/1000.0] [batch_t 0.611 (0.607)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 03:17:18,461 - Train: 2.44% [13200/541000] [24.4/1000.0] [batch_t 0.604 (0.611)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-02 03:18:19,161 - Train: 2.46% [13300/541000] [24.6/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 03:19:19,848 - Train: 2.48% [13400/541000] [24.8/1000.0] [batch_t 0.610 (0.607)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 03:20:20,562 - Train: 2.50% [13500/541000] [25.0/1000.0] [batch_t 0.602 (0.607)] [data_t 0.002] [optim_t 0.601] [lr 0.005000] 2024-04-02 03:20:35,752 - ==> Total time: 2:20:30 Eta: 3 days, 19:19:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 03:21:23,909 - Train: 2.51% [13600/541000] [25.1/1000.0] [batch_t 0.608 (0.624)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 03:22:24,923 - Train: 2.53% [13700/541000] [25.3/1000.0] [batch_t 0.611 (0.610)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 03:23:26,091 - Train: 2.55% [13800/541000] [25.5/1000.0] [batch_t 0.604 (0.611)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 03:24:26,785 - Train: 2.57% [13900/541000] [25.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 03:25:27,456 - Train: 2.59% [14000/541000] [25.9/1000.0] [batch_t 0.611 (0.607)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 03:26:07,528 - ==> Total time: 2:26:01 Eta: 3 days, 19:10:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 03:26:31,451 - Train: 2.61% [14100/541000] [26.1/1000.0] [batch_t 0.605 (0.654)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 03:27:32,174 - Train: 2.62% [14200/541000] [26.2/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 03:28:33,127 - Train: 2.64% [14300/541000] [26.4/1000.0] [batch_t 0.610 (0.609)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 03:29:33,832 - Train: 2.66% [14400/541000] [26.6/1000.0] [batch_t 0.601 (0.607)] [data_t 0.002] [optim_t 0.600] [lr 0.005000] 2024-04-02 03:30:34,516 - Train: 2.68% [14500/541000] [26.8/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 03:31:35,228 - Train: 2.70% [14600/541000] [27.0/1000.0] [batch_t 0.612 (0.607)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 03:31:39,479 - ==> Total time: 2:31:33 Eta: 3 days, 19:01:58 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 03:32:39,038 - Train: 2.72% [14700/541000] [27.2/1000.0] [batch_t 0.603 (0.621)] [data_t 0.002] [optim_t 0.601] [lr 0.005000] 2024-04-02 03:33:39,779 - Train: 2.74% [14800/541000] [27.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 03:34:40,638 - Train: 2.75% [14900/541000] [27.5/1000.0] [batch_t 0.612 (0.608)] [data_t 0.002] [optim_t 0.611] [lr 0.005000] 2024-04-02 03:35:41,345 - Train: 2.77% [15000/541000] [27.7/1000.0] [batch_t 0.603 (0.607)] [data_t 0.002] [optim_t 0.601] [lr 0.005000] 2024-04-02 03:36:42,088 - Train: 2.79% [15100/541000] [27.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 03:37:11,249 - ==> Total time: 2:37:05 Eta: 3 days, 18:53:26 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 03:37:45,360 - Train: 2.81% [15200/541000] [28.1/1000.0] [batch_t 0.608 (0.626)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 03:38:46,121 - Train: 2.83% [15300/541000] [28.3/1000.0] [batch_t 0.603 (0.607)] [data_t 0.002] [optim_t 0.601] [lr 0.005000] 2024-04-02 03:39:46,832 - Train: 2.85% [15400/541000] [28.5/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 03:40:47,723 - Train: 2.87% [15500/541000] [28.7/1000.0] [batch_t 0.611 (0.609)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 03:41:48,424 - Train: 2.88% [15600/541000] [28.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 03:42:42,495 - ==> Total time: 2:42:36 Eta: 3 days, 18:44:49 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 03:42:52,056 - Train: 2.90% [15700/541000] [29.0/1000.0] [batch_t 0.611 (0.706)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 03:43:52,784 - Train: 2.92% [15800/541000] [29.2/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 03:44:53,486 - Train: 2.94% [15900/541000] [29.4/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 03:45:54,233 - Train: 2.96% [16000/541000] [29.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 03:46:54,961 - Train: 2.98% [16100/541000] [29.8/1000.0] [batch_t 0.610 (0.607)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 03:47:55,820 - Train: 2.99% [16200/541000] [29.9/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 03:48:14,041 - ==> Total time: 2:48:08 Eta: 3 days, 18:36:34 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 03:48:58,647 - Train: 3.01% [16300/541000] [30.1/1000.0] [batch_t 0.607 (0.618)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 03:49:59,347 - Train: 3.03% [16400/541000] [30.3/1000.0] [batch_t 0.611 (0.607)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 03:51:00,341 - Train: 3.05% [16500/541000] [30.5/1000.0] [batch_t 0.604 (0.610)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 03:52:01,395 - Train: 3.07% [16600/541000] [30.7/1000.0] [batch_t 0.607 (0.610)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 03:53:03,170 - Train: 3.09% [16700/541000] [30.9/1000.0] [batch_t 0.611 (0.618)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 03:53:46,233 - ==> Total time: 2:53:40 Eta: 3 days, 18:28:50 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 03:54:05,967 - Train: 3.11% [16800/541000] [31.1/1000.0] [batch_t 0.601 (0.634)] [data_t 0.002] [optim_t 0.599] [lr 0.005000] 2024-04-02 03:55:06,681 - Train: 3.12% [16900/541000] [31.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 03:56:07,771 - Train: 3.14% [17000/541000] [31.4/1000.0] [batch_t 0.611 (0.611)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 03:57:08,578 - Train: 3.16% [17100/541000] [31.6/1000.0] [batch_t 0.599 (0.608)] [data_t 0.002] [optim_t 0.598] [lr 0.005000] 2024-04-02 03:58:09,301 - Train: 3.18% [17200/541000] [31.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 03:59:10,180 - Train: 3.20% [17300/541000] [32.0/1000.0] [batch_t 0.611 (0.609)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 03:59:17,459 - ==> Total time: 2:59:11 Eta: 3 days, 18:20:45 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 04:00:13,395 - Train: 3.22% [17400/541000] [32.2/1000.0] [batch_t 0.606 (0.617)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 04:01:14,096 - Train: 3.23% [17500/541000] [32.3/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 04:02:16,200 - Train: 3.25% [17600/541000] [32.5/1000.0] [batch_t 0.611 (0.621)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 04:03:16,853 - Train: 3.27% [17700/541000] [32.7/1000.0] [batch_t 0.606 (0.606)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 04:04:17,514 - Train: 3.29% [17800/541000] [32.9/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 04:04:49,678 - ==> Total time: 3:04:44 Eta: 3 days, 18:13:18 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 04:05:23,364 - Train: 3.31% [17900/541000] [33.1/1000.0] [batch_t 0.609 (0.655)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 04:06:24,207 - Train: 3.33% [18000/541000] [33.3/1000.0] [batch_t 0.603 (0.608)] [data_t 0.002] [optim_t 0.601] [lr 0.005000] 2024-04-02 04:07:24,951 - Train: 3.35% [18100/541000] [33.5/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 04:08:25,650 - Train: 3.36% [18200/541000] [33.6/1000.0] [batch_t 0.610 (0.607)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 04:09:26,403 - Train: 3.38% [18300/541000] [33.8/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 04:10:24,199 - ==> Total time: 3:10:18 Eta: 3 days, 18:07:03 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 04:10:31,301 - Train: 3.40% [18400/541000] [34.0/1000.0] [batch_t 0.605 (0.769)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 04:11:31,997 - Train: 3.42% [18500/541000] [34.2/1000.0] [batch_t 0.611 (0.607)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 04:12:33,008 - Train: 3.44% [18600/541000] [34.4/1000.0] [batch_t 0.603 (0.610)] [data_t 0.002] [optim_t 0.601] [lr 0.005000] 2024-04-02 04:13:34,340 - Train: 3.46% [18700/541000] [34.6/1000.0] [batch_t 0.607 (0.613)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 04:14:35,041 - Train: 3.48% [18800/541000] [34.8/1000.0] [batch_t 0.612 (0.607)] [data_t 0.002] [optim_t 0.610] [lr 0.005000] 2024-04-02 04:15:36,761 - Train: 3.49% [18900/541000] [34.9/1000.0] [batch_t 0.604 (0.617)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-02 04:15:58,006 - ==> Total time: 3:15:52 Eta: 3 days, 18:00:31 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 04:16:40,007 - Train: 3.51% [19000/541000] [35.1/1000.0] [batch_t 0.604 (0.622)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-02 04:17:42,135 - Train: 3.53% [19100/541000] [35.3/1000.0] [batch_t 0.609 (0.621)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 04:18:42,842 - Train: 3.55% [19200/541000] [35.5/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 04:19:43,566 - Train: 3.57% [19300/541000] [35.7/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 04:20:44,652 - Train: 3.59% [19400/541000] [35.9/1000.0] [batch_t 0.610 (0.611)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 04:21:30,894 - ==> Total time: 3:21:25 Eta: 3 days, 17:53:38 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 04:21:48,405 - Train: 3.60% [19500/541000] [36.0/1000.0] [batch_t 0.604 (0.661)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-02 04:22:49,228 - Train: 3.62% [19600/541000] [36.2/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-02 04:23:50,055 - Train: 3.64% [19700/541000] [36.4/1000.0] [batch_t 0.611 (0.608)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 04:24:51,145 - Train: 3.66% [19800/541000] [36.6/1000.0] [batch_t 0.607 (0.611)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 04:25:52,150 - Train: 3.68% [19900/541000] [36.8/1000.0] [batch_t 0.608 (0.610)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 04:26:52,902 - Train: 3.70% [20000/541000] [37.0/1000.0] [batch_t 0.610 (0.607)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 04:27:03,231 - ==> Total time: 3:26:57 Eta: 3 days, 17:46:35 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 04:27:58,194 - Train: 3.72% [20100/541000] [37.2/1000.0] [batch_t 0.606 (0.626)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 04:28:58,932 - Train: 3.73% [20200/541000] [37.3/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 04:29:59,786 - Train: 3.75% [20300/541000] [37.5/1000.0] [batch_t 0.610 (0.608)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 04:31:00,580 - Train: 3.77% [20400/541000] [37.7/1000.0] [batch_t 0.604 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 04:32:01,313 - Train: 3.79% [20500/541000] [37.9/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 04:32:36,602 - ==> Total time: 3:32:31 Eta: 3 days, 17:40:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 04:33:05,269 - Train: 3.81% [20600/541000] [38.1/1000.0] [batch_t 0.610 (0.632)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 04:34:05,995 - Train: 3.83% [20700/541000] [38.3/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 04:35:07,484 - Train: 3.84% [20800/541000] [38.4/1000.0] [batch_t 1.248 (0.615)] [data_t 0.644] [optim_t 0.604] [lr 0.005000] 2024-04-02 04:36:08,209 - Train: 3.86% [20900/541000] [38.6/1000.0] [batch_t 0.611 (0.607)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 04:37:08,965 - Train: 3.88% [21000/541000] [38.8/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 04:38:09,079 - ==> Total time: 3:38:03 Eta: 3 days, 17:33:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 04:38:12,300 - Train: 3.90% [21100/541000] [39.0/1000.0] [batch_t 1.470 (1.470)] [data_t 0.861] [optim_t 0.608] [lr 0.005000] 2024-04-02 04:39:13,013 - Train: 3.92% [21200/541000] [39.2/1000.0] [batch_t 0.612 (0.607)] [data_t 0.002] [optim_t 0.610] [lr 0.005000] 2024-04-02 04:40:13,761 - Train: 3.94% [21300/541000] [39.4/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 04:41:14,523 - Train: 3.96% [21400/541000] [39.6/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 04:42:15,402 - Train: 3.97% [21500/541000] [39.7/1000.0] [batch_t 0.610 (0.609)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 04:43:16,148 - Train: 3.99% [21600/541000] [39.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 04:43:40,427 - ==> Total time: 3:43:34 Eta: 3 days, 17:25:56 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 04:44:19,960 - Train: 4.01% [21700/541000] [40.1/1000.0] [batch_t 0.607 (0.623)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 04:45:20,718 - Train: 4.03% [21800/541000] [40.3/1000.0] [batch_t 0.613 (0.607)] [data_t 0.002] [optim_t 0.610] [lr 0.005000] 2024-04-02 04:46:21,450 - Train: 4.05% [21900/541000] [40.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 04:47:22,314 - Train: 4.07% [22000/541000] [40.7/1000.0] [batch_t 0.607 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 04:48:23,094 - Train: 4.09% [22100/541000] [40.9/1000.0] [batch_t 0.610 (0.608)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 04:49:12,307 - ==> Total time: 3:49:06 Eta: 3 days, 17:18:59 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 04:49:26,893 - Train: 4.10% [22200/541000] [41.0/1000.0] [batch_t 0.604 (0.674)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-02 04:50:27,652 - Train: 4.12% [22300/541000] [41.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 04:51:28,442 - Train: 4.14% [22400/541000] [41.4/1000.0] [batch_t 0.611 (0.608)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 04:52:29,172 - Train: 4.16% [22500/541000] [41.6/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 04:53:30,084 - Train: 4.18% [22600/541000] [41.8/1000.0] [batch_t 0.605 (0.609)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 04:54:30,847 - Train: 4.20% [22700/541000] [42.0/1000.0] [batch_t 0.612 (0.608)] [data_t 0.002] [optim_t 0.610] [lr 0.005000] 2024-04-02 04:54:44,207 - ==> Total time: 3:54:38 Eta: 3 days, 17:12:07 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 04:55:34,030 - Train: 4.21% [22800/541000] [42.1/1000.0] [batch_t 0.605 (0.615)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 04:56:34,863 - Train: 4.23% [22900/541000] [42.3/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 04:57:35,642 - Train: 4.25% [23000/541000] [42.5/1000.0] [batch_t 0.612 (0.608)] [data_t 0.002] [optim_t 0.610] [lr 0.005000] 2024-04-02 04:58:36,396 - Train: 4.27% [23100/541000] [42.7/1000.0] [batch_t 0.603 (0.607)] [data_t 0.002] [optim_t 0.601] [lr 0.005000] 2024-04-02 04:59:37,307 - Train: 4.29% [23200/541000] [42.9/1000.0] [batch_t 0.605 (0.609)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 05:00:15,558 - ==> Total time: 4:00:10 Eta: 3 days, 17:05:06 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 05:00:40,759 - Train: 4.31% [23300/541000] [43.1/1000.0] [batch_t 0.611 (0.633)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 05:01:41,516 - Train: 4.33% [23400/541000] [43.3/1000.0] [batch_t 0.604 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 05:02:42,256 - Train: 4.34% [23500/541000] [43.4/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 05:03:42,995 - Train: 4.36% [23600/541000] [43.6/1000.0] [batch_t 0.610 (0.607)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 05:04:43,865 - Train: 4.38% [23700/541000] [43.8/1000.0] [batch_t 0.605 (0.609)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 05:05:44,616 - Train: 4.40% [23800/541000] [44.0/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 05:05:47,051 - ==> Total time: 4:05:41 Eta: 3 days, 16:58:12 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 05:06:47,907 - Train: 4.42% [23900/541000] [44.2/1000.0] [batch_t 0.610 (0.615)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 05:07:48,625 - Train: 4.44% [24000/541000] [44.4/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 05:08:49,414 - Train: 4.45% [24100/541000] [44.5/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 05:09:50,156 - Train: 4.47% [24200/541000] [44.7/1000.0] [batch_t 0.612 (0.607)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 05:10:51,054 - Train: 4.49% [24300/541000] [44.9/1000.0] [batch_t 0.605 (0.609)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 05:11:18,396 - ==> Total time: 4:11:12 Eta: 3 days, 16:51:19 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 05:11:54,483 - Train: 4.51% [24400/541000] [45.1/1000.0] [batch_t 0.607 (0.620)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 05:12:55,259 - Train: 4.53% [24500/541000] [45.3/1000.0] [batch_t 0.612 (0.608)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 05:13:56,012 - Train: 4.55% [24600/541000] [45.5/1000.0] [batch_t 0.604 (0.607)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-02 05:14:56,726 - Train: 4.57% [24700/541000] [45.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 05:15:57,476 - Train: 4.58% [24800/541000] [45.8/1000.0] [batch_t 0.610 (0.607)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 05:16:49,870 - ==> Total time: 4:16:44 Eta: 3 days, 16:44:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 05:17:00,510 - Train: 4.60% [24900/541000] [46.0/1000.0] [batch_t 0.603 (0.652)] [data_t 0.002] [optim_t 0.601] [lr 0.005000] 2024-04-02 05:18:01,287 - Train: 4.62% [25000/541000] [46.2/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 05:19:02,048 - Train: 4.64% [25100/541000] [46.4/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 05:20:03,170 - Train: 4.66% [25200/541000] [46.6/1000.0] [batch_t 0.606 (0.611)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 05:21:03,910 - Train: 4.68% [25300/541000] [46.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 05:22:04,694 - Train: 4.70% [25400/541000] [47.0/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 05:22:22,038 - ==> Total time: 4:22:16 Eta: 3 days, 16:38:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 05:23:08,996 - Train: 4.71% [25500/541000] [47.1/1000.0] [batch_t 0.606 (0.621)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 05:24:09,742 - Train: 4.73% [25600/541000] [47.3/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 05:25:10,519 - Train: 4.75% [25700/541000] [47.5/1000.0] [batch_t 0.612 (0.608)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 05:26:11,970 - Train: 4.77% [25800/541000] [47.7/1000.0] [batch_t 0.607 (0.614)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 05:27:12,695 - Train: 4.79% [25900/541000] [47.9/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 05:27:54,013 - ==> Total time: 4:27:48 Eta: 3 days, 16:31:31 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 05:28:16,581 - Train: 4.81% [26000/541000] [48.1/1000.0] [batch_t 0.611 (0.642)] [data_t 0.003] [optim_t 0.608] [lr 0.005000] 2024-04-02 05:29:17,363 - Train: 4.82% [26100/541000] [48.2/1000.0] [batch_t 0.605 (0.608)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-02 05:30:18,144 - Train: 4.84% [26200/541000] [48.4/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 05:31:18,909 - Train: 4.86% [26300/541000] [48.6/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 05:32:20,151 - Train: 4.88% [26400/541000] [48.8/1000.0] [batch_t 0.605 (0.612)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 05:33:20,933 - Train: 4.90% [26500/541000] [49.0/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 05:33:26,399 - ==> Total time: 4:33:20 Eta: 3 days, 16:25:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 05:34:24,309 - Train: 4.92% [26600/541000] [49.2/1000.0] [batch_t 0.611 (0.615)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 05:35:25,110 - Train: 4.94% [26700/541000] [49.4/1000.0] [batch_t 0.605 (0.608)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-02 05:36:25,872 - Train: 4.95% [26800/541000] [49.5/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 05:37:26,615 - Train: 4.97% [26900/541000] [49.7/1000.0] [batch_t 0.611 (0.607)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 05:38:27,370 - Train: 4.99% [27000/541000] [49.9/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-02 05:39:19,131 - Test: 36.76% [50/136] [batch_t 0.211 (0.421)] 2024-04-02 05:39:30,997 - Test: 73.53% [100/136] [batch_t 0.212 (0.329)] 2024-04-02 05:39:38,640 - Test: 100.00% [136/136] [batch_t 0.270 (0.298)] 2024-04-02 05:40:26,940 - ==> Metric Time for candle : 0.002 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.508 (mAUROC_px) 8.052 (mAP_px) 0.930 (mF1_max_px) 25.130 (mAUPRO_px) 0.317 (mF1_px_0.2_0.8_0.1) 0.326 (mAcc_px_0.2_0.8_0.1) 0.328 (mIoU_px_0.2_0.8_0.1) 0.952 (mIoU_max_px) 2024-04-02 05:41:04,257 - ==> Metric Time for capsules : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 8.286 (mAUROC_px) 6.287 (mAP_px) 0.756 (mF1_max_px) 20.263 (mAUPRO_px) 0.254 (mF1_px_0.2_0.8_0.1) 0.254 (mAcc_px_0.2_0.8_0.1) 0.254 (mIoU_px_0.2_0.8_0.1) 0.742 (mIoU_max_px) 2024-04-02 05:41:39,894 - ==> Metric Time for cashew : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.770 (mAUROC_px) 5.983 (mAP_px) 0.715 (mF1_max_px) 19.508 (mAUPRO_px) 0.244 (mF1_px_0.2_0.8_0.1) 0.243 (mAcc_px_0.2_0.8_0.1) 0.244 (mIoU_px_0.2_0.8_0.1) 0.719 (mIoU_max_px) 2024-04-02 05:42:16,360 - ==> Metric Time for chewinggum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.650 (mAUROC_px) 5.872 (mAP_px) 0.703 (mF1_max_px) 20.625 (mAUPRO_px) 0.237 (mF1_px_0.2_0.8_0.1) 0.239 (mAcc_px_0.2_0.8_0.1) 0.240 (mIoU_px_0.2_0.8_0.1) 0.695 (mIoU_max_px) 2024-04-02 05:42:52,504 - ==> Metric Time for fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.623 (mAUROC_px) 5.641 (mAP_px) 0.710 (mF1_max_px) 20.558 (mAUPRO_px) 0.240 (mF1_px_0.2_0.8_0.1) 0.240 (mAcc_px_0.2_0.8_0.1) 0.234 (mIoU_px_0.2_0.8_0.1) 0.691 (mIoU_max_px) 2024-04-02 05:43:38,882 - ==> Metric Time for macaroni1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.372 (mAUROC_px) 7.806 (mAP_px) 0.923 (mF1_max_px) 25.120 (mAUPRO_px) 0.321 (mF1_px_0.2_0.8_0.1) 0.321 (mAcc_px_0.2_0.8_0.1) 0.316 (mIoU_px_0.2_0.8_0.1) 0.928 (mIoU_max_px) 2024-04-02 05:44:25,036 - ==> Metric Time for macaroni2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.415 (mAUROC_px) 7.731 (mAP_px) 0.950 (mF1_max_px) 24.886 (mAUPRO_px) 0.317 (mF1_px_0.2_0.8_0.1) 0.317 (mAcc_px_0.2_0.8_0.1) 0.320 (mIoU_px_0.2_0.8_0.1) 0.939 (mIoU_max_px) 2024-04-02 05:45:12,585 - ==> Metric Time for pcb1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.550 (mAUROC_px) 8.020 (mAP_px) 0.938 (mF1_max_px) 25.849 (mAUPRO_px) 0.326 (mF1_px_0.2_0.8_0.1) 0.322 (mAcc_px_0.2_0.8_0.1) 0.324 (mIoU_px_0.2_0.8_0.1) 0.941 (mIoU_max_px) 2024-04-02 05:46:00,053 - ==> Metric Time for pcb2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.527 (mAUROC_px) 8.082 (mAP_px) 0.935 (mF1_max_px) 25.745 (mAUPRO_px) 0.323 (mF1_px_0.2_0.8_0.1) 0.325 (mAcc_px_0.2_0.8_0.1) 0.321 (mIoU_px_0.2_0.8_0.1) 0.931 (mIoU_max_px) 2024-04-02 05:46:47,443 - ==> Metric Time for pcb3 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.513 (mAUROC_px) 7.882 (mAP_px) 0.937 (mF1_max_px) 25.879 (mAUPRO_px) 0.324 (mF1_px_0.2_0.8_0.1) 0.325 (mAcc_px_0.2_0.8_0.1) 0.325 (mIoU_px_0.2_0.8_0.1) 0.930 (mIoU_max_px) 2024-04-02 05:47:36,695 - ==> Metric Time for pcb4 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.637 (mAUROC_px) 8.128 (mAP_px) 0.941 (mF1_max_px) 27.373 (mAUPRO_px) 0.318 (mF1_px_0.2_0.8_0.1) 0.317 (mAcc_px_0.2_0.8_0.1) 0.328 (mIoU_px_0.2_0.8_0.1) 0.928 (mIoU_max_px) 2024-04-02 05:48:12,727 - ==> Metric Time for pipe_fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.741 (mAUROC_px) 5.799 (mAP_px) 0.722 (mF1_max_px) 20.149 (mAUPRO_px) 0.238 (mF1_px_0.2_0.8_0.1) 0.235 (mAcc_px_0.2_0.8_0.1) 0.240 (mIoU_px_0.2_0.8_0.1) 0.695 (mIoU_max_px) 2024-04-02 05:48:12,747 - | Name | mAUROC_sp_max | mAUROC_sp_max (Max) | mAP_sp_max | mAP_sp_max (Max) | mF1_max_sp_max | mF1_max_sp_max (Max) | mAUROC_px | mAUROC_px (Max) | mAP_px | mAP_px (Max) | mF1_max_px | mF1_max_px (Max) | mAUPRO_px | mAUPRO_px (Max) | mF1_px_0.2_0.8_0.1 | mF1_px_0.2_0.8_0.1 (Max) | mAcc_px_0.2_0.8_0.1 | mAcc_px_0.2_0.8_0.1 (Max) | mIoU_px_0.2_0.8_0.1 | mIoU_px_0.2_0.8_0.1 (Max) | mIoU_max_px | mIoU_max_px (Max) | |:----------:|:---------------:|:---------------------:|:------------:|:------------------:|:----------------:|:----------------------:|:-----------:|:------------------:|:--------:|:------------------:|:------------:|:------------------:|:-----------:|:------------------:|:--------------------:|:--------------------------:|:---------------------:|:---------------------------:|:---------------------:|:---------------------------:|:-------------:|:-------------------:| | candle | 97.100 | 97.100 (50 epoch) | 97.346 | 97.346 (50 epoch) | 91.542 | 91.542 (50 epoch) | 98.970 | 98.970 (50 epoch) | 21.600 | 21.600 (50 epoch) | 30.681 | 30.681 (50 epoch) | 95.763 | 95.763 (50 epoch) | 18.878 | 18.878 (50 epoch) | 36.606 | 36.606 (50 epoch) | 10.735 | 10.735 (50 epoch) | 18.121 | 18.121 (50 epoch) | | capsules | 89.233 | 89.233 (50 epoch) | 92.963 | 92.963 (50 epoch) | 86.538 | 86.538 (50 epoch) | 98.851 | 98.851 (50 epoch) | 51.277 | 51.277 (50 epoch) | 51.425 | 51.425 (50 epoch) | 90.636 | 90.636 (50 epoch) | 29.177 | 29.177 (50 epoch) | 62.576 | 62.576 (50 epoch) | 18.374 | 18.374 (50 epoch) | 34.612 | 34.612 (50 epoch) | | cashew | 92.660 | 92.660 (50 epoch) | 96.572 | 96.572 (50 epoch) | 90.452 | 90.452 (50 epoch) | 94.652 | 94.652 (50 epoch) | 46.784 | 46.784 (50 epoch) | 51.994 | 51.994 (50 epoch) | 88.493 | 88.493 (50 epoch) | 26.577 | 26.577 (50 epoch) | 30.723 | 30.723 (50 epoch) | 16.556 | 16.556 (50 epoch) | 35.130 | 35.130 (50 epoch) | | chewinggum | 97.370 | 97.370 (50 epoch) | 98.755 | 98.755 (50 epoch) | 94.301 | 94.301 (50 epoch) | 98.136 | 98.136 (50 epoch) | 59.823 | 59.823 (50 epoch) | 61.088 | 61.088 (50 epoch) | 79.773 | 79.773 (50 epoch) | 35.748 | 35.748 (50 epoch) | 65.611 | 65.611 (50 epoch) | 23.754 | 23.754 (50 epoch) | 43.976 | 43.976 (50 epoch) | | fryum | 94.280 | 94.280 (50 epoch) | 97.384 | 97.384 (50 epoch) | 89.302 | 89.302 (50 epoch) | 96.890 | 96.890 (50 epoch) | 48.087 | 48.087 (50 epoch) | 51.089 | 51.089 (50 epoch) | 91.111 | 91.111 (50 epoch) | 26.312 | 26.312 (50 epoch) | 27.708 | 27.708 (50 epoch) | 16.200 | 16.200 (50 epoch) | 34.308 | 34.308 (50 epoch) | | macaroni1 | 89.695 | 89.695 (50 epoch) | 88.248 | 88.248 (50 epoch) | 80.189 | 80.189 (50 epoch) | 99.428 | 99.428 (50 epoch) | 17.810 | 17.810 (50 epoch) | 24.995 | 24.995 (50 epoch) | 94.570 | 94.570 (50 epoch) | 15.720 | 15.720 (50 epoch) | 40.765 | 40.765 (50 epoch) | 8.704 | 8.704 (50 epoch) | 14.283 | 14.283 (50 epoch) | | macaroni2 | 82.589 | 82.589 (50 epoch) | 81.196 | 81.196 (50 epoch) | 75.622 | 75.622 (50 epoch) | 99.377 | 99.377 (50 epoch) | 11.272 | 11.272 (50 epoch) | 17.568 | 17.568 (50 epoch) | 95.808 | 95.808 (50 epoch) | 10.086 | 10.086 (50 epoch) | 49.562 | 49.562 (50 epoch) | 5.433 | 5.433 (50 epoch) | 9.630 | 9.630 (50 epoch) | | pcb1 | 95.170 | 95.170 (50 epoch) | 92.928 | 92.928 (50 epoch) | 91.542 | 91.542 (50 epoch) | 99.761 | 99.761 (50 epoch) | 75.837 | 75.837 (50 epoch) | 71.602 | 71.602 (50 epoch) | 93.660 | 93.660 (50 epoch) | 44.333 | 44.333 (50 epoch) | 62.362 | 62.362 (50 epoch) | 30.903 | 30.903 (50 epoch) | 55.766 | 55.766 (50 epoch) | | pcb2 | 92.790 | 92.790 (50 epoch) | 92.126 | 92.126 (50 epoch) | 87.234 | 87.234 (50 epoch) | 98.771 | 98.771 (50 epoch) | 14.130 | 14.130 (50 epoch) | 22.634 | 22.634 (50 epoch) | 87.709 | 87.709 (50 epoch) | 15.482 | 15.482 (50 epoch) | 47.412 | 47.412 (50 epoch) | 8.481 | 8.481 (50 epoch) | 12.761 | 12.761 (50 epoch) | | pcb3 | 92.426 | 92.426 (50 epoch) | 92.968 | 92.968 (50 epoch) | 85.294 | 85.294 (50 epoch) | 98.976 | 98.976 (50 epoch) | 18.037 | 18.037 (50 epoch) | 26.937 | 26.937 (50 epoch) | 90.129 | 90.129 (50 epoch) | 18.893 | 18.893 (50 epoch) | 48.341 | 48.341 (50 epoch) | 10.601 | 10.601 (50 epoch) | 15.565 | 15.565 (50 epoch) | | pcb4 | 99.881 | 99.881 (50 epoch) | 99.885 | 99.885 (50 epoch) | 99.000 | 99.000 (50 epoch) | 98.340 | 98.340 (50 epoch) | 45.472 | 45.472 (50 epoch) | 45.003 | 45.003 (50 epoch) | 87.073 | 87.073 (50 epoch) | 26.466 | 26.466 (50 epoch) | 39.716 | 39.716 (50 epoch) | 16.151 | 16.151 (50 epoch) | 29.035 | 29.035 (50 epoch) | | pipe_fryum | 98.780 | 98.780 (50 epoch) | 99.391 | 99.391 (50 epoch) | 96.585 | 96.585 (50 epoch) | 99.009 | 99.009 (50 epoch) | 52.106 | 52.106 (50 epoch) | 57.743 | 57.743 (50 epoch) | 94.075 | 94.075 (50 epoch) | 33.316 | 33.316 (50 epoch) | 42.769 | 42.769 (50 epoch) | 21.412 | 21.412 (50 epoch) | 40.591 | 40.591 (50 epoch) | | Avg | 93.498 | 93.498 (50 epoch) | 94.147 | 94.147 (50 epoch) | 88.967 | 88.967 (50 epoch) | 98.430 | 98.430 (50 epoch) | 38.519 | 38.519 (50 epoch) | 42.730 | 42.730 (50 epoch) | 90.733 | 90.733 (50 epoch) | 25.082 | 25.082 (50 epoch) | 46.179 | 46.179 (50 epoch) | 15.609 | 15.609 (50 epoch) | 28.648 | 28.648 (50 epoch) | 2024-04-02 05:48:12,985 - ==> Total time: 4:48:07 Eta: 3 days, 19:14:21 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 05:48:49,724 - Train: 5.01% [27100/541000] [50.1/1000.0] [batch_t 0.608 (0.687)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 05:49:50,550 - Train: 5.03% [27200/541000] [50.3/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 05:50:51,372 - Train: 5.05% [27300/541000] [50.5/1000.0] [batch_t 0.602 (0.608)] [data_t 0.002] [optim_t 0.600] [lr 0.005000] 2024-04-02 05:51:52,257 - Train: 5.06% [27400/541000] [50.6/1000.0] [batch_t 0.605 (0.609)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 05:52:53,087 - Train: 5.08% [27500/541000] [50.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 05:53:48,478 - ==> Total time: 4:53:42 Eta: 3 days, 19:05:24 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 05:53:58,125 - Train: 5.10% [27600/541000] [51.0/1000.0] [batch_t 0.611 (0.850)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 05:54:59,470 - Train: 5.12% [27700/541000] [51.2/1000.0] [batch_t 0.609 (0.613)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 05:56:00,333 - Train: 5.14% [27800/541000] [51.4/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 05:57:01,160 - Train: 5.16% [27900/541000] [51.6/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 05:58:02,043 - Train: 5.18% [28000/541000] [51.8/1000.0] [batch_t 0.609 (0.609)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 05:59:02,872 - Train: 5.19% [28100/541000] [51.9/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 05:59:22,336 - ==> Total time: 4:59:16 Eta: 3 days, 18:56:05 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 06:00:06,928 - Train: 5.21% [28200/541000] [52.1/1000.0] [batch_t 0.610 (0.625)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-02 06:01:07,934 - Train: 5.23% [28300/541000] [52.3/1000.0] [batch_t 0.609 (0.610)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 06:02:08,774 - Train: 5.25% [28400/541000] [52.5/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 06:03:09,600 - Train: 5.27% [28500/541000] [52.7/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 06:04:10,479 - Train: 5.29% [28600/541000] [52.9/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 06:04:54,896 - ==> Total time: 5:04:49 Eta: 3 days, 18:46:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 06:05:13,831 - Train: 5.30% [28700/541000] [53.0/1000.0] [batch_t 0.607 (0.633)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 06:06:14,796 - Train: 5.32% [28800/541000] [53.2/1000.0] [batch_t 0.609 (0.610)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 06:07:15,634 - Train: 5.34% [28900/541000] [53.4/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 06:08:16,453 - Train: 5.36% [29000/541000] [53.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 06:09:17,262 - Train: 5.38% [29100/541000] [53.8/1000.0] [batch_t 0.610 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 06:10:18,105 - Train: 5.40% [29200/541000] [54.0/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 06:10:26,625 - ==> Total time: 5:10:21 Eta: 3 days, 18:36:53 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 06:11:21,488 - Train: 5.42% [29300/541000] [54.2/1000.0] [batch_t 0.610 (0.615)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 06:12:22,913 - Train: 5.43% [29400/541000] [54.3/1000.0] [batch_t 0.609 (0.614)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-02 06:13:23,760 - Train: 5.45% [29500/541000] [54.5/1000.0] [batch_t 0.605 (0.608)] [data_t 0.003] [optim_t 0.602] [lr 0.005000] 2024-04-02 06:14:24,583 - Train: 5.47% [29600/541000] [54.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 06:15:25,426 - Train: 5.49% [29700/541000] [54.9/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 06:15:58,901 - ==> Total time: 5:15:53 Eta: 3 days, 18:27:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 06:16:28,679 - Train: 5.51% [29800/541000] [55.1/1000.0] [batch_t 0.609 (0.622)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 06:17:29,639 - Train: 5.53% [29900/541000] [55.3/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 06:18:30,493 - Train: 5.55% [30000/541000] [55.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 06:19:31,322 - Train: 5.56% [30100/541000] [55.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 06:20:32,153 - Train: 5.58% [30200/541000] [55.8/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 06:21:30,578 - ==> Total time: 5:21:25 Eta: 3 days, 18:18:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 06:21:35,304 - Train: 5.60% [30300/541000] [56.0/1000.0] [batch_t 0.607 (0.770)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 06:22:36,145 - Train: 5.62% [30400/541000] [56.2/1000.0] [batch_t 0.610 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-02 06:23:37,049 - Train: 5.64% [30500/541000] [56.4/1000.0] [batch_t 0.611 (0.609)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 06:24:37,863 - Train: 5.66% [30600/541000] [56.6/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 06:25:38,675 - Train: 5.67% [30700/541000] [56.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 06:26:39,513 - Train: 5.69% [30800/541000] [56.9/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 06:27:02,030 - ==> Total time: 5:26:56 Eta: 3 days, 18:08:52 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 06:27:46,187 - Train: 5.71% [30900/541000] [57.1/1000.0] [batch_t 0.606 (0.630)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 06:28:46,989 - Train: 5.73% [31000/541000] [57.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 06:29:47,947 - Train: 5.75% [31100/541000] [57.5/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 06:30:48,807 - Train: 5.77% [31200/541000] [57.7/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 06:31:49,828 - Train: 5.79% [31300/541000] [57.9/1000.0] [batch_t 0.606 (0.610)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 06:32:37,266 - ==> Total time: 5:32:31 Eta: 3 days, 18:00:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 06:32:53,119 - Train: 5.80% [31400/541000] [58.0/1000.0] [batch_t 0.610 (0.635)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 06:33:53,927 - Train: 5.82% [31500/541000] [58.2/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 06:34:54,755 - Train: 5.84% [31600/541000] [58.4/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 06:35:55,710 - Train: 5.86% [31700/541000] [58.6/1000.0] [batch_t 0.612 (0.609)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 06:36:56,760 - Train: 5.88% [31800/541000] [58.8/1000.0] [batch_t 0.607 (0.610)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 06:37:57,601 - Train: 5.90% [31900/541000] [59.0/1000.0] [batch_t 0.610 (0.608)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 06:38:09,183 - ==> Total time: 5:38:03 Eta: 3 days, 17:51:46 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 06:39:01,021 - Train: 5.91% [32000/541000] [59.1/1000.0] [batch_t 0.609 (0.617)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 06:40:01,823 - Train: 5.93% [32100/541000] [59.3/1000.0] [batch_t 0.605 (0.608)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 06:41:02,774 - Train: 5.95% [32200/541000] [59.5/1000.0] [batch_t 0.605 (0.609)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 06:42:03,589 - Train: 5.97% [32300/541000] [59.7/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 06:43:04,698 - Train: 5.99% [32400/541000] [59.9/1000.0] [batch_t 0.610 (0.611)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 06:43:41,192 - ==> Total time: 5:43:35 Eta: 3 days, 17:42:58 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 06:44:07,992 - Train: 6.01% [32500/541000] [60.1/1000.0] [batch_t 0.609 (0.621)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 06:45:08,809 - Train: 6.03% [32600/541000] [60.3/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 06:46:09,666 - Train: 6.04% [32700/541000] [60.4/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 06:47:10,629 - Train: 6.06% [32800/541000] [60.6/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 06:48:11,465 - Train: 6.08% [32900/541000] [60.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 06:49:12,317 - Train: 6.10% [33000/541000] [61.0/1000.0] [batch_t 0.605 (0.608)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 06:49:12,929 - ==> Total time: 5:49:07 Eta: 3 days, 17:34:12 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 06:50:15,637 - Train: 6.12% [33100/541000] [61.2/1000.0] [batch_t 0.609 (0.614)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 06:51:16,447 - Train: 6.14% [33200/541000] [61.4/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 06:52:17,438 - Train: 6.16% [33300/541000] [61.6/1000.0] [batch_t 0.606 (0.610)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 06:53:18,274 - Train: 6.17% [33400/541000] [61.7/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 06:54:19,114 - Train: 6.19% [33500/541000] [61.9/1000.0] [batch_t 0.605 (0.608)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 06:54:44,682 - ==> Total time: 5:54:39 Eta: 3 days, 17:25:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 06:55:22,481 - Train: 6.21% [33600/541000] [62.1/1000.0] [batch_t 0.609 (0.617)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 06:56:23,340 - Train: 6.23% [33700/541000] [62.3/1000.0] [batch_t 0.613 (0.608)] [data_t 0.002] [optim_t 0.610] [lr 0.005000] 2024-04-02 06:57:24,721 - Train: 6.25% [33800/541000] [62.5/1000.0] [batch_t 0.610 (0.614)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 06:58:25,724 - Train: 6.27% [33900/541000] [62.7/1000.0] [batch_t 0.605 (0.610)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 06:59:26,574 - Train: 6.28% [34000/541000] [62.8/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 07:00:17,060 - ==> Total time: 6:00:11 Eta: 3 days, 17:17:08 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 07:00:29,908 - Train: 6.30% [34100/541000] [63.0/1000.0] [batch_t 0.609 (0.638)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 07:01:30,736 - Train: 6.32% [34200/541000] [63.2/1000.0] [batch_t 0.610 (0.608)] [data_t 0.003] [optim_t 0.608] [lr 0.005000] 2024-04-02 07:02:32,971 - Train: 6.34% [34300/541000] [63.4/1000.0] [batch_t 0.607 (0.622)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 07:03:33,953 - Train: 6.36% [34400/541000] [63.6/1000.0] [batch_t 0.610 (0.610)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 07:04:34,984 - Train: 6.38% [34500/541000] [63.8/1000.0] [batch_t 0.609 (0.610)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 07:05:35,812 - Train: 6.40% [34600/541000] [64.0/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-02 07:05:50,415 - ==> Total time: 6:05:44 Eta: 3 days, 17:09:03 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 07:06:38,925 - Train: 6.41% [34700/541000] [64.1/1000.0] [batch_t 0.608 (0.615)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 07:07:39,764 - Train: 6.43% [34800/541000] [64.3/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 07:08:40,589 - Train: 6.45% [34900/541000] [64.5/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 07:09:41,516 - Train: 6.47% [35000/541000] [64.7/1000.0] [batch_t 0.610 (0.609)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 07:10:42,344 - Train: 6.49% [35100/541000] [64.9/1000.0] [batch_t 0.605 (0.608)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-02 07:11:21,878 - ==> Total time: 6:11:16 Eta: 3 days, 17:00:36 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 07:11:45,542 - Train: 6.51% [35200/541000] [65.1/1000.0] [batch_t 0.605 (0.630)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 07:12:46,363 - Train: 6.52% [35300/541000] [65.2/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 07:13:47,705 - Train: 6.54% [35400/541000] [65.4/1000.0] [batch_t 0.610 (0.613)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 07:14:48,522 - Train: 6.56% [35500/541000] [65.6/1000.0] [batch_t 0.611 (0.608)] [data_t 0.003] [optim_t 0.608] [lr 0.005000] 2024-04-02 07:15:49,499 - Train: 6.58% [35600/541000] [65.8/1000.0] [batch_t 0.605 (0.610)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 07:16:50,303 - Train: 6.60% [35700/541000] [66.0/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 07:16:53,958 - ==> Total time: 6:16:48 Eta: 3 days, 16:52:23 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 07:17:53,723 - Train: 6.62% [35800/541000] [66.2/1000.0] [batch_t 0.608 (0.615)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 07:18:54,546 - Train: 6.64% [35900/541000] [66.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 07:19:55,381 - Train: 6.65% [36000/541000] [66.5/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 07:20:56,356 - Train: 6.67% [36100/541000] [66.7/1000.0] [batch_t 0.608 (0.610)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 07:21:57,199 - Train: 6.69% [36200/541000] [66.9/1000.0] [batch_t 0.610 (0.608)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 07:22:25,807 - ==> Total time: 6:22:20 Eta: 3 days, 16:44:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 07:23:00,649 - Train: 6.71% [36300/541000] [67.1/1000.0] [batch_t 0.610 (0.621)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 07:24:01,498 - Train: 6.73% [36400/541000] [67.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 07:25:02,353 - Train: 6.75% [36500/541000] [67.5/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 07:26:03,170 - Train: 6.77% [36600/541000] [67.7/1000.0] [batch_t 0.610 (0.608)] [data_t 0.003] [optim_t 0.608] [lr 0.005000] 2024-04-02 07:27:04,156 - Train: 6.78% [36700/541000] [67.8/1000.0] [batch_t 0.610 (0.610)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 07:27:57,705 - ==> Total time: 6:27:52 Eta: 3 days, 16:36:05 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 07:28:07,771 - Train: 6.80% [36800/541000] [68.0/1000.0] [batch_t 0.609 (0.683)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 07:29:08,618 - Train: 6.82% [36900/541000] [68.2/1000.0] [batch_t 0.612 (0.608)] [data_t 0.003] [optim_t 0.609] [lr 0.005000] 2024-04-02 07:30:09,472 - Train: 6.84% [37000/541000] [68.4/1000.0] [batch_t 0.611 (0.608)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 07:31:10,302 - Train: 6.86% [37100/541000] [68.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 07:32:11,256 - Train: 6.88% [37200/541000] [68.8/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 07:33:12,351 - Train: 6.89% [37300/541000] [68.9/1000.0] [batch_t 0.606 (0.611)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 07:33:29,990 - ==> Total time: 6:33:24 Eta: 3 days, 16:28:08 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 07:34:15,630 - Train: 6.91% [37400/541000] [69.1/1000.0] [batch_t 0.607 (0.617)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 07:35:16,488 - Train: 6.93% [37500/541000] [69.3/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 07:36:17,326 - Train: 6.95% [37600/541000] [69.5/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 07:37:18,175 - Train: 6.97% [37700/541000] [69.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 07:38:19,887 - Train: 6.99% [37800/541000] [69.9/1000.0] [batch_t 0.608 (0.617)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 07:39:02,575 - ==> Total time: 6:38:57 Eta: 3 days, 16:20:20 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 07:39:25,901 - Train: 7.01% [37900/541000] [70.1/1000.0] [batch_t 0.608 (0.632)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 07:40:26,697 - Train: 7.02% [38000/541000] [70.2/1000.0] [batch_t 0.610 (0.608)] [data_t 0.003] [optim_t 0.608] [lr 0.005000] 2024-04-02 07:41:27,526 - Train: 7.04% [38100/541000] [70.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 07:42:28,371 - Train: 7.06% [38200/541000] [70.6/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 07:43:29,197 - Train: 7.08% [38300/541000] [70.8/1000.0] [batch_t 0.610 (0.608)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 07:44:30,094 - Train: 7.10% [38400/541000] [71.0/1000.0] [batch_t 0.610 (0.609)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 07:44:36,784 - ==> Total time: 6:44:31 Eta: 3 days, 16:12:57 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 07:45:33,912 - Train: 7.12% [38500/541000] [71.2/1000.0] [batch_t 0.607 (0.622)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 07:46:34,774 - Train: 7.13% [38600/541000] [71.3/1000.0] [batch_t 0.607 (0.609)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 07:47:35,608 - Train: 7.15% [38700/541000] [71.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 07:48:36,456 - Train: 7.17% [38800/541000] [71.7/1000.0] [batch_t 0.605 (0.608)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 07:49:37,304 - Train: 7.19% [38900/541000] [71.9/1000.0] [batch_t 0.610 (0.608)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 07:50:08,936 - ==> Total time: 6:50:03 Eta: 3 days, 16:05:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 07:50:41,024 - Train: 7.21% [39000/541000] [72.1/1000.0] [batch_t 0.608 (0.626)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 07:51:41,844 - Train: 7.23% [39100/541000] [72.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 07:52:42,807 - Train: 7.25% [39200/541000] [72.5/1000.0] [batch_t 0.609 (0.610)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 07:53:43,643 - Train: 7.26% [39300/541000] [72.6/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 07:54:44,491 - Train: 7.28% [39400/541000] [72.8/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 07:55:41,052 - ==> Total time: 6:55:35 Eta: 3 days, 15:57:26 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 07:55:48,489 - Train: 7.30% [39500/541000] [73.0/1000.0] [batch_t 0.608 (0.795)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 07:56:49,317 - Train: 7.32% [39600/541000] [73.2/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 07:57:50,274 - Train: 7.34% [39700/541000] [73.4/1000.0] [batch_t 0.609 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 07:58:51,386 - Train: 7.36% [39800/541000] [73.6/1000.0] [batch_t 0.608 (0.611)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 07:59:52,225 - Train: 7.38% [39900/541000] [73.8/1000.0] [batch_t 0.605 (0.608)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 08:00:53,042 - Train: 7.39% [40000/541000] [73.9/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 08:01:13,724 - ==> Total time: 7:01:08 Eta: 3 days, 15:49:53 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 08:01:56,537 - Train: 7.41% [40100/541000] [74.1/1000.0] [batch_t 0.609 (0.620)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 08:02:57,507 - Train: 7.43% [40200/541000] [74.3/1000.0] [batch_t 0.605 (0.610)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 08:03:58,368 - Train: 7.45% [40300/541000] [74.5/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-02 08:04:59,267 - Train: 7.47% [40400/541000] [74.7/1000.0] [batch_t 0.610 (0.609)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-02 08:06:00,150 - Train: 7.49% [40500/541000] [74.9/1000.0] [batch_t 0.609 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 08:06:45,822 - ==> Total time: 7:06:40 Eta: 3 days, 15:42:16 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 08:07:03,513 - Train: 7.50% [40600/541000] [75.0/1000.0] [batch_t 0.609 (0.642)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-02 08:08:04,589 - Train: 7.52% [40700/541000] [75.2/1000.0] [batch_t 0.610 (0.611)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-02 08:09:05,410 - Train: 7.54% [40800/541000] [75.4/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 08:10:06,232 - Train: 7.56% [40900/541000] [75.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 08:11:07,080 - Train: 7.58% [41000/541000] [75.8/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 08:12:08,261 - Train: 7.60% [41100/541000] [76.0/1000.0] [batch_t 0.611 (0.612)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 08:12:18,001 - ==> Total time: 7:12:12 Eta: 3 days, 15:34:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 08:13:12,061 - Train: 7.62% [41200/541000] [76.2/1000.0] [batch_t 0.607 (0.624)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 08:14:12,855 - Train: 7.63% [41300/541000] [76.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 08:15:13,661 - Train: 7.65% [41400/541000] [76.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 08:16:14,774 - Train: 7.67% [41500/541000] [76.7/1000.0] [batch_t 0.608 (0.611)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 08:17:15,580 - Train: 7.69% [41600/541000] [76.9/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 08:17:50,243 - ==> Total time: 7:17:44 Eta: 3 days, 15:27:15 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 08:18:19,281 - Train: 7.71% [41700/541000] [77.1/1000.0] [batch_t 0.610 (0.625)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 08:19:20,234 - Train: 7.73% [41800/541000] [77.3/1000.0] [batch_t 0.609 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 08:20:21,083 - Train: 7.74% [41900/541000] [77.4/1000.0] [batch_t 0.610 (0.608)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 08:21:21,893 - Train: 7.76% [42000/541000] [77.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 08:22:22,730 - Train: 7.78% [42100/541000] [77.8/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 08:23:22,343 - ==> Total time: 7:23:16 Eta: 3 days, 15:19:47 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 08:23:26,562 - Train: 7.80% [42200/541000] [78.0/1000.0] [batch_t 0.610 (0.986)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 08:24:27,382 - Train: 7.82% [42300/541000] [78.2/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 08:25:28,322 - Train: 7.84% [42400/541000] [78.4/1000.0] [batch_t 0.606 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 08:26:29,192 - Train: 7.86% [42500/541000] [78.6/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-02 08:27:30,067 - Train: 7.87% [42600/541000] [78.7/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 08:28:31,493 - Train: 7.89% [42700/541000] [78.9/1000.0] [batch_t 0.608 (0.614)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 08:28:55,204 - ==> Total time: 7:28:49 Eta: 3 days, 15:12:31 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 08:29:34,943 - Train: 7.91% [42800/541000] [79.1/1000.0] [batch_t 0.608 (0.621)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 08:30:35,795 - Train: 7.93% [42900/541000] [79.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 08:31:36,747 - Train: 7.95% [43000/541000] [79.5/1000.0] [batch_t 0.609 (0.609)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 08:32:37,578 - Train: 7.97% [43100/541000] [79.7/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 08:33:38,387 - Train: 7.99% [43200/541000] [79.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 08:34:27,056 - ==> Total time: 7:34:21 Eta: 3 days, 15:05:07 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 08:34:42,396 - Train: 8.00% [43300/541000] [80.0/1000.0] [batch_t 0.609 (0.677)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 08:35:43,190 - Train: 8.02% [43400/541000] [80.2/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 08:36:44,028 - Train: 8.04% [43500/541000] [80.4/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 08:37:44,964 - Train: 8.06% [43600/541000] [80.6/1000.0] [batch_t 0.606 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 08:38:45,737 - Train: 8.08% [43700/541000] [80.8/1000.0] [batch_t 0.610 (0.608)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 08:39:47,052 - Train: 8.10% [43800/541000] [81.0/1000.0] [batch_t 0.608 (0.613)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 08:39:59,826 - ==> Total time: 7:39:54 Eta: 3 days, 14:57:55 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 08:40:53,856 - Train: 8.11% [43900/541000] [81.1/1000.0] [batch_t 0.606 (0.660)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 08:41:54,633 - Train: 8.13% [44000/541000] [81.3/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 08:42:55,457 - Train: 8.15% [44100/541000] [81.5/1000.0] [batch_t 0.610 (0.608)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 08:43:56,388 - Train: 8.17% [44200/541000] [81.7/1000.0] [batch_t 0.609 (0.609)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 08:44:57,158 - Train: 8.19% [44300/541000] [81.9/1000.0] [batch_t 0.610 (0.608)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 08:45:34,839 - ==> Total time: 7:45:29 Eta: 3 days, 14:51:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 08:46:00,524 - Train: 8.21% [44400/541000] [82.1/1000.0] [batch_t 0.609 (0.627)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 08:47:01,315 - Train: 8.23% [44500/541000] [82.3/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 08:48:02,096 - Train: 8.24% [44600/541000] [82.4/1000.0] [batch_t 0.610 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 08:49:02,868 - Train: 8.26% [44700/541000] [82.6/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 08:50:03,786 - Train: 8.28% [44800/541000] [82.8/1000.0] [batch_t 0.607 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 08:51:04,576 - Train: 8.30% [44900/541000] [83.0/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 08:51:06,403 - ==> Total time: 7:51:00 Eta: 3 days, 14:43:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 08:52:08,243 - Train: 8.32% [45000/541000] [83.2/1000.0] [batch_t 0.606 (0.615)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 08:53:11,635 - Train: 8.34% [45100/541000] [83.4/1000.0] [batch_t 0.605 (0.634)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 08:54:15,724 - Train: 8.35% [45200/541000] [83.5/1000.0] [batch_t 0.610 (0.641)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 08:55:16,603 - Train: 8.37% [45300/541000] [83.7/1000.0] [batch_t 0.610 (0.609)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 08:56:17,394 - Train: 8.39% [45400/541000] [83.9/1000.0] [batch_t 0.610 (0.608)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 08:56:44,159 - ==> Total time: 7:56:38 Eta: 3 days, 14:37:40 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 08:57:20,439 - Train: 8.41% [45500/541000] [84.1/1000.0] [batch_t 0.606 (0.616)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 08:58:21,238 - Train: 8.43% [45600/541000] [84.3/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 08:59:22,052 - Train: 8.45% [45700/541000] [84.5/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 09:00:22,870 - Train: 8.47% [45800/541000] [84.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 09:01:23,790 - Train: 8.48% [45900/541000] [84.8/1000.0] [batch_t 0.609 (0.609)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 09:02:15,470 - ==> Total time: 8:02:09 Eta: 3 days, 14:30:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 09:02:27,438 - Train: 8.50% [46000/541000] [85.0/1000.0] [batch_t 0.607 (0.648)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 09:03:28,264 - Train: 8.52% [46100/541000] [85.2/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 09:04:29,110 - Train: 8.54% [46200/541000] [85.4/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 09:05:29,930 - Train: 8.56% [46300/541000] [85.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 09:06:30,738 - Train: 8.58% [46400/541000] [85.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 09:07:31,627 - Train: 8.60% [46500/541000] [86.0/1000.0] [batch_t 0.607 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 09:07:47,426 - ==> Total time: 8:07:41 Eta: 3 days, 14:23:12 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 09:08:34,896 - Train: 8.61% [46600/541000] [86.1/1000.0] [batch_t 0.608 (0.617)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 09:09:35,743 - Train: 8.63% [46700/541000] [86.3/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 09:10:36,553 - Train: 8.65% [46800/541000] [86.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 09:11:37,806 - Train: 8.67% [46900/541000] [86.7/1000.0] [batch_t 0.606 (0.612)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 09:12:38,664 - Train: 8.69% [47000/541000] [86.9/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 09:13:19,430 - ==> Total time: 8:13:13 Eta: 3 days, 14:16:05 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 09:13:42,385 - Train: 8.71% [47100/541000] [87.1/1000.0] [batch_t 0.608 (0.627)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 09:14:43,199 - Train: 8.72% [47200/541000] [87.2/1000.0] [batch_t 0.610 (0.608)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 09:15:44,034 - Train: 8.74% [47300/541000] [87.4/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 09:16:44,864 - Train: 8.76% [47400/541000] [87.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 09:17:45,679 - Train: 8.78% [47500/541000] [87.8/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 09:18:46,481 - Train: 8.80% [47600/541000] [88.0/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 09:18:51,349 - ==> Total time: 8:18:45 Eta: 3 days, 14:09:00 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 09:19:50,368 - Train: 8.82% [47700/541000] [88.2/1000.0] [batch_t 0.609 (0.616)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 09:20:51,156 - Train: 8.84% [47800/541000] [88.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 09:21:51,980 - Train: 8.85% [47900/541000] [88.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 09:22:52,779 - Train: 8.87% [48000/541000] [88.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 09:23:53,601 - Train: 8.89% [48100/541000] [88.9/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 09:24:23,401 - ==> Total time: 8:24:17 Eta: 3 days, 14:01:57 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 09:24:56,931 - Train: 8.91% [48200/541000] [89.1/1000.0] [batch_t 0.607 (0.621)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 09:25:57,884 - Train: 8.93% [48300/541000] [89.3/1000.0] [batch_t 0.611 (0.609)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 09:26:58,763 - Train: 8.95% [48400/541000] [89.5/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 09:27:59,600 - Train: 8.96% [48500/541000] [89.6/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 09:29:00,421 - Train: 8.98% [48600/541000] [89.8/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 09:29:55,130 - ==> Total time: 8:29:49 Eta: 3 days, 13:54:54 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 09:30:03,760 - Train: 9.00% [48700/541000] [90.0/1000.0] [batch_t 0.607 (0.650)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 09:31:04,609 - Train: 9.02% [48800/541000] [90.2/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 09:32:05,452 - Train: 9.04% [48900/541000] [90.4/1000.0] [batch_t 0.610 (0.608)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 09:33:06,423 - Train: 9.06% [49000/541000] [90.6/1000.0] [batch_t 0.608 (0.610)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 09:34:07,280 - Train: 9.08% [49100/541000] [90.8/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 09:35:08,059 - Train: 9.09% [49200/541000] [90.9/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 09:35:26,921 - ==> Total time: 8:35:21 Eta: 3 days, 13:47:53 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 09:36:12,154 - Train: 9.11% [49300/541000] [91.1/1000.0] [batch_t 0.608 (0.625)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 09:37:13,008 - Train: 9.13% [49400/541000] [91.3/1000.0] [batch_t 0.610 (0.608)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 09:38:13,861 - Train: 9.15% [49500/541000] [91.5/1000.0] [batch_t 0.605 (0.608)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 09:39:14,686 - Train: 9.17% [49600/541000] [91.7/1000.0] [batch_t 0.612 (0.608)] [data_t 0.002] [optim_t 0.610] [lr 0.005000] 2024-04-02 09:40:15,631 - Train: 9.19% [49700/541000] [91.9/1000.0] [batch_t 0.607 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 09:40:59,481 - ==> Total time: 8:40:53 Eta: 3 days, 13:41:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 09:41:19,035 - Train: 9.21% [49800/541000] [92.1/1000.0] [batch_t 0.608 (0.633)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-02 09:42:19,886 - Train: 9.22% [49900/541000] [92.2/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 09:43:20,737 - Train: 9.24% [50000/541000] [92.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-02 09:44:21,588 - Train: 9.26% [50100/541000] [92.6/1000.0] [batch_t 0.610 (0.608)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 09:45:22,536 - Train: 9.28% [50200/541000] [92.8/1000.0] [batch_t 0.606 (0.609)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 09:46:23,339 - Train: 9.30% [50300/541000] [93.0/1000.0] [batch_t 0.605 (0.608)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 09:46:31,259 - ==> Total time: 8:46:25 Eta: 3 days, 13:34:05 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 09:47:26,918 - Train: 9.32% [50400/541000] [93.2/1000.0] [batch_t 0.606 (0.615)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 09:48:27,766 - Train: 9.33% [50500/541000] [93.3/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 09:49:28,586 - Train: 9.35% [50600/541000] [93.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 09:50:29,606 - Train: 9.37% [50700/541000] [93.7/1000.0] [batch_t 0.609 (0.610)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 09:51:30,435 - Train: 9.39% [50800/541000] [93.9/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 09:52:03,282 - ==> Total time: 8:51:57 Eta: 3 days, 13:27:12 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 09:52:36,175 - Train: 9.41% [50900/541000] [94.1/1000.0] [batch_t 0.607 (0.625)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 09:53:37,026 - Train: 9.43% [51000/541000] [94.3/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 09:54:37,908 - Train: 9.45% [51100/541000] [94.5/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 09:55:40,303 - Train: 9.46% [51200/541000] [94.6/1000.0] [batch_t 0.609 (0.624)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-02 09:56:45,954 - Train: 9.48% [51300/541000] [94.8/1000.0] [batch_t 0.609 (0.656)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 09:57:45,807 - ==> Total time: 8:57:40 Eta: 3 days, 13:22:01 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 09:57:53,210 - Train: 9.50% [51400/541000] [95.0/1000.0] [batch_t 0.609 (0.992)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 09:58:54,053 - Train: 9.52% [51500/541000] [95.2/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 09:59:56,257 - Train: 9.54% [51600/541000] [95.4/1000.0] [batch_t 0.607 (0.622)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-02 10:00:57,219 - Train: 9.56% [51700/541000] [95.6/1000.0] [batch_t 0.609 (0.610)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 10:01:58,630 - Train: 9.57% [51800/541000] [95.7/1000.0] [batch_t 0.610 (0.614)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 10:03:28,788 - Train: 9.59% [51900/541000] [95.9/1000.0] [batch_t 0.608 (0.901)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 10:03:54,254 - ==> Total time: 9:03:48 Eta: 3 days, 13:20:53 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 10:04:50,266 - Train: 9.61% [52000/541000] [96.1/1000.0] [batch_t 0.608 (0.819)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 10:06:12,216 - Train: 9.63% [52100/541000] [96.3/1000.0] [batch_t 0.608 (0.819)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 10:07:15,900 - Train: 9.65% [52200/541000] [96.5/1000.0] [batch_t 0.606 (0.637)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 10:08:39,462 - Train: 9.67% [52300/541000] [96.7/1000.0] [batch_t 0.607 (0.836)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 10:09:50,414 - Train: 9.69% [52400/541000] [96.9/1000.0] [batch_t 0.609 (0.708)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 10:10:43,230 - ==> Total time: 9:10:37 Eta: 3 days, 13:25:56 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 10:11:02,331 - Train: 9.70% [52500/541000] [97.0/1000.0] [batch_t 0.607 (0.683)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 10:12:25,488 - Train: 9.72% [52600/541000] [97.2/1000.0] [batch_t 2.242 (0.831)] [data_t 1.637] [optim_t 0.604] [lr 0.005000] 2024-04-02 10:13:58,291 - Train: 9.74% [52700/541000] [97.4/1000.0] [batch_t 1.554 (0.928)] [data_t 0.950] [optim_t 0.604] [lr 0.005000] 2024-04-02 10:15:03,444 - Train: 9.76% [52800/541000] [97.6/1000.0] [batch_t 0.606 (0.651)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 10:16:47,200 - Train: 9.78% [52900/541000] [97.8/1000.0] [batch_t 0.608 (1.037)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 10:17:52,575 - Train: 9.80% [53000/541000] [98.0/1000.0] [batch_t 0.607 (0.654)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 10:18:03,514 - ==> Total time: 9:17:57 Eta: 3 days, 13:35:33 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 10:19:00,329 - Train: 9.82% [53100/541000] [98.2/1000.0] [batch_t 0.606 (0.625)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 10:20:36,150 - Train: 9.83% [53200/541000] [98.3/1000.0] [batch_t 2.987 (0.958)] [data_t 2.380] [optim_t 0.607] [lr 0.005000] 2024-04-02 10:22:36,384 - Train: 9.85% [53300/541000] [98.5/1000.0] [batch_t 0.957 (1.202)] [data_t 0.352] [optim_t 0.604] [lr 0.005000] 2024-04-02 10:24:04,863 - Train: 9.87% [53400/541000] [98.7/1000.0] [batch_t 3.517 (0.885)] [data_t 2.910] [optim_t 0.607] [lr 0.005000] 2024-04-02 10:26:32,527 - Train: 9.89% [53500/541000] [98.9/1000.0] [batch_t 0.607 (1.477)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 10:27:41,254 - ==> Total time: 9:27:35 Eta: 3 days, 14:05:41 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 10:28:16,696 - Train: 9.91% [53600/541000] [99.1/1000.0] [batch_t 0.609 (0.810)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 10:29:48,115 - Train: 9.93% [53700/541000] [99.3/1000.0] [batch_t 0.608 (0.914)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 10:31:28,971 - Train: 9.94% [53800/541000] [99.4/1000.0] [batch_t 2.427 (1.008)] [data_t 1.823] [optim_t 0.604] [lr 0.005000] 2024-04-02 10:33:59,041 - Train: 9.96% [53900/541000] [99.6/1000.0] [batch_t 0.608 (1.501)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 10:34:59,832 - Train: 9.98% [54000/541000] [99.8/1000.0] [batch_t 0.610 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 10:36:02,920 - Train: 10.00% [54100/541000] [100.0/1000.0] [batch_t 0.608 (0.631)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 10:36:52,509 - Test: 36.76% [50/136] [batch_t 0.219 (0.948)] 2024-04-02 10:37:40,558 - Test: 73.53% [100/136] [batch_t 0.215 (0.955)] 2024-04-02 10:38:46,441 - Test: 100.00% [136/136] [batch_t 0.052 (1.186)] 2024-04-02 10:39:34,293 - ==> Metric Time for candle : 0.002 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.298 (mAUROC_px) 7.956 (mAP_px) 0.920 (mF1_max_px) 25.070 (mAUPRO_px) 0.315 (mF1_px_0.2_0.8_0.1) 0.313 (mAcc_px_0.2_0.8_0.1) 0.313 (mIoU_px_0.2_0.8_0.1) 0.909 (mIoU_max_px) 2024-04-02 10:40:11,034 - ==> Metric Time for capsules : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 8.098 (mAUROC_px) 6.190 (mAP_px) 0.717 (mF1_max_px) 20.078 (mAUPRO_px) 0.247 (mF1_px_0.2_0.8_0.1) 0.246 (mAcc_px_0.2_0.8_0.1) 0.242 (mIoU_px_0.2_0.8_0.1) 0.709 (mIoU_max_px) 2024-04-02 10:40:47,785 - ==> Metric Time for cashew : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 8.537 (mAUROC_px) 6.609 (mAP_px) 0.676 (mF1_max_px) 19.340 (mAUPRO_px) 0.233 (mF1_px_0.2_0.8_0.1) 0.232 (mAcc_px_0.2_0.8_0.1) 0.236 (mIoU_px_0.2_0.8_0.1) 0.681 (mIoU_max_px) 2024-04-02 10:41:23,785 - ==> Metric Time for chewinggum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.571 (mAUROC_px) 5.785 (mAP_px) 0.671 (mF1_max_px) 20.142 (mAUPRO_px) 0.295 (mF1_px_0.2_0.8_0.1) 0.271 (mAcc_px_0.2_0.8_0.1) 0.268 (mIoU_px_0.2_0.8_0.1) 0.789 (mIoU_max_px) 2024-04-02 10:42:03,500 - ==> Metric Time for fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 8.103 (mAUROC_px) 6.018 (mAP_px) 0.742 (mF1_max_px) 23.120 (mAUPRO_px) 0.252 (mF1_px_0.2_0.8_0.1) 0.263 (mAcc_px_0.2_0.8_0.1) 0.256 (mIoU_px_0.2_0.8_0.1) 0.732 (mIoU_max_px) 2024-04-02 10:42:51,740 - ==> Metric Time for macaroni1 : 0.002 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 11.136 (mAUROC_px) 8.338 (mAP_px) 0.988 (mF1_max_px) 25.636 (mAUPRO_px) 0.311 (mF1_px_0.2_0.8_0.1) 0.308 (mAcc_px_0.2_0.8_0.1) 0.311 (mIoU_px_0.2_0.8_0.1) 0.911 (mIoU_max_px) 2024-04-02 10:43:39,460 - ==> Metric Time for macaroni2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.372 (mAUROC_px) 7.943 (mAP_px) 0.946 (mF1_max_px) 26.305 (mAUPRO_px) 0.317 (mF1_px_0.2_0.8_0.1) 0.316 (mAcc_px_0.2_0.8_0.1) 0.319 (mIoU_px_0.2_0.8_0.1) 0.917 (mIoU_max_px) 2024-04-02 10:44:28,040 - ==> Metric Time for pcb1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.585 (mAUROC_px) 8.026 (mAP_px) 0.923 (mF1_max_px) 26.897 (mAUPRO_px) 0.313 (mF1_px_0.2_0.8_0.1) 0.312 (mAcc_px_0.2_0.8_0.1) 0.316 (mIoU_px_0.2_0.8_0.1) 0.918 (mIoU_max_px) 2024-04-02 10:45:16,661 - ==> Metric Time for pcb2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.596 (mAUROC_px) 8.134 (mAP_px) 0.906 (mF1_max_px) 26.839 (mAUPRO_px) 0.317 (mF1_px_0.2_0.8_0.1) 0.315 (mAcc_px_0.2_0.8_0.1) 0.310 (mIoU_px_0.2_0.8_0.1) 0.915 (mIoU_max_px) 2024-04-02 10:46:04,892 - ==> Metric Time for pcb3 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.660 (mAUROC_px) 8.001 (mAP_px) 0.924 (mF1_max_px) 26.495 (mAUPRO_px) 0.310 (mF1_px_0.2_0.8_0.1) 0.314 (mAcc_px_0.2_0.8_0.1) 0.318 (mIoU_px_0.2_0.8_0.1) 0.919 (mIoU_max_px) 2024-04-02 10:46:54,981 - ==> Metric Time for pcb4 : 0.002 (mAUROC_sp_max) 0.001 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.675 (mAUROC_px) 8.108 (mAP_px) 0.923 (mF1_max_px) 28.156 (mAUPRO_px) 0.321 (mF1_px_0.2_0.8_0.1) 0.348 (mAcc_px_0.2_0.8_0.1) 0.322 (mIoU_px_0.2_0.8_0.1) 0.931 (mIoU_max_px) 2024-04-02 10:47:31,546 - ==> Metric Time for pipe_fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.764 (mAUROC_px) 5.808 (mAP_px) 0.688 (mF1_max_px) 20.646 (mAUPRO_px) 0.247 (mF1_px_0.2_0.8_0.1) 0.243 (mAcc_px_0.2_0.8_0.1) 0.239 (mIoU_px_0.2_0.8_0.1) 0.704 (mIoU_max_px) 2024-04-02 10:47:31,567 - | Name | mAUROC_sp_max | mAUROC_sp_max (Max) | mAP_sp_max | mAP_sp_max (Max) | mF1_max_sp_max | mF1_max_sp_max (Max) | mAUROC_px | mAUROC_px (Max) | mAP_px | mAP_px (Max) | mF1_max_px | mF1_max_px (Max) | mAUPRO_px | mAUPRO_px (Max) | mF1_px_0.2_0.8_0.1 | mF1_px_0.2_0.8_0.1 (Max) | mAcc_px_0.2_0.8_0.1 | mAcc_px_0.2_0.8_0.1 (Max) | mIoU_px_0.2_0.8_0.1 | mIoU_px_0.2_0.8_0.1 (Max) | mIoU_max_px | mIoU_max_px (Max) | |:----------:|:---------------:|:---------------------:|:------------:|:------------------:|:----------------:|:----------------------:|:-----------:|:------------------:|:--------:|:------------------:|:------------:|:------------------:|:-----------:|:------------------:|:--------------------:|:--------------------------:|:---------------------:|:---------------------------:|:---------------------:|:---------------------------:|:-------------:|:-------------------:| | candle | 96.750 | 97.100 (50 epoch) | 96.930 | 97.346 (50 epoch) | 90.099 | 91.542 (50 epoch) | 99.012 | 99.012 (100 epoch) | 23.162 | 23.162 (100 epoch) | 32.438 | 32.438 (100 epoch) | 95.489 | 95.763 (50 epoch) | 19.803 | 19.803 (100 epoch) | 36.722 | 36.722 (100 epoch) | 11.357 | 11.357 (100 epoch) | 19.359 | 19.359 (100 epoch) | | capsules | 91.750 | 91.750 (100 epoch) | 94.964 | 94.964 (100 epoch) | 88.780 | 88.780 (100 epoch) | 99.123 | 99.123 (100 epoch) | 61.256 | 61.256 (100 epoch) | 59.766 | 59.766 (100 epoch) | 91.811 | 91.811 (100 epoch) | 33.186 | 33.186 (100 epoch) | 61.994 | 62.576 (50 epoch) | 21.701 | 21.701 (100 epoch) | 42.619 | 42.619 (100 epoch) | | cashew | 94.500 | 94.500 (100 epoch) | 97.332 | 97.332 (100 epoch) | 91.089 | 91.089 (100 epoch) | 94.313 | 94.652 (50 epoch) | 46.765 | 46.784 (50 epoch) | 51.376 | 51.994 (50 epoch) | 87.830 | 88.493 (50 epoch) | 26.061 | 26.577 (50 epoch) | 29.192 | 30.723 (50 epoch) | 16.159 | 16.556 (50 epoch) | 34.568 | 35.130 (50 epoch) | | chewinggum | 97.743 | 97.743 (100 epoch) | 98.928 | 98.928 (100 epoch) | 94.241 | 94.301 (50 epoch) | 98.145 | 98.145 (100 epoch) | 57.478 | 59.823 (50 epoch) | 59.888 | 61.088 (50 epoch) | 79.699 | 79.773 (50 epoch) | 33.887 | 35.748 (50 epoch) | 63.101 | 65.611 (50 epoch) | 22.371 | 23.754 (50 epoch) | 42.743 | 43.976 (50 epoch) | | fryum | 95.160 | 95.160 (100 epoch) | 97.701 | 97.701 (100 epoch) | 90.498 | 90.498 (100 epoch) | 96.900 | 96.900 (100 epoch) | 47.767 | 48.087 (50 epoch) | 51.860 | 51.860 (100 epoch) | 91.622 | 91.622 (100 epoch) | 25.106 | 26.312 (50 epoch) | 26.131 | 27.708 (50 epoch) | 15.500 | 16.200 (50 epoch) | 35.008 | 35.008 (100 epoch) | | macaroni1 | 91.568 | 91.568 (100 epoch) | 89.846 | 89.846 (100 epoch) | 81.592 | 81.592 (100 epoch) | 99.519 | 99.519 (100 epoch) | 17.475 | 17.810 (50 epoch) | 27.607 | 27.607 (100 epoch) | 95.182 | 95.182 (100 epoch) | 14.621 | 15.720 (50 epoch) | 38.317 | 40.765 (50 epoch) | 8.109 | 8.704 (50 epoch) | 16.014 | 16.014 (100 epoch) | | macaroni2 | 81.631 | 82.589 (50 epoch) | 78.009 | 81.196 (50 epoch) | 73.846 | 75.622 (50 epoch) | 99.452 | 99.452 (100 epoch) | 9.195 | 11.272 (50 epoch) | 16.106 | 17.568 (50 epoch) | 96.235 | 96.235 (100 epoch) | 8.249 | 10.086 (50 epoch) | 48.862 | 49.562 (50 epoch) | 4.409 | 5.433 (50 epoch) | 8.758 | 9.630 (50 epoch) | | pcb1 | 95.370 | 95.370 (100 epoch) | 92.969 | 92.969 (100 epoch) | 91.589 | 91.589 (100 epoch) | 99.770 | 99.770 (100 epoch) | 77.101 | 77.101 (100 epoch) | 72.418 | 72.418 (100 epoch) | 92.751 | 93.660 (50 epoch) | 44.590 | 44.590 (100 epoch) | 61.388 | 62.362 (50 epoch) | 31.173 | 31.173 (100 epoch) | 56.761 | 56.761 (100 epoch) | | pcb2 | 94.240 | 94.240 (100 epoch) | 93.727 | 93.727 (100 epoch) | 89.340 | 89.340 (100 epoch) | 98.899 | 98.899 (100 epoch) | 13.296 | 14.130 (50 epoch) | 23.380 | 23.380 (100 epoch) | 89.590 | 89.590 (100 epoch) | 15.650 | 15.650 (100 epoch) | 45.849 | 47.412 (50 epoch) | 8.591 | 8.591 (100 epoch) | 13.237 | 13.237 (100 epoch) | | pcb3 | 93.683 | 93.683 (100 epoch) | 94.115 | 94.115 (100 epoch) | 86.700 | 86.700 (100 epoch) | 99.073 | 99.073 (100 epoch) | 18.325 | 18.325 (100 epoch) | 27.439 | 27.439 (100 epoch) | 89.149 | 90.129 (50 epoch) | 19.567 | 19.567 (100 epoch) | 48.268 | 48.341 (50 epoch) | 11.002 | 11.002 (100 epoch) | 15.901 | 15.901 (100 epoch) | | pcb4 | 99.901 | 99.901 (100 epoch) | 99.901 | 99.901 (100 epoch) | 98.507 | 99.000 (50 epoch) | 98.609 | 98.609 (100 epoch) | 46.993 | 46.993 (100 epoch) | 46.928 | 46.928 (100 epoch) | 87.575 | 87.575 (100 epoch) | 26.907 | 26.907 (100 epoch) | 39.164 | 39.716 (50 epoch) | 16.517 | 16.517 (100 epoch) | 30.658 | 30.658 (100 epoch) | | pipe_fryum | 98.720 | 98.780 (50 epoch) | 99.345 | 99.391 (50 epoch) | 97.030 | 97.030 (100 epoch) | 99.080 | 99.080 (100 epoch) | 53.518 | 53.518 (100 epoch) | 58.465 | 58.465 (100 epoch) | 95.085 | 95.085 (100 epoch) | 34.338 | 34.338 (100 epoch) | 43.014 | 43.014 (100 epoch) | 22.178 | 22.178 (100 epoch) | 41.308 | 41.308 (100 epoch) | | Avg | 94.251 | 94.251 (100 epoch) | 94.481 | 94.481 (100 epoch) | 89.443 | 89.443 (100 epoch) | 98.491 | 98.491 (100 epoch) | 39.361 | 39.361 (100 epoch) | 43.973 | 43.973 (100 epoch) | 91.002 | 91.002 (100 epoch) | 25.164 | 25.164 (100 epoch) | 45.167 | 46.179 (50 epoch) | 15.755 | 15.755 (100 epoch) | 29.744 | 29.744 (100 epoch) | 2024-04-02 10:47:31,906 - ==> Total time: 9:47:26 Eta: 3 days, 16:06:57 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 10:49:59,640 - Train: 10.02% [54200/541000] [100.2/1000.0] [batch_t 1.435 (1.448)] [data_t 0.003] [optim_t 1.432] [lr 0.005000] 2024-04-02 10:52:22,829 - Train: 10.04% [54300/541000] [100.4/1000.0] [batch_t 1.445 (1.432)] [data_t 0.002] [optim_t 1.443] [lr 0.005000] 2024-04-02 10:54:46,089 - Train: 10.06% [54400/541000] [100.6/1000.0] [batch_t 1.440 (1.433)] [data_t 0.003] [optim_t 1.437] [lr 0.005000] 2024-04-02 10:57:09,389 - Train: 10.07% [54500/541000] [100.7/1000.0] [batch_t 1.428 (1.433)] [data_t 0.002] [optim_t 1.425] [lr 0.005000] 2024-04-02 10:59:32,737 - Train: 10.09% [54600/541000] [100.9/1000.0] [batch_t 1.441 (1.433)] [data_t 0.002] [optim_t 1.439] [lr 0.005000] 2024-04-02 11:00:31,504 - ==> Total time: 10:00:25 Eta: 3 days, 17:04:26 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 11:02:01,727 - Train: 10.11% [54700/541000] [101.1/1000.0] [batch_t 1.429 (1.471)] [data_t 0.002] [optim_t 1.426] [lr 0.005000] 2024-04-02 11:04:24,789 - Train: 10.13% [54800/541000] [101.3/1000.0] [batch_t 1.446 (1.431)] [data_t 0.002] [optim_t 1.444] [lr 0.005000] 2024-04-02 11:06:47,980 - Train: 10.15% [54900/541000] [101.5/1000.0] [batch_t 1.427 (1.432)] [data_t 0.003] [optim_t 1.424] [lr 0.005000] 2024-04-02 11:08:16,460 - Train: 10.17% [55000/541000] [101.7/1000.0] [batch_t 0.608 (0.885)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 11:09:17,332 - Train: 10.18% [55100/541000] [101.8/1000.0] [batch_t 0.607 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 11:10:07,128 - ==> Total time: 10:10:01 Eta: 3 days, 17:30:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 11:10:22,209 - Train: 10.20% [55200/541000] [102.0/1000.0] [batch_t 0.607 (0.710)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 11:11:22,988 - Train: 10.22% [55300/541000] [102.2/1000.0] [batch_t 0.610 (0.608)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 11:12:23,706 - Train: 10.24% [55400/541000] [102.4/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 11:13:24,453 - Train: 10.26% [55500/541000] [102.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 11:14:29,748 - Train: 10.28% [55600/541000] [102.8/1000.0] [batch_t 0.606 (0.653)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 11:15:35,924 - Train: 10.30% [55700/541000] [103.0/1000.0] [batch_t 0.608 (0.662)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 11:15:50,293 - ==> Total time: 10:15:44 Eta: 3 days, 17:22:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 11:16:47,622 - Train: 10.31% [55800/541000] [103.1/1000.0] [batch_t 0.607 (0.706)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 11:18:00,588 - Train: 10.33% [55900/541000] [103.3/1000.0] [batch_t 0.608 (0.730)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 11:19:06,553 - Train: 10.35% [56000/541000] [103.5/1000.0] [batch_t 0.607 (0.660)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 11:20:07,594 - Train: 10.37% [56100/541000] [103.7/1000.0] [batch_t 0.904 (0.610)] [data_t 0.300] [optim_t 0.604] [lr 0.005000] 2024-04-02 11:21:08,323 - Train: 10.39% [56200/541000] [103.9/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 11:21:47,200 - ==> Total time: 10:21:41 Eta: 3 days, 17:16:08 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 11:22:12,412 - Train: 10.41% [56300/541000] [104.1/1000.0] [batch_t 0.607 (0.629)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 11:23:13,142 - Train: 10.43% [56400/541000] [104.3/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 11:24:14,230 - Train: 10.44% [56500/541000] [104.4/1000.0] [batch_t 0.609 (0.611)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-02 11:25:14,954 - Train: 10.46% [56600/541000] [104.6/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 11:26:15,694 - Train: 10.48% [56700/541000] [104.8/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 11:27:16,434 - Train: 10.50% [56800/541000] [105.0/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 11:27:19,475 - ==> Total time: 10:27:13 Eta: 3 days, 17:06:24 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 11:28:26,961 - Train: 10.52% [56900/541000] [105.2/1000.0] [batch_t 0.608 (0.683)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 11:29:28,253 - Train: 10.54% [57000/541000] [105.4/1000.0] [batch_t 0.606 (0.613)] [data_t 0.003] [optim_t 0.603] [lr 0.005000] 2024-04-02 11:30:30,595 - Train: 10.55% [57100/541000] [105.5/1000.0] [batch_t 0.608 (0.623)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 11:31:31,392 - Train: 10.57% [57200/541000] [105.7/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 11:32:38,124 - Train: 10.59% [57300/541000] [105.9/1000.0] [batch_t 0.608 (0.667)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 11:33:07,046 - ==> Total time: 10:33:01 Eta: 3 days, 16:58:54 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 11:33:42,689 - Train: 10.61% [57400/541000] [106.1/1000.0] [batch_t 0.608 (0.624)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 11:34:43,580 - Train: 10.63% [57500/541000] [106.3/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 11:35:44,353 - Train: 10.65% [57600/541000] [106.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 11:36:45,089 - Train: 10.67% [57700/541000] [106.7/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 11:37:45,859 - Train: 10.68% [57800/541000] [106.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 11:38:38,687 - ==> Total time: 10:38:33 Eta: 3 days, 16:49:13 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 11:38:49,004 - Train: 10.70% [57900/541000] [107.0/1000.0] [batch_t 0.608 (0.648)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 11:39:49,737 - Train: 10.72% [58000/541000] [107.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 11:40:50,657 - Train: 10.74% [58100/541000] [107.4/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 11:41:51,387 - Train: 10.76% [58200/541000] [107.6/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 11:42:52,133 - Train: 10.78% [58300/541000] [107.8/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 11:43:53,361 - Train: 10.79% [58400/541000] [107.9/1000.0] [batch_t 0.607 (0.612)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 11:44:10,368 - ==> Total time: 10:44:04 Eta: 3 days, 16:39:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 11:44:57,034 - Train: 10.81% [58500/541000] [108.1/1000.0] [batch_t 0.608 (0.617)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 11:45:57,744 - Train: 10.83% [58600/541000] [108.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 11:46:58,652 - Train: 10.85% [58700/541000] [108.5/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 11:47:59,383 - Train: 10.87% [58800/541000] [108.7/1000.0] [batch_t 0.611 (0.607)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 11:49:00,134 - Train: 10.89% [58900/541000] [108.9/1000.0] [batch_t 0.608 (0.607)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-02 11:49:42,058 - ==> Total time: 10:49:36 Eta: 3 days, 16:30:06 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 11:50:03,709 - Train: 10.91% [59000/541000] [109.1/1000.0] [batch_t 0.608 (0.632)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 11:51:04,474 - Train: 10.92% [59100/541000] [109.2/1000.0] [batch_t 0.610 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-02 11:52:05,281 - Train: 10.94% [59200/541000] [109.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 11:53:06,256 - Train: 10.96% [59300/541000] [109.6/1000.0] [batch_t 0.605 (0.610)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-02 11:54:07,073 - Train: 10.98% [59400/541000] [109.8/1000.0] [batch_t 0.613 (0.608)] [data_t 0.003] [optim_t 0.610] [lr 0.005000] 2024-04-02 11:55:07,894 - Train: 11.00% [59500/541000] [110.0/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-02 11:55:13,984 - ==> Total time: 10:55:08 Eta: 3 days, 16:20:40 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 11:56:25,879 - Train: 11.02% [59600/541000] [110.2/1000.0] [batch_t 0.607 (0.778)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 11:57:49,988 - Train: 11.04% [59700/541000] [110.4/1000.0] [batch_t 0.607 (0.841)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 11:58:50,740 - Train: 11.05% [59800/541000] [110.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 11:59:51,610 - Train: 11.07% [59900/541000] [110.7/1000.0] [batch_t 0.607 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 12:00:52,392 - Train: 11.09% [60000/541000] [110.9/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 12:01:23,383 - ==> Total time: 11:01:17 Eta: 3 days, 16:16:20 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 12:01:55,992 - Train: 11.11% [60100/541000] [111.1/1000.0] [batch_t 0.606 (0.623)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 12:02:56,720 - Train: 11.13% [60200/541000] [111.3/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 12:03:57,435 - Train: 11.15% [60300/541000] [111.5/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 12:04:58,191 - Train: 11.16% [60400/541000] [111.6/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 12:05:59,035 - Train: 11.18% [60500/541000] [111.8/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 12:06:54,919 - ==> Total time: 11:06:49 Eta: 3 days, 16:06:57 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 12:07:03,149 - Train: 11.20% [60600/541000] [112.0/1000.0] [batch_t 0.607 (0.777)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 12:08:05,688 - Train: 11.22% [60700/541000] [112.2/1000.0] [batch_t 0.607 (0.625)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 12:09:12,696 - Train: 11.24% [60800/541000] [112.4/1000.0] [batch_t 0.607 (0.670)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 12:10:22,165 - Train: 11.26% [60900/541000] [112.6/1000.0] [batch_t 0.607 (0.695)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 12:11:25,722 - Train: 11.28% [61000/541000] [112.8/1000.0] [batch_t 0.606 (0.635)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 12:12:26,630 - Train: 11.29% [61100/541000] [112.9/1000.0] [batch_t 0.607 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 12:12:46,666 - ==> Total time: 11:12:41 Eta: 3 days, 16:00:16 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 12:13:30,484 - Train: 11.31% [61200/541000] [113.1/1000.0] [batch_t 0.609 (0.620)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 12:14:31,209 - Train: 11.33% [61300/541000] [113.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 12:15:31,923 - Train: 11.35% [61400/541000] [113.5/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 12:16:32,631 - Train: 11.37% [61500/541000] [113.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 12:17:33,394 - Train: 11.39% [61600/541000] [113.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 12:18:18,445 - ==> Total time: 11:18:12 Eta: 3 days, 15:51:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 12:18:36,949 - Train: 11.40% [61700/541000] [114.0/1000.0] [batch_t 0.608 (0.636)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 12:19:37,663 - Train: 11.42% [61800/541000] [114.2/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 12:20:38,375 - Train: 11.44% [61900/541000] [114.4/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 12:21:39,109 - Train: 11.46% [62000/541000] [114.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 12:22:39,847 - Train: 11.48% [62100/541000] [114.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 12:23:40,589 - Train: 11.50% [62200/541000] [115.0/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 12:23:49,696 - ==> Total time: 11:23:44 Eta: 3 days, 15:41:47 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 12:24:43,845 - Train: 11.52% [62300/541000] [115.2/1000.0] [batch_t 0.608 (0.616)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 12:25:44,568 - Train: 11.53% [62400/541000] [115.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 12:26:45,288 - Train: 11.55% [62500/541000] [115.5/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 12:27:46,016 - Train: 11.57% [62600/541000] [115.7/1000.0] [batch_t 0.611 (0.607)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 12:28:46,730 - Train: 11.59% [62700/541000] [115.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 12:29:20,711 - ==> Total time: 11:29:15 Eta: 3 days, 15:32:34 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 12:29:49,505 - Train: 11.61% [62800/541000] [116.1/1000.0] [batch_t 0.607 (0.619)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-02 12:30:50,398 - Train: 11.63% [62900/541000] [116.3/1000.0] [batch_t 0.606 (0.609)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 12:31:51,119 - Train: 11.65% [63000/541000] [116.5/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 12:32:51,827 - Train: 11.66% [63100/541000] [116.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 12:33:52,545 - Train: 11.68% [63200/541000] [116.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 12:34:51,452 - ==> Total time: 11:34:45 Eta: 3 days, 15:23:23 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 12:34:56,459 - Train: 11.70% [63300/541000] [117.0/1000.0] [batch_t 0.607 (0.941)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 12:35:57,222 - Train: 11.72% [63400/541000] [117.2/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-02 12:36:59,486 - Train: 11.74% [63500/541000] [117.4/1000.0] [batch_t 0.605 (0.623)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-02 12:38:00,242 - Train: 11.76% [63600/541000] [117.6/1000.0] [batch_t 0.608 (0.607)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-02 12:39:00,990 - Train: 11.77% [63700/541000] [117.7/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 12:40:01,694 - Train: 11.79% [63800/541000] [117.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 12:40:25,214 - ==> Total time: 11:40:19 Eta: 3 days, 15:14:39 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 12:41:05,529 - Train: 11.81% [63900/541000] [118.1/1000.0] [batch_t 0.606 (0.616)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 12:42:06,311 - Train: 11.83% [64000/541000] [118.3/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 12:43:07,047 - Train: 11.85% [64100/541000] [118.5/1000.0] [batch_t 0.610 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 12:44:07,956 - Train: 11.87% [64200/541000] [118.7/1000.0] [batch_t 0.614 (0.609)] [data_t 0.003] [optim_t 0.611] [lr 0.005000] 2024-04-02 12:45:08,762 - Train: 11.89% [64300/541000] [118.9/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-02 12:45:56,740 - ==> Total time: 11:45:51 Eta: 3 days, 15:05:41 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 12:46:12,294 - Train: 11.90% [64400/541000] [119.0/1000.0] [batch_t 0.608 (0.639)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 12:47:13,023 - Train: 11.92% [64500/541000] [119.2/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 12:48:13,769 - Train: 11.94% [64600/541000] [119.4/1000.0] [batch_t 0.612 (0.607)] [data_t 0.002] [optim_t 0.610] [lr 0.005000] 2024-04-02 12:49:14,650 - Train: 11.96% [64700/541000] [119.6/1000.0] [batch_t 0.606 (0.609)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 12:50:15,868 - Train: 11.98% [64800/541000] [119.8/1000.0] [batch_t 0.605 (0.612)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 12:51:16,576 - Train: 12.00% [64900/541000] [120.0/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 12:51:28,715 - ==> Total time: 11:51:23 Eta: 3 days, 14:56:49 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 12:52:22,767 - Train: 12.01% [65000/541000] [120.1/1000.0] [batch_t 0.607 (0.627)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 12:53:23,497 - Train: 12.03% [65100/541000] [120.3/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 12:54:24,262 - Train: 12.05% [65200/541000] [120.5/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 12:55:25,210 - Train: 12.07% [65300/541000] [120.7/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 12:56:25,938 - Train: 12.09% [65400/541000] [120.9/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 12:57:02,982 - ==> Total time: 11:56:57 Eta: 3 days, 14:48:18 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 12:57:31,123 - Train: 12.11% [65500/541000] [121.1/1000.0] [batch_t 0.607 (0.636)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 12:58:31,836 - Train: 12.13% [65600/541000] [121.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 12:59:32,535 - Train: 12.14% [65700/541000] [121.4/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 13:01:02,630 - Train: 12.16% [65800/541000] [121.6/1000.0] [batch_t 0.606 (0.901)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 13:03:16,265 - Train: 12.18% [65900/541000] [121.8/1000.0] [batch_t 0.608 (1.336)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 13:04:50,524 - Train: 12.20% [66000/541000] [122.0/1000.0] [batch_t 0.607 (0.943)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 13:04:51,741 - ==> Total time: 12:04:46 Eta: 3 days, 14:55:57 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 13:05:54,577 - Train: 12.22% [66100/541000] [122.2/1000.0] [batch_t 0.607 (0.616)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 13:06:55,365 - Train: 12.24% [66200/541000] [122.4/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 13:07:56,107 - Train: 12.26% [66300/541000] [122.6/1000.0] [batch_t 0.609 (0.607)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-02 13:08:57,064 - Train: 12.27% [66400/541000] [122.7/1000.0] [batch_t 0.607 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 13:09:57,772 - Train: 12.29% [66500/541000] [122.9/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 13:10:23,881 - ==> Total time: 12:10:18 Eta: 3 days, 14:47:08 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 13:11:01,318 - Train: 12.31% [66600/541000] [123.1/1000.0] [batch_t 0.608 (0.617)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 13:12:02,078 - Train: 12.33% [66700/541000] [123.3/1000.0] [batch_t 0.608 (0.607)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-02 13:13:02,829 - Train: 12.35% [66800/541000] [123.5/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 13:14:03,533 - Train: 12.37% [66900/541000] [123.7/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 13:15:04,478 - Train: 12.38% [67000/541000] [123.8/1000.0] [batch_t 0.607 (0.609)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 13:16:09,289 - ==> Total time: 12:16:03 Eta: 3 days, 14:39:55 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 13:16:21,938 - Train: 12.40% [67100/541000] [124.0/1000.0] [batch_t 0.606 (0.666)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 13:17:22,621 - Train: 12.42% [67200/541000] [124.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 13:18:23,358 - Train: 12.44% [67300/541000] [124.4/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 13:19:24,093 - Train: 12.46% [67400/541000] [124.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 13:20:25,049 - Train: 12.48% [67500/541000] [124.8/1000.0] [batch_t 0.606 (0.609)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 13:21:25,794 - Train: 12.50% [67600/541000] [125.0/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 13:21:40,969 - ==> Total time: 12:21:35 Eta: 3 days, 14:31:07 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 13:22:28,997 - Train: 12.51% [67700/541000] [125.1/1000.0] [batch_t 0.608 (0.613)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 13:23:29,727 - Train: 12.53% [67800/541000] [125.3/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 13:24:30,417 - Train: 12.55% [67900/541000] [125.5/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 13:25:31,147 - Train: 12.57% [68000/541000] [125.7/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 13:26:31,980 - Train: 12.59% [68100/541000] [125.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 13:27:12,039 - ==> Total time: 12:27:06 Eta: 3 days, 14:22:19 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 13:27:36,042 - Train: 12.61% [68200/541000] [126.1/1000.0] [batch_t 0.607 (0.647)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 13:28:36,753 - Train: 12.62% [68300/541000] [126.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 13:29:37,450 - Train: 12.64% [68400/541000] [126.4/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 13:30:38,162 - Train: 12.66% [68500/541000] [126.6/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 13:31:38,874 - Train: 12.68% [68600/541000] [126.8/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 13:32:39,725 - Train: 12.70% [68700/541000] [127.0/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-02 13:32:43,973 - ==> Total time: 12:32:38 Eta: 3 days, 14:13:39 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 13:33:43,401 - Train: 12.72% [68800/541000] [127.2/1000.0] [batch_t 0.607 (0.617)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 13:34:44,123 - Train: 12.74% [68900/541000] [127.4/1000.0] [batch_t 0.608 (0.607)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-02 13:35:44,837 - Train: 12.75% [69000/541000] [127.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 13:36:45,536 - Train: 12.77% [69100/541000] [127.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 13:37:46,257 - Train: 12.79% [69200/541000] [127.9/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 13:38:15,400 - ==> Total time: 12:38:09 Eta: 3 days, 14:04:59 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 13:38:49,446 - Train: 12.81% [69300/541000] [128.1/1000.0] [batch_t 0.606 (0.619)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 13:39:50,164 - Train: 12.83% [69400/541000] [128.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 13:40:50,845 - Train: 12.85% [69500/541000] [128.5/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 13:41:51,581 - Train: 12.87% [69600/541000] [128.7/1000.0] [batch_t 0.612 (0.607)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 13:42:52,305 - Train: 12.88% [69700/541000] [128.8/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 13:43:46,314 - ==> Total time: 12:43:40 Eta: 3 days, 13:56:18 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 13:43:55,688 - Train: 12.90% [69800/541000] [129.0/1000.0] [batch_t 0.608 (0.662)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 13:44:56,518 - Train: 12.92% [69900/541000] [129.2/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 13:45:57,239 - Train: 12.94% [70000/541000] [129.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 13:46:57,937 - Train: 12.96% [70100/541000] [129.6/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 13:47:58,629 - Train: 12.98% [70200/541000] [129.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 13:48:59,336 - Train: 12.99% [70300/541000] [129.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 13:49:17,555 - ==> Total time: 12:49:12 Eta: 3 days, 13:47:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 13:50:02,417 - Train: 13.01% [70400/541000] [130.1/1000.0] [batch_t 0.607 (0.617)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-02 13:51:03,282 - Train: 13.03% [70500/541000] [130.3/1000.0] [batch_t 0.607 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 13:52:03,959 - Train: 13.05% [70600/541000] [130.5/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 13:53:04,661 - Train: 13.07% [70700/541000] [130.7/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 13:54:05,335 - Train: 13.09% [70800/541000] [130.9/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 13:54:48,445 - ==> Total time: 12:54:42 Eta: 3 days, 13:39:08 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 13:55:08,293 - Train: 13.11% [70900/541000] [131.1/1000.0] [batch_t 0.606 (0.621)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 13:56:09,014 - Train: 13.12% [71000/541000] [131.2/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 13:57:09,843 - Train: 13.14% [71100/541000] [131.4/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 13:58:10,539 - Train: 13.16% [71200/541000] [131.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 13:59:11,253 - Train: 13.18% [71300/541000] [131.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 14:00:11,947 - Train: 13.20% [71400/541000] [132.0/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 14:00:19,235 - ==> Total time: 13:00:13 Eta: 3 days, 13:30:35 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 14:01:15,315 - Train: 13.22% [71500/541000] [132.2/1000.0] [batch_t 0.608 (0.616)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 14:02:16,028 - Train: 13.23% [71600/541000] [132.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 14:03:16,863 - Train: 13.25% [71700/541000] [132.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 14:04:17,532 - Train: 13.27% [71800/541000] [132.7/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 14:05:18,231 - Train: 13.29% [71900/541000] [132.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 14:05:50,404 - ==> Total time: 13:05:44 Eta: 3 days, 13:22:07 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 14:06:21,782 - Train: 13.31% [72000/541000] [133.1/1000.0] [batch_t 0.607 (0.629)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 14:07:22,501 - Train: 13.33% [72100/541000] [133.3/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 14:08:23,204 - Train: 13.35% [72200/541000] [133.5/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 14:09:24,063 - Train: 13.36% [72300/541000] [133.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 14:10:24,783 - Train: 13.38% [72400/541000] [133.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 14:11:21,852 - ==> Total time: 13:11:16 Eta: 3 days, 13:13:44 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 14:11:27,760 - Train: 13.40% [72500/541000] [134.0/1000.0] [batch_t 0.608 (0.694)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 14:12:28,507 - Train: 13.42% [72600/541000] [134.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 14:13:29,222 - Train: 13.44% [72700/541000] [134.4/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 14:14:29,941 - Train: 13.46% [72800/541000] [134.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 14:15:30,796 - Train: 13.48% [72900/541000] [134.8/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 14:16:31,508 - Train: 13.49% [73000/541000] [134.9/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 14:16:52,763 - ==> Total time: 13:16:47 Eta: 3 days, 13:05:20 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 14:17:35,121 - Train: 13.51% [73100/541000] [135.1/1000.0] [batch_t 0.607 (0.620)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 14:18:36,011 - Train: 13.53% [73200/541000] [135.3/1000.0] [batch_t 0.606 (0.609)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 14:19:36,721 - Train: 13.55% [73300/541000] [135.5/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 14:20:37,560 - Train: 13.57% [73400/541000] [135.7/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-02 14:21:38,280 - Train: 13.59% [73500/541000] [135.9/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 14:22:24,591 - ==> Total time: 13:22:19 Eta: 3 days, 12:57:04 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 14:22:41,586 - Train: 13.60% [73600/541000] [136.0/1000.0] [batch_t 0.607 (0.632)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 14:23:42,283 - Train: 13.62% [73700/541000] [136.2/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 14:24:42,988 - Train: 13.64% [73800/541000] [136.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 14:25:43,720 - Train: 13.66% [73900/541000] [136.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 14:26:44,553 - Train: 13.68% [74000/541000] [136.8/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 14:27:45,263 - Train: 13.70% [74100/541000] [137.0/1000.0] [batch_t 0.607 (0.607)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-02 14:27:55,590 - ==> Total time: 13:27:50 Eta: 3 days, 12:48:45 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 14:28:48,377 - Train: 13.72% [74200/541000] [137.2/1000.0] [batch_t 0.609 (0.615)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 14:29:49,357 - Train: 13.73% [74300/541000] [137.3/1000.0] [batch_t 0.608 (0.610)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 14:30:50,065 - Train: 13.75% [74400/541000] [137.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 14:31:50,777 - Train: 13.77% [74500/541000] [137.7/1000.0] [batch_t 0.611 (0.607)] [data_t 0.003] [optim_t 0.609] [lr 0.005000] 2024-04-02 14:32:51,589 - Train: 13.79% [74600/541000] [137.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 14:33:26,798 - ==> Total time: 13:33:21 Eta: 3 days, 12:40:30 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 14:33:54,479 - Train: 13.81% [74700/541000] [138.1/1000.0] [batch_t 0.607 (0.620)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 14:34:55,203 - Train: 13.83% [74800/541000] [138.3/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 14:35:55,918 - Train: 13.84% [74900/541000] [138.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 14:36:56,631 - Train: 13.86% [75000/541000] [138.6/1000.0] [batch_t 0.610 (0.607)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 14:37:57,333 - Train: 13.88% [75100/541000] [138.8/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 14:38:57,446 - ==> Total time: 13:38:51 Eta: 3 days, 12:32:14 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 14:39:01,209 - Train: 13.90% [75200/541000] [139.0/1000.0] [batch_t 1.569 (1.569)] [data_t 0.963] [optim_t 0.606] [lr 0.005000] 2024-04-02 14:40:01,931 - Train: 13.92% [75300/541000] [139.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 14:41:02,654 - Train: 13.94% [75400/541000] [139.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 14:42:03,436 - Train: 13.96% [75500/541000] [139.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 14:43:04,453 - Train: 13.97% [75600/541000] [139.7/1000.0] [batch_t 0.608 (0.610)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 14:44:05,157 - Train: 13.99% [75700/541000] [139.9/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 14:44:29,438 - ==> Total time: 13:44:23 Eta: 3 days, 12:24:09 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 14:45:08,688 - Train: 14.01% [75800/541000] [140.1/1000.0] [batch_t 0.611 (0.618)] [data_t 0.003] [optim_t 0.608] [lr 0.005000] 2024-04-02 14:46:09,580 - Train: 14.03% [75900/541000] [140.3/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-02 14:47:10,279 - Train: 14.05% [76000/541000] [140.5/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 14:48:11,015 - Train: 14.07% [76100/541000] [140.7/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 14:49:12,761 - Train: 14.09% [76200/541000] [140.9/1000.0] [batch_t 0.608 (0.617)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-02 14:50:01,949 - ==> Total time: 13:49:56 Eta: 3 days, 12:16:09 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 14:50:20,741 - Train: 14.10% [76300/541000] [141.0/1000.0] [batch_t 0.606 (0.651)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 14:51:21,459 - Train: 14.12% [76400/541000] [141.2/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 14:52:22,741 - Train: 14.14% [76500/541000] [141.4/1000.0] [batch_t 0.608 (0.613)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 14:53:23,479 - Train: 14.16% [76600/541000] [141.6/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 14:54:24,196 - Train: 14.18% [76700/541000] [141.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 14:55:24,932 - Train: 14.20% [76800/541000] [142.0/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 14:55:38,300 - ==> Total time: 13:55:32 Eta: 3 days, 12:08:34 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 14:56:28,330 - Train: 14.21% [76900/541000] [142.1/1000.0] [batch_t 0.607 (0.616)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 14:57:41,167 - Train: 14.23% [77000/541000] [142.3/1000.0] [batch_t 1.065 (0.728)] [data_t 0.455] [optim_t 0.610] [lr 0.005000] 2024-04-02 14:59:32,038 - Train: 14.25% [77100/541000] [142.5/1000.0] [batch_t 0.608 (1.109)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 15:00:32,767 - Train: 14.27% [77200/541000] [142.7/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 15:01:45,386 - Train: 14.29% [77300/541000] [142.9/1000.0] [batch_t 0.607 (0.726)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 15:02:23,633 - ==> Total time: 14:02:18 Eta: 3 days, 12:07:55 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 15:02:49,136 - Train: 14.31% [77400/541000] [143.1/1000.0] [batch_t 0.605 (0.628)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 15:03:50,773 - Train: 14.33% [77500/541000] [143.3/1000.0] [batch_t 0.603 (0.616)] [data_t 0.002] [optim_t 0.601] [lr 0.005000] 2024-04-02 15:04:51,657 - Train: 14.34% [77600/541000] [143.4/1000.0] [batch_t 0.609 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 15:05:53,195 - Train: 14.36% [77700/541000] [143.6/1000.0] [batch_t 0.609 (0.615)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 15:06:53,931 - Train: 14.38% [77800/541000] [143.8/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 15:07:54,664 - Train: 14.40% [77900/541000] [144.0/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 15:07:57,099 - ==> Total time: 14:07:51 Eta: 3 days, 12:00:03 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 15:08:58,289 - Train: 14.42% [78000/541000] [144.2/1000.0] [batch_t 0.605 (0.616)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 15:09:59,695 - Train: 14.44% [78100/541000] [144.4/1000.0] [batch_t 0.606 (0.614)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 15:11:00,791 - Train: 14.45% [78200/541000] [144.5/1000.0] [batch_t 0.607 (0.611)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 15:12:04,075 - Train: 14.47% [78300/541000] [144.7/1000.0] [batch_t 2.063 (0.633)] [data_t 1.455] [optim_t 0.608] [lr 0.005000] 2024-04-02 15:13:20,167 - Train: 14.49% [78400/541000] [144.9/1000.0] [batch_t 1.825 (0.761)] [data_t 1.219] [optim_t 0.606] [lr 0.005000] 2024-04-02 15:13:54,178 - ==> Total time: 14:13:48 Eta: 3 days, 11:54:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 15:14:34,495 - Train: 14.51% [78500/541000] [145.1/1000.0] [batch_t 2.809 (0.695)] [data_t 2.201] [optim_t 0.608] [lr 0.005000] 2024-04-02 15:15:42,551 - Train: 14.53% [78600/541000] [145.3/1000.0] [batch_t 0.607 (0.680)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 15:16:43,443 - Train: 14.55% [78700/541000] [145.5/1000.0] [batch_t 0.605 (0.609)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 15:17:44,163 - Train: 14.57% [78800/541000] [145.7/1000.0] [batch_t 0.610 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 15:18:44,909 - Train: 14.58% [78900/541000] [145.8/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 15:19:37,126 - ==> Total time: 14:19:31 Eta: 3 days, 11:47:38 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 15:19:48,184 - Train: 14.60% [79000/541000] [146.0/1000.0] [batch_t 0.607 (0.675)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 15:20:49,695 - Train: 14.62% [79100/541000] [146.2/1000.0] [batch_t 0.608 (0.615)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 15:21:50,410 - Train: 14.64% [79200/541000] [146.4/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 15:22:51,271 - Train: 14.66% [79300/541000] [146.6/1000.0] [batch_t 0.604 (0.609)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-02 15:23:51,990 - Train: 14.68% [79400/541000] [146.8/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 15:24:52,682 - Train: 14.70% [79500/541000] [147.0/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 15:25:09,074 - ==> Total time: 14:25:03 Eta: 3 days, 11:39:41 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 15:25:56,699 - Train: 14.71% [79600/541000] [147.1/1000.0] [batch_t 0.607 (0.622)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 15:26:57,401 - Train: 14.73% [79700/541000] [147.3/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 15:27:58,433 - Train: 14.75% [79800/541000] [147.5/1000.0] [batch_t 0.607 (0.610)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 15:28:59,175 - Train: 14.77% [79900/541000] [147.7/1000.0] [batch_t 0.611 (0.607)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 15:30:00,007 - Train: 14.79% [80000/541000] [147.9/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 15:30:41,284 - ==> Total time: 14:30:35 Eta: 3 days, 11:31:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 15:31:02,802 - Train: 14.81% [80100/541000] [148.1/1000.0] [batch_t 0.605 (0.620)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 15:32:03,523 - Train: 14.82% [80200/541000] [148.2/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 15:33:04,258 - Train: 14.84% [80300/541000] [148.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 15:34:04,973 - Train: 14.86% [80400/541000] [148.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 15:35:05,976 - Train: 14.88% [80500/541000] [148.8/1000.0] [batch_t 0.608 (0.610)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 15:36:06,872 - Train: 14.90% [80600/541000] [149.0/1000.0] [batch_t 0.609 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-02 15:36:12,348 - ==> Total time: 14:36:06 Eta: 3 days, 11:23:50 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 15:37:10,495 - Train: 14.92% [80700/541000] [149.2/1000.0] [batch_t 0.608 (0.617)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-02 15:38:11,273 - Train: 14.94% [80800/541000] [149.4/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-02 15:39:12,115 - Train: 14.95% [80900/541000] [149.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-02 15:40:12,934 - Train: 14.97% [81000/541000] [149.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-02 15:41:14,962 - Train: 14.99% [81100/541000] [149.9/1000.0] [batch_t 0.608 (0.620)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 15:41:58,219 - Test: 36.76% [50/136] [batch_t 0.212 (0.242)] 2024-04-02 15:42:13,385 - Test: 73.53% [100/136] [batch_t 0.209 (0.272)] 2024-04-02 15:42:21,014 - Test: 100.00% [136/136] [batch_t 0.053 (0.256)] 2024-04-02 15:43:08,235 - ==> Metric Time for candle : 0.002 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.382 (mAUROC_px) 7.915 (mAP_px) 0.910 (mF1_max_px) 24.453 (mAUPRO_px) 0.302 (mF1_px_0.2_0.8_0.1) 0.298 (mAcc_px_0.2_0.8_0.1) 0.311 (mIoU_px_0.2_0.8_0.1) 0.907 (mIoU_max_px) 2024-04-02 15:43:45,061 - ==> Metric Time for capsules : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 8.148 (mAUROC_px) 6.293 (mAP_px) 0.709 (mF1_max_px) 20.024 (mAUPRO_px) 0.243 (mF1_px_0.2_0.8_0.1) 0.243 (mAcc_px_0.2_0.8_0.1) 0.240 (mIoU_px_0.2_0.8_0.1) 0.702 (mIoU_max_px) 2024-04-02 15:44:20,239 - ==> Metric Time for cashew : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.647 (mAUROC_px) 5.898 (mAP_px) 0.655 (mF1_max_px) 19.413 (mAUPRO_px) 0.230 (mF1_px_0.2_0.8_0.1) 0.228 (mAcc_px_0.2_0.8_0.1) 0.231 (mIoU_px_0.2_0.8_0.1) 0.669 (mIoU_max_px) 2024-04-02 15:44:55,567 - ==> Metric Time for chewinggum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.598 (mAUROC_px) 5.803 (mAP_px) 0.678 (mF1_max_px) 19.659 (mAUPRO_px) 0.231 (mF1_px_0.2_0.8_0.1) 0.234 (mAcc_px_0.2_0.8_0.1) 0.230 (mIoU_px_0.2_0.8_0.1) 0.686 (mIoU_max_px) 2024-04-02 15:45:34,712 - ==> Metric Time for fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.658 (mAUROC_px) 5.821 (mAP_px) 0.771 (mF1_max_px) 23.175 (mAUPRO_px) 0.257 (mF1_px_0.2_0.8_0.1) 0.260 (mAcc_px_0.2_0.8_0.1) 0.253 (mIoU_px_0.2_0.8_0.1) 0.739 (mIoU_max_px) 2024-04-02 15:46:22,894 - ==> Metric Time for macaroni1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.707 (mAUROC_px) 8.052 (mAP_px) 0.934 (mF1_max_px) 26.266 (mAUPRO_px) 0.321 (mF1_px_0.2_0.8_0.1) 0.316 (mAcc_px_0.2_0.8_0.1) 0.323 (mIoU_px_0.2_0.8_0.1) 0.972 (mIoU_max_px) 2024-04-02 15:47:12,444 - ==> Metric Time for macaroni2 : 0.002 (mAUROC_sp_max) 0.001 (mAP_sp_max) 0.001 (mF1_max_sp_max) 12.067 (mAUROC_px) 7.968 (mAP_px) 0.937 (mF1_max_px) 26.387 (mAUPRO_px) 0.315 (mF1_px_0.2_0.8_0.1) 0.314 (mAcc_px_0.2_0.8_0.1) 0.316 (mIoU_px_0.2_0.8_0.1) 0.918 (mIoU_max_px) 2024-04-02 15:48:01,730 - ==> Metric Time for pcb1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.735 (mAUROC_px) 8.127 (mAP_px) 0.926 (mF1_max_px) 27.347 (mAUPRO_px) 0.319 (mF1_px_0.2_0.8_0.1) 0.315 (mAcc_px_0.2_0.8_0.1) 0.314 (mIoU_px_0.2_0.8_0.1) 0.915 (mIoU_max_px) 2024-04-02 15:48:50,327 - ==> Metric Time for pcb2 : 0.002 (mAUROC_sp_max) 0.001 (mAP_sp_max) 0.001 (mF1_max_sp_max) 10.695 (mAUROC_px) 8.144 (mAP_px) 0.931 (mF1_max_px) 26.665 (mAUPRO_px) 0.315 (mF1_px_0.2_0.8_0.1) 0.313 (mAcc_px_0.2_0.8_0.1) 0.314 (mIoU_px_0.2_0.8_0.1) 0.913 (mIoU_max_px) 2024-04-02 15:49:38,340 - ==> Metric Time for pcb3 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.683 (mAUROC_px) 8.007 (mAP_px) 0.971 (mF1_max_px) 26.203 (mAUPRO_px) 0.313 (mF1_px_0.2_0.8_0.1) 0.318 (mAcc_px_0.2_0.8_0.1) 0.315 (mIoU_px_0.2_0.8_0.1) 0.916 (mIoU_max_px) 2024-04-02 15:50:27,702 - ==> Metric Time for pcb4 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.608 (mAUROC_px) 8.044 (mAP_px) 0.920 (mF1_max_px) 27.658 (mAUPRO_px) 0.313 (mF1_px_0.2_0.8_0.1) 0.311 (mAcc_px_0.2_0.8_0.1) 0.308 (mIoU_px_0.2_0.8_0.1) 0.910 (mIoU_max_px) 2024-04-02 15:51:03,794 - ==> Metric Time for pipe_fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.656 (mAUROC_px) 5.765 (mAP_px) 0.664 (mF1_max_px) 20.459 (mAUPRO_px) 0.226 (mF1_px_0.2_0.8_0.1) 0.232 (mAcc_px_0.2_0.8_0.1) 0.227 (mIoU_px_0.2_0.8_0.1) 0.644 (mIoU_max_px) 2024-04-02 15:51:03,814 - | Name | mAUROC_sp_max | mAUROC_sp_max (Max) | mAP_sp_max | mAP_sp_max (Max) | mF1_max_sp_max | mF1_max_sp_max (Max) | mAUROC_px | mAUROC_px (Max) | mAP_px | mAP_px (Max) | mF1_max_px | mF1_max_px (Max) | mAUPRO_px | mAUPRO_px (Max) | mF1_px_0.2_0.8_0.1 | mF1_px_0.2_0.8_0.1 (Max) | mAcc_px_0.2_0.8_0.1 | mAcc_px_0.2_0.8_0.1 (Max) | mIoU_px_0.2_0.8_0.1 | mIoU_px_0.2_0.8_0.1 (Max) | mIoU_max_px | mIoU_max_px (Max) | |:----------:|:---------------:|:---------------------:|:------------:|:------------------:|:----------------:|:----------------------:|:-----------:|:------------------:|:--------:|:------------------:|:------------:|:------------------:|:-----------:|:------------------:|:--------------------:|:--------------------------:|:---------------------:|:---------------------------:|:---------------------:|:---------------------------:|:-------------:|:-------------------:| | candle | 96.770 | 97.100 (50 epoch) | 96.975 | 97.346 (50 epoch) | 90.640 | 91.542 (50 epoch) | 99.018 | 99.018 (150 epoch) | 23.463 | 23.463 (150 epoch) | 32.350 | 32.438 (100 epoch) | 95.225 | 95.763 (50 epoch) | 19.845 | 19.845 (150 epoch) | 35.575 | 36.722 (100 epoch) | 11.386 | 11.386 (150 epoch) | 19.296 | 19.359 (100 epoch) | | capsules | 90.317 | 91.750 (100 epoch) | 94.136 | 94.964 (100 epoch) | 88.679 | 88.780 (100 epoch) | 99.122 | 99.123 (100 epoch) | 60.204 | 61.256 (100 epoch) | 59.678 | 59.766 (100 epoch) | 91.493 | 91.811 (100 epoch) | 32.552 | 33.186 (100 epoch) | 58.647 | 62.576 (50 epoch) | 21.221 | 21.701 (100 epoch) | 42.530 | 42.619 (100 epoch) | | cashew | 94.800 | 94.800 (150 epoch) | 97.492 | 97.492 (150 epoch) | 92.228 | 92.228 (150 epoch) | 94.706 | 94.706 (150 epoch) | 47.821 | 47.821 (150 epoch) | 52.507 | 52.507 (150 epoch) | 88.960 | 88.960 (150 epoch) | 28.279 | 28.279 (150 epoch) | 34.522 | 34.522 (150 epoch) | 17.655 | 17.655 (150 epoch) | 35.600 | 35.600 (150 epoch) | | chewinggum | 96.507 | 97.743 (100 epoch) | 98.444 | 98.928 (100 epoch) | 94.180 | 94.301 (50 epoch) | 98.059 | 98.145 (100 epoch) | 55.538 | 59.823 (50 epoch) | 58.958 | 61.088 (50 epoch) | 78.921 | 79.773 (50 epoch) | 30.603 | 35.748 (50 epoch) | 54.848 | 65.611 (50 epoch) | 20.026 | 23.754 (50 epoch) | 41.801 | 43.976 (50 epoch) | | fryum | 95.220 | 95.220 (150 epoch) | 97.737 | 97.737 (150 epoch) | 91.163 | 91.163 (150 epoch) | 97.089 | 97.089 (150 epoch) | 48.983 | 48.983 (150 epoch) | 52.093 | 52.093 (150 epoch) | 91.201 | 91.622 (100 epoch) | 25.863 | 26.312 (50 epoch) | 26.972 | 27.708 (50 epoch) | 16.058 | 16.200 (50 epoch) | 35.220 | 35.220 (150 epoch) | | macaroni1 | 90.282 | 91.568 (100 epoch) | 88.540 | 89.846 (100 epoch) | 80.000 | 81.592 (100 epoch) | 99.474 | 99.519 (100 epoch) | 18.184 | 18.184 (150 epoch) | 28.252 | 28.252 (150 epoch) | 95.494 | 95.494 (150 epoch) | 14.732 | 15.720 (50 epoch) | 36.325 | 40.765 (50 epoch) | 8.205 | 8.704 (50 epoch) | 16.450 | 16.450 (150 epoch) | | macaroni2 | 79.786 | 82.589 (50 epoch) | 74.328 | 81.196 (50 epoch) | 74.107 | 75.622 (50 epoch) | 99.360 | 99.452 (100 epoch) | 7.216 | 11.272 (50 epoch) | 14.325 | 17.568 (50 epoch) | 96.253 | 96.253 (150 epoch) | 6.751 | 10.086 (50 epoch) | 47.592 | 49.562 (50 epoch) | 3.565 | 5.433 (50 epoch) | 7.715 | 9.630 (50 epoch) | | pcb1 | 96.320 | 96.320 (150 epoch) | 94.447 | 94.447 (150 epoch) | 93.720 | 93.720 (150 epoch) | 99.771 | 99.771 (150 epoch) | 79.877 | 79.877 (150 epoch) | 73.861 | 73.861 (150 epoch) | 93.333 | 93.660 (50 epoch) | 46.703 | 46.703 (150 epoch) | 62.099 | 62.362 (50 epoch) | 32.976 | 32.976 (150 epoch) | 58.556 | 58.556 (150 epoch) | | pcb2 | 94.820 | 94.820 (150 epoch) | 94.529 | 94.529 (150 epoch) | 90.000 | 90.000 (150 epoch) | 98.932 | 98.932 (150 epoch) | 13.355 | 14.130 (50 epoch) | 23.728 | 23.728 (150 epoch) | 88.028 | 89.590 (100 epoch) | 16.106 | 16.106 (150 epoch) | 47.820 | 47.820 (150 epoch) | 8.862 | 8.862 (150 epoch) | 13.461 | 13.461 (150 epoch) | | pcb3 | 94.614 | 94.614 (150 epoch) | 94.865 | 94.865 (150 epoch) | 88.235 | 88.235 (150 epoch) | 99.124 | 99.124 (150 epoch) | 20.249 | 20.249 (150 epoch) | 27.912 | 27.912 (150 epoch) | 89.697 | 90.129 (50 epoch) | 20.042 | 20.042 (150 epoch) | 46.163 | 48.341 (50 epoch) | 11.295 | 11.295 (150 epoch) | 16.219 | 16.219 (150 epoch) | | pcb4 | 99.832 | 99.901 (100 epoch) | 99.833 | 99.901 (100 epoch) | 98.492 | 99.000 (50 epoch) | 98.617 | 98.617 (150 epoch) | 46.845 | 46.993 (100 epoch) | 46.750 | 46.928 (100 epoch) | 88.438 | 88.438 (150 epoch) | 25.996 | 26.907 (100 epoch) | 36.508 | 39.716 (50 epoch) | 15.937 | 16.517 (100 epoch) | 30.505 | 30.658 (100 epoch) | | pipe_fryum | 98.840 | 98.840 (150 epoch) | 99.400 | 99.400 (150 epoch) | 97.030 | 97.030 (100 epoch) | 99.034 | 99.080 (100 epoch) | 52.529 | 53.518 (100 epoch) | 57.239 | 58.465 (100 epoch) | 95.525 | 95.525 (150 epoch) | 33.796 | 34.338 (100 epoch) | 41.107 | 43.014 (100 epoch) | 21.729 | 22.178 (100 epoch) | 40.095 | 41.308 (100 epoch) | | Avg | 94.009 | 94.251 (100 epoch) | 94.227 | 94.481 (100 epoch) | 89.873 | 89.873 (150 epoch) | 98.526 | 98.526 (150 epoch) | 39.522 | 39.522 (150 epoch) | 43.971 | 43.973 (100 epoch) | 91.047 | 91.047 (150 epoch) | 25.106 | 25.164 (100 epoch) | 44.015 | 46.179 (50 epoch) | 15.743 | 15.755 (100 epoch) | 29.787 | 29.787 (150 epoch) | 2024-04-02 15:51:04,155 - ==> Total time: 14:50:58 Eta: 3 days, 12:08:52 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 15:52:20,935 - Train: 15.01% [81200/541000] [150.1/1000.0] [batch_t 1.430 (1.457)] [data_t 0.003] [optim_t 1.427] [lr 0.005000] 2024-04-02 15:54:43,930 - Train: 15.03% [81300/541000] [150.3/1000.0] [batch_t 1.425 (1.430)] [data_t 0.002] [optim_t 1.423] [lr 0.005000] 2024-04-02 15:57:07,044 - Train: 15.05% [81400/541000] [150.5/1000.0] [batch_t 1.434 (1.431)] [data_t 0.003] [optim_t 1.431] [lr 0.005000] 2024-04-02 15:59:30,152 - Train: 15.06% [81500/541000] [150.6/1000.0] [batch_t 1.424 (1.431)] [data_t 0.002] [optim_t 1.422] [lr 0.005000] 2024-04-02 16:01:53,134 - Train: 15.08% [81600/541000] [150.8/1000.0] [batch_t 1.421 (1.430)] [data_t 0.003] [optim_t 1.418] [lr 0.005000] 2024-04-02 16:04:03,307 - ==> Total time: 15:03:57 Eta: 3 days, 12:42:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 16:04:20,923 - Train: 15.10% [81700/541000] [151.0/1000.0] [batch_t 1.424 (1.528)] [data_t 0.002] [optim_t 1.422] [lr 0.005000] 2024-04-02 16:06:43,836 - Train: 15.12% [81800/541000] [151.2/1000.0] [batch_t 1.423 (1.429)] [data_t 0.002] [optim_t 1.421] [lr 0.005000] 2024-04-02 16:09:06,963 - Train: 15.14% [81900/541000] [151.4/1000.0] [batch_t 1.426 (1.431)] [data_t 0.003] [optim_t 1.423] [lr 0.005000] 2024-04-02 16:11:30,013 - Train: 15.16% [82000/541000] [151.6/1000.0] [batch_t 1.434 (1.430)] [data_t 0.003] [optim_t 1.431] [lr 0.005000] 2024-04-02 16:13:53,044 - Train: 15.18% [82100/541000] [151.8/1000.0] [batch_t 1.438 (1.430)] [data_t 0.002] [optim_t 1.436] [lr 0.005000] 2024-04-02 16:16:16,215 - Train: 15.19% [82200/541000] [151.9/1000.0] [batch_t 1.430 (1.432)] [data_t 0.002] [optim_t 1.428] [lr 0.005000] 2024-04-02 16:17:01,980 - ==> Total time: 15:16:56 Eta: 3 days, 13:15:33 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 16:18:45,617 - Train: 15.21% [82300/541000] [152.1/1000.0] [batch_t 1.429 (1.449)] [data_t 0.003] [optim_t 1.426] [lr 0.005000] 2024-04-02 16:21:08,667 - Train: 15.23% [82400/541000] [152.3/1000.0] [batch_t 1.433 (1.430)] [data_t 0.002] [optim_t 1.431] [lr 0.005000] 2024-04-02 16:23:31,732 - Train: 15.25% [82500/541000] [152.5/1000.0] [batch_t 1.440 (1.431)] [data_t 0.003] [optim_t 1.437] [lr 0.005000] 2024-04-02 16:25:54,748 - Train: 15.27% [82600/541000] [152.7/1000.0] [batch_t 1.434 (1.430)] [data_t 0.002] [optim_t 1.432] [lr 0.005000] 2024-04-02 16:28:17,647 - Train: 15.29% [82700/541000] [152.9/1000.0] [batch_t 1.439 (1.429)] [data_t 0.002] [optim_t 1.437] [lr 0.005000] 2024-04-02 16:30:02,113 - ==> Total time: 15:29:56 Eta: 3 days, 13:48:06 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 16:30:49,220 - Train: 15.30% [82800/541000] [153.0/1000.0] [batch_t 1.435 (1.489)] [data_t 0.002] [optim_t 1.433] [lr 0.005000] 2024-04-02 16:33:12,275 - Train: 15.32% [82900/541000] [153.2/1000.0] [batch_t 1.431 (1.430)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-02 16:35:36,919 - Train: 15.34% [83000/541000] [153.4/1000.0] [batch_t 1.438 (1.446)] [data_t 0.002] [optim_t 1.435] [lr 0.005000] 2024-04-02 16:37:59,891 - Train: 15.36% [83100/541000] [153.6/1000.0] [batch_t 1.420 (1.430)] [data_t 0.003] [optim_t 1.417] [lr 0.005000] 2024-04-02 16:40:57,257 - Train: 15.38% [83200/541000] [153.8/1000.0] [batch_t 1.425 (1.774)] [data_t 0.003] [optim_t 1.422] [lr 0.005000] 2024-04-02 16:43:20,211 - Train: 15.40% [83300/541000] [154.0/1000.0] [batch_t 1.451 (1.429)] [data_t 0.003] [optim_t 1.448] [lr 0.005000] 2024-04-02 16:43:40,236 - ==> Total time: 15:43:34 Eta: 3 days, 14:23:33 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 16:45:48,012 - Train: 15.42% [83400/541000] [154.2/1000.0] [batch_t 1.430 (1.461)] [data_t 0.003] [optim_t 1.427] [lr 0.005000] 2024-04-02 16:48:10,983 - Train: 15.43% [83500/541000] [154.3/1000.0] [batch_t 1.424 (1.430)] [data_t 0.003] [optim_t 1.421] [lr 0.005000] 2024-04-02 16:50:34,003 - Train: 15.45% [83600/541000] [154.5/1000.0] [batch_t 1.434 (1.430)] [data_t 0.003] [optim_t 1.431] [lr 0.005000] 2024-04-02 16:52:57,182 - Train: 15.47% [83700/541000] [154.7/1000.0] [batch_t 1.444 (1.432)] [data_t 0.002] [optim_t 1.441] [lr 0.005000] 2024-04-02 16:55:20,358 - Train: 15.49% [83800/541000] [154.9/1000.0] [batch_t 1.424 (1.432)] [data_t 0.003] [optim_t 1.421] [lr 0.005000] 2024-04-02 16:56:39,045 - ==> Total time: 15:56:33 Eta: 3 days, 14:54:47 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 16:57:47,624 - Train: 15.51% [83900/541000] [155.1/1000.0] [batch_t 1.425 (1.470)] [data_t 0.003] [optim_t 1.422] [lr 0.005000] 2024-04-02 17:00:10,623 - Train: 15.53% [84000/541000] [155.3/1000.0] [batch_t 1.446 (1.430)] [data_t 0.002] [optim_t 1.444] [lr 0.005000] 2024-04-02 17:02:35,829 - Train: 15.55% [84100/541000] [155.5/1000.0] [batch_t 1.423 (1.452)] [data_t 0.003] [optim_t 1.420] [lr 0.005000] 2024-04-02 17:04:58,907 - Train: 15.56% [84200/541000] [155.6/1000.0] [batch_t 1.450 (1.431)] [data_t 0.002] [optim_t 1.448] [lr 0.005000] 2024-04-02 17:07:21,800 - Train: 15.58% [84300/541000] [155.8/1000.0] [batch_t 1.427 (1.429)] [data_t 0.002] [optim_t 1.424] [lr 0.005000] 2024-04-02 17:08:33,283 - ==> Total time: 16:08:27 Eta: 3 days, 15:19:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 17:08:39,057 - Train: 15.60% [84400/541000] [156.0/1000.0] [batch_t 0.608 (0.864)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 17:09:39,793 - Train: 15.62% [84500/541000] [156.2/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 17:10:40,956 - Train: 15.64% [84600/541000] [156.4/1000.0] [batch_t 0.608 (0.612)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 17:11:41,712 - Train: 15.66% [84700/541000] [156.6/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 17:12:42,654 - Train: 15.67% [84800/541000] [156.7/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 17:13:43,424 - Train: 15.69% [84900/541000] [156.9/1000.0] [batch_t 0.610 (0.608)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 17:14:05,897 - ==> Total time: 16:14:00 Eta: 3 days, 15:09:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 17:14:47,031 - Train: 15.71% [85000/541000] [157.1/1000.0] [batch_t 0.607 (0.616)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 17:15:47,771 - Train: 15.73% [85100/541000] [157.3/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 17:16:48,885 - Train: 15.75% [85200/541000] [157.5/1000.0] [batch_t 0.610 (0.611)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 17:17:49,657 - Train: 15.77% [85300/541000] [157.7/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 17:18:50,357 - Train: 15.79% [85400/541000] [157.9/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 17:19:37,733 - ==> Total time: 16:19:32 Eta: 3 days, 15:00:03 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 17:19:55,599 - Train: 15.80% [85500/541000] [158.0/1000.0] [batch_t 0.607 (0.732)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 17:20:56,292 - Train: 15.82% [85600/541000] [158.2/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 17:21:57,057 - Train: 15.84% [85700/541000] [158.4/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 17:22:57,746 - Train: 15.86% [85800/541000] [158.6/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 17:23:58,724 - Train: 15.88% [85900/541000] [158.8/1000.0] [batch_t 0.607 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 17:25:00,013 - Train: 15.90% [86000/541000] [159.0/1000.0] [batch_t 0.607 (0.613)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 17:25:13,776 - ==> Total time: 16:25:08 Eta: 3 days, 14:50:41 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 17:26:08,186 - Train: 15.91% [86100/541000] [159.1/1000.0] [batch_t 1.297 (0.637)] [data_t 0.693] [optim_t 0.603] [lr 0.005000] 2024-04-02 17:27:11,095 - Train: 15.93% [86200/541000] [159.3/1000.0] [batch_t 0.603 (0.629)] [data_t 0.002] [optim_t 0.601] [lr 0.005000] 2024-04-02 17:28:12,756 - Train: 15.95% [86300/541000] [159.5/1000.0] [batch_t 0.607 (0.617)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 17:29:17,707 - Train: 15.97% [86400/541000] [159.7/1000.0] [batch_t 0.607 (0.649)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 17:30:19,065 - Train: 15.99% [86500/541000] [159.9/1000.0] [batch_t 0.608 (0.613)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 17:30:55,578 - ==> Total time: 16:30:50 Eta: 3 days, 14:41:52 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 17:31:24,045 - Train: 16.01% [86600/541000] [160.1/1000.0] [batch_t 0.607 (0.651)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 17:32:24,745 - Train: 16.03% [86700/541000] [160.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 17:33:26,058 - Train: 16.04% [86800/541000] [160.4/1000.0] [batch_t 0.607 (0.613)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 17:34:26,782 - Train: 16.06% [86900/541000] [160.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 17:35:27,500 - Train: 16.08% [87000/541000] [160.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 17:36:28,299 - Train: 16.10% [87100/541000] [161.0/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 17:36:28,912 - ==> Total time: 16:36:23 Eta: 3 days, 14:32:21 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 17:37:32,246 - Train: 16.12% [87200/541000] [161.2/1000.0] [batch_t 0.606 (0.620)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 17:38:32,983 - Train: 16.14% [87300/541000] [161.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 17:39:33,924 - Train: 16.16% [87400/541000] [161.6/1000.0] [batch_t 0.606 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 17:40:34,654 - Train: 16.17% [87500/541000] [161.7/1000.0] [batch_t 0.608 (0.607)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-02 17:41:35,395 - Train: 16.19% [87600/541000] [161.9/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 17:42:00,914 - ==> Total time: 16:41:55 Eta: 3 days, 14:22:47 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 17:42:43,034 - Train: 16.21% [87700/541000] [162.1/1000.0] [batch_t 0.607 (0.648)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 17:43:43,750 - Train: 16.23% [87800/541000] [162.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 17:44:49,611 - Train: 16.25% [87900/541000] [162.5/1000.0] [batch_t 0.607 (0.659)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 17:45:50,335 - Train: 16.27% [88000/541000] [162.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 17:46:51,550 - Train: 16.28% [88100/541000] [162.8/1000.0] [batch_t 0.607 (0.612)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 17:47:41,970 - ==> Total time: 16:47:36 Eta: 3 days, 14:14:01 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 17:47:54,745 - Train: 16.30% [88200/541000] [163.0/1000.0] [batch_t 0.606 (0.630)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 17:48:55,508 - Train: 16.32% [88300/541000] [163.2/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 17:50:03,758 - Train: 16.34% [88400/541000] [163.4/1000.0] [batch_t 0.606 (0.682)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 17:51:04,503 - Train: 16.36% [88500/541000] [163.6/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 17:52:05,171 - Train: 16.38% [88600/541000] [163.8/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 17:53:05,909 - Train: 16.40% [88700/541000] [164.0/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 17:53:20,490 - ==> Total time: 16:53:14 Eta: 3 days, 14:05:05 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 17:54:09,218 - Train: 16.41% [88800/541000] [164.1/1000.0] [batch_t 0.608 (0.616)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 17:55:09,936 - Train: 16.43% [88900/541000] [164.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 17:56:10,644 - Train: 16.45% [89000/541000] [164.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 17:57:11,555 - Train: 16.47% [89100/541000] [164.7/1000.0] [batch_t 0.607 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 17:58:12,320 - Train: 16.49% [89200/541000] [164.9/1000.0] [batch_t 0.611 (0.608)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 17:58:51,828 - ==> Total time: 16:58:46 Eta: 3 days, 13:55:35 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 17:59:15,963 - Train: 16.51% [89300/541000] [165.1/1000.0] [batch_t 0.608 (0.630)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 18:00:17,381 - Train: 16.52% [89400/541000] [165.2/1000.0] [batch_t 0.606 (0.614)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 18:01:18,521 - Train: 16.54% [89500/541000] [165.4/1000.0] [batch_t 0.606 (0.611)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 18:02:20,064 - Train: 16.56% [89600/541000] [165.6/1000.0] [batch_t 0.608 (0.615)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-02 18:03:20,779 - Train: 16.58% [89700/541000] [165.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 18:04:21,493 - Train: 16.60% [89800/541000] [166.0/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 18:04:25,141 - ==> Total time: 17:04:19 Eta: 3 days, 13:46:18 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 18:05:25,355 - Train: 16.62% [89900/541000] [166.2/1000.0] [batch_t 0.608 (0.620)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 18:06:26,093 - Train: 16.64% [90000/541000] [166.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 18:07:26,817 - Train: 16.65% [90100/541000] [166.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 18:08:32,934 - Train: 16.67% [90200/541000] [166.7/1000.0] [batch_t 0.607 (0.661)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 18:09:33,637 - Train: 16.69% [90300/541000] [166.9/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 18:10:02,166 - ==> Total time: 17:09:56 Eta: 3 days, 13:37:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 18:10:39,501 - Train: 16.71% [90400/541000] [167.1/1000.0] [batch_t 0.607 (0.618)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 18:11:40,227 - Train: 16.73% [90500/541000] [167.3/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 18:12:44,309 - Train: 16.75% [90600/541000] [167.5/1000.0] [batch_t 0.607 (0.641)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 18:13:45,084 - Train: 16.77% [90700/541000] [167.7/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 18:14:45,797 - Train: 16.78% [90800/541000] [167.8/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 18:15:43,108 - ==> Total time: 17:15:37 Eta: 3 days, 13:28:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 18:15:52,957 - Train: 16.80% [90900/541000] [168.0/1000.0] [batch_t 0.607 (0.663)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 18:16:53,925 - Train: 16.82% [91000/541000] [168.2/1000.0] [batch_t 0.608 (0.610)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 18:17:54,804 - Train: 16.84% [91100/541000] [168.4/1000.0] [batch_t 0.606 (0.609)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-02 18:18:55,629 - Train: 16.86% [91200/541000] [168.6/1000.0] [batch_t 0.605 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 18:19:58,742 - Train: 16.88% [91300/541000] [168.8/1000.0] [batch_t 0.607 (0.631)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 18:20:59,435 - Train: 16.89% [91400/541000] [168.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 18:21:17,044 - ==> Total time: 17:21:11 Eta: 3 days, 13:19:42 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 18:22:02,160 - Train: 16.91% [91500/541000] [169.1/1000.0] [batch_t 0.607 (0.613)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 18:23:03,073 - Train: 16.93% [91600/541000] [169.3/1000.0] [batch_t 0.610 (0.609)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 18:24:03,766 - Train: 16.95% [91700/541000] [169.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 18:25:04,511 - Train: 16.97% [91800/541000] [169.7/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-02 18:26:05,273 - Train: 16.99% [91900/541000] [169.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 18:26:47,755 - ==> Total time: 17:26:42 Eta: 3 days, 13:10:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 18:27:08,443 - Train: 17.01% [92000/541000] [170.1/1000.0] [batch_t 0.608 (0.621)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-02 18:28:09,147 - Train: 17.02% [92100/541000] [170.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 18:29:09,857 - Train: 17.04% [92200/541000] [170.4/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 18:30:10,752 - Train: 17.06% [92300/541000] [170.6/1000.0] [batch_t 0.607 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 18:31:11,441 - Train: 17.08% [92400/541000] [170.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 18:32:12,158 - Train: 17.10% [92500/541000] [171.0/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 18:32:18,853 - ==> Total time: 17:32:13 Eta: 3 days, 13:01:07 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 18:33:15,403 - Train: 17.12% [92600/541000] [171.2/1000.0] [batch_t 0.608 (0.615)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 18:34:16,072 - Train: 17.13% [92700/541000] [171.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-02 18:35:16,927 - Train: 17.15% [92800/541000] [171.5/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 18:36:17,633 - Train: 17.17% [92900/541000] [171.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 18:37:18,339 - Train: 17.19% [93000/541000] [171.9/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 18:37:49,924 - ==> Total time: 17:37:44 Eta: 3 days, 12:51:54 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 18:38:32,649 - Train: 17.21% [93100/541000] [172.1/1000.0] [batch_t 0.608 (0.852)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 18:39:33,375 - Train: 17.23% [93200/541000] [172.3/1000.0] [batch_t 0.611 (0.607)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-02 18:40:34,249 - Train: 17.25% [93300/541000] [172.5/1000.0] [batch_t 0.607 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 18:41:34,966 - Train: 17.26% [93400/541000] [172.6/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 18:42:35,713 - Train: 17.28% [93500/541000] [172.8/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 18:43:41,617 - ==> Total time: 17:43:36 Eta: 3 days, 12:44:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 18:43:48,407 - Train: 17.30% [93600/541000] [173.0/1000.0] [batch_t 0.607 (0.719)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 18:44:49,103 - Train: 17.32% [93700/541000] [173.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 18:45:49,855 - Train: 17.34% [93800/541000] [173.4/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 18:46:50,672 - Train: 17.36% [93900/541000] [173.6/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-02 18:47:51,433 - Train: 17.38% [94000/541000] [173.8/1000.0] [batch_t 0.610 (0.608)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-02 18:48:52,125 - Train: 17.39% [94100/541000] [173.9/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 18:49:12,756 - ==> Total time: 17:49:07 Eta: 3 days, 12:35:14 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 18:49:55,508 - Train: 17.41% [94200/541000] [174.1/1000.0] [batch_t 0.609 (0.619)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-02 18:50:56,212 - Train: 17.43% [94300/541000] [174.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 18:51:56,998 - Train: 17.45% [94400/541000] [174.5/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-02 18:52:57,677 - Train: 17.47% [94500/541000] [174.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 18:53:58,419 - Train: 17.49% [94600/541000] [174.9/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 18:54:44,184 - ==> Total time: 17:54:38 Eta: 3 days, 12:26:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 18:55:01,743 - Train: 17.50% [94700/541000] [175.0/1000.0] [batch_t 0.607 (0.632)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-02 18:56:02,487 - Train: 17.52% [94800/541000] [175.2/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 18:58:25,760 - Train: 17.54% [94900/541000] [175.4/1000.0] [batch_t 0.608 (1.433)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-02 19:02:19,886 - Train: 17.56% [95000/541000] [175.6/1000.0] [batch_t 1.439 (2.341)] [data_t 0.002] [optim_t 1.437] [lr 0.005000] 2024-04-02 19:05:19,999 - Train: 17.58% [95100/541000] [175.8/1000.0] [batch_t 2.448 (1.801)] [data_t 1.004] [optim_t 1.443] [lr 0.005000] 2024-04-02 19:07:59,861 - Train: 17.60% [95200/541000] [176.0/1000.0] [batch_t 1.440 (1.599)] [data_t 0.003] [optim_t 1.437] [lr 0.005000] 2024-04-02 19:08:29,671 - ==> Total time: 18:08:24 Eta: 3 days, 12:55:41 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 19:11:46,023 - Train: 17.62% [95300/541000] [176.2/1000.0] [batch_t 1.448 (2.233)] [data_t 0.003] [optim_t 1.445] [lr 0.005000] 2024-04-02 19:14:56,218 - Train: 17.63% [95400/541000] [176.3/1000.0] [batch_t 1.431 (1.902)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-02 19:17:19,007 - Train: 17.65% [95500/541000] [176.5/1000.0] [batch_t 1.421 (1.428)] [data_t 0.003] [optim_t 1.419] [lr 0.005000] 2024-04-02 19:19:42,324 - Train: 17.67% [95600/541000] [176.7/1000.0] [batch_t 1.423 (1.433)] [data_t 0.003] [optim_t 1.420] [lr 0.005000] 2024-04-02 19:22:05,596 - Train: 17.69% [95700/541000] [176.9/1000.0] [batch_t 1.443 (1.433)] [data_t 0.003] [optim_t 1.440] [lr 0.005000] 2024-04-02 19:23:27,304 - ==> Total time: 18:23:21 Eta: 3 days, 13:30:19 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 19:24:37,525 - Train: 17.71% [95800/541000] [177.1/1000.0] [batch_t 1.431 (1.531)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-02 19:27:00,806 - Train: 17.73% [95900/541000] [177.3/1000.0] [batch_t 1.449 (1.433)] [data_t 0.003] [optim_t 1.445] [lr 0.005000] 2024-04-02 19:29:26,036 - Train: 17.74% [96000/541000] [177.4/1000.0] [batch_t 1.454 (1.452)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-02 19:31:51,605 - Train: 17.76% [96100/541000] [177.6/1000.0] [batch_t 1.455 (1.456)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-02 19:34:15,903 - Train: 17.78% [96200/541000] [177.8/1000.0] [batch_t 1.426 (1.443)] [data_t 0.003] [optim_t 1.423] [lr 0.005000] 2024-04-02 19:36:36,371 - ==> Total time: 18:36:30 Eta: 3 days, 13:56:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 19:36:42,583 - Train: 17.80% [96300/541000] [178.0/1000.0] [batch_t 1.430 (1.976)] [data_t 0.002] [optim_t 1.428] [lr 0.005000] 2024-04-02 19:39:05,769 - Train: 17.82% [96400/541000] [178.2/1000.0] [batch_t 1.417 (1.432)] [data_t 0.002] [optim_t 1.414] [lr 0.005000] 2024-04-02 19:41:29,011 - Train: 17.84% [96500/541000] [178.4/1000.0] [batch_t 1.444 (1.432)] [data_t 0.003] [optim_t 1.441] [lr 0.005000] 2024-04-02 19:43:52,185 - Train: 17.86% [96600/541000] [178.6/1000.0] [batch_t 1.433 (1.432)] [data_t 0.003] [optim_t 1.430] [lr 0.005000] 2024-04-02 19:46:15,501 - Train: 17.87% [96700/541000] [178.7/1000.0] [batch_t 1.421 (1.433)] [data_t 0.002] [optim_t 1.419] [lr 0.005000] 2024-04-02 19:48:38,605 - Train: 17.89% [96800/541000] [178.9/1000.0] [batch_t 1.427 (1.431)] [data_t 0.003] [optim_t 1.424] [lr 0.005000] 2024-04-02 19:49:34,502 - ==> Total time: 18:49:28 Eta: 3 days, 14:20:28 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 19:51:05,209 - Train: 17.91% [96900/541000] [179.1/1000.0] [batch_t 1.435 (1.449)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-02 19:53:28,339 - Train: 17.93% [97000/541000] [179.3/1000.0] [batch_t 1.437 (1.431)] [data_t 0.003] [optim_t 1.434] [lr 0.005000] 2024-04-02 19:55:51,672 - Train: 17.95% [97100/541000] [179.5/1000.0] [batch_t 1.425 (1.433)] [data_t 0.003] [optim_t 1.423] [lr 0.005000] 2024-04-02 19:58:15,024 - Train: 17.97% [97200/541000] [179.7/1000.0] [batch_t 1.440 (1.433)] [data_t 0.003] [optim_t 1.437] [lr 0.005000] 2024-04-02 20:00:38,233 - Train: 17.99% [97300/541000] [179.9/1000.0] [batch_t 1.428 (1.432)] [data_t 0.002] [optim_t 1.426] [lr 0.005000] 2024-04-02 20:02:32,802 - ==> Total time: 19:02:27 Eta: 3 days, 14:44:30 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 20:03:06,969 - Train: 18.00% [97400/541000] [180.0/1000.0] [batch_t 1.448 (1.538)] [data_t 0.003] [optim_t 1.445] [lr 0.005000] 2024-04-02 20:05:30,319 - Train: 18.02% [97500/541000] [180.2/1000.0] [batch_t 1.430 (1.433)] [data_t 0.002] [optim_t 1.428] [lr 0.005000] 2024-04-02 20:07:53,553 - Train: 18.04% [97600/541000] [180.4/1000.0] [batch_t 1.427 (1.432)] [data_t 0.003] [optim_t 1.424] [lr 0.005000] 2024-04-02 20:10:16,813 - Train: 18.06% [97700/541000] [180.6/1000.0] [batch_t 1.426 (1.433)] [data_t 0.003] [optim_t 1.423] [lr 0.005000] 2024-04-02 20:12:39,971 - Train: 18.08% [97800/541000] [180.8/1000.0] [batch_t 1.435 (1.431)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-02 20:15:03,199 - Train: 18.10% [97900/541000] [181.0/1000.0] [batch_t 1.430 (1.432)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-02 20:15:33,298 - ==> Total time: 19:15:27 Eta: 3 days, 15:08:18 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 20:17:33,897 - Train: 18.11% [98000/541000] [181.1/1000.0] [batch_t 1.436 (1.482)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-02 20:19:57,225 - Train: 18.13% [98100/541000] [181.3/1000.0] [batch_t 1.436 (1.433)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-02 20:22:20,528 - Train: 18.15% [98200/541000] [181.5/1000.0] [batch_t 1.437 (1.433)] [data_t 0.003] [optim_t 1.434] [lr 0.005000] 2024-04-02 20:24:43,687 - Train: 18.17% [98300/541000] [181.7/1000.0] [batch_t 1.427 (1.431)] [data_t 0.003] [optim_t 1.424] [lr 0.005000] 2024-04-02 20:27:06,952 - Train: 18.19% [98400/541000] [181.9/1000.0] [batch_t 1.424 (1.433)] [data_t 0.003] [optim_t 1.421] [lr 0.005000] 2024-04-02 20:28:35,776 - ==> Total time: 19:28:30 Eta: 3 days, 15:31:50 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 20:29:36,366 - Train: 18.21% [98500/541000] [182.1/1000.0] [batch_t 1.431 (1.496)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-02 20:31:59,466 - Train: 18.23% [98600/541000] [182.3/1000.0] [batch_t 1.426 (1.431)] [data_t 0.003] [optim_t 1.423] [lr 0.005000] 2024-04-02 20:34:22,579 - Train: 18.24% [98700/541000] [182.4/1000.0] [batch_t 1.441 (1.431)] [data_t 0.003] [optim_t 1.438] [lr 0.005000] 2024-04-02 20:36:45,836 - Train: 18.26% [98800/541000] [182.6/1000.0] [batch_t 1.431 (1.432)] [data_t 0.002] [optim_t 1.429] [lr 0.005000] 2024-04-02 20:39:08,934 - Train: 18.28% [98900/541000] [182.8/1000.0] [batch_t 1.439 (1.431)] [data_t 0.003] [optim_t 1.437] [lr 0.005000] 2024-04-02 20:41:32,199 - Train: 18.30% [99000/541000] [183.0/1000.0] [batch_t 1.432 (1.433)] [data_t 0.003] [optim_t 1.429] [lr 0.005000] 2024-04-02 20:41:36,519 - ==> Total time: 19:41:30 Eta: 3 days, 15:54:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 20:43:58,890 - Train: 18.32% [99100/541000] [183.2/1000.0] [batch_t 1.427 (1.444)] [data_t 0.003] [optim_t 1.425] [lr 0.005000] 2024-04-02 20:46:21,969 - Train: 18.34% [99200/541000] [183.4/1000.0] [batch_t 1.427 (1.431)] [data_t 0.002] [optim_t 1.424] [lr 0.005000] 2024-04-02 20:48:48,879 - Train: 18.35% [99300/541000] [183.5/1000.0] [batch_t 1.418 (1.469)] [data_t 0.003] [optim_t 1.415] [lr 0.005000] 2024-04-02 20:51:12,038 - Train: 18.37% [99400/541000] [183.7/1000.0] [batch_t 1.440 (1.431)] [data_t 0.003] [optim_t 1.437] [lr 0.005000] 2024-04-02 20:53:35,803 - Train: 18.39% [99500/541000] [183.9/1000.0] [batch_t 1.421 (1.438)] [data_t 0.002] [optim_t 1.419] [lr 0.005000] 2024-04-02 20:54:38,836 - ==> Total time: 19:54:33 Eta: 3 days, 16:17:35 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 20:56:02,335 - Train: 18.41% [99600/541000] [184.1/1000.0] [batch_t 1.442 (1.448)] [data_t 0.002] [optim_t 1.440] [lr 0.005000] 2024-04-02 20:58:25,636 - Train: 18.43% [99700/541000] [184.3/1000.0] [batch_t 1.437 (1.433)] [data_t 0.003] [optim_t 1.434] [lr 0.005000] 2024-04-02 21:00:48,931 - Train: 18.45% [99800/541000] [184.5/1000.0] [batch_t 1.434 (1.433)] [data_t 0.003] [optim_t 1.432] [lr 0.005000] 2024-04-02 21:03:12,283 - Train: 18.47% [99900/541000] [184.7/1000.0] [batch_t 1.437 (1.433)] [data_t 0.002] [optim_t 1.435] [lr 0.005000] 2024-04-02 21:05:35,508 - Train: 18.48% [100000/541000] [184.8/1000.0] [batch_t 1.431 (1.432)] [data_t 0.002] [optim_t 1.428] [lr 0.005000] 2024-04-02 21:07:37,276 - ==> Total time: 20:07:31 Eta: 3 days, 16:39:39 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 21:08:01,670 - Train: 18.50% [100100/541000] [185.0/1000.0] [batch_t 1.423 (1.492)] [data_t 0.002] [optim_t 1.421] [lr 0.005000] 2024-04-02 21:10:24,871 - Train: 18.52% [100200/541000] [185.2/1000.0] [batch_t 1.427 (1.432)] [data_t 0.002] [optim_t 1.425] [lr 0.005000] 2024-04-02 21:12:48,233 - Train: 18.54% [100300/541000] [185.4/1000.0] [batch_t 1.422 (1.434)] [data_t 0.002] [optim_t 1.419] [lr 0.005000] 2024-04-02 21:15:11,448 - Train: 18.56% [100400/541000] [185.6/1000.0] [batch_t 1.445 (1.432)] [data_t 0.003] [optim_t 1.442] [lr 0.005000] 2024-04-02 21:17:34,750 - Train: 18.58% [100500/541000] [185.8/1000.0] [batch_t 1.428 (1.433)] [data_t 0.002] [optim_t 1.426] [lr 0.005000] 2024-04-02 21:19:57,949 - Train: 18.60% [100600/541000] [186.0/1000.0] [batch_t 1.439 (1.432)] [data_t 0.003] [optim_t 1.436] [lr 0.005000] 2024-04-02 21:20:35,284 - ==> Total time: 20:20:29 Eta: 3 days, 17:01:18 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 21:22:25,276 - Train: 18.61% [100700/541000] [186.1/1000.0] [batch_t 1.446 (1.446)] [data_t 0.002] [optim_t 1.443] [lr 0.005000] 2024-04-02 21:24:48,635 - Train: 18.63% [100800/541000] [186.3/1000.0] [batch_t 1.434 (1.433)] [data_t 0.003] [optim_t 1.431] [lr 0.005000] 2024-04-02 21:27:11,731 - Train: 18.65% [100900/541000] [186.5/1000.0] [batch_t 1.434 (1.431)] [data_t 0.003] [optim_t 1.431] [lr 0.005000] 2024-04-02 21:29:35,094 - Train: 18.67% [101000/541000] [186.7/1000.0] [batch_t 1.423 (1.434)] [data_t 0.003] [optim_t 1.421] [lr 0.005000] 2024-04-02 21:31:58,216 - Train: 18.69% [101100/541000] [186.9/1000.0] [batch_t 1.430 (1.431)] [data_t 0.002] [optim_t 1.427] [lr 0.005000] 2024-04-02 21:33:34,152 - ==> Total time: 20:33:28 Eta: 3 days, 17:22:39 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 21:34:25,942 - Train: 18.71% [101200/541000] [187.1/1000.0] [batch_t 1.420 (1.483)] [data_t 0.003] [optim_t 1.417] [lr 0.005000] 2024-04-02 21:36:49,083 - Train: 18.72% [101300/541000] [187.2/1000.0] [batch_t 1.425 (1.431)] [data_t 0.003] [optim_t 1.422] [lr 0.005000] 2024-04-02 21:39:12,327 - Train: 18.74% [101400/541000] [187.4/1000.0] [batch_t 1.431 (1.432)] [data_t 0.002] [optim_t 1.429] [lr 0.005000] 2024-04-02 21:41:40,665 - Train: 18.76% [101500/541000] [187.6/1000.0] [batch_t 1.423 (1.483)] [data_t 0.003] [optim_t 1.420] [lr 0.005000] 2024-04-02 21:44:03,893 - Train: 18.78% [101600/541000] [187.8/1000.0] [batch_t 1.434 (1.432)] [data_t 0.002] [optim_t 1.432] [lr 0.005000] 2024-04-02 21:46:26,944 - Train: 18.80% [101700/541000] [188.0/1000.0] [batch_t 1.428 (1.430)] [data_t 0.003] [optim_t 1.426] [lr 0.005000] 2024-04-02 21:46:38,379 - ==> Total time: 20:46:32 Eta: 3 days, 17:44:01 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 21:48:53,899 - Train: 18.82% [101800/541000] [188.2/1000.0] [batch_t 1.436 (1.445)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-02 21:51:16,907 - Train: 18.84% [101900/541000] [188.4/1000.0] [batch_t 1.442 (1.430)] [data_t 0.003] [optim_t 1.439] [lr 0.005000] 2024-04-02 21:53:40,029 - Train: 18.85% [102000/541000] [188.5/1000.0] [batch_t 1.441 (1.431)] [data_t 0.003] [optim_t 1.438] [lr 0.005000] 2024-04-02 21:56:03,381 - Train: 18.87% [102100/541000] [188.7/1000.0] [batch_t 1.436 (1.433)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-02 21:58:26,514 - Train: 18.89% [102200/541000] [188.9/1000.0] [batch_t 1.431 (1.431)] [data_t 0.003] [optim_t 1.429] [lr 0.005000] 2024-04-02 21:59:36,669 - ==> Total time: 20:59:31 Eta: 3 days, 18:04:36 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 22:00:52,600 - Train: 18.91% [102300/541000] [189.1/1000.0] [batch_t 1.441 (1.448)] [data_t 0.003] [optim_t 1.438] [lr 0.005000] 2024-04-02 22:03:15,860 - Train: 18.93% [102400/541000] [189.3/1000.0] [batch_t 1.439 (1.433)] [data_t 0.002] [optim_t 1.437] [lr 0.005000] 2024-04-02 22:05:39,170 - Train: 18.95% [102500/541000] [189.5/1000.0] [batch_t 1.433 (1.433)] [data_t 0.002] [optim_t 1.431] [lr 0.005000] 2024-04-02 22:08:02,262 - Train: 18.96% [102600/541000] [189.6/1000.0] [batch_t 1.424 (1.431)] [data_t 0.002] [optim_t 1.422] [lr 0.005000] 2024-04-02 22:10:25,603 - Train: 18.98% [102700/541000] [189.8/1000.0] [batch_t 1.425 (1.433)] [data_t 0.003] [optim_t 1.422] [lr 0.005000] 2024-04-02 22:12:34,567 - ==> Total time: 21:12:29 Eta: 3 days, 18:24:47 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 22:12:52,013 - Train: 19.00% [102800/541000] [190.0/1000.0] [batch_t 1.431 (1.540)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-02 22:15:15,216 - Train: 19.02% [102900/541000] [190.2/1000.0] [batch_t 1.423 (1.432)] [data_t 0.003] [optim_t 1.420] [lr 0.005000] 2024-04-02 22:17:38,365 - Train: 19.04% [103000/541000] [190.4/1000.0] [batch_t 1.423 (1.431)] [data_t 0.002] [optim_t 1.420] [lr 0.005000] 2024-04-02 22:20:01,462 - Train: 19.06% [103100/541000] [190.6/1000.0] [batch_t 1.436 (1.431)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-02 22:22:24,655 - Train: 19.08% [103200/541000] [190.8/1000.0] [batch_t 1.430 (1.432)] [data_t 0.003] [optim_t 1.427] [lr 0.005000] 2024-04-02 22:24:49,493 - Train: 19.09% [103300/541000] [190.9/1000.0] [batch_t 1.439 (1.448)] [data_t 0.003] [optim_t 1.436] [lr 0.005000] 2024-04-02 22:25:33,888 - ==> Total time: 21:25:28 Eta: 3 days, 18:44:44 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 22:27:15,437 - Train: 19.11% [103400/541000] [191.1/1000.0] [batch_t 1.432 (1.441)] [data_t 0.003] [optim_t 1.429] [lr 0.005000] 2024-04-02 22:29:39,563 - Train: 19.13% [103500/541000] [191.3/1000.0] [batch_t 1.428 (1.441)] [data_t 0.003] [optim_t 1.425] [lr 0.005000] 2024-04-02 22:32:02,780 - Train: 19.15% [103600/541000] [191.5/1000.0] [batch_t 1.441 (1.432)] [data_t 0.003] [optim_t 1.438] [lr 0.005000] 2024-04-02 22:34:26,002 - Train: 19.17% [103700/541000] [191.7/1000.0] [batch_t 1.440 (1.432)] [data_t 0.003] [optim_t 1.437] [lr 0.005000] 2024-04-02 22:36:49,216 - Train: 19.19% [103800/541000] [191.9/1000.0] [batch_t 1.426 (1.432)] [data_t 0.002] [optim_t 1.424] [lr 0.005000] 2024-04-02 22:38:32,273 - ==> Total time: 21:38:26 Eta: 3 days, 19:04:17 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 22:39:15,907 - Train: 19.21% [103900/541000] [192.1/1000.0] [batch_t 1.429 (1.472)] [data_t 0.002] [optim_t 1.426] [lr 0.005000] 2024-04-02 22:41:39,040 - Train: 19.22% [104000/541000] [192.2/1000.0] [batch_t 1.432 (1.431)] [data_t 0.003] [optim_t 1.429] [lr 0.005000] 2024-04-02 22:44:02,162 - Train: 19.24% [104100/541000] [192.4/1000.0] [batch_t 1.431 (1.431)] [data_t 0.002] [optim_t 1.429] [lr 0.005000] 2024-04-02 22:46:25,421 - Train: 19.26% [104200/541000] [192.6/1000.0] [batch_t 1.426 (1.432)] [data_t 0.002] [optim_t 1.424] [lr 0.005000] 2024-04-02 22:48:48,557 - Train: 19.28% [104300/541000] [192.8/1000.0] [batch_t 1.436 (1.431)] [data_t 0.003] [optim_t 1.434] [lr 0.005000] 2024-04-02 22:51:11,719 - Train: 19.30% [104400/541000] [193.0/1000.0] [batch_t 1.441 (1.432)] [data_t 0.003] [optim_t 1.439] [lr 0.005000] 2024-04-02 22:51:30,322 - ==> Total time: 21:51:24 Eta: 3 days, 19:23:28 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 22:53:38,028 - Train: 19.32% [104500/541000] [193.2/1000.0] [batch_t 1.437 (1.443)] [data_t 0.003] [optim_t 1.435] [lr 0.005000] 2024-04-02 22:56:01,216 - Train: 19.33% [104600/541000] [193.3/1000.0] [batch_t 1.424 (1.432)] [data_t 0.002] [optim_t 1.422] [lr 0.005000] 2024-04-02 22:58:24,471 - Train: 19.35% [104700/541000] [193.5/1000.0] [batch_t 1.438 (1.432)] [data_t 0.002] [optim_t 1.436] [lr 0.005000] 2024-04-02 23:00:47,681 - Train: 19.37% [104800/541000] [193.7/1000.0] [batch_t 1.440 (1.432)] [data_t 0.002] [optim_t 1.438] [lr 0.005000] 2024-04-02 23:03:10,924 - Train: 19.39% [104900/541000] [193.9/1000.0] [batch_t 1.441 (1.432)] [data_t 0.003] [optim_t 1.439] [lr 0.005000] 2024-04-02 23:04:28,223 - ==> Total time: 22:04:22 Eta: 3 days, 19:42:18 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 23:05:36,421 - Train: 19.41% [105000/541000] [194.1/1000.0] [batch_t 1.435 (1.446)] [data_t 0.003] [optim_t 1.432] [lr 0.005000] 2024-04-02 23:07:59,686 - Train: 19.43% [105100/541000] [194.3/1000.0] [batch_t 1.432 (1.433)] [data_t 0.003] [optim_t 1.429] [lr 0.005000] 2024-04-02 23:10:22,708 - Train: 19.45% [105200/541000] [194.5/1000.0] [batch_t 1.436 (1.430)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-02 23:12:45,873 - Train: 19.46% [105300/541000] [194.6/1000.0] [batch_t 1.421 (1.432)] [data_t 0.002] [optim_t 1.419] [lr 0.005000] 2024-04-02 23:15:08,974 - Train: 19.48% [105400/541000] [194.8/1000.0] [batch_t 1.426 (1.431)] [data_t 0.002] [optim_t 1.423] [lr 0.005000] 2024-04-02 23:17:30,930 - ==> Total time: 22:17:25 Eta: 3 days, 20:01:09 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 23:17:40,915 - Train: 19.50% [105500/541000] [195.0/1000.0] [batch_t 1.453 (1.620)] [data_t 0.003] [optim_t 1.450] [lr 0.005000] 2024-04-02 23:20:06,510 - Train: 19.52% [105600/541000] [195.2/1000.0] [batch_t 1.457 (1.456)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-02 23:22:30,975 - Train: 19.54% [105700/541000] [195.4/1000.0] [batch_t 1.430 (1.445)] [data_t 0.002] [optim_t 1.428] [lr 0.005000] 2024-04-02 23:24:54,004 - Train: 19.56% [105800/541000] [195.6/1000.0] [batch_t 1.412 (1.430)] [data_t 0.003] [optim_t 1.409] [lr 0.005000] 2024-04-02 23:27:17,099 - Train: 19.57% [105900/541000] [195.7/1000.0] [batch_t 1.428 (1.431)] [data_t 0.003] [optim_t 1.426] [lr 0.005000] 2024-04-02 23:29:40,261 - Train: 19.59% [106000/541000] [195.9/1000.0] [batch_t 1.427 (1.432)] [data_t 0.002] [optim_t 1.424] [lr 0.005000] 2024-04-02 23:30:31,778 - ==> Total time: 22:30:26 Eta: 3 days, 20:19:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 23:32:05,911 - Train: 19.61% [106100/541000] [196.1/1000.0] [batch_t 1.427 (1.443)] [data_t 0.002] [optim_t 1.424] [lr 0.005000] 2024-04-02 23:34:29,112 - Train: 19.63% [106200/541000] [196.3/1000.0] [batch_t 1.427 (1.432)] [data_t 0.002] [optim_t 1.424] [lr 0.005000] 2024-04-02 23:36:52,195 - Train: 19.65% [106300/541000] [196.5/1000.0] [batch_t 1.430 (1.431)] [data_t 0.003] [optim_t 1.427] [lr 0.005000] 2024-04-02 23:39:15,369 - Train: 19.67% [106400/541000] [196.7/1000.0] [batch_t 1.446 (1.432)] [data_t 0.003] [optim_t 1.443] [lr 0.005000] 2024-04-02 23:41:38,535 - Train: 19.69% [106500/541000] [196.9/1000.0] [batch_t 1.444 (1.432)] [data_t 0.003] [optim_t 1.441] [lr 0.005000] 2024-04-02 23:43:28,747 - ==> Total time: 22:43:23 Eta: 3 days, 20:37:21 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 23:44:04,810 - Train: 19.70% [106600/541000] [197.0/1000.0] [batch_t 1.436 (1.475)] [data_t 0.002] [optim_t 1.433] [lr 0.005000] 2024-04-02 23:46:27,823 - Train: 19.72% [106700/541000] [197.2/1000.0] [batch_t 1.425 (1.430)] [data_t 0.003] [optim_t 1.423] [lr 0.005000] 2024-04-02 23:48:50,812 - Train: 19.74% [106800/541000] [197.4/1000.0] [batch_t 1.419 (1.430)] [data_t 0.003] [optim_t 1.416] [lr 0.005000] 2024-04-02 23:51:13,914 - Train: 19.76% [106900/541000] [197.6/1000.0] [batch_t 1.425 (1.431)] [data_t 0.003] [optim_t 1.422] [lr 0.005000] 2024-04-02 23:53:36,859 - Train: 19.78% [107000/541000] [197.8/1000.0] [batch_t 1.438 (1.429)] [data_t 0.003] [optim_t 1.436] [lr 0.005000] 2024-04-02 23:55:59,989 - Train: 19.80% [107100/541000] [198.0/1000.0] [batch_t 1.426 (1.431)] [data_t 0.003] [optim_t 1.423] [lr 0.005000] 2024-04-02 23:56:25,719 - ==> Total time: 22:56:20 Eta: 3 days, 20:54:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-02 23:58:25,309 - Train: 19.82% [107200/541000] [198.2/1000.0] [batch_t 1.426 (1.437)] [data_t 0.002] [optim_t 1.424] [lr 0.005000] 2024-04-03 00:00:48,304 - Train: 19.83% [107300/541000] [198.3/1000.0] [batch_t 1.426 (1.430)] [data_t 0.002] [optim_t 1.424] [lr 0.005000] 2024-04-03 00:03:11,588 - Train: 19.85% [107400/541000] [198.5/1000.0] [batch_t 1.438 (1.433)] [data_t 0.002] [optim_t 1.435] [lr 0.005000] 2024-04-03 00:05:34,840 - Train: 19.87% [107500/541000] [198.7/1000.0] [batch_t 1.431 (1.432)] [data_t 0.002] [optim_t 1.429] [lr 0.005000] 2024-04-03 00:07:58,205 - Train: 19.89% [107600/541000] [198.9/1000.0] [batch_t 1.424 (1.434)] [data_t 0.003] [optim_t 1.421] [lr 0.005000] 2024-04-03 00:09:22,770 - ==> Total time: 23:09:17 Eta: 3 days, 21:12:03 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 00:10:24,478 - Train: 19.91% [107700/541000] [199.1/1000.0] [batch_t 1.438 (1.451)] [data_t 0.003] [optim_t 1.435] [lr 0.005000] 2024-04-03 00:12:47,707 - Train: 19.93% [107800/541000] [199.3/1000.0] [batch_t 1.436 (1.432)] [data_t 0.002] [optim_t 1.434] [lr 0.005000] 2024-04-03 00:15:11,065 - Train: 19.94% [107900/541000] [199.4/1000.0] [batch_t 1.436 (1.433)] [data_t 0.002] [optim_t 1.433] [lr 0.005000] 2024-04-03 00:17:34,460 - Train: 19.96% [108000/541000] [199.6/1000.0] [batch_t 1.440 (1.434)] [data_t 0.003] [optim_t 1.438] [lr 0.005000] 2024-04-03 00:19:57,755 - Train: 19.98% [108100/541000] [199.8/1000.0] [batch_t 1.426 (1.433)] [data_t 0.002] [optim_t 1.423] [lr 0.005000] 2024-04-03 00:22:21,017 - Train: 20.00% [108200/541000] [200.0/1000.0] [batch_t 1.426 (1.433)] [data_t 0.002] [optim_t 1.424] [lr 0.005000] 2024-04-03 00:22:43,359 - Test: 36.76% [50/136] [batch_t 0.422 (0.436)] 2024-04-03 00:23:04,832 - Test: 73.53% [100/136] [batch_t 0.417 (0.433)] 2024-04-03 00:23:19,637 - Test: 100.00% [136/136] [batch_t 0.073 (0.427)] 2024-04-03 00:24:08,602 - ==> Metric Time for candle : 0.002 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.646 (mAUROC_px) 8.078 (mAP_px) 0.985 (mF1_max_px) 25.542 (mAUPRO_px) 0.326 (mF1_px_0.2_0.8_0.1) 0.342 (mAcc_px_0.2_0.8_0.1) 0.324 (mIoU_px_0.2_0.8_0.1) 0.944 (mIoU_max_px) 2024-04-03 00:24:46,516 - ==> Metric Time for capsules : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 8.476 (mAUROC_px) 6.396 (mAP_px) 0.768 (mF1_max_px) 20.536 (mAUPRO_px) 0.255 (mF1_px_0.2_0.8_0.1) 0.256 (mAcc_px_0.2_0.8_0.1) 0.258 (mIoU_px_0.2_0.8_0.1) 0.741 (mIoU_max_px) 2024-04-03 00:25:23,059 - ==> Metric Time for cashew : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.897 (mAUROC_px) 6.064 (mAP_px) 0.712 (mF1_max_px) 20.238 (mAUPRO_px) 0.240 (mF1_px_0.2_0.8_0.1) 0.243 (mAcc_px_0.2_0.8_0.1) 0.245 (mIoU_px_0.2_0.8_0.1) 0.692 (mIoU_max_px) 2024-04-03 00:25:59,728 - ==> Metric Time for chewinggum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.802 (mAUROC_px) 6.177 (mAP_px) 0.704 (mF1_max_px) 20.349 (mAUPRO_px) 0.243 (mF1_px_0.2_0.8_0.1) 0.241 (mAcc_px_0.2_0.8_0.1) 0.242 (mIoU_px_0.2_0.8_0.1) 0.703 (mIoU_max_px) 2024-04-03 00:26:36,838 - ==> Metric Time for fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.869 (mAUROC_px) 5.759 (mAP_px) 0.686 (mF1_max_px) 21.162 (mAUPRO_px) 0.243 (mF1_px_0.2_0.8_0.1) 0.239 (mAcc_px_0.2_0.8_0.1) 0.241 (mIoU_px_0.2_0.8_0.1) 0.699 (mIoU_max_px) 2024-04-03 00:27:24,073 - ==> Metric Time for macaroni1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.710 (mAUROC_px) 8.005 (mAP_px) 0.953 (mF1_max_px) 25.374 (mAUPRO_px) 0.323 (mF1_px_0.2_0.8_0.1) 0.324 (mAcc_px_0.2_0.8_0.1) 0.326 (mIoU_px_0.2_0.8_0.1) 0.947 (mIoU_max_px) 2024-04-03 00:28:10,969 - ==> Metric Time for macaroni2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.674 (mAUROC_px) 8.011 (mAP_px) 0.952 (mF1_max_px) 25.064 (mAUPRO_px) 0.322 (mF1_px_0.2_0.8_0.1) 0.326 (mAcc_px_0.2_0.8_0.1) 0.325 (mIoU_px_0.2_0.8_0.1) 0.941 (mIoU_max_px) 2024-04-03 00:28:58,898 - ==> Metric Time for pcb1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.600 (mAUROC_px) 8.090 (mAP_px) 0.948 (mF1_max_px) 26.100 (mAUPRO_px) 0.320 (mF1_px_0.2_0.8_0.1) 0.324 (mAcc_px_0.2_0.8_0.1) 0.327 (mIoU_px_0.2_0.8_0.1) 0.941 (mIoU_max_px) 2024-04-03 00:29:50,871 - ==> Metric Time for pcb2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.773 (mAUROC_px) 8.232 (mAP_px) 0.950 (mF1_max_px) 29.455 (mAUPRO_px) 0.382 (mF1_px_0.2_0.8_0.1) 0.389 (mAcc_px_0.2_0.8_0.1) 0.386 (mIoU_px_0.2_0.8_0.1) 1.122 (mIoU_max_px) 2024-04-03 00:30:40,327 - ==> Metric Time for pcb3 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.655 (mAUROC_px) 8.189 (mAP_px) 1.038 (mF1_max_px) 27.195 (mAUPRO_px) 0.347 (mF1_px_0.2_0.8_0.1) 0.353 (mAcc_px_0.2_0.8_0.1) 0.354 (mIoU_px_0.2_0.8_0.1) 1.026 (mIoU_max_px) 2024-04-03 00:31:30,899 - ==> Metric Time for pcb4 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.760 (mAUROC_px) 8.140 (mAP_px) 1.030 (mF1_max_px) 28.240 (mAUPRO_px) 0.369 (mF1_px_0.2_0.8_0.1) 0.359 (mAcc_px_0.2_0.8_0.1) 0.350 (mIoU_px_0.2_0.8_0.1) 1.028 (mIoU_max_px) 2024-04-03 00:32:08,075 - ==> Metric Time for pipe_fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.830 (mAUROC_px) 5.917 (mAP_px) 0.790 (mF1_max_px) 20.874 (mAUPRO_px) 0.259 (mF1_px_0.2_0.8_0.1) 0.262 (mAcc_px_0.2_0.8_0.1) 0.265 (mIoU_px_0.2_0.8_0.1) 0.752 (mIoU_max_px) 2024-04-03 00:32:08,095 - | Name | mAUROC_sp_max | mAUROC_sp_max (Max) | mAP_sp_max | mAP_sp_max (Max) | mF1_max_sp_max | mF1_max_sp_max (Max) | mAUROC_px | mAUROC_px (Max) | mAP_px | mAP_px (Max) | mF1_max_px | mF1_max_px (Max) | mAUPRO_px | mAUPRO_px (Max) | mF1_px_0.2_0.8_0.1 | mF1_px_0.2_0.8_0.1 (Max) | mAcc_px_0.2_0.8_0.1 | mAcc_px_0.2_0.8_0.1 (Max) | mIoU_px_0.2_0.8_0.1 | mIoU_px_0.2_0.8_0.1 (Max) | mIoU_max_px | mIoU_max_px (Max) | |:----------:|:---------------:|:---------------------:|:------------:|:------------------:|:----------------:|:----------------------:|:-----------:|:------------------:|:--------:|:------------------:|:------------:|:------------------:|:-----------:|:------------------:|:--------------------:|:--------------------------:|:---------------------:|:---------------------------:|:---------------------:|:---------------------------:|:-------------:|:-------------------:| | candle | 96.580 | 97.100 (50 epoch) | 96.698 | 97.346 (50 epoch) | 89.756 | 91.542 (50 epoch) | 99.039 | 99.039 (200 epoch) | 23.265 | 23.463 (150 epoch) | 32.654 | 32.654 (200 epoch) | 95.151 | 95.763 (50 epoch) | 19.569 | 19.845 (150 epoch) | 34.687 | 36.722 (100 epoch) | 11.227 | 11.386 (150 epoch) | 19.513 | 19.513 (200 epoch) | | capsules | 89.767 | 91.750 (100 epoch) | 93.910 | 94.964 (100 epoch) | 86.916 | 88.780 (100 epoch) | 99.047 | 99.123 (100 epoch) | 58.721 | 61.256 (100 epoch) | 59.342 | 59.766 (100 epoch) | 90.944 | 91.811 (100 epoch) | 31.745 | 33.186 (100 epoch) | 56.312 | 62.576 (50 epoch) | 20.591 | 21.701 (100 epoch) | 42.188 | 42.619 (100 epoch) | | cashew | 94.380 | 94.800 (150 epoch) | 97.290 | 97.492 (150 epoch) | 92.386 | 92.386 (200 epoch) | 94.904 | 94.904 (200 epoch) | 49.028 | 49.028 (200 epoch) | 52.806 | 52.806 (200 epoch) | 88.165 | 88.960 (150 epoch) | 28.047 | 28.279 (150 epoch) | 33.337 | 34.522 (150 epoch) | 17.536 | 17.655 (150 epoch) | 35.875 | 35.875 (200 epoch) | | chewinggum | 96.880 | 97.743 (100 epoch) | 98.568 | 98.928 (100 epoch) | 93.684 | 94.301 (50 epoch) | 98.064 | 98.145 (100 epoch) | 53.306 | 59.823 (50 epoch) | 57.907 | 61.088 (50 epoch) | 78.864 | 79.773 (50 epoch) | 28.946 | 35.748 (50 epoch) | 52.645 | 65.611 (50 epoch) | 18.802 | 23.754 (50 epoch) | 40.753 | 43.976 (50 epoch) | | fryum | 95.060 | 95.220 (150 epoch) | 97.628 | 97.737 (150 epoch) | 90.995 | 91.163 (150 epoch) | 97.022 | 97.089 (150 epoch) | 48.485 | 48.983 (150 epoch) | 52.388 | 52.388 (200 epoch) | 91.069 | 91.622 (100 epoch) | 25.067 | 26.312 (50 epoch) | 25.472 | 27.708 (50 epoch) | 15.553 | 16.200 (50 epoch) | 35.490 | 35.490 (200 epoch) | | macaroni1 | 90.384 | 91.568 (100 epoch) | 88.977 | 89.846 (100 epoch) | 81.111 | 81.592 (100 epoch) | 99.505 | 99.519 (100 epoch) | 18.719 | 18.719 (200 epoch) | 29.377 | 29.377 (200 epoch) | 95.326 | 95.494 (150 epoch) | 15.007 | 15.720 (50 epoch) | 36.478 | 40.765 (50 epoch) | 8.365 | 8.704 (50 epoch) | 17.217 | 17.217 (200 epoch) | | macaroni2 | 80.351 | 82.589 (50 epoch) | 74.750 | 81.196 (50 epoch) | 74.227 | 75.622 (50 epoch) | 99.394 | 99.452 (100 epoch) | 7.429 | 11.272 (50 epoch) | 14.742 | 17.568 (50 epoch) | 95.709 | 96.253 (150 epoch) | 6.942 | 10.086 (50 epoch) | 47.957 | 49.562 (50 epoch) | 3.664 | 5.433 (50 epoch) | 7.957 | 9.630 (50 epoch) | | pcb1 | 96.610 | 96.610 (200 epoch) | 95.251 | 95.251 (200 epoch) | 93.333 | 93.720 (150 epoch) | 99.784 | 99.784 (200 epoch) | 79.725 | 79.877 (150 epoch) | 73.608 | 73.861 (150 epoch) | 92.683 | 93.660 (50 epoch) | 46.891 | 46.891 (200 epoch) | 60.343 | 62.362 (50 epoch) | 32.945 | 32.976 (150 epoch) | 58.238 | 58.556 (150 epoch) | | pcb2 | 94.990 | 94.990 (200 epoch) | 94.755 | 94.755 (200 epoch) | 89.899 | 90.000 (150 epoch) | 98.970 | 98.970 (200 epoch) | 13.506 | 14.130 (50 epoch) | 23.967 | 23.967 (200 epoch) | 89.413 | 89.590 (100 epoch) | 16.150 | 16.150 (200 epoch) | 46.904 | 47.820 (150 epoch) | 8.892 | 8.892 (200 epoch) | 13.615 | 13.615 (200 epoch) | | pcb3 | 94.941 | 94.941 (200 epoch) | 95.114 | 95.114 (200 epoch) | 87.963 | 88.235 (150 epoch) | 99.117 | 99.124 (150 epoch) | 19.900 | 20.249 (150 epoch) | 27.757 | 27.912 (150 epoch) | 91.649 | 91.649 (200 epoch) | 19.617 | 20.042 (150 epoch) | 44.657 | 48.341 (50 epoch) | 11.047 | 11.295 (150 epoch) | 16.115 | 16.219 (150 epoch) | | pcb4 | 99.802 | 99.901 (100 epoch) | 99.801 | 99.901 (100 epoch) | 97.561 | 99.000 (50 epoch) | 98.641 | 98.641 (200 epoch) | 47.434 | 47.434 (200 epoch) | 46.752 | 46.928 (100 epoch) | 88.549 | 88.549 (200 epoch) | 26.925 | 26.925 (200 epoch) | 37.130 | 39.716 (50 epoch) | 16.537 | 16.537 (200 epoch) | 30.508 | 30.658 (100 epoch) | | pipe_fryum | 98.800 | 98.840 (150 epoch) | 99.363 | 99.400 (150 epoch) | 97.561 | 97.561 (200 epoch) | 99.030 | 99.080 (100 epoch) | 52.924 | 53.518 (100 epoch) | 56.768 | 58.465 (100 epoch) | 95.057 | 95.525 (150 epoch) | 33.450 | 34.338 (100 epoch) | 40.312 | 43.014 (100 epoch) | 21.513 | 22.178 (100 epoch) | 39.634 | 41.308 (100 epoch) | | Avg | 94.045 | 94.251 (100 epoch) | 94.342 | 94.481 (100 epoch) | 89.616 | 89.873 (150 epoch) | 98.543 | 98.543 (200 epoch) | 39.370 | 39.522 (150 epoch) | 44.006 | 44.006 (200 epoch) | 91.048 | 91.048 (200 epoch) | 24.863 | 25.164 (100 epoch) | 43.019 | 46.179 (50 epoch) | 15.556 | 15.755 (100 epoch) | 29.759 | 29.787 (150 epoch) | 2024-04-03 00:32:08,415 - ==> Total time: 23:32:02 Eta: 3 days, 22:08:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 00:34:35,043 - Train: 20.02% [108300/541000] [200.2/1000.0] [batch_t 1.426 (1.440)] [data_t 0.002] [optim_t 1.423] [lr 0.005000] 2024-04-03 00:36:58,391 - Train: 20.04% [108400/541000] [200.4/1000.0] [batch_t 1.434 (1.433)] [data_t 0.003] [optim_t 1.431] [lr 0.005000] 2024-04-03 00:39:21,731 - Train: 20.06% [108500/541000] [200.6/1000.0] [batch_t 1.416 (1.433)] [data_t 0.002] [optim_t 1.413] [lr 0.005000] 2024-04-03 00:41:45,131 - Train: 20.07% [108600/541000] [200.7/1000.0] [batch_t 1.433 (1.434)] [data_t 0.002] [optim_t 1.430] [lr 0.005000] 2024-04-03 00:44:08,448 - Train: 20.09% [108700/541000] [200.9/1000.0] [batch_t 1.422 (1.433)] [data_t 0.003] [optim_t 1.419] [lr 0.005000] 2024-04-03 00:45:07,153 - ==> Total time: 23:45:01 Eta: 3 days, 22:24:39 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 00:46:34,728 - Train: 20.11% [108800/541000] [201.1/1000.0] [batch_t 1.427 (1.446)] [data_t 0.002] [optim_t 1.425] [lr 0.005000] 2024-04-03 00:48:57,991 - Train: 20.13% [108900/541000] [201.3/1000.0] [batch_t 1.438 (1.433)] [data_t 0.003] [optim_t 1.435] [lr 0.005000] 2024-04-03 00:51:21,399 - Train: 20.15% [109000/541000] [201.5/1000.0] [batch_t 1.444 (1.434)] [data_t 0.003] [optim_t 1.441] [lr 0.005000] 2024-04-03 00:53:44,625 - Train: 20.17% [109100/541000] [201.7/1000.0] [batch_t 1.439 (1.432)] [data_t 0.003] [optim_t 1.437] [lr 0.005000] 2024-04-03 00:56:07,918 - Train: 20.18% [109200/541000] [201.8/1000.0] [batch_t 1.427 (1.433)] [data_t 0.003] [optim_t 1.424] [lr 0.005000] 2024-04-03 00:58:05,413 - ==> Total time: 23:57:59 Eta: 3 days, 22:40:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 00:58:34,976 - Train: 20.20% [109300/541000] [202.0/1000.0] [batch_t 1.434 (1.465)] [data_t 0.003] [optim_t 1.431] [lr 0.005000] 2024-04-03 01:00:58,331 - Train: 20.22% [109400/541000] [202.2/1000.0] [batch_t 1.438 (1.433)] [data_t 0.003] [optim_t 1.435] [lr 0.005000] 2024-04-03 01:03:21,713 - Train: 20.24% [109500/541000] [202.4/1000.0] [batch_t 1.441 (1.434)] [data_t 0.002] [optim_t 1.438] [lr 0.005000] 2024-04-03 01:05:44,935 - Train: 20.26% [109600/541000] [202.6/1000.0] [batch_t 1.441 (1.432)] [data_t 0.003] [optim_t 1.439] [lr 0.005000] 2024-04-03 01:08:08,412 - Train: 20.28% [109700/541000] [202.8/1000.0] [batch_t 1.423 (1.435)] [data_t 0.003] [optim_t 1.420] [lr 0.005000] 2024-04-03 01:10:31,806 - Train: 20.30% [109800/541000] [203.0/1000.0] [batch_t 1.425 (1.434)] [data_t 0.002] [optim_t 1.423] [lr 0.005000] 2024-04-03 01:11:04,774 - ==> Total time: 1 day, 0:10:59 Eta: 3 days, 22:56:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 01:12:58,570 - Train: 20.31% [109900/541000] [203.1/1000.0] [batch_t 1.433 (1.441)] [data_t 0.003] [optim_t 1.431] [lr 0.005000] 2024-04-03 01:15:21,893 - Train: 20.33% [110000/541000] [203.3/1000.0] [batch_t 1.436 (1.433)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-03 01:17:45,186 - Train: 20.35% [110100/541000] [203.5/1000.0] [batch_t 1.430 (1.433)] [data_t 0.002] [optim_t 1.428] [lr 0.005000] 2024-04-03 01:20:08,579 - Train: 20.37% [110200/541000] [203.7/1000.0] [batch_t 1.422 (1.434)] [data_t 0.003] [optim_t 1.419] [lr 0.005000] 2024-04-03 01:22:32,029 - Train: 20.39% [110300/541000] [203.9/1000.0] [batch_t 1.421 (1.434)] [data_t 0.003] [optim_t 1.418] [lr 0.005000] 2024-04-03 01:24:03,798 - ==> Total time: 1 day, 0:23:58 Eta: 3 days, 23:12:21 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 01:25:00,295 - Train: 20.41% [110400/541000] [204.1/1000.0] [batch_t 1.423 (1.466)] [data_t 0.003] [optim_t 1.420] [lr 0.005000] 2024-04-03 01:27:23,650 - Train: 20.43% [110500/541000] [204.3/1000.0] [batch_t 1.438 (1.433)] [data_t 0.003] [optim_t 1.435] [lr 0.005000] 2024-04-03 01:29:46,950 - Train: 20.44% [110600/541000] [204.4/1000.0] [batch_t 1.427 (1.433)] [data_t 0.002] [optim_t 1.425] [lr 0.005000] 2024-04-03 01:32:10,320 - Train: 20.46% [110700/541000] [204.6/1000.0] [batch_t 1.441 (1.434)] [data_t 0.003] [optim_t 1.439] [lr 0.005000] 2024-04-03 01:34:33,623 - Train: 20.48% [110800/541000] [204.8/1000.0] [batch_t 1.432 (1.433)] [data_t 0.002] [optim_t 1.429] [lr 0.005000] 2024-04-03 01:36:57,030 - Train: 20.50% [110900/541000] [205.0/1000.0] [batch_t 1.441 (1.434)] [data_t 0.003] [optim_t 1.439] [lr 0.005000] 2024-04-03 01:37:04,207 - ==> Total time: 1 day, 0:36:58 Eta: 3 days, 23:27:47 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 01:39:23,128 - Train: 20.52% [111000/541000] [205.2/1000.0] [batch_t 1.431 (1.440)] [data_t 0.002] [optim_t 1.429] [lr 0.005000] 2024-04-03 01:41:46,409 - Train: 20.54% [111100/541000] [205.4/1000.0] [batch_t 1.434 (1.433)] [data_t 0.002] [optim_t 1.431] [lr 0.005000] 2024-04-03 01:44:09,837 - Train: 20.55% [111200/541000] [205.5/1000.0] [batch_t 1.428 (1.434)] [data_t 0.003] [optim_t 1.426] [lr 0.005000] 2024-04-03 01:46:33,091 - Train: 20.57% [111300/541000] [205.7/1000.0] [batch_t 1.432 (1.432)] [data_t 0.003] [optim_t 1.430] [lr 0.005000] 2024-04-03 01:48:56,417 - Train: 20.59% [111400/541000] [205.9/1000.0] [batch_t 1.435 (1.433)] [data_t 0.003] [optim_t 1.432] [lr 0.005000] 2024-04-03 01:50:02,262 - ==> Total time: 1 day, 0:49:56 Eta: 3 days, 23:42:47 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 01:51:23,642 - Train: 20.61% [111500/541000] [206.1/1000.0] [batch_t 1.441 (1.445)] [data_t 0.003] [optim_t 1.439] [lr 0.005000] 2024-04-03 01:53:46,877 - Train: 20.63% [111600/541000] [206.3/1000.0] [batch_t 1.439 (1.432)] [data_t 0.003] [optim_t 1.436] [lr 0.005000] 2024-04-03 01:56:10,144 - Train: 20.65% [111700/541000] [206.5/1000.0] [batch_t 1.439 (1.433)] [data_t 0.003] [optim_t 1.436] [lr 0.005000] 2024-04-03 01:58:33,544 - Train: 20.67% [111800/541000] [206.7/1000.0] [batch_t 1.427 (1.434)] [data_t 0.002] [optim_t 1.424] [lr 0.005000] 2024-04-03 02:00:56,862 - Train: 20.68% [111900/541000] [206.8/1000.0] [batch_t 1.438 (1.433)] [data_t 0.003] [optim_t 1.435] [lr 0.005000] 2024-04-03 02:03:01,534 - ==> Total time: 1 day, 1:02:55 Eta: 3 days, 23:57:36 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 02:03:24,034 - Train: 20.70% [112000/541000] [207.0/1000.0] [batch_t 1.427 (1.480)] [data_t 0.003] [optim_t 1.424] [lr 0.005000] 2024-04-03 02:05:47,390 - Train: 20.72% [112100/541000] [207.2/1000.0] [batch_t 1.442 (1.433)] [data_t 0.003] [optim_t 1.439] [lr 0.005000] 2024-04-03 02:08:10,754 - Train: 20.74% [112200/541000] [207.4/1000.0] [batch_t 1.422 (1.434)] [data_t 0.002] [optim_t 1.420] [lr 0.005000] 2024-04-03 02:10:34,006 - Train: 20.76% [112300/541000] [207.6/1000.0] [batch_t 1.447 (1.432)] [data_t 0.002] [optim_t 1.444] [lr 0.005000] 2024-04-03 02:12:57,401 - Train: 20.78% [112400/541000] [207.8/1000.0] [batch_t 1.432 (1.434)] [data_t 0.003] [optim_t 1.430] [lr 0.005000] 2024-04-03 02:15:20,787 - Train: 20.79% [112500/541000] [207.9/1000.0] [batch_t 1.447 (1.434)] [data_t 0.003] [optim_t 1.444] [lr 0.005000] 2024-04-03 02:16:00,870 - ==> Total time: 1 day, 1:15:55 Eta: 4 days, 0:12:09 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 02:17:48,404 - Train: 20.81% [112600/541000] [208.1/1000.0] [batch_t 1.425 (1.441)] [data_t 0.002] [optim_t 1.423] [lr 0.005000] 2024-04-03 02:20:11,769 - Train: 20.83% [112700/541000] [208.3/1000.0] [batch_t 1.436 (1.434)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-03 02:22:35,069 - Train: 20.85% [112800/541000] [208.5/1000.0] [batch_t 1.429 (1.433)] [data_t 0.002] [optim_t 1.427] [lr 0.005000] 2024-04-03 02:24:58,478 - Train: 20.87% [112900/541000] [208.7/1000.0] [batch_t 1.431 (1.434)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-03 02:27:21,614 - Train: 20.89% [113000/541000] [208.9/1000.0] [batch_t 1.422 (1.431)] [data_t 0.002] [optim_t 1.420] [lr 0.005000] 2024-04-03 02:29:00,461 - ==> Total time: 1 day, 1:28:54 Eta: 4 days, 0:26:28 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 02:29:50,147 - Train: 20.91% [113100/541000] [209.1/1000.0] [batch_t 1.421 (1.533)] [data_t 0.003] [optim_t 1.418] [lr 0.005000] 2024-04-03 02:32:13,478 - Train: 20.92% [113200/541000] [209.2/1000.0] [batch_t 1.432 (1.433)] [data_t 0.002] [optim_t 1.430] [lr 0.005000] 2024-04-03 02:34:36,889 - Train: 20.94% [113300/541000] [209.4/1000.0] [batch_t 1.424 (1.434)] [data_t 0.003] [optim_t 1.421] [lr 0.005000] 2024-04-03 02:37:00,246 - Train: 20.96% [113400/541000] [209.6/1000.0] [batch_t 1.441 (1.433)] [data_t 0.003] [optim_t 1.438] [lr 0.005000] 2024-04-03 02:39:23,598 - Train: 20.98% [113500/541000] [209.8/1000.0] [batch_t 1.432 (1.433)] [data_t 0.003] [optim_t 1.429] [lr 0.005000] 2024-04-03 02:41:46,998 - Train: 21.00% [113600/541000] [210.0/1000.0] [batch_t 1.441 (1.434)] [data_t 0.003] [optim_t 1.438] [lr 0.005000] 2024-04-03 02:42:01,324 - ==> Total time: 1 day, 1:41:55 Eta: 4 days, 0:40:35 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 02:44:14,319 - Train: 21.02% [113700/541000] [210.2/1000.0] [batch_t 1.433 (1.437)] [data_t 0.002] [optim_t 1.430] [lr 0.005000] 2024-04-03 02:46:37,631 - Train: 21.04% [113800/541000] [210.4/1000.0] [batch_t 1.440 (1.433)] [data_t 0.003] [optim_t 1.437] [lr 0.005000] 2024-04-03 02:49:00,788 - Train: 21.05% [113900/541000] [210.5/1000.0] [batch_t 1.426 (1.431)] [data_t 0.003] [optim_t 1.423] [lr 0.005000] 2024-04-03 02:51:24,156 - Train: 21.07% [114000/541000] [210.7/1000.0] [batch_t 1.433 (1.434)] [data_t 0.003] [optim_t 1.430] [lr 0.005000] 2024-04-03 02:53:47,310 - Train: 21.09% [114100/541000] [210.9/1000.0] [batch_t 1.422 (1.431)] [data_t 0.003] [optim_t 1.420] [lr 0.005000] 2024-04-03 02:55:00,478 - ==> Total time: 1 day, 1:54:54 Eta: 4 days, 0:54:21 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 02:56:14,737 - Train: 21.11% [114200/541000] [211.1/1000.0] [batch_t 1.433 (1.478)] [data_t 0.003] [optim_t 1.430] [lr 0.005000] 2024-04-03 02:58:38,078 - Train: 21.13% [114300/541000] [211.3/1000.0] [batch_t 1.425 (1.433)] [data_t 0.003] [optim_t 1.423] [lr 0.005000] 2024-04-03 03:01:01,406 - Train: 21.15% [114400/541000] [211.5/1000.0] [batch_t 1.422 (1.433)] [data_t 0.003] [optim_t 1.419] [lr 0.005000] 2024-04-03 03:03:24,838 - Train: 21.16% [114500/541000] [211.6/1000.0] [batch_t 1.437 (1.434)] [data_t 0.003] [optim_t 1.434] [lr 0.005000] 2024-04-03 03:05:49,945 - Train: 21.18% [114600/541000] [211.8/1000.0] [batch_t 1.456 (1.451)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-03 03:08:04,077 - ==> Total time: 1 day, 2:07:58 Eta: 4 days, 1:08:08 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 03:08:19,703 - Train: 21.20% [114700/541000] [212.0/1000.0] [batch_t 1.458 (1.546)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-03 03:10:43,853 - Train: 21.22% [114800/541000] [212.2/1000.0] [batch_t 1.426 (1.441)] [data_t 0.002] [optim_t 1.424] [lr 0.005000] 2024-04-03 03:13:07,083 - Train: 21.24% [114900/541000] [212.4/1000.0] [batch_t 1.441 (1.432)] [data_t 0.003] [optim_t 1.438] [lr 0.005000] 2024-04-03 03:15:30,421 - Train: 21.26% [115000/541000] [212.6/1000.0] [batch_t 1.436 (1.433)] [data_t 0.002] [optim_t 1.434] [lr 0.005000] 2024-04-03 03:17:53,827 - Train: 21.28% [115100/541000] [212.8/1000.0] [batch_t 1.442 (1.434)] [data_t 0.003] [optim_t 1.439] [lr 0.005000] 2024-04-03 03:20:17,187 - Train: 21.29% [115200/541000] [212.9/1000.0] [batch_t 1.428 (1.433)] [data_t 0.003] [optim_t 1.425] [lr 0.005000] 2024-04-03 03:21:04,387 - ==> Total time: 1 day, 2:20:58 Eta: 4 days, 1:21:27 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 03:22:43,701 - Train: 21.31% [115300/541000] [213.1/1000.0] [batch_t 1.436 (1.443)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-03 03:25:07,043 - Train: 21.33% [115400/541000] [213.3/1000.0] [batch_t 1.436 (1.433)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-03 03:27:30,269 - Train: 21.35% [115500/541000] [213.5/1000.0] [batch_t 1.431 (1.432)] [data_t 0.002] [optim_t 1.429] [lr 0.005000] 2024-04-03 03:29:53,491 - Train: 21.37% [115600/541000] [213.7/1000.0] [batch_t 1.421 (1.432)] [data_t 0.002] [optim_t 1.419] [lr 0.005000] 2024-04-03 03:32:16,844 - Train: 21.39% [115700/541000] [213.9/1000.0] [batch_t 1.425 (1.433)] [data_t 0.002] [optim_t 1.423] [lr 0.005000] 2024-04-03 03:34:02,811 - ==> Total time: 1 day, 2:33:57 Eta: 4 days, 1:34:25 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 03:34:43,694 - Train: 21.40% [115800/541000] [214.0/1000.0] [batch_t 1.438 (1.460)] [data_t 0.003] [optim_t 1.435] [lr 0.005000] 2024-04-03 03:37:07,014 - Train: 21.42% [115900/541000] [214.2/1000.0] [batch_t 1.437 (1.433)] [data_t 0.003] [optim_t 1.435] [lr 0.005000] 2024-04-03 03:39:30,269 - Train: 21.44% [116000/541000] [214.4/1000.0] [batch_t 1.437 (1.432)] [data_t 0.003] [optim_t 1.434] [lr 0.005000] 2024-04-03 03:41:53,538 - Train: 21.46% [116100/541000] [214.6/1000.0] [batch_t 1.440 (1.433)] [data_t 0.003] [optim_t 1.437] [lr 0.005000] 2024-04-03 03:44:16,833 - Train: 21.48% [116200/541000] [214.8/1000.0] [batch_t 1.431 (1.433)] [data_t 0.002] [optim_t 1.429] [lr 0.005000] 2024-04-03 03:46:40,212 - Train: 21.50% [116300/541000] [215.0/1000.0] [batch_t 1.431 (1.434)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-03 03:47:01,660 - ==> Total time: 1 day, 2:46:56 Eta: 4 days, 1:47:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 03:49:08,613 - Train: 21.52% [116400/541000] [215.2/1000.0] [batch_t 1.446 (1.444)] [data_t 0.003] [optim_t 1.443] [lr 0.005000] 2024-04-03 03:51:31,856 - Train: 21.53% [116500/541000] [215.3/1000.0] [batch_t 1.429 (1.432)] [data_t 0.002] [optim_t 1.426] [lr 0.005000] 2024-04-03 03:53:55,157 - Train: 21.55% [116600/541000] [215.5/1000.0] [batch_t 1.439 (1.433)] [data_t 0.003] [optim_t 1.436] [lr 0.005000] 2024-04-03 03:56:18,559 - Train: 21.57% [116700/541000] [215.7/1000.0] [batch_t 1.423 (1.434)] [data_t 0.003] [optim_t 1.420] [lr 0.005000] 2024-04-03 03:58:41,903 - Train: 21.59% [116800/541000] [215.9/1000.0] [batch_t 1.421 (1.433)] [data_t 0.003] [optim_t 1.418] [lr 0.005000] 2024-04-03 04:00:02,130 - ==> Total time: 1 day, 2:59:56 Eta: 4 days, 1:59:47 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 04:01:10,666 - Train: 21.61% [116900/541000] [216.1/1000.0] [batch_t 1.431 (1.451)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-03 04:03:34,065 - Train: 21.63% [117000/541000] [216.3/1000.0] [batch_t 1.433 (1.434)] [data_t 0.003] [optim_t 1.430] [lr 0.005000] 2024-04-03 04:05:57,484 - Train: 21.65% [117100/541000] [216.5/1000.0] [batch_t 1.439 (1.434)] [data_t 0.003] [optim_t 1.436] [lr 0.005000] 2024-04-03 04:08:20,818 - Train: 21.66% [117200/541000] [216.6/1000.0] [batch_t 1.431 (1.433)] [data_t 0.003] [optim_t 1.429] [lr 0.005000] 2024-04-03 04:10:44,146 - Train: 21.68% [117300/541000] [216.8/1000.0] [batch_t 1.425 (1.433)] [data_t 0.002] [optim_t 1.423] [lr 0.005000] 2024-04-03 04:13:03,146 - ==> Total time: 1 day, 3:12:57 Eta: 4 days, 2:12:12 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 04:13:10,252 - Train: 21.70% [117400/541000] [217.0/1000.0] [batch_t 1.437 (1.649)] [data_t 0.003] [optim_t 1.434] [lr 0.005000] 2024-04-03 04:15:33,503 - Train: 21.72% [117500/541000] [217.2/1000.0] [batch_t 1.443 (1.432)] [data_t 0.003] [optim_t 1.440] [lr 0.005000] 2024-04-03 04:17:56,827 - Train: 21.74% [117600/541000] [217.4/1000.0] [batch_t 1.427 (1.433)] [data_t 0.002] [optim_t 1.424] [lr 0.005000] 2024-04-03 04:20:20,168 - Train: 21.76% [117700/541000] [217.6/1000.0] [batch_t 1.430 (1.433)] [data_t 0.002] [optim_t 1.428] [lr 0.005000] 2024-04-03 04:22:43,525 - Train: 21.77% [117800/541000] [217.7/1000.0] [batch_t 1.427 (1.433)] [data_t 0.002] [optim_t 1.425] [lr 0.005000] 2024-04-03 04:25:06,876 - Train: 21.79% [117900/541000] [217.9/1000.0] [batch_t 1.431 (1.433)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-03 04:26:01,330 - ==> Total time: 1 day, 3:25:55 Eta: 4 days, 2:24:12 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 04:27:35,712 - Train: 21.81% [118000/541000] [218.1/1000.0] [batch_t 1.435 (1.457)] [data_t 0.003] [optim_t 1.432] [lr 0.005000] 2024-04-03 04:29:59,120 - Train: 21.83% [118100/541000] [218.3/1000.0] [batch_t 1.437 (1.434)] [data_t 0.003] [optim_t 1.435] [lr 0.005000] 2024-04-03 04:32:22,425 - Train: 21.85% [118200/541000] [218.5/1000.0] [batch_t 1.437 (1.433)] [data_t 0.002] [optim_t 1.435] [lr 0.005000] 2024-04-03 04:34:45,843 - Train: 21.87% [118300/541000] [218.7/1000.0] [batch_t 1.433 (1.434)] [data_t 0.003] [optim_t 1.430] [lr 0.005000] 2024-04-03 04:37:09,106 - Train: 21.89% [118400/541000] [218.9/1000.0] [batch_t 1.432 (1.433)] [data_t 0.002] [optim_t 1.430] [lr 0.005000] 2024-04-03 04:39:02,345 - ==> Total time: 1 day, 3:38:56 Eta: 4 days, 2:36:09 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 04:39:37,534 - Train: 21.90% [118500/541000] [219.0/1000.0] [batch_t 1.421 (1.467)] [data_t 0.003] [optim_t 1.418] [lr 0.005000] 2024-04-03 04:42:01,063 - Train: 21.92% [118600/541000] [219.2/1000.0] [batch_t 1.434 (1.435)] [data_t 0.003] [optim_t 1.431] [lr 0.005000] 2024-04-03 04:44:24,490 - Train: 21.94% [118700/541000] [219.4/1000.0] [batch_t 1.431 (1.434)] [data_t 0.002] [optim_t 1.429] [lr 0.005000] 2024-04-03 04:46:47,758 - Train: 21.96% [118800/541000] [219.6/1000.0] [batch_t 1.424 (1.433)] [data_t 0.003] [optim_t 1.421] [lr 0.005000] 2024-04-03 04:49:11,156 - Train: 21.98% [118900/541000] [219.8/1000.0] [batch_t 1.435 (1.434)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-03 04:51:34,484 - Train: 22.00% [119000/541000] [220.0/1000.0] [batch_t 1.428 (1.433)] [data_t 0.002] [optim_t 1.425] [lr 0.005000] 2024-04-03 04:52:03,134 - ==> Total time: 1 day, 3:51:57 Eta: 4 days, 2:47:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 04:54:02,022 - Train: 22.01% [119100/541000] [220.1/1000.0] [batch_t 1.433 (1.444)] [data_t 0.003] [optim_t 1.430] [lr 0.005000] 2024-04-03 04:56:25,477 - Train: 22.03% [119200/541000] [220.3/1000.0] [batch_t 1.422 (1.434)] [data_t 0.003] [optim_t 1.419] [lr 0.005000] 2024-04-03 04:58:48,798 - Train: 22.05% [119300/541000] [220.5/1000.0] [batch_t 1.443 (1.433)] [data_t 0.002] [optim_t 1.441] [lr 0.005000] 2024-04-03 05:01:12,209 - Train: 22.07% [119400/541000] [220.7/1000.0] [batch_t 1.435 (1.434)] [data_t 0.003] [optim_t 1.432] [lr 0.005000] 2024-04-03 05:03:35,455 - Train: 22.09% [119500/541000] [220.9/1000.0] [batch_t 1.431 (1.432)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-03 05:05:02,889 - ==> Total time: 1 day, 4:04:57 Eta: 4 days, 2:59:16 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 05:06:02,282 - Train: 22.11% [119600/541000] [221.1/1000.0] [batch_t 1.432 (1.450)] [data_t 0.002] [optim_t 1.430] [lr 0.005000] 2024-04-03 05:08:25,683 - Train: 22.13% [119700/541000] [221.3/1000.0] [batch_t 1.440 (1.434)] [data_t 0.002] [optim_t 1.438] [lr 0.005000] 2024-04-03 05:10:49,090 - Train: 22.14% [119800/541000] [221.4/1000.0] [batch_t 1.428 (1.434)] [data_t 0.002] [optim_t 1.426] [lr 0.005000] 2024-04-03 05:13:12,408 - Train: 22.16% [119900/541000] [221.6/1000.0] [batch_t 1.437 (1.433)] [data_t 0.003] [optim_t 1.434] [lr 0.005000] 2024-04-03 05:15:35,736 - Train: 22.18% [120000/541000] [221.8/1000.0] [batch_t 1.423 (1.433)] [data_t 0.003] [optim_t 1.421] [lr 0.005000] 2024-04-03 05:17:59,136 - Train: 22.20% [120100/541000] [222.0/1000.0] [batch_t 1.447 (1.434)] [data_t 0.002] [optim_t 1.445] [lr 0.005000] 2024-04-03 05:18:02,040 - ==> Total time: 1 day, 4:17:56 Eta: 4 days, 3:10:26 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 05:20:27,311 - Train: 22.22% [120200/541000] [222.2/1000.0] [batch_t 1.433 (1.439)] [data_t 0.003] [optim_t 1.431] [lr 0.005000] 2024-04-03 05:22:50,490 - Train: 22.24% [120300/541000] [222.4/1000.0] [batch_t 1.417 (1.432)] [data_t 0.002] [optim_t 1.415] [lr 0.005000] 2024-04-03 05:25:13,600 - Train: 22.26% [120400/541000] [222.6/1000.0] [batch_t 1.433 (1.431)] [data_t 0.003] [optim_t 1.430] [lr 0.005000] 2024-04-03 05:27:36,947 - Train: 22.27% [120500/541000] [222.7/1000.0] [batch_t 1.441 (1.433)] [data_t 0.003] [optim_t 1.438] [lr 0.005000] 2024-04-03 05:30:00,290 - Train: 22.29% [120600/541000] [222.9/1000.0] [batch_t 1.435 (1.433)] [data_t 0.003] [optim_t 1.432] [lr 0.005000] 2024-04-03 05:31:01,839 - ==> Total time: 1 day, 4:30:56 Eta: 4 days, 3:21:25 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 05:32:27,878 - Train: 22.31% [120700/541000] [223.1/1000.0] [batch_t 1.427 (1.444)] [data_t 0.002] [optim_t 1.425] [lr 0.005000] 2024-04-03 05:34:51,056 - Train: 22.33% [120800/541000] [223.3/1000.0] [batch_t 1.440 (1.432)] [data_t 0.003] [optim_t 1.436] [lr 0.005000] 2024-04-03 05:37:14,438 - Train: 22.35% [120900/541000] [223.5/1000.0] [batch_t 1.442 (1.434)] [data_t 0.003] [optim_t 1.440] [lr 0.005000] 2024-04-03 05:39:37,792 - Train: 22.37% [121000/541000] [223.7/1000.0] [batch_t 1.441 (1.433)] [data_t 0.003] [optim_t 1.438] [lr 0.005000] 2024-04-03 05:42:00,989 - Train: 22.38% [121100/541000] [223.8/1000.0] [batch_t 1.436 (1.432)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-03 05:44:01,370 - ==> Total time: 1 day, 4:43:55 Eta: 4 days, 3:32:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 05:44:30,673 - Train: 22.40% [121200/541000] [224.0/1000.0] [batch_t 1.439 (1.543)] [data_t 0.003] [optim_t 1.436] [lr 0.005000] 2024-04-03 05:46:54,014 - Train: 22.42% [121300/541000] [224.2/1000.0] [batch_t 1.417 (1.433)] [data_t 0.003] [optim_t 1.414] [lr 0.005000] 2024-04-03 05:49:17,342 - Train: 22.44% [121400/541000] [224.4/1000.0] [batch_t 1.437 (1.433)] [data_t 0.002] [optim_t 1.435] [lr 0.005000] 2024-04-03 05:51:40,659 - Train: 22.46% [121500/541000] [224.6/1000.0] [batch_t 1.435 (1.433)] [data_t 0.003] [optim_t 1.432] [lr 0.005000] 2024-04-03 05:54:03,989 - Train: 22.48% [121600/541000] [224.8/1000.0] [batch_t 1.431 (1.433)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-03 05:56:27,254 - Train: 22.50% [121700/541000] [225.0/1000.0] [batch_t 1.423 (1.433)] [data_t 0.002] [optim_t 1.421] [lr 0.005000] 2024-04-03 05:57:03,070 - ==> Total time: 1 day, 4:56:57 Eta: 4 days, 3:42:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 05:58:54,559 - Train: 22.51% [121800/541000] [225.1/1000.0] [batch_t 1.444 (1.447)] [data_t 0.003] [optim_t 1.442] [lr 0.005000] 2024-04-03 06:01:17,932 - Train: 22.53% [121900/541000] [225.3/1000.0] [batch_t 1.435 (1.434)] [data_t 0.003] [optim_t 1.432] [lr 0.005000] 2024-04-03 06:03:41,299 - Train: 22.55% [122000/541000] [225.5/1000.0] [batch_t 1.433 (1.434)] [data_t 0.002] [optim_t 1.430] [lr 0.005000] 2024-04-03 06:06:04,444 - Train: 22.57% [122100/541000] [225.7/1000.0] [batch_t 1.426 (1.431)] [data_t 0.002] [optim_t 1.424] [lr 0.005000] 2024-04-03 06:08:27,758 - Train: 22.59% [122200/541000] [225.9/1000.0] [batch_t 1.430 (1.433)] [data_t 0.003] [optim_t 1.427] [lr 0.005000] 2024-04-03 06:10:02,300 - ==> Total time: 1 day, 5:09:56 Eta: 4 days, 3:53:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 06:10:56,137 - Train: 22.61% [122300/541000] [226.1/1000.0] [batch_t 1.442 (1.462)] [data_t 0.003] [optim_t 1.440] [lr 0.005000] 2024-04-03 06:13:19,467 - Train: 22.62% [122400/541000] [226.2/1000.0] [batch_t 1.441 (1.433)] [data_t 0.003] [optim_t 1.438] [lr 0.005000] 2024-04-03 06:15:42,731 - Train: 22.64% [122500/541000] [226.4/1000.0] [batch_t 1.427 (1.433)] [data_t 0.003] [optim_t 1.424] [lr 0.005000] 2024-04-03 06:18:06,167 - Train: 22.66% [122600/541000] [226.6/1000.0] [batch_t 1.443 (1.434)] [data_t 0.002] [optim_t 1.441] [lr 0.005000] 2024-04-03 06:20:29,466 - Train: 22.68% [122700/541000] [226.8/1000.0] [batch_t 1.428 (1.433)] [data_t 0.003] [optim_t 1.425] [lr 0.005000] 2024-04-03 06:22:52,712 - Train: 22.70% [122800/541000] [227.0/1000.0] [batch_t 1.443 (1.432)] [data_t 0.003] [optim_t 1.440] [lr 0.005000] 2024-04-03 06:23:02,764 - ==> Total time: 1 day, 5:22:57 Eta: 4 days, 4:03:21 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 06:25:19,082 - Train: 22.72% [122900/541000] [227.2/1000.0] [batch_t 1.436 (1.441)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-03 06:27:42,283 - Train: 22.74% [123000/541000] [227.4/1000.0] [batch_t 1.440 (1.432)] [data_t 0.003] [optim_t 1.438] [lr 0.005000] 2024-04-03 06:30:05,746 - Train: 22.75% [123100/541000] [227.5/1000.0] [batch_t 1.436 (1.435)] [data_t 0.002] [optim_t 1.434] [lr 0.005000] 2024-04-03 06:32:29,265 - Train: 22.77% [123200/541000] [227.7/1000.0] [batch_t 1.430 (1.435)] [data_t 0.003] [optim_t 1.427] [lr 0.005000] 2024-04-03 06:34:52,607 - Train: 22.79% [123300/541000] [227.9/1000.0] [batch_t 1.427 (1.433)] [data_t 0.002] [optim_t 1.424] [lr 0.005000] 2024-04-03 06:36:01,436 - ==> Total time: 1 day, 5:35:55 Eta: 4 days, 4:13:14 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 06:37:20,401 - Train: 22.81% [123400/541000] [228.1/1000.0] [batch_t 1.436 (1.453)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-03 06:39:43,715 - Train: 22.83% [123500/541000] [228.3/1000.0] [batch_t 1.426 (1.433)] [data_t 0.003] [optim_t 1.423] [lr 0.005000] 2024-04-03 06:42:06,903 - Train: 22.85% [123600/541000] [228.5/1000.0] [batch_t 1.424 (1.432)] [data_t 0.003] [optim_t 1.421] [lr 0.005000] 2024-04-03 06:44:30,286 - Train: 22.87% [123700/541000] [228.7/1000.0] [batch_t 1.440 (1.434)] [data_t 0.002] [optim_t 1.438] [lr 0.005000] 2024-04-03 06:46:53,688 - Train: 22.88% [123800/541000] [228.8/1000.0] [batch_t 1.434 (1.434)] [data_t 0.003] [optim_t 1.431] [lr 0.005000] 2024-04-03 06:49:01,227 - ==> Total time: 1 day, 5:48:55 Eta: 4 days, 4:22:59 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 06:49:21,659 - Train: 22.90% [123900/541000] [229.0/1000.0] [batch_t 1.430 (1.511)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-03 06:51:44,998 - Train: 22.92% [124000/541000] [229.2/1000.0] [batch_t 1.427 (1.433)] [data_t 0.002] [optim_t 1.425] [lr 0.005000] 2024-04-03 06:54:10,689 - Train: 22.94% [124100/541000] [229.4/1000.0] [batch_t 1.458 (1.457)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-03 06:56:36,384 - Train: 22.96% [124200/541000] [229.6/1000.0] [batch_t 1.458 (1.457)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-03 06:58:59,608 - Train: 22.98% [124300/541000] [229.8/1000.0] [batch_t 1.431 (1.432)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-03 07:01:22,923 - Train: 22.99% [124400/541000] [229.9/1000.0] [batch_t 1.429 (1.433)] [data_t 0.003] [optim_t 1.426] [lr 0.005000] 2024-04-03 07:02:05,905 - ==> Total time: 1 day, 6:02:00 Eta: 4 days, 4:32:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 07:03:49,184 - Train: 23.01% [124500/541000] [230.1/1000.0] [batch_t 1.419 (1.445)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-03 07:06:12,476 - Train: 23.03% [124600/541000] [230.3/1000.0] [batch_t 1.429 (1.433)] [data_t 0.003] [optim_t 1.426] [lr 0.005000] 2024-04-03 07:08:35,772 - Train: 23.05% [124700/541000] [230.5/1000.0] [batch_t 1.437 (1.433)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-03 07:10:59,070 - Train: 23.07% [124800/541000] [230.7/1000.0] [batch_t 1.430 (1.433)] [data_t 0.003] [optim_t 1.427] [lr 0.005000] 2024-04-03 07:13:22,309 - Train: 23.09% [124900/541000] [230.9/1000.0] [batch_t 1.432 (1.432)] [data_t 0.003] [optim_t 1.429] [lr 0.005000] 2024-04-03 07:15:04,124 - ==> Total time: 1 day, 6:14:58 Eta: 4 days, 4:42:03 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 07:15:49,260 - Train: 23.11% [125000/541000] [231.1/1000.0] [batch_t 1.435 (1.461)] [data_t 0.003] [optim_t 1.432] [lr 0.005000] 2024-04-03 07:18:12,591 - Train: 23.12% [125100/541000] [231.2/1000.0] [batch_t 1.435 (1.433)] [data_t 0.003] [optim_t 1.432] [lr 0.005000] 2024-04-03 07:20:35,901 - Train: 23.14% [125200/541000] [231.4/1000.0] [batch_t 1.431 (1.433)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-03 07:22:59,260 - Train: 23.16% [125300/541000] [231.6/1000.0] [batch_t 1.426 (1.433)] [data_t 0.002] [optim_t 1.424] [lr 0.005000] 2024-04-03 07:25:22,601 - Train: 23.18% [125400/541000] [231.8/1000.0] [batch_t 1.437 (1.433)] [data_t 0.002] [optim_t 1.434] [lr 0.005000] 2024-04-03 07:27:45,945 - Train: 23.20% [125500/541000] [232.0/1000.0] [batch_t 1.444 (1.433)] [data_t 0.003] [optim_t 1.441] [lr 0.005000] 2024-04-03 07:28:03,147 - ==> Total time: 1 day, 6:27:57 Eta: 4 days, 4:51:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 07:30:12,633 - Train: 23.22% [125600/541000] [232.2/1000.0] [batch_t 1.437 (1.441)] [data_t 0.002] [optim_t 1.435] [lr 0.005000] 2024-04-03 07:32:35,985 - Train: 23.23% [125700/541000] [232.3/1000.0] [batch_t 1.434 (1.433)] [data_t 0.003] [optim_t 1.431] [lr 0.005000] 2024-04-03 07:34:59,328 - Train: 23.25% [125800/541000] [232.5/1000.0] [batch_t 1.434 (1.433)] [data_t 0.003] [optim_t 1.432] [lr 0.005000] 2024-04-03 07:37:22,706 - Train: 23.27% [125900/541000] [232.7/1000.0] [batch_t 1.430 (1.434)] [data_t 0.002] [optim_t 1.428] [lr 0.005000] 2024-04-03 07:39:45,937 - Train: 23.29% [126000/541000] [232.9/1000.0] [batch_t 1.443 (1.432)] [data_t 0.002] [optim_t 1.441] [lr 0.005000] 2024-04-03 07:41:01,932 - ==> Total time: 1 day, 6:40:56 Eta: 4 days, 5:00:05 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 07:42:13,076 - Train: 23.31% [126100/541000] [233.1/1000.0] [batch_t 1.440 (1.445)] [data_t 0.003] [optim_t 1.438] [lr 0.005000] 2024-04-03 07:44:36,260 - Train: 23.33% [126200/541000] [233.3/1000.0] [batch_t 1.428 (1.432)] [data_t 0.002] [optim_t 1.426] [lr 0.005000] 2024-04-03 07:46:59,563 - Train: 23.35% [126300/541000] [233.5/1000.0] [batch_t 1.427 (1.433)] [data_t 0.003] [optim_t 1.425] [lr 0.005000] 2024-04-03 07:49:22,864 - Train: 23.36% [126400/541000] [233.6/1000.0] [batch_t 1.450 (1.433)] [data_t 0.002] [optim_t 1.448] [lr 0.005000] 2024-04-03 07:51:46,195 - Train: 23.38% [126500/541000] [233.8/1000.0] [batch_t 1.433 (1.433)] [data_t 0.003] [optim_t 1.430] [lr 0.005000] 2024-04-03 07:54:00,955 - ==> Total time: 1 day, 6:53:55 Eta: 4 days, 5:08:49 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 07:54:14,025 - Train: 23.40% [126600/541000] [234.0/1000.0] [batch_t 1.437 (1.533)] [data_t 0.003] [optim_t 1.434] [lr 0.005000] 2024-04-03 07:56:37,376 - Train: 23.42% [126700/541000] [234.2/1000.0] [batch_t 1.443 (1.433)] [data_t 0.003] [optim_t 1.440] [lr 0.005000] 2024-04-03 07:59:00,804 - Train: 23.44% [126800/541000] [234.4/1000.0] [batch_t 1.431 (1.434)] [data_t 0.003] [optim_t 1.429] [lr 0.005000] 2024-04-03 08:01:24,022 - Train: 23.46% [126900/541000] [234.6/1000.0] [batch_t 1.430 (1.432)] [data_t 0.003] [optim_t 1.427] [lr 0.005000] 2024-04-03 08:03:47,252 - Train: 23.48% [127000/541000] [234.8/1000.0] [batch_t 1.433 (1.432)] [data_t 0.003] [optim_t 1.430] [lr 0.005000] 2024-04-03 08:06:10,504 - Train: 23.49% [127100/541000] [234.9/1000.0] [batch_t 1.430 (1.432)] [data_t 0.003] [optim_t 1.427] [lr 0.005000] 2024-04-03 08:07:00,629 - ==> Total time: 1 day, 7:06:55 Eta: 4 days, 5:17:24 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 08:08:39,156 - Train: 23.51% [127200/541000] [235.1/1000.0] [batch_t 1.422 (1.459)] [data_t 0.002] [optim_t 1.419] [lr 0.005000] 2024-04-03 08:11:02,387 - Train: 23.53% [127300/541000] [235.3/1000.0] [batch_t 1.439 (1.432)] [data_t 0.002] [optim_t 1.436] [lr 0.005000] 2024-04-03 08:13:25,700 - Train: 23.55% [127400/541000] [235.5/1000.0] [batch_t 1.427 (1.433)] [data_t 0.002] [optim_t 1.425] [lr 0.005000] 2024-04-03 08:15:48,987 - Train: 23.57% [127500/541000] [235.7/1000.0] [batch_t 1.437 (1.433)] [data_t 0.003] [optim_t 1.435] [lr 0.005000] 2024-04-03 08:18:12,219 - Train: 23.59% [127600/541000] [235.9/1000.0] [batch_t 1.445 (1.432)] [data_t 0.002] [optim_t 1.443] [lr 0.005000] 2024-04-03 08:20:01,068 - ==> Total time: 1 day, 7:19:55 Eta: 4 days, 5:25:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 08:20:40,944 - Train: 23.60% [127700/541000] [236.0/1000.0] [batch_t 1.429 (1.505)] [data_t 0.003] [optim_t 1.426] [lr 0.005000] 2024-04-03 08:23:04,262 - Train: 23.62% [127800/541000] [236.2/1000.0] [batch_t 1.426 (1.433)] [data_t 0.003] [optim_t 1.423] [lr 0.005000] 2024-04-03 08:25:27,540 - Train: 23.64% [127900/541000] [236.4/1000.0] [batch_t 1.430 (1.433)] [data_t 0.002] [optim_t 1.428] [lr 0.005000] 2024-04-03 08:27:50,794 - Train: 23.66% [128000/541000] [236.6/1000.0] [batch_t 1.440 (1.432)] [data_t 0.003] [optim_t 1.437] [lr 0.005000] 2024-04-03 08:30:14,070 - Train: 23.68% [128100/541000] [236.8/1000.0] [batch_t 1.430 (1.433)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-03 08:32:37,225 - Train: 23.70% [128200/541000] [237.0/1000.0] [batch_t 1.427 (1.431)] [data_t 0.003] [optim_t 1.424] [lr 0.005000] 2024-04-03 08:33:01,549 - ==> Total time: 1 day, 7:32:55 Eta: 4 days, 5:34:07 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 08:35:05,377 - Train: 23.72% [128300/541000] [237.2/1000.0] [batch_t 1.441 (1.443)] [data_t 0.003] [optim_t 1.438] [lr 0.005000] 2024-04-03 08:37:28,506 - Train: 23.73% [128400/541000] [237.3/1000.0] [batch_t 1.446 (1.431)] [data_t 0.003] [optim_t 1.443] [lr 0.005000] 2024-04-03 08:39:51,773 - Train: 23.75% [128500/541000] [237.5/1000.0] [batch_t 1.442 (1.433)] [data_t 0.003] [optim_t 1.439] [lr 0.005000] 2024-04-03 08:42:14,936 - Train: 23.77% [128600/541000] [237.7/1000.0] [batch_t 1.436 (1.432)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-03 08:44:38,271 - Train: 23.79% [128700/541000] [237.9/1000.0] [batch_t 1.441 (1.433)] [data_t 0.002] [optim_t 1.439] [lr 0.005000] 2024-04-03 08:46:01,322 - ==> Total time: 1 day, 7:45:55 Eta: 4 days, 5:42:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 08:47:06,152 - Train: 23.81% [128800/541000] [238.1/1000.0] [batch_t 1.432 (1.456)] [data_t 0.003] [optim_t 1.429] [lr 0.005000] 2024-04-03 08:49:29,493 - Train: 23.83% [128900/541000] [238.3/1000.0] [batch_t 1.429 (1.433)] [data_t 0.003] [optim_t 1.426] [lr 0.005000] 2024-04-03 08:51:52,672 - Train: 23.84% [129000/541000] [238.4/1000.0] [batch_t 1.431 (1.432)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-03 08:54:15,881 - Train: 23.86% [129100/541000] [238.6/1000.0] [batch_t 1.435 (1.432)] [data_t 0.003] [optim_t 1.432] [lr 0.005000] 2024-04-03 08:56:39,258 - Train: 23.88% [129200/541000] [238.8/1000.0] [batch_t 1.435 (1.434)] [data_t 0.003] [optim_t 1.432] [lr 0.005000] 2024-04-03 08:59:01,022 - ==> Total time: 1 day, 7:58:55 Eta: 4 days, 5:50:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 08:59:07,797 - Train: 23.90% [129300/541000] [239.0/1000.0] [batch_t 3.208 (3.208)] [data_t 1.780] [optim_t 1.428] [lr 0.005000] 2024-04-03 09:01:30,962 - Train: 23.92% [129400/541000] [239.2/1000.0] [batch_t 1.445 (1.432)] [data_t 0.003] [optim_t 1.442] [lr 0.005000] 2024-04-03 09:03:54,062 - Train: 23.94% [129500/541000] [239.4/1000.0] [batch_t 1.440 (1.431)] [data_t 0.003] [optim_t 1.437] [lr 0.005000] 2024-04-03 09:06:17,304 - Train: 23.96% [129600/541000] [239.6/1000.0] [batch_t 1.425 (1.432)] [data_t 0.002] [optim_t 1.423] [lr 0.005000] 2024-04-03 09:08:40,447 - Train: 23.97% [129700/541000] [239.7/1000.0] [batch_t 1.426 (1.431)] [data_t 0.003] [optim_t 1.423] [lr 0.005000] 2024-04-03 09:11:03,706 - Train: 23.99% [129800/541000] [239.9/1000.0] [batch_t 1.432 (1.432)] [data_t 0.003] [optim_t 1.430] [lr 0.005000] 2024-04-03 09:12:00,999 - ==> Total time: 1 day, 8:11:55 Eta: 4 days, 5:57:45 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 09:13:32,165 - Train: 24.01% [129900/541000] [240.1/1000.0] [batch_t 1.436 (1.463)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-03 09:15:55,532 - Train: 24.03% [130000/541000] [240.3/1000.0] [batch_t 1.422 (1.434)] [data_t 0.002] [optim_t 1.420] [lr 0.005000] 2024-04-03 09:18:18,880 - Train: 24.05% [130100/541000] [240.5/1000.0] [batch_t 1.436 (1.433)] [data_t 0.002] [optim_t 1.434] [lr 0.005000] 2024-04-03 09:20:42,182 - Train: 24.07% [130200/541000] [240.7/1000.0] [batch_t 1.432 (1.433)] [data_t 0.003] [optim_t 1.429] [lr 0.005000] 2024-04-03 09:23:05,394 - Train: 24.09% [130300/541000] [240.9/1000.0] [batch_t 1.456 (1.432)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-03 09:25:01,239 - ==> Total time: 1 day, 8:24:55 Eta: 4 days, 6:05:18 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 09:25:32,940 - Train: 24.10% [130400/541000] [241.0/1000.0] [batch_t 1.438 (1.491)] [data_t 0.003] [optim_t 1.435] [lr 0.005000] 2024-04-03 09:27:56,111 - Train: 24.12% [130500/541000] [241.2/1000.0] [batch_t 1.437 (1.432)] [data_t 0.002] [optim_t 1.434] [lr 0.005000] 2024-04-03 09:30:19,175 - Train: 24.14% [130600/541000] [241.4/1000.0] [batch_t 1.431 (1.431)] [data_t 0.002] [optim_t 1.428] [lr 0.005000] 2024-04-03 09:32:42,356 - Train: 24.16% [130700/541000] [241.6/1000.0] [batch_t 1.442 (1.432)] [data_t 0.003] [optim_t 1.439] [lr 0.005000] 2024-04-03 09:35:05,507 - Train: 24.18% [130800/541000] [241.8/1000.0] [batch_t 1.421 (1.431)] [data_t 0.003] [optim_t 1.418] [lr 0.005000] 2024-04-03 09:37:28,653 - Train: 24.20% [130900/541000] [242.0/1000.0] [batch_t 1.429 (1.431)] [data_t 0.003] [optim_t 1.427] [lr 0.005000] 2024-04-03 09:38:00,107 - ==> Total time: 1 day, 8:37:54 Eta: 4 days, 6:12:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 09:39:55,490 - Train: 24.21% [131000/541000] [242.1/1000.0] [batch_t 1.426 (1.457)] [data_t 0.003] [optim_t 1.423] [lr 0.005000] 2024-04-03 09:42:18,610 - Train: 24.23% [131100/541000] [242.3/1000.0] [batch_t 1.436 (1.431)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-03 09:44:41,638 - Train: 24.25% [131200/541000] [242.5/1000.0] [batch_t 1.427 (1.430)] [data_t 0.002] [optim_t 1.425] [lr 0.005000] 2024-04-03 09:47:04,720 - Train: 24.27% [131300/541000] [242.7/1000.0] [batch_t 1.438 (1.431)] [data_t 0.003] [optim_t 1.435] [lr 0.005000] 2024-04-03 09:49:27,879 - Train: 24.29% [131400/541000] [242.9/1000.0] [batch_t 1.435 (1.432)] [data_t 0.003] [optim_t 1.432] [lr 0.005000] 2024-04-03 09:50:58,091 - ==> Total time: 1 day, 8:50:52 Eta: 4 days, 6:19:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 09:51:53,884 - Train: 24.31% [131500/541000] [243.1/1000.0] [batch_t 1.435 (1.452)] [data_t 0.003] [optim_t 1.432] [lr 0.005000] 2024-04-03 09:54:17,050 - Train: 24.33% [131600/541000] [243.3/1000.0] [batch_t 1.434 (1.432)] [data_t 0.003] [optim_t 1.431] [lr 0.005000] 2024-04-03 09:56:40,204 - Train: 24.34% [131700/541000] [243.4/1000.0] [batch_t 1.435 (1.431)] [data_t 0.002] [optim_t 1.433] [lr 0.005000] 2024-04-03 09:59:03,334 - Train: 24.36% [131800/541000] [243.6/1000.0] [batch_t 1.443 (1.431)] [data_t 0.002] [optim_t 1.441] [lr 0.005000] 2024-04-03 10:01:26,409 - Train: 24.38% [131900/541000] [243.8/1000.0] [batch_t 1.436 (1.431)] [data_t 0.002] [optim_t 1.433] [lr 0.005000] 2024-04-03 10:03:49,477 - Train: 24.40% [132000/541000] [244.0/1000.0] [batch_t 1.424 (1.431)] [data_t 0.002] [optim_t 1.422] [lr 0.005000] 2024-04-03 10:03:55,186 - ==> Total time: 1 day, 9:03:49 Eta: 4 days, 6:26:36 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 10:06:14,933 - Train: 24.42% [132100/541000] [244.2/1000.0] [batch_t 1.433 (1.439)] [data_t 0.003] [optim_t 1.430] [lr 0.005000] 2024-04-03 10:08:38,063 - Train: 24.44% [132200/541000] [244.4/1000.0] [batch_t 1.424 (1.431)] [data_t 0.002] [optim_t 1.421] [lr 0.005000] 2024-04-03 10:11:01,183 - Train: 24.45% [132300/541000] [244.5/1000.0] [batch_t 1.431 (1.431)] [data_t 0.002] [optim_t 1.428] [lr 0.005000] 2024-04-03 10:13:24,311 - Train: 24.47% [132400/541000] [244.7/1000.0] [batch_t 1.446 (1.431)] [data_t 0.002] [optim_t 1.444] [lr 0.005000] 2024-04-03 10:15:47,550 - Train: 24.49% [132500/541000] [244.9/1000.0] [batch_t 1.436 (1.432)] [data_t 0.003] [optim_t 1.434] [lr 0.005000] 2024-04-03 10:16:52,020 - ==> Total time: 1 day, 9:16:46 Eta: 4 days, 6:33:19 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 10:18:13,546 - Train: 24.51% [132600/541000] [245.1/1000.0] [batch_t 1.430 (1.448)] [data_t 0.002] [optim_t 1.428] [lr 0.005000] 2024-04-03 10:20:41,797 - Train: 24.53% [132700/541000] [245.3/1000.0] [batch_t 1.432 (1.482)] [data_t 0.003] [optim_t 1.429] [lr 0.005000] 2024-04-03 10:23:04,955 - Train: 24.55% [132800/541000] [245.5/1000.0] [batch_t 1.426 (1.431)] [data_t 0.003] [optim_t 1.424] [lr 0.005000] 2024-04-03 10:25:28,151 - Train: 24.57% [132900/541000] [245.7/1000.0] [batch_t 1.427 (1.432)] [data_t 0.002] [optim_t 1.425] [lr 0.005000] 2024-04-03 10:27:51,142 - Train: 24.58% [133000/541000] [245.8/1000.0] [batch_t 1.430 (1.430)] [data_t 0.002] [optim_t 1.428] [lr 0.005000] 2024-04-03 10:29:54,160 - ==> Total time: 1 day, 9:29:48 Eta: 4 days, 6:40:08 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 10:30:17,375 - Train: 24.60% [133100/541000] [246.0/1000.0] [batch_t 1.431 (1.494)] [data_t 0.003] [optim_t 1.429] [lr 0.005000] 2024-04-03 10:32:40,425 - Train: 24.62% [133200/541000] [246.2/1000.0] [batch_t 1.438 (1.430)] [data_t 0.003] [optim_t 1.435] [lr 0.005000] 2024-04-03 10:35:03,614 - Train: 24.64% [133300/541000] [246.4/1000.0] [batch_t 1.431 (1.432)] [data_t 0.003] [optim_t 1.429] [lr 0.005000] 2024-04-03 10:37:26,709 - Train: 24.66% [133400/541000] [246.6/1000.0] [batch_t 1.436 (1.431)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-03 10:39:50,686 - Train: 24.68% [133500/541000] [246.8/1000.0] [batch_t 1.461 (1.440)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-03 10:42:16,233 - Train: 24.70% [133600/541000] [247.0/1000.0] [batch_t 1.451 (1.455)] [data_t 0.003] [optim_t 1.448] [lr 0.005000] 2024-04-03 10:42:55,553 - ==> Total time: 1 day, 9:42:49 Eta: 4 days, 6:46:46 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 10:44:43,691 - Train: 24.71% [133700/541000] [247.1/1000.0] [batch_t 1.430 (1.454)] [data_t 0.003] [optim_t 1.427] [lr 0.005000] 2024-04-03 10:47:06,871 - Train: 24.73% [133800/541000] [247.3/1000.0] [batch_t 1.415 (1.432)] [data_t 0.003] [optim_t 1.413] [lr 0.005000] 2024-04-03 10:49:29,989 - Train: 24.75% [133900/541000] [247.5/1000.0] [batch_t 1.439 (1.431)] [data_t 0.003] [optim_t 1.436] [lr 0.005000] 2024-04-03 10:51:52,885 - Train: 24.77% [134000/541000] [247.7/1000.0] [batch_t 1.423 (1.429)] [data_t 0.002] [optim_t 1.421] [lr 0.005000] 2024-04-03 10:54:15,996 - Train: 24.79% [134100/541000] [247.9/1000.0] [batch_t 1.423 (1.431)] [data_t 0.003] [optim_t 1.421] [lr 0.005000] 2024-04-03 10:55:53,305 - ==> Total time: 1 day, 9:55:47 Eta: 4 days, 6:53:03 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 10:56:41,784 - Train: 24.81% [134200/541000] [248.1/1000.0] [batch_t 1.432 (1.449)] [data_t 0.003] [optim_t 1.429] [lr 0.005000] 2024-04-03 10:59:05,105 - Train: 24.82% [134300/541000] [248.2/1000.0] [batch_t 1.429 (1.433)] [data_t 0.002] [optim_t 1.426] [lr 0.005000] 2024-04-03 11:01:28,282 - Train: 24.84% [134400/541000] [248.4/1000.0] [batch_t 1.425 (1.432)] [data_t 0.002] [optim_t 1.422] [lr 0.005000] 2024-04-03 11:03:51,457 - Train: 24.86% [134500/541000] [248.6/1000.0] [batch_t 1.435 (1.432)] [data_t 0.003] [optim_t 1.432] [lr 0.005000] 2024-04-03 11:06:14,656 - Train: 24.88% [134600/541000] [248.8/1000.0] [batch_t 1.421 (1.432)] [data_t 0.002] [optim_t 1.418] [lr 0.005000] 2024-04-03 11:08:37,899 - Train: 24.90% [134700/541000] [249.0/1000.0] [batch_t 1.430 (1.432)] [data_t 0.003] [optim_t 1.427] [lr 0.005000] 2024-04-03 11:08:50,751 - ==> Total time: 1 day, 10:08:45 Eta: 4 days, 6:59:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 11:11:03,354 - Train: 24.92% [134800/541000] [249.2/1000.0] [batch_t 1.430 (1.435)] [data_t 0.002] [optim_t 1.428] [lr 0.005000] 2024-04-03 11:13:26,415 - Train: 24.94% [134900/541000] [249.4/1000.0] [batch_t 1.425 (1.431)] [data_t 0.003] [optim_t 1.422] [lr 0.005000] 2024-04-03 11:15:49,700 - Train: 24.95% [135000/541000] [249.5/1000.0] [batch_t 1.436 (1.433)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-03 11:18:12,800 - Train: 24.97% [135100/541000] [249.7/1000.0] [batch_t 1.436 (1.431)] [data_t 0.002] [optim_t 1.434] [lr 0.005000] 2024-04-03 11:20:35,788 - Train: 24.99% [135200/541000] [249.9/1000.0] [batch_t 1.437 (1.430)] [data_t 0.003] [optim_t 1.435] [lr 0.005000] 2024-04-03 11:22:12,539 - Test: 36.76% [50/136] [batch_t 0.421 (0.494)] 2024-04-03 11:22:33,491 - Test: 73.53% [100/136] [batch_t 0.420 (0.457)] 2024-04-03 11:22:48,194 - Test: 100.00% [136/136] [batch_t 0.073 (0.444)] 2024-04-03 11:23:36,506 - ==> Metric Time for candle : 0.002 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.150 (mAUROC_px) 7.805 (mAP_px) 0.900 (mF1_max_px) 25.493 (mAUPRO_px) 0.375 (mF1_px_0.2_0.8_0.1) 0.388 (mAcc_px_0.2_0.8_0.1) 0.369 (mIoU_px_0.2_0.8_0.1) 1.091 (mIoU_max_px) 2024-04-03 11:24:14,680 - ==> Metric Time for capsules : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 8.210 (mAUROC_px) 6.290 (mAP_px) 0.788 (mF1_max_px) 21.076 (mAUPRO_px) 0.257 (mF1_px_0.2_0.8_0.1) 0.274 (mAcc_px_0.2_0.8_0.1) 0.276 (mIoU_px_0.2_0.8_0.1) 0.766 (mIoU_max_px) 2024-04-03 11:24:50,901 - ==> Metric Time for cashew : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.661 (mAUROC_px) 5.877 (mAP_px) 0.721 (mF1_max_px) 20.267 (mAUPRO_px) 0.256 (mF1_px_0.2_0.8_0.1) 0.258 (mAcc_px_0.2_0.8_0.1) 0.252 (mIoU_px_0.2_0.8_0.1) 0.709 (mIoU_max_px) 2024-04-03 11:25:27,349 - ==> Metric Time for chewinggum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.554 (mAUROC_px) 5.806 (mAP_px) 0.726 (mF1_max_px) 20.727 (mAUPRO_px) 0.249 (mF1_px_0.2_0.8_0.1) 0.248 (mAcc_px_0.2_0.8_0.1) 0.244 (mIoU_px_0.2_0.8_0.1) 0.677 (mIoU_max_px) 2024-04-03 11:26:04,026 - ==> Metric Time for fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.535 (mAUROC_px) 5.651 (mAP_px) 0.684 (mF1_max_px) 21.247 (mAUPRO_px) 0.229 (mF1_px_0.2_0.8_0.1) 0.225 (mAcc_px_0.2_0.8_0.1) 0.226 (mIoU_px_0.2_0.8_0.1) 0.667 (mIoU_max_px) 2024-04-03 11:26:50,637 - ==> Metric Time for macaroni1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.431 (mAUROC_px) 7.729 (mAP_px) 0.918 (mF1_max_px) 25.402 (mAUPRO_px) 0.312 (mF1_px_0.2_0.8_0.1) 0.321 (mAcc_px_0.2_0.8_0.1) 0.315 (mIoU_px_0.2_0.8_0.1) 0.902 (mIoU_max_px) 2024-04-03 11:27:37,068 - ==> Metric Time for macaroni2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.249 (mAUROC_px) 7.699 (mAP_px) 0.936 (mF1_max_px) 25.412 (mAUPRO_px) 0.311 (mF1_px_0.2_0.8_0.1) 0.314 (mAcc_px_0.2_0.8_0.1) 0.315 (mIoU_px_0.2_0.8_0.1) 0.906 (mIoU_max_px) 2024-04-03 11:28:24,752 - ==> Metric Time for pcb1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.355 (mAUROC_px) 7.772 (mAP_px) 0.911 (mF1_max_px) 26.514 (mAUPRO_px) 0.310 (mF1_px_0.2_0.8_0.1) 0.319 (mAcc_px_0.2_0.8_0.1) 0.321 (mIoU_px_0.2_0.8_0.1) 0.896 (mIoU_max_px) 2024-04-03 11:29:12,676 - ==> Metric Time for pcb2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.444 (mAUROC_px) 8.027 (mAP_px) 0.902 (mF1_max_px) 26.398 (mAUPRO_px) 0.315 (mF1_px_0.2_0.8_0.1) 0.317 (mAcc_px_0.2_0.8_0.1) 0.314 (mIoU_px_0.2_0.8_0.1) 0.918 (mIoU_max_px) 2024-04-03 11:30:00,060 - ==> Metric Time for pcb3 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.312 (mAUROC_px) 7.703 (mAP_px) 0.923 (mF1_max_px) 26.312 (mAUPRO_px) 0.316 (mF1_px_0.2_0.8_0.1) 0.313 (mAcc_px_0.2_0.8_0.1) 0.311 (mIoU_px_0.2_0.8_0.1) 0.909 (mIoU_max_px) 2024-04-03 11:30:49,343 - ==> Metric Time for pcb4 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.534 (mAUROC_px) 7.889 (mAP_px) 0.914 (mF1_max_px) 27.804 (mAUPRO_px) 0.316 (mF1_px_0.2_0.8_0.1) 0.311 (mAcc_px_0.2_0.8_0.1) 0.320 (mIoU_px_0.2_0.8_0.1) 0.908 (mIoU_max_px) 2024-04-03 11:31:25,401 - ==> Metric Time for pipe_fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.550 (mAUROC_px) 5.685 (mAP_px) 0.668 (mF1_max_px) 20.566 (mAUPRO_px) 0.232 (mF1_px_0.2_0.8_0.1) 0.231 (mAcc_px_0.2_0.8_0.1) 0.234 (mIoU_px_0.2_0.8_0.1) 0.672 (mIoU_max_px) 2024-04-03 11:31:25,422 - | Name | mAUROC_sp_max | mAUROC_sp_max (Max) | mAP_sp_max | mAP_sp_max (Max) | mF1_max_sp_max | mF1_max_sp_max (Max) | mAUROC_px | mAUROC_px (Max) | mAP_px | mAP_px (Max) | mF1_max_px | mF1_max_px (Max) | mAUPRO_px | mAUPRO_px (Max) | mF1_px_0.2_0.8_0.1 | mF1_px_0.2_0.8_0.1 (Max) | mAcc_px_0.2_0.8_0.1 | mAcc_px_0.2_0.8_0.1 (Max) | mIoU_px_0.2_0.8_0.1 | mIoU_px_0.2_0.8_0.1 (Max) | mIoU_max_px | mIoU_max_px (Max) | |:----------:|:---------------:|:---------------------:|:------------:|:------------------:|:----------------:|:----------------------:|:-----------:|:------------------:|:--------:|:------------------:|:------------:|:------------------:|:-----------:|:------------------:|:--------------------:|:--------------------------:|:---------------------:|:---------------------------:|:---------------------:|:---------------------------:|:-------------:|:-------------------:| | candle | 96.640 | 97.100 (50 epoch) | 96.862 | 97.346 (50 epoch) | 90.452 | 91.542 (50 epoch) | 99.098 | 99.098 (250 epoch) | 23.141 | 23.463 (150 epoch) | 31.984 | 32.654 (200 epoch) | 95.717 | 95.763 (50 epoch) | 20.613 | 20.613 (250 epoch) | 37.776 | 37.776 (250 epoch) | 11.820 | 11.820 (250 epoch) | 19.037 | 19.513 (200 epoch) | | capsules | 89.783 | 91.750 (100 epoch) | 93.726 | 94.964 (100 epoch) | 87.442 | 88.780 (100 epoch) | 99.064 | 99.123 (100 epoch) | 58.711 | 61.256 (100 epoch) | 59.868 | 59.868 (250 epoch) | 91.475 | 91.811 (100 epoch) | 31.356 | 33.186 (100 epoch) | 56.611 | 62.576 (50 epoch) | 20.384 | 21.701 (100 epoch) | 42.722 | 42.722 (250 epoch) | | cashew | 94.040 | 94.800 (150 epoch) | 97.102 | 97.492 (150 epoch) | 91.000 | 92.386 (200 epoch) | 95.614 | 95.614 (250 epoch) | 50.952 | 50.952 (250 epoch) | 53.875 | 53.875 (250 epoch) | 88.033 | 88.960 (150 epoch) | 28.203 | 28.279 (150 epoch) | 33.314 | 34.522 (150 epoch) | 17.734 | 17.734 (250 epoch) | 36.869 | 36.869 (250 epoch) | | chewinggum | 97.096 | 97.743 (100 epoch) | 98.681 | 98.928 (100 epoch) | 94.180 | 94.301 (50 epoch) | 97.970 | 98.145 (100 epoch) | 52.616 | 59.823 (50 epoch) | 56.916 | 61.088 (50 epoch) | 79.410 | 79.773 (50 epoch) | 28.921 | 35.748 (50 epoch) | 53.713 | 65.611 (50 epoch) | 18.689 | 23.754 (50 epoch) | 39.778 | 43.976 (50 epoch) | | fryum | 95.420 | 95.420 (250 epoch) | 97.826 | 97.826 (250 epoch) | 92.308 | 92.308 (250 epoch) | 97.138 | 97.138 (250 epoch) | 49.934 | 49.934 (250 epoch) | 52.992 | 52.992 (250 epoch) | 91.242 | 91.622 (100 epoch) | 24.924 | 26.312 (50 epoch) | 25.949 | 27.708 (50 epoch) | 15.526 | 16.200 (50 epoch) | 36.047 | 36.047 (250 epoch) | | macaroni1 | 90.870 | 91.568 (100 epoch) | 89.283 | 89.846 (100 epoch) | 82.000 | 82.000 (250 epoch) | 99.515 | 99.519 (100 epoch) | 18.694 | 18.719 (200 epoch) | 28.613 | 29.377 (200 epoch) | 95.899 | 95.899 (250 epoch) | 15.454 | 15.720 (50 epoch) | 38.000 | 40.765 (50 epoch) | 8.612 | 8.704 (50 epoch) | 16.695 | 17.217 (200 epoch) | | macaroni2 | 79.363 | 82.589 (50 epoch) | 74.334 | 81.196 (50 epoch) | 72.906 | 75.622 (50 epoch) | 99.167 | 99.452 (100 epoch) | 6.126 | 11.272 (50 epoch) | 13.206 | 17.568 (50 epoch) | 94.765 | 96.253 (150 epoch) | 5.924 | 10.086 (50 epoch) | 46.544 | 49.562 (50 epoch) | 3.109 | 5.433 (50 epoch) | 7.070 | 9.630 (50 epoch) | | pcb1 | 96.480 | 96.610 (200 epoch) | 94.970 | 95.251 (200 epoch) | 94.581 | 94.581 (250 epoch) | 99.786 | 99.786 (250 epoch) | 82.024 | 82.024 (250 epoch) | 75.268 | 75.268 (250 epoch) | 93.130 | 93.660 (50 epoch) | 47.196 | 47.196 (250 epoch) | 59.842 | 62.362 (50 epoch) | 33.507 | 33.507 (250 epoch) | 60.344 | 60.344 (250 epoch) | | pcb2 | 94.720 | 94.990 (200 epoch) | 94.334 | 94.755 (200 epoch) | 89.899 | 90.000 (150 epoch) | 98.978 | 98.978 (250 epoch) | 13.077 | 14.130 (50 epoch) | 23.977 | 23.977 (250 epoch) | 90.754 | 90.754 (250 epoch) | 15.800 | 16.150 (200 epoch) | 46.218 | 47.820 (150 epoch) | 8.692 | 8.892 (200 epoch) | 13.621 | 13.621 (250 epoch) | | pcb3 | 95.158 | 95.158 (250 epoch) | 95.221 | 95.221 (250 epoch) | 88.991 | 88.991 (250 epoch) | 99.122 | 99.124 (150 epoch) | 18.767 | 20.249 (150 epoch) | 27.558 | 27.912 (150 epoch) | 91.527 | 91.649 (200 epoch) | 19.283 | 20.042 (150 epoch) | 43.768 | 48.341 (50 epoch) | 10.843 | 11.295 (150 epoch) | 15.981 | 16.219 (150 epoch) | | pcb4 | 99.713 | 99.901 (100 epoch) | 99.722 | 99.901 (100 epoch) | 97.462 | 99.000 (50 epoch) | 98.727 | 98.727 (250 epoch) | 47.966 | 47.966 (250 epoch) | 47.709 | 47.709 (250 epoch) | 90.729 | 90.729 (250 epoch) | 26.291 | 26.925 (200 epoch) | 35.441 | 39.716 (50 epoch) | 16.177 | 16.537 (200 epoch) | 31.328 | 31.328 (250 epoch) | | pipe_fryum | 98.780 | 98.840 (150 epoch) | 99.360 | 99.400 (150 epoch) | 97.561 | 97.561 (200 epoch) | 99.096 | 99.096 (250 epoch) | 54.401 | 54.401 (250 epoch) | 57.537 | 58.465 (100 epoch) | 95.244 | 95.525 (150 epoch) | 34.684 | 34.684 (250 epoch) | 41.729 | 43.014 (100 epoch) | 22.399 | 22.399 (250 epoch) | 40.387 | 41.308 (100 epoch) | | Avg | 94.005 | 94.251 (100 epoch) | 94.285 | 94.481 (100 epoch) | 89.899 | 89.899 (250 epoch) | 98.606 | 98.606 (250 epoch) | 39.701 | 39.701 (250 epoch) | 44.125 | 44.125 (250 epoch) | 91.494 | 91.494 (250 epoch) | 24.887 | 25.164 (100 epoch) | 43.242 | 46.179 (50 epoch) | 15.624 | 15.755 (100 epoch) | 29.990 | 29.990 (250 epoch) | 2024-04-03 11:31:25,763 - ==> Total time: 1 day, 10:31:20 Eta: 4 days, 7:34:00 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 11:32:40,201 - Train: 25.01% [135300/541000] [250.1/1000.0] [batch_t 1.440 (1.446)] [data_t 0.003] [optim_t 1.437] [lr 0.005000] 2024-04-03 11:35:03,357 - Train: 25.03% [135400/541000] [250.3/1000.0] [batch_t 1.435 (1.431)] [data_t 0.003] [optim_t 1.432] [lr 0.005000] 2024-04-03 11:37:26,543 - Train: 25.05% [135500/541000] [250.5/1000.0] [batch_t 1.444 (1.432)] [data_t 0.002] [optim_t 1.441] [lr 0.005000] 2024-04-03 11:39:49,668 - Train: 25.06% [135600/541000] [250.6/1000.0] [batch_t 1.430 (1.431)] [data_t 0.003] [optim_t 1.427] [lr 0.005000] 2024-04-03 11:42:12,865 - Train: 25.08% [135700/541000] [250.8/1000.0] [batch_t 1.420 (1.432)] [data_t 0.003] [optim_t 1.417] [lr 0.005000] 2024-04-03 11:46:21,968 - ==> Total time: 1 day, 10:46:16 Eta: 4 days, 7:45:34 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 11:47:13,849 - Train: 25.10% [135800/541000] [251.0/1000.0] [batch_t 8.376 (4.209)] [data_t 6.924] [optim_t 1.452] [lr 0.005000] 2024-04-03 11:51:43,246 - Train: 25.12% [135900/541000] [251.2/1000.0] [batch_t 5.544 (2.694)] [data_t 4.115] [optim_t 1.429] [lr 0.005000] 2024-04-03 11:55:33,216 - Train: 25.14% [136000/541000] [251.4/1000.0] [batch_t 1.434 (2.300)] [data_t 0.002] [optim_t 1.431] [lr 0.005000] 2024-04-03 11:57:56,381 - Train: 25.16% [136100/541000] [251.6/1000.0] [batch_t 1.426 (1.432)] [data_t 0.003] [optim_t 1.424] [lr 0.005000] 2024-04-03 12:00:19,398 - Train: 25.18% [136200/541000] [251.8/1000.0] [batch_t 1.427 (1.430)] [data_t 0.003] [optim_t 1.424] [lr 0.005000] 2024-04-03 12:02:42,501 - Train: 25.19% [136300/541000] [251.9/1000.0] [batch_t 1.434 (1.431)] [data_t 0.003] [optim_t 1.431] [lr 0.005000] 2024-04-03 12:03:28,278 - ==> Total time: 1 day, 11:03:22 Eta: 4 days, 8:03:21 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 12:05:09,046 - Train: 25.21% [136400/541000] [252.1/1000.0] [batch_t 1.446 (1.449)] [data_t 0.003] [optim_t 1.442] [lr 0.005000] 2024-04-03 12:07:32,054 - Train: 25.23% [136500/541000] [252.3/1000.0] [batch_t 1.416 (1.430)] [data_t 0.003] [optim_t 1.414] [lr 0.005000] 2024-04-03 12:09:55,210 - Train: 25.25% [136600/541000] [252.5/1000.0] [batch_t 1.436 (1.431)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-03 12:12:18,301 - Train: 25.27% [136700/541000] [252.7/1000.0] [batch_t 1.430 (1.431)] [data_t 0.002] [optim_t 1.427] [lr 0.005000] 2024-04-03 12:14:41,336 - Train: 25.29% [136800/541000] [252.9/1000.0] [batch_t 1.435 (1.430)] [data_t 0.003] [optim_t 1.432] [lr 0.005000] 2024-04-03 12:16:25,585 - ==> Total time: 1 day, 11:16:20 Eta: 4 days, 8:08:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 12:17:06,231 - Train: 25.30% [136900/541000] [253.0/1000.0] [batch_t 1.431 (1.440)] [data_t 0.002] [optim_t 1.429] [lr 0.005000] 2024-04-03 12:19:29,288 - Train: 25.32% [137000/541000] [253.2/1000.0] [batch_t 1.442 (1.430)] [data_t 0.003] [optim_t 1.439] [lr 0.005000] 2024-04-03 12:21:52,390 - Train: 25.34% [137100/541000] [253.4/1000.0] [batch_t 1.435 (1.431)] [data_t 0.003] [optim_t 1.432] [lr 0.005000] 2024-04-03 12:24:15,495 - Train: 25.36% [137200/541000] [253.6/1000.0] [batch_t 1.428 (1.431)] [data_t 0.002] [optim_t 1.426] [lr 0.005000] 2024-04-03 12:26:38,517 - Train: 25.38% [137300/541000] [253.8/1000.0] [batch_t 1.434 (1.430)] [data_t 0.003] [optim_t 1.431] [lr 0.005000] 2024-04-03 12:29:01,566 - Train: 25.40% [137400/541000] [254.0/1000.0] [batch_t 1.427 (1.430)] [data_t 0.002] [optim_t 1.425] [lr 0.005000] 2024-04-03 12:29:21,617 - ==> Total time: 1 day, 11:29:16 Eta: 4 days, 8:13:40 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 12:31:26,715 - Train: 25.42% [137500/541000] [254.2/1000.0] [batch_t 1.435 (1.435)] [data_t 0.002] [optim_t 1.433] [lr 0.005000] 2024-04-03 12:33:49,780 - Train: 25.43% [137600/541000] [254.3/1000.0] [batch_t 1.435 (1.431)] [data_t 0.003] [optim_t 1.432] [lr 0.005000] 2024-04-03 12:36:13,001 - Train: 25.45% [137700/541000] [254.5/1000.0] [batch_t 1.433 (1.432)] [data_t 0.003] [optim_t 1.430] [lr 0.005000] 2024-04-03 12:38:35,907 - Train: 25.47% [137800/541000] [254.7/1000.0] [batch_t 1.423 (1.429)] [data_t 0.002] [optim_t 1.420] [lr 0.005000] 2024-04-03 12:40:59,095 - Train: 25.49% [137900/541000] [254.9/1000.0] [batch_t 1.442 (1.432)] [data_t 0.003] [optim_t 1.439] [lr 0.005000] 2024-04-03 12:42:17,825 - ==> Total time: 1 day, 11:42:12 Eta: 4 days, 8:18:35 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 12:43:24,564 - Train: 25.51% [138000/541000] [255.1/1000.0] [batch_t 1.422 (1.441)] [data_t 0.002] [optim_t 1.420] [lr 0.005000] 2024-04-03 12:45:47,670 - Train: 25.53% [138100/541000] [255.3/1000.0] [batch_t 1.436 (1.431)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-03 12:48:10,772 - Train: 25.55% [138200/541000] [255.5/1000.0] [batch_t 1.433 (1.431)] [data_t 0.003] [optim_t 1.430] [lr 0.005000] 2024-04-03 12:50:33,916 - Train: 25.56% [138300/541000] [255.6/1000.0] [batch_t 1.441 (1.431)] [data_t 0.003] [optim_t 1.438] [lr 0.005000] 2024-04-03 12:52:56,958 - Train: 25.58% [138400/541000] [255.8/1000.0] [batch_t 1.429 (1.430)] [data_t 0.002] [optim_t 1.426] [lr 0.005000] 2024-04-03 12:55:14,352 - ==> Total time: 1 day, 11:55:08 Eta: 4 days, 8:23:23 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 12:55:22,393 - Train: 25.60% [138500/541000] [256.0/1000.0] [batch_t 1.432 (1.573)] [data_t 0.003] [optim_t 1.429] [lr 0.005000] 2024-04-03 12:57:45,464 - Train: 25.62% [138600/541000] [256.2/1000.0] [batch_t 1.436 (1.431)] [data_t 0.002] [optim_t 1.433] [lr 0.005000] 2024-04-03 13:00:08,622 - Train: 25.64% [138700/541000] [256.4/1000.0] [batch_t 1.430 (1.431)] [data_t 0.003] [optim_t 1.427] [lr 0.005000] 2024-04-03 13:02:31,778 - Train: 25.66% [138800/541000] [256.6/1000.0] [batch_t 1.436 (1.431)] [data_t 0.002] [optim_t 1.434] [lr 0.005000] 2024-04-03 13:04:54,825 - Train: 25.67% [138900/541000] [256.7/1000.0] [batch_t 1.428 (1.430)] [data_t 0.003] [optim_t 1.425] [lr 0.005000] 2024-04-03 13:07:17,868 - Train: 25.69% [139000/541000] [256.9/1000.0] [batch_t 1.435 (1.430)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-03 13:08:10,806 - ==> Total time: 1 day, 12:08:05 Eta: 4 days, 8:28:03 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 13:09:43,123 - Train: 25.71% [139100/541000] [257.1/1000.0] [batch_t 1.426 (1.438)] [data_t 0.003] [optim_t 1.424] [lr 0.005000] 2024-04-03 13:12:06,336 - Train: 25.73% [139200/541000] [257.3/1000.0] [batch_t 1.433 (1.432)] [data_t 0.003] [optim_t 1.430] [lr 0.005000] 2024-04-03 13:14:29,455 - Train: 25.75% [139300/541000] [257.5/1000.0] [batch_t 1.431 (1.431)] [data_t 0.002] [optim_t 1.429] [lr 0.005000] 2024-04-03 13:16:52,534 - Train: 25.77% [139400/541000] [257.7/1000.0] [batch_t 1.442 (1.431)] [data_t 0.003] [optim_t 1.439] [lr 0.005000] 2024-04-03 13:19:15,664 - Train: 25.79% [139500/541000] [257.9/1000.0] [batch_t 1.437 (1.431)] [data_t 0.003] [optim_t 1.434] [lr 0.005000] 2024-04-03 13:21:07,334 - ==> Total time: 1 day, 12:21:01 Eta: 4 days, 8:32:34 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 13:21:41,025 - Train: 25.80% [139600/541000] [258.0/1000.0] [batch_t 1.438 (1.447)] [data_t 0.002] [optim_t 1.435] [lr 0.005000] 2024-04-03 13:24:04,159 - Train: 25.82% [139700/541000] [258.2/1000.0] [batch_t 1.422 (1.431)] [data_t 0.002] [optim_t 1.420] [lr 0.005000] 2024-04-03 13:26:27,357 - Train: 25.84% [139800/541000] [258.4/1000.0] [batch_t 1.431 (1.432)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-03 13:28:50,519 - Train: 25.86% [139900/541000] [258.6/1000.0] [batch_t 1.422 (1.432)] [data_t 0.003] [optim_t 1.420] [lr 0.005000] 2024-04-03 13:31:13,586 - Train: 25.88% [140000/541000] [258.8/1000.0] [batch_t 1.435 (1.431)] [data_t 0.003] [optim_t 1.432] [lr 0.005000] 2024-04-03 13:33:36,643 - Train: 25.90% [140100/541000] [259.0/1000.0] [batch_t 1.422 (1.430)] [data_t 0.002] [optim_t 1.420] [lr 0.005000] 2024-04-03 13:34:03,856 - ==> Total time: 1 day, 12:33:58 Eta: 4 days, 8:36:57 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 13:36:01,993 - Train: 25.91% [140200/541000] [259.1/1000.0] [batch_t 1.430 (1.436)] [data_t 0.003] [optim_t 1.427] [lr 0.005000] 2024-04-03 13:38:25,092 - Train: 25.93% [140300/541000] [259.3/1000.0] [batch_t 1.427 (1.431)] [data_t 0.002] [optim_t 1.424] [lr 0.005000] 2024-04-03 13:40:48,258 - Train: 25.95% [140400/541000] [259.5/1000.0] [batch_t 1.421 (1.432)] [data_t 0.002] [optim_t 1.419] [lr 0.005000] 2024-04-03 13:43:11,363 - Train: 25.97% [140500/541000] [259.7/1000.0] [batch_t 1.429 (1.431)] [data_t 0.003] [optim_t 1.426] [lr 0.005000] 2024-04-03 13:45:34,325 - Train: 25.99% [140600/541000] [259.9/1000.0] [batch_t 1.440 (1.430)] [data_t 0.002] [optim_t 1.438] [lr 0.005000] 2024-04-03 13:47:00,181 - ==> Total time: 1 day, 12:46:54 Eta: 4 days, 8:41:12 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 13:48:00,845 - Train: 26.01% [140700/541000] [260.1/1000.0] [batch_t 1.434 (1.473)] [data_t 0.003] [optim_t 1.432] [lr 0.005000] 2024-04-03 13:50:23,944 - Train: 26.03% [140800/541000] [260.3/1000.0] [batch_t 1.439 (1.431)] [data_t 0.003] [optim_t 1.436] [lr 0.005000] 2024-04-03 13:52:47,083 - Train: 26.04% [140900/541000] [260.4/1000.0] [batch_t 1.434 (1.431)] [data_t 0.003] [optim_t 1.431] [lr 0.005000] 2024-04-03 13:55:10,127 - Train: 26.06% [141000/541000] [260.6/1000.0] [batch_t 1.436 (1.430)] [data_t 0.002] [optim_t 1.434] [lr 0.005000] 2024-04-03 13:57:33,200 - Train: 26.08% [141100/541000] [260.8/1000.0] [batch_t 1.430 (1.431)] [data_t 0.003] [optim_t 1.427] [lr 0.005000] 2024-04-03 13:59:56,286 - Train: 26.10% [141200/541000] [261.0/1000.0] [batch_t 1.431 (1.431)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-03 13:59:57,726 - ==> Total time: 1 day, 12:59:52 Eta: 4 days, 8:45:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 14:02:21,511 - Train: 26.12% [141300/541000] [261.2/1000.0] [batch_t 1.421 (1.435)] [data_t 0.002] [optim_t 1.419] [lr 0.005000] 2024-04-03 14:04:44,578 - Train: 26.14% [141400/541000] [261.4/1000.0] [batch_t 1.436 (1.431)] [data_t 0.003] [optim_t 1.434] [lr 0.005000] 2024-04-03 14:07:07,582 - Train: 26.16% [141500/541000] [261.6/1000.0] [batch_t 1.424 (1.430)] [data_t 0.002] [optim_t 1.422] [lr 0.005000] 2024-04-03 14:09:30,620 - Train: 26.17% [141600/541000] [261.7/1000.0] [batch_t 1.428 (1.430)] [data_t 0.002] [optim_t 1.425] [lr 0.005000] 2024-04-03 14:11:53,719 - Train: 26.19% [141700/541000] [261.9/1000.0] [batch_t 1.431 (1.431)] [data_t 0.002] [optim_t 1.429] [lr 0.005000] 2024-04-03 14:12:53,788 - ==> Total time: 1 day, 13:12:48 Eta: 4 days, 8:49:20 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 14:14:18,951 - Train: 26.21% [141800/541000] [262.1/1000.0] [batch_t 1.422 (1.438)] [data_t 0.003] [optim_t 1.419] [lr 0.005000] 2024-04-03 14:16:41,899 - Train: 26.23% [141900/541000] [262.3/1000.0] [batch_t 1.431 (1.429)] [data_t 0.002] [optim_t 1.428] [lr 0.005000] 2024-04-03 14:20:28,363 - Train: 26.25% [142000/541000] [262.5/1000.0] [batch_t 1.429 (2.265)] [data_t 0.003] [optim_t 1.426] [lr 0.005000] 2024-04-03 14:22:51,424 - Train: 26.27% [142100/541000] [262.7/1000.0] [batch_t 1.428 (1.431)] [data_t 0.002] [optim_t 1.426] [lr 0.005000] 2024-04-03 14:25:14,439 - Train: 26.28% [142200/541000] [262.8/1000.0] [batch_t 1.425 (1.430)] [data_t 0.003] [optim_t 1.422] [lr 0.005000] 2024-04-03 14:27:13,124 - ==> Total time: 1 day, 13:27:07 Eta: 4 days, 8:57:04 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 14:27:40,235 - Train: 26.30% [142300/541000] [263.0/1000.0] [batch_t 1.440 (1.482)] [data_t 0.003] [optim_t 1.437] [lr 0.005000] 2024-04-03 14:30:03,371 - Train: 26.32% [142400/541000] [263.2/1000.0] [batch_t 1.431 (1.431)] [data_t 0.002] [optim_t 1.428] [lr 0.005000] 2024-04-03 14:32:28,874 - Train: 26.34% [142500/541000] [263.4/1000.0] [batch_t 1.455 (1.455)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-03 14:35:52,245 - Train: 26.36% [142600/541000] [263.6/1000.0] [batch_t 5.421 (2.034)] [data_t 3.988] [optim_t 1.433] [lr 0.005000] 2024-04-03 14:39:19,792 - Train: 26.38% [142700/541000] [263.8/1000.0] [batch_t 1.452 (2.075)] [data_t 0.003] [optim_t 1.448] [lr 0.005000] 2024-04-03 14:41:46,582 - Train: 26.40% [142800/541000] [264.0/1000.0] [batch_t 1.426 (1.468)] [data_t 0.002] [optim_t 1.424] [lr 0.005000] 2024-04-03 14:42:20,912 - ==> Total time: 1 day, 13:42:15 Eta: 4 days, 9:06:53 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 14:44:13,135 - Train: 26.41% [142900/541000] [264.1/1000.0] [batch_t 1.439 (1.449)] [data_t 0.002] [optim_t 1.436] [lr 0.005000] 2024-04-03 14:46:36,190 - Train: 26.43% [143000/541000] [264.3/1000.0] [batch_t 1.430 (1.430)] [data_t 0.002] [optim_t 1.428] [lr 0.005000] 2024-04-03 14:48:59,140 - Train: 26.45% [143100/541000] [264.5/1000.0] [batch_t 1.426 (1.429)] [data_t 0.002] [optim_t 1.423] [lr 0.005000] 2024-04-03 14:51:22,206 - Train: 26.47% [143200/541000] [264.7/1000.0] [batch_t 1.424 (1.431)] [data_t 0.002] [optim_t 1.421] [lr 0.005000] 2024-04-03 14:53:45,217 - Train: 26.49% [143300/541000] [264.9/1000.0] [batch_t 1.434 (1.430)] [data_t 0.002] [optim_t 1.432] [lr 0.005000] 2024-04-03 14:55:18,253 - ==> Total time: 1 day, 13:55:12 Eta: 4 days, 9:10:29 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 14:56:10,996 - Train: 26.51% [143400/541000] [265.1/1000.0] [batch_t 1.430 (1.450)] [data_t 0.003] [optim_t 1.427] [lr 0.005000] 2024-04-03 14:58:34,022 - Train: 26.52% [143500/541000] [265.2/1000.0] [batch_t 1.434 (1.430)] [data_t 0.002] [optim_t 1.432] [lr 0.005000] 2024-04-03 15:00:57,078 - Train: 26.54% [143600/541000] [265.4/1000.0] [batch_t 1.437 (1.430)] [data_t 0.003] [optim_t 1.434] [lr 0.005000] 2024-04-03 15:03:20,200 - Train: 26.56% [143700/541000] [265.6/1000.0] [batch_t 1.426 (1.431)] [data_t 0.003] [optim_t 1.423] [lr 0.005000] 2024-04-03 15:05:43,320 - Train: 26.58% [143800/541000] [265.8/1000.0] [batch_t 1.425 (1.431)] [data_t 0.003] [optim_t 1.422] [lr 0.005000] 2024-04-03 15:08:06,372 - Train: 26.60% [143900/541000] [266.0/1000.0] [batch_t 1.441 (1.430)] [data_t 0.002] [optim_t 1.439] [lr 0.005000] 2024-04-03 15:08:14,989 - ==> Total time: 1 day, 14:08:09 Eta: 4 days, 9:13:56 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 15:10:32,037 - Train: 26.62% [144000/541000] [266.2/1000.0] [batch_t 1.435 (1.439)] [data_t 0.003] [optim_t 1.432] [lr 0.005000] 2024-04-03 15:12:55,173 - Train: 26.64% [144100/541000] [266.4/1000.0] [batch_t 1.428 (1.431)] [data_t 0.003] [optim_t 1.426] [lr 0.005000] 2024-04-03 15:15:18,237 - Train: 26.65% [144200/541000] [266.5/1000.0] [batch_t 1.446 (1.431)] [data_t 0.002] [optim_t 1.444] [lr 0.005000] 2024-04-03 15:17:41,258 - Train: 26.67% [144300/541000] [266.7/1000.0] [batch_t 1.431 (1.430)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-03 15:20:04,319 - Train: 26.69% [144400/541000] [266.9/1000.0] [batch_t 1.425 (1.431)] [data_t 0.003] [optim_t 1.423] [lr 0.005000] 2024-04-03 15:21:11,534 - ==> Total time: 1 day, 14:21:05 Eta: 4 days, 9:17:15 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 15:22:30,175 - Train: 26.71% [144500/541000] [267.1/1000.0] [batch_t 1.432 (1.442)] [data_t 0.003] [optim_t 1.430] [lr 0.005000] 2024-04-03 15:24:53,320 - Train: 26.73% [144600/541000] [267.3/1000.0] [batch_t 1.427 (1.431)] [data_t 0.002] [optim_t 1.425] [lr 0.005000] 2024-04-03 15:27:16,390 - Train: 26.75% [144700/541000] [267.5/1000.0] [batch_t 1.425 (1.431)] [data_t 0.002] [optim_t 1.423] [lr 0.005000] 2024-04-03 15:29:39,397 - Train: 26.77% [144800/541000] [267.7/1000.0] [batch_t 1.422 (1.430)] [data_t 0.002] [optim_t 1.420] [lr 0.005000] 2024-04-03 15:32:02,356 - Train: 26.78% [144900/541000] [267.8/1000.0] [batch_t 1.436 (1.430)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-03 15:34:08,202 - ==> Total time: 1 day, 14:34:02 Eta: 4 days, 9:20:26 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 15:34:27,992 - Train: 26.80% [145000/541000] [268.0/1000.0] [batch_t 1.432 (1.481)] [data_t 0.002] [optim_t 1.429] [lr 0.005000] 2024-04-03 15:36:51,225 - Train: 26.82% [145100/541000] [268.2/1000.0] [batch_t 1.428 (1.432)] [data_t 0.003] [optim_t 1.426] [lr 0.005000] 2024-04-03 15:39:14,321 - Train: 26.84% [145200/541000] [268.4/1000.0] [batch_t 1.423 (1.431)] [data_t 0.003] [optim_t 1.420] [lr 0.005000] 2024-04-03 15:41:37,414 - Train: 26.86% [145300/541000] [268.6/1000.0] [batch_t 1.426 (1.431)] [data_t 0.003] [optim_t 1.424] [lr 0.005000] 2024-04-03 15:44:00,698 - Train: 26.88% [145400/541000] [268.8/1000.0] [batch_t 1.434 (1.433)] [data_t 0.002] [optim_t 1.431] [lr 0.005000] 2024-04-03 15:46:23,638 - Train: 26.89% [145500/541000] [268.9/1000.0] [batch_t 1.441 (1.429)] [data_t 0.003] [optim_t 1.438] [lr 0.005000] 2024-04-03 15:47:05,128 - ==> Total time: 1 day, 14:46:59 Eta: 4 days, 9:23:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 15:48:49,077 - Train: 26.91% [145600/541000] [269.1/1000.0] [batch_t 1.427 (1.436)] [data_t 0.002] [optim_t 1.425] [lr 0.005000] 2024-04-03 15:51:12,290 - Train: 26.93% [145700/541000] [269.3/1000.0] [batch_t 1.436 (1.432)] [data_t 0.002] [optim_t 1.434] [lr 0.005000] 2024-04-03 15:53:35,348 - Train: 26.95% [145800/541000] [269.5/1000.0] [batch_t 1.431 (1.430)] [data_t 0.002] [optim_t 1.429] [lr 0.005000] 2024-04-03 15:55:58,429 - Train: 26.97% [145900/541000] [269.7/1000.0] [batch_t 1.424 (1.431)] [data_t 0.002] [optim_t 1.422] [lr 0.005000] 2024-04-03 15:58:21,517 - Train: 26.99% [146000/541000] [269.9/1000.0] [batch_t 1.432 (1.431)] [data_t 0.002] [optim_t 1.429] [lr 0.005000] 2024-04-03 16:00:01,616 - ==> Total time: 1 day, 14:59:56 Eta: 4 days, 9:26:29 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 16:00:49,308 - Train: 27.01% [146100/541000] [270.1/1000.0] [batch_t 1.437 (1.451)] [data_t 0.003] [optim_t 1.434] [lr 0.005000] 2024-04-03 16:03:12,289 - Train: 27.02% [146200/541000] [270.2/1000.0] [batch_t 1.440 (1.430)] [data_t 0.003] [optim_t 1.437] [lr 0.005000] 2024-04-03 16:05:35,511 - Train: 27.04% [146300/541000] [270.4/1000.0] [batch_t 1.435 (1.432)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-03 16:07:58,475 - Train: 27.06% [146400/541000] [270.6/1000.0] [batch_t 1.431 (1.430)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-03 16:10:21,559 - Train: 27.08% [146500/541000] [270.8/1000.0] [batch_t 1.434 (1.431)] [data_t 0.003] [optim_t 1.432] [lr 0.005000] 2024-04-03 16:12:44,640 - Train: 27.10% [146600/541000] [271.0/1000.0] [batch_t 1.428 (1.431)] [data_t 0.003] [optim_t 1.425] [lr 0.005000] 2024-04-03 16:13:00,397 - ==> Total time: 1 day, 15:12:54 Eta: 4 days, 9:29:25 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 16:15:11,949 - Train: 27.12% [146700/541000] [271.2/1000.0] [batch_t 1.422 (1.455)] [data_t 0.003] [optim_t 1.420] [lr 0.005000] 2024-04-03 16:17:35,023 - Train: 27.13% [146800/541000] [271.3/1000.0] [batch_t 1.435 (1.431)] [data_t 0.003] [optim_t 1.432] [lr 0.005000] 2024-04-03 16:19:58,120 - Train: 27.15% [146900/541000] [271.5/1000.0] [batch_t 1.440 (1.431)] [data_t 0.003] [optim_t 1.437] [lr 0.005000] 2024-04-03 16:22:21,171 - Train: 27.17% [147000/541000] [271.7/1000.0] [batch_t 1.437 (1.430)] [data_t 0.003] [optim_t 1.434] [lr 0.005000] 2024-04-03 16:24:44,297 - Train: 27.19% [147100/541000] [271.9/1000.0] [batch_t 1.441 (1.431)] [data_t 0.002] [optim_t 1.439] [lr 0.005000] 2024-04-03 16:25:58,615 - ==> Total time: 1 day, 15:25:53 Eta: 4 days, 9:32:13 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 16:27:09,465 - Train: 27.21% [147200/541000] [272.1/1000.0] [batch_t 1.424 (1.440)] [data_t 0.002] [optim_t 1.421] [lr 0.005000] 2024-04-03 16:29:32,546 - Train: 27.23% [147300/541000] [272.3/1000.0] [batch_t 1.438 (1.431)] [data_t 0.003] [optim_t 1.435] [lr 0.005000] 2024-04-03 16:31:55,621 - Train: 27.25% [147400/541000] [272.5/1000.0] [batch_t 1.432 (1.431)] [data_t 0.003] [optim_t 1.429] [lr 0.005000] 2024-04-03 16:34:18,729 - Train: 27.26% [147500/541000] [272.6/1000.0] [batch_t 1.421 (1.431)] [data_t 0.003] [optim_t 1.418] [lr 0.005000] 2024-04-03 16:36:41,779 - Train: 27.28% [147600/541000] [272.8/1000.0] [batch_t 1.433 (1.430)] [data_t 0.003] [optim_t 1.430] [lr 0.005000] 2024-04-03 16:38:54,833 - ==> Total time: 1 day, 15:38:49 Eta: 4 days, 9:34:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 16:39:07,052 - Train: 27.30% [147700/541000] [273.0/1000.0] [batch_t 1.434 (1.494)] [data_t 0.003] [optim_t 1.431] [lr 0.005000] 2024-04-03 16:41:30,228 - Train: 27.32% [147800/541000] [273.2/1000.0] [batch_t 1.426 (1.432)] [data_t 0.003] [optim_t 1.423] [lr 0.005000] 2024-04-03 16:43:53,248 - Train: 27.34% [147900/541000] [273.4/1000.0] [batch_t 1.427 (1.430)] [data_t 0.003] [optim_t 1.425] [lr 0.005000] 2024-04-03 16:46:16,381 - Train: 27.36% [148000/541000] [273.6/1000.0] [batch_t 1.429 (1.431)] [data_t 0.002] [optim_t 1.427] [lr 0.005000] 2024-04-03 16:48:39,463 - Train: 27.38% [148100/541000] [273.8/1000.0] [batch_t 1.430 (1.431)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-03 16:51:02,459 - Train: 27.39% [148200/541000] [273.9/1000.0] [batch_t 1.420 (1.430)] [data_t 0.003] [optim_t 1.418] [lr 0.005000] 2024-04-03 16:51:51,027 - ==> Total time: 1 day, 15:51:45 Eta: 4 days, 9:37:17 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 16:53:27,880 - Train: 27.41% [148300/541000] [274.1/1000.0] [batch_t 1.433 (1.436)] [data_t 0.003] [optim_t 1.431] [lr 0.005000] 2024-04-03 16:55:50,898 - Train: 27.43% [148400/541000] [274.3/1000.0] [batch_t 1.440 (1.430)] [data_t 0.003] [optim_t 1.437] [lr 0.005000] 2024-04-03 16:58:13,921 - Train: 27.45% [148500/541000] [274.5/1000.0] [batch_t 1.421 (1.430)] [data_t 0.003] [optim_t 1.418] [lr 0.005000] 2024-04-03 17:00:37,075 - Train: 27.47% [148600/541000] [274.7/1000.0] [batch_t 1.437 (1.431)] [data_t 0.002] [optim_t 1.435] [lr 0.005000] 2024-04-03 17:03:00,156 - Train: 27.49% [148700/541000] [274.9/1000.0] [batch_t 1.436 (1.431)] [data_t 0.002] [optim_t 1.434] [lr 0.005000] 2024-04-03 17:04:47,478 - ==> Total time: 1 day, 16:04:41 Eta: 4 days, 9:39:39 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 17:05:25,271 - Train: 27.50% [148800/541000] [275.0/1000.0] [batch_t 1.427 (1.443)] [data_t 0.003] [optim_t 1.424] [lr 0.005000] 2024-04-03 17:07:48,361 - Train: 27.52% [148900/541000] [275.2/1000.0] [batch_t 1.440 (1.431)] [data_t 0.002] [optim_t 1.437] [lr 0.005000] 2024-04-03 17:10:11,419 - Train: 27.54% [149000/541000] [275.4/1000.0] [batch_t 1.435 (1.430)] [data_t 0.002] [optim_t 1.432] [lr 0.005000] 2024-04-03 17:12:34,434 - Train: 27.56% [149100/541000] [275.6/1000.0] [batch_t 1.433 (1.430)] [data_t 0.003] [optim_t 1.430] [lr 0.005000] 2024-04-03 17:14:57,578 - Train: 27.58% [149200/541000] [275.8/1000.0] [batch_t 1.436 (1.431)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-03 17:17:20,721 - Train: 27.60% [149300/541000] [276.0/1000.0] [batch_t 1.426 (1.431)] [data_t 0.003] [optim_t 1.424] [lr 0.005000] 2024-04-03 17:17:43,627 - ==> Total time: 1 day, 16:17:38 Eta: 4 days, 9:41:54 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 17:19:45,775 - Train: 27.62% [149400/541000] [276.2/1000.0] [batch_t 1.435 (1.434)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-03 17:22:08,832 - Train: 27.63% [149500/541000] [276.3/1000.0] [batch_t 1.430 (1.430)] [data_t 0.002] [optim_t 1.427] [lr 0.005000] 2024-04-03 17:24:38,050 - Train: 27.65% [149600/541000] [276.5/1000.0] [batch_t 1.436 (1.492)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-03 17:27:01,214 - Train: 27.67% [149700/541000] [276.7/1000.0] [batch_t 1.426 (1.432)] [data_t 0.003] [optim_t 1.423] [lr 0.005000] 2024-04-03 17:29:24,381 - Train: 27.69% [149800/541000] [276.9/1000.0] [batch_t 1.425 (1.432)] [data_t 0.003] [optim_t 1.423] [lr 0.005000] 2024-04-03 17:30:45,932 - ==> Total time: 1 day, 16:30:40 Eta: 4 days, 9:44:19 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 17:31:50,225 - Train: 27.71% [149900/541000] [277.1/1000.0] [batch_t 1.435 (1.444)] [data_t 0.002] [optim_t 1.433] [lr 0.005000] 2024-04-03 17:34:13,196 - Train: 27.73% [150000/541000] [277.3/1000.0] [batch_t 1.426 (1.430)] [data_t 0.002] [optim_t 1.423] [lr 0.005000] 2024-04-03 17:36:36,289 - Train: 27.74% [150100/541000] [277.4/1000.0] [batch_t 1.422 (1.431)] [data_t 0.002] [optim_t 1.420] [lr 0.005000] 2024-04-03 17:38:59,308 - Train: 27.76% [150200/541000] [277.6/1000.0] [batch_t 1.454 (1.430)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-03 17:41:22,297 - Train: 27.78% [150300/541000] [277.8/1000.0] [batch_t 1.420 (1.430)] [data_t 0.002] [optim_t 1.418] [lr 0.005000] 2024-04-03 17:43:42,591 - ==> Total time: 1 day, 16:43:37 Eta: 4 days, 9:46:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 17:43:48,087 - Train: 27.80% [150400/541000] [278.0/1000.0] [batch_t 1.427 (1.750)] [data_t 0.002] [optim_t 1.424] [lr 0.005000] 2024-04-03 17:46:11,055 - Train: 27.82% [150500/541000] [278.2/1000.0] [batch_t 1.437 (1.430)] [data_t 0.002] [optim_t 1.435] [lr 0.005000] 2024-04-03 17:48:34,081 - Train: 27.84% [150600/541000] [278.4/1000.0] [batch_t 1.432 (1.430)] [data_t 0.002] [optim_t 1.430] [lr 0.005000] 2024-04-03 17:50:57,154 - Train: 27.86% [150700/541000] [278.6/1000.0] [batch_t 1.422 (1.431)] [data_t 0.002] [optim_t 1.420] [lr 0.005000] 2024-04-03 17:53:20,287 - Train: 27.87% [150800/541000] [278.7/1000.0] [batch_t 1.436 (1.431)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-03 17:55:43,371 - Train: 27.89% [150900/541000] [278.9/1000.0] [batch_t 1.432 (1.431)] [data_t 0.002] [optim_t 1.430] [lr 0.005000] 2024-04-03 17:56:39,191 - ==> Total time: 1 day, 16:56:33 Eta: 4 days, 9:48:18 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 17:58:08,791 - Train: 27.91% [151000/541000] [279.1/1000.0] [batch_t 1.446 (1.444)] [data_t 0.003] [optim_t 1.443] [lr 0.005000] 2024-04-03 18:00:31,939 - Train: 27.93% [151100/541000] [279.3/1000.0] [batch_t 1.432 (1.431)] [data_t 0.003] [optim_t 1.429] [lr 0.005000] 2024-04-03 18:02:55,077 - Train: 27.95% [151200/541000] [279.5/1000.0] [batch_t 1.430 (1.431)] [data_t 0.003] [optim_t 1.427] [lr 0.005000] 2024-04-03 18:05:18,182 - Train: 27.97% [151300/541000] [279.7/1000.0] [batch_t 1.421 (1.431)] [data_t 0.003] [optim_t 1.418] [lr 0.005000] 2024-04-03 18:07:41,336 - Train: 27.99% [151400/541000] [279.9/1000.0] [batch_t 1.427 (1.431)] [data_t 0.003] [optim_t 1.424] [lr 0.005000] 2024-04-03 18:09:35,745 - ==> Total time: 1 day, 17:09:30 Eta: 4 days, 9:50:09 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 18:10:06,515 - Train: 28.00% [151500/541000] [280.0/1000.0] [batch_t 1.455 (1.453)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-03 18:12:29,692 - Train: 28.02% [151600/541000] [280.2/1000.0] [batch_t 1.434 (1.432)] [data_t 0.003] [optim_t 1.431] [lr 0.005000] 2024-04-03 18:14:52,949 - Train: 28.04% [151700/541000] [280.4/1000.0] [batch_t 1.436 (1.432)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-03 18:17:16,133 - Train: 28.06% [151800/541000] [280.6/1000.0] [batch_t 1.439 (1.432)] [data_t 0.003] [optim_t 1.437] [lr 0.005000] 2024-04-03 18:19:39,237 - Train: 28.08% [151900/541000] [280.8/1000.0] [batch_t 1.435 (1.431)] [data_t 0.003] [optim_t 1.432] [lr 0.005000] 2024-04-03 18:22:02,299 - Train: 28.10% [152000/541000] [281.0/1000.0] [batch_t 1.435 (1.431)] [data_t 0.003] [optim_t 1.432] [lr 0.005000] 2024-04-03 18:22:32,370 - ==> Total time: 1 day, 17:22:26 Eta: 4 days, 9:51:53 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 18:24:28,201 - Train: 28.11% [152100/541000] [281.1/1000.0] [batch_t 1.451 (1.444)] [data_t 0.003] [optim_t 1.448] [lr 0.005000] 2024-04-03 18:26:53,747 - Train: 28.13% [152200/541000] [281.3/1000.0] [batch_t 1.460 (1.455)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-03 18:29:18,563 - Train: 28.15% [152300/541000] [281.5/1000.0] [batch_t 1.434 (1.448)] [data_t 0.003] [optim_t 1.431] [lr 0.005000] 2024-04-03 18:31:41,680 - Train: 28.17% [152400/541000] [281.7/1000.0] [batch_t 1.426 (1.431)] [data_t 0.003] [optim_t 1.423] [lr 0.005000] 2024-04-03 18:34:04,672 - Train: 28.19% [152500/541000] [281.9/1000.0] [batch_t 1.429 (1.430)] [data_t 0.002] [optim_t 1.427] [lr 0.005000] 2024-04-03 18:35:33,325 - ==> Total time: 1 day, 17:35:27 Eta: 4 days, 9:53:41 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 18:36:30,200 - Train: 28.21% [152600/541000] [282.1/1000.0] [batch_t 1.436 (1.443)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-03 18:38:53,355 - Train: 28.23% [152700/541000] [282.3/1000.0] [batch_t 1.433 (1.431)] [data_t 0.003] [optim_t 1.430] [lr 0.005000] 2024-04-03 18:41:16,399 - Train: 28.24% [152800/541000] [282.4/1000.0] [batch_t 1.426 (1.430)] [data_t 0.002] [optim_t 1.423] [lr 0.005000] 2024-04-03 18:43:39,509 - Train: 28.26% [152900/541000] [282.6/1000.0] [batch_t 1.433 (1.431)] [data_t 0.003] [optim_t 1.431] [lr 0.005000] 2024-04-03 18:46:02,678 - Train: 28.28% [153000/541000] [282.8/1000.0] [batch_t 1.422 (1.432)] [data_t 0.002] [optim_t 1.420] [lr 0.005000] 2024-04-03 18:48:25,711 - Train: 28.30% [153100/541000] [283.0/1000.0] [batch_t 1.430 (1.430)] [data_t 0.002] [optim_t 1.428] [lr 0.005000] 2024-04-03 18:48:30,003 - ==> Total time: 1 day, 17:48:24 Eta: 4 days, 9:55:13 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 18:50:51,011 - Train: 28.32% [153200/541000] [283.2/1000.0] [batch_t 1.433 (1.434)] [data_t 0.002] [optim_t 1.430] [lr 0.005000] 2024-04-03 18:53:14,132 - Train: 28.34% [153300/541000] [283.4/1000.0] [batch_t 1.421 (1.431)] [data_t 0.002] [optim_t 1.419] [lr 0.005000] 2024-04-03 18:55:37,270 - Train: 28.35% [153400/541000] [283.5/1000.0] [batch_t 1.431 (1.431)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-03 18:58:00,177 - Train: 28.37% [153500/541000] [283.7/1000.0] [batch_t 1.418 (1.429)] [data_t 0.003] [optim_t 1.415] [lr 0.005000] 2024-04-03 19:00:23,372 - Train: 28.39% [153600/541000] [283.9/1000.0] [batch_t 1.436 (1.432)] [data_t 0.002] [optim_t 1.434] [lr 0.005000] 2024-04-03 19:01:26,393 - ==> Total time: 1 day, 18:01:20 Eta: 4 days, 9:56:38 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 19:02:48,757 - Train: 28.41% [153700/541000] [284.1/1000.0] [batch_t 1.417 (1.436)] [data_t 0.003] [optim_t 1.414] [lr 0.005000] 2024-04-03 19:05:11,844 - Train: 28.43% [153800/541000] [284.3/1000.0] [batch_t 1.439 (1.431)] [data_t 0.002] [optim_t 1.437] [lr 0.005000] 2024-04-03 19:07:34,921 - Train: 28.45% [153900/541000] [284.5/1000.0] [batch_t 1.433 (1.431)] [data_t 0.003] [optim_t 1.430] [lr 0.005000] 2024-04-03 19:09:57,975 - Train: 28.47% [154000/541000] [284.7/1000.0] [batch_t 1.431 (1.430)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-03 19:12:21,113 - Train: 28.48% [154100/541000] [284.8/1000.0] [batch_t 1.417 (1.431)] [data_t 0.002] [optim_t 1.415] [lr 0.005000] 2024-04-03 19:14:22,709 - ==> Total time: 1 day, 18:14:17 Eta: 4 days, 9:57:56 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 19:14:46,880 - Train: 28.50% [154200/541000] [285.0/1000.0] [batch_t 1.432 (1.467)] [data_t 0.003] [optim_t 1.429] [lr 0.005000] 2024-04-03 19:17:10,153 - Train: 28.52% [154300/541000] [285.2/1000.0] [batch_t 1.441 (1.433)] [data_t 0.003] [optim_t 1.438] [lr 0.005000] 2024-04-03 19:19:33,310 - Train: 28.54% [154400/541000] [285.4/1000.0] [batch_t 1.435 (1.431)] [data_t 0.002] [optim_t 1.433] [lr 0.005000] 2024-04-03 19:21:56,641 - Train: 28.56% [154500/541000] [285.6/1000.0] [batch_t 1.440 (1.433)] [data_t 0.003] [optim_t 1.437] [lr 0.005000] 2024-04-03 19:24:19,885 - Train: 28.58% [154600/541000] [285.8/1000.0] [batch_t 1.436 (1.432)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-03 19:26:43,086 - Train: 28.60% [154700/541000] [286.0/1000.0] [batch_t 1.420 (1.432)] [data_t 0.002] [optim_t 1.418] [lr 0.005000] 2024-04-03 19:27:20,331 - ==> Total time: 1 day, 18:27:14 Eta: 4 days, 9:59:12 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 19:29:08,923 - Train: 28.61% [154800/541000] [286.1/1000.0] [batch_t 1.428 (1.438)] [data_t 0.002] [optim_t 1.425] [lr 0.005000] 2024-04-03 19:31:32,301 - Train: 28.63% [154900/541000] [286.3/1000.0] [batch_t 1.436 (1.434)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-03 19:33:55,483 - Train: 28.65% [155000/541000] [286.5/1000.0] [batch_t 1.428 (1.432)] [data_t 0.002] [optim_t 1.425] [lr 0.005000] 2024-04-03 19:36:18,688 - Train: 28.67% [155100/541000] [286.7/1000.0] [batch_t 1.437 (1.432)] [data_t 0.002] [optim_t 1.434] [lr 0.005000] 2024-04-03 19:38:41,897 - Train: 28.69% [155200/541000] [286.9/1000.0] [batch_t 1.442 (1.432)] [data_t 0.003] [optim_t 1.439] [lr 0.005000] 2024-04-03 19:40:17,878 - ==> Total time: 1 day, 18:40:12 Eta: 4 days, 10:00:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 19:41:07,809 - Train: 28.71% [155300/541000] [287.1/1000.0] [batch_t 1.434 (1.453)] [data_t 0.002] [optim_t 1.432] [lr 0.005000] 2024-04-03 19:43:31,024 - Train: 28.72% [155400/541000] [287.2/1000.0] [batch_t 1.436 (1.432)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-03 19:45:54,403 - Train: 28.74% [155500/541000] [287.4/1000.0] [batch_t 1.439 (1.434)] [data_t 0.003] [optim_t 1.437] [lr 0.005000] 2024-04-03 19:48:17,799 - Train: 28.76% [155600/541000] [287.6/1000.0] [batch_t 1.434 (1.434)] [data_t 0.003] [optim_t 1.432] [lr 0.005000] 2024-04-03 19:50:40,924 - Train: 28.78% [155700/541000] [287.8/1000.0] [batch_t 1.426 (1.431)] [data_t 0.004] [optim_t 1.422] [lr 0.005000] 2024-04-03 19:53:04,035 - Train: 28.80% [155800/541000] [288.0/1000.0] [batch_t 1.420 (1.431)] [data_t 0.003] [optim_t 1.417] [lr 0.005000] 2024-04-03 19:53:15,459 - ==> Total time: 1 day, 18:53:09 Eta: 4 days, 10:01:26 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 19:55:29,266 - Train: 28.82% [155900/541000] [288.2/1000.0] [batch_t 1.441 (1.435)] [data_t 0.003] [optim_t 1.438] [lr 0.005000] 2024-04-03 19:57:52,450 - Train: 28.84% [156000/541000] [288.4/1000.0] [batch_t 1.427 (1.432)] [data_t 0.003] [optim_t 1.425] [lr 0.005000] 2024-04-03 20:00:15,604 - Train: 28.85% [156100/541000] [288.5/1000.0] [batch_t 1.441 (1.431)] [data_t 0.003] [optim_t 1.438] [lr 0.005000] 2024-04-03 20:02:38,888 - Train: 28.87% [156200/541000] [288.7/1000.0] [batch_t 1.433 (1.433)] [data_t 0.003] [optim_t 1.430] [lr 0.005000] 2024-04-03 20:05:02,128 - Train: 28.89% [156300/541000] [288.9/1000.0] [batch_t 1.443 (1.432)] [data_t 0.003] [optim_t 1.440] [lr 0.005000] 2024-04-03 20:06:12,340 - ==> Total time: 1 day, 19:06:06 Eta: 4 days, 10:02:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 20:07:27,558 - Train: 28.91% [156400/541000] [289.1/1000.0] [batch_t 1.436 (1.440)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-03 20:09:50,660 - Train: 28.93% [156500/541000] [289.3/1000.0] [batch_t 1.443 (1.431)] [data_t 0.002] [optim_t 1.441] [lr 0.005000] 2024-04-03 20:12:13,858 - Train: 28.95% [156600/541000] [289.5/1000.0] [batch_t 1.436 (1.432)] [data_t 0.002] [optim_t 1.434] [lr 0.005000] 2024-04-03 20:14:37,072 - Train: 28.96% [156700/541000] [289.6/1000.0] [batch_t 1.434 (1.432)] [data_t 0.002] [optim_t 1.431] [lr 0.005000] 2024-04-03 20:17:00,230 - Train: 28.98% [156800/541000] [289.8/1000.0] [batch_t 1.436 (1.431)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-03 20:19:09,056 - ==> Total time: 1 day, 19:19:03 Eta: 4 days, 10:03:12 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 20:19:25,644 - Train: 29.00% [156900/541000] [290.0/1000.0] [batch_t 1.431 (1.481)] [data_t 0.002] [optim_t 1.429] [lr 0.005000] 2024-04-03 20:21:48,815 - Train: 29.02% [157000/541000] [290.2/1000.0] [batch_t 1.434 (1.432)] [data_t 0.003] [optim_t 1.431] [lr 0.005000] 2024-04-03 20:24:12,127 - Train: 29.04% [157100/541000] [290.4/1000.0] [batch_t 1.440 (1.433)] [data_t 0.003] [optim_t 1.437] [lr 0.005000] 2024-04-03 20:26:35,381 - Train: 29.06% [157200/541000] [290.6/1000.0] [batch_t 1.431 (1.432)] [data_t 0.002] [optim_t 1.429] [lr 0.005000] 2024-04-03 20:28:58,669 - Train: 29.08% [157300/541000] [290.8/1000.0] [batch_t 1.425 (1.433)] [data_t 0.002] [optim_t 1.423] [lr 0.005000] 2024-04-03 20:31:21,968 - Train: 29.09% [157400/541000] [290.9/1000.0] [batch_t 1.441 (1.433)] [data_t 0.003] [optim_t 1.439] [lr 0.005000] 2024-04-03 20:32:06,358 - ==> Total time: 1 day, 19:32:00 Eta: 4 days, 10:03:58 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 20:33:47,607 - Train: 29.11% [157500/541000] [291.1/1000.0] [batch_t 1.433 (1.440)] [data_t 0.003] [optim_t 1.430] [lr 0.005000] 2024-04-03 20:36:10,872 - Train: 29.13% [157600/541000] [291.3/1000.0] [batch_t 1.432 (1.433)] [data_t 0.003] [optim_t 1.429] [lr 0.005000] 2024-04-03 20:38:34,047 - Train: 29.15% [157700/541000] [291.5/1000.0] [batch_t 1.422 (1.432)] [data_t 0.003] [optim_t 1.419] [lr 0.005000] 2024-04-03 20:40:57,247 - Train: 29.17% [157800/541000] [291.7/1000.0] [batch_t 1.443 (1.432)] [data_t 0.002] [optim_t 1.441] [lr 0.005000] 2024-04-03 20:43:20,527 - Train: 29.19% [157900/541000] [291.9/1000.0] [batch_t 1.435 (1.433)] [data_t 0.002] [optim_t 1.432] [lr 0.005000] 2024-04-03 20:45:03,731 - ==> Total time: 1 day, 19:44:58 Eta: 4 days, 10:04:39 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 20:45:46,469 - Train: 29.21% [158000/541000] [292.1/1000.0] [batch_t 1.431 (1.454)] [data_t 0.002] [optim_t 1.429] [lr 0.005000] 2024-04-03 20:48:09,665 - Train: 29.22% [158100/541000] [292.2/1000.0] [batch_t 1.441 (1.432)] [data_t 0.003] [optim_t 1.438] [lr 0.005000] 2024-04-03 20:50:32,784 - Train: 29.24% [158200/541000] [292.4/1000.0] [batch_t 1.438 (1.431)] [data_t 0.003] [optim_t 1.435] [lr 0.005000] 2024-04-03 20:52:55,936 - Train: 29.26% [158300/541000] [292.6/1000.0] [batch_t 1.427 (1.431)] [data_t 0.002] [optim_t 1.425] [lr 0.005000] 2024-04-03 20:55:19,152 - Train: 29.28% [158400/541000] [292.8/1000.0] [batch_t 1.445 (1.432)] [data_t 0.003] [optim_t 1.443] [lr 0.005000] 2024-04-03 20:57:42,375 - Train: 29.30% [158500/541000] [293.0/1000.0] [batch_t 1.438 (1.432)] [data_t 0.003] [optim_t 1.435] [lr 0.005000] 2024-04-03 20:58:01,036 - ==> Total time: 1 day, 19:57:55 Eta: 4 days, 10:05:13 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 21:00:10,817 - Train: 29.32% [158600/541000] [293.2/1000.0] [batch_t 1.428 (1.446)] [data_t 0.003] [optim_t 1.425] [lr 0.005000] 2024-04-03 21:02:34,099 - Train: 29.33% [158700/541000] [293.3/1000.0] [batch_t 1.428 (1.433)] [data_t 0.003] [optim_t 1.425] [lr 0.005000] 2024-04-03 21:04:57,456 - Train: 29.35% [158800/541000] [293.5/1000.0] [batch_t 1.427 (1.433)] [data_t 0.002] [optim_t 1.425] [lr 0.005000] 2024-04-03 21:07:20,708 - Train: 29.37% [158900/541000] [293.7/1000.0] [batch_t 1.426 (1.432)] [data_t 0.002] [optim_t 1.424] [lr 0.005000] 2024-04-03 21:09:43,963 - Train: 29.39% [159000/541000] [293.9/1000.0] [batch_t 1.433 (1.432)] [data_t 0.002] [optim_t 1.431] [lr 0.005000] 2024-04-03 21:11:01,305 - ==> Total time: 1 day, 20:10:55 Eta: 4 days, 10:05:50 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 21:12:12,343 - Train: 29.41% [159100/541000] [294.1/1000.0] [batch_t 1.440 (1.451)] [data_t 0.003] [optim_t 1.437] [lr 0.005000] 2024-04-03 21:14:35,637 - Train: 29.43% [159200/541000] [294.3/1000.0] [batch_t 1.432 (1.433)] [data_t 0.002] [optim_t 1.429] [lr 0.005000] 2024-04-03 21:16:58,977 - Train: 29.45% [159300/541000] [294.5/1000.0] [batch_t 1.424 (1.433)] [data_t 0.003] [optim_t 1.422] [lr 0.005000] 2024-04-03 21:19:22,152 - Train: 29.46% [159400/541000] [294.6/1000.0] [batch_t 1.428 (1.432)] [data_t 0.002] [optim_t 1.426] [lr 0.005000] 2024-04-03 21:21:45,358 - Train: 29.48% [159500/541000] [294.8/1000.0] [batch_t 1.423 (1.432)] [data_t 0.003] [optim_t 1.420] [lr 0.005000] 2024-04-03 21:24:01,557 - ==> Total time: 1 day, 20:23:56 Eta: 4 days, 10:06:20 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 21:24:13,635 - Train: 29.50% [159600/541000] [295.0/1000.0] [batch_t 1.428 (1.598)] [data_t 0.002] [optim_t 1.425] [lr 0.005000] 2024-04-03 21:26:36,778 - Train: 29.52% [159700/541000] [295.2/1000.0] [batch_t 1.442 (1.431)] [data_t 0.003] [optim_t 1.439] [lr 0.005000] 2024-04-03 21:29:00,051 - Train: 29.54% [159800/541000] [295.4/1000.0] [batch_t 1.436 (1.433)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-03 21:31:23,207 - Train: 29.56% [159900/541000] [295.6/1000.0] [batch_t 1.427 (1.431)] [data_t 0.003] [optim_t 1.425] [lr 0.005000] 2024-04-03 21:33:46,457 - Train: 29.57% [160000/541000] [295.7/1000.0] [batch_t 1.441 (1.432)] [data_t 0.003] [optim_t 1.438] [lr 0.005000] 2024-04-03 21:36:09,668 - Train: 29.59% [160100/541000] [295.9/1000.0] [batch_t 1.437 (1.432)] [data_t 0.003] [optim_t 1.434] [lr 0.005000] 2024-04-03 21:37:01,282 - ==> Total time: 1 day, 20:36:55 Eta: 4 days, 10:06:44 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 21:38:37,449 - Train: 29.61% [160200/541000] [296.1/1000.0] [batch_t 1.423 (1.446)] [data_t 0.002] [optim_t 1.421] [lr 0.005000] 2024-04-03 21:41:00,723 - Train: 29.63% [160300/541000] [296.3/1000.0] [batch_t 1.435 (1.433)] [data_t 0.003] [optim_t 1.432] [lr 0.005000] 2024-04-03 21:43:24,015 - Train: 29.65% [160400/541000] [296.5/1000.0] [batch_t 1.441 (1.433)] [data_t 0.003] [optim_t 1.438] [lr 0.005000] 2024-04-03 21:45:47,332 - Train: 29.67% [160500/541000] [296.7/1000.0] [batch_t 1.441 (1.433)] [data_t 0.002] [optim_t 1.439] [lr 0.005000] 2024-04-03 21:48:10,514 - Train: 29.69% [160600/541000] [296.9/1000.0] [batch_t 1.442 (1.432)] [data_t 0.003] [optim_t 1.439] [lr 0.005000] 2024-04-03 21:50:00,806 - ==> Total time: 1 day, 20:49:55 Eta: 4 days, 10:07:03 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 21:50:38,171 - Train: 29.70% [160700/541000] [297.0/1000.0] [batch_t 1.428 (1.524)] [data_t 0.002] [optim_t 1.425] [lr 0.005000] 2024-04-03 21:53:01,272 - Train: 29.72% [160800/541000] [297.2/1000.0] [batch_t 1.420 (1.431)] [data_t 0.003] [optim_t 1.417] [lr 0.005000] 2024-04-03 21:55:24,457 - Train: 29.74% [160900/541000] [297.4/1000.0] [batch_t 1.437 (1.432)] [data_t 0.003] [optim_t 1.434] [lr 0.005000] 2024-04-03 21:57:47,737 - Train: 29.76% [161000/541000] [297.6/1000.0] [batch_t 1.433 (1.433)] [data_t 0.003] [optim_t 1.430] [lr 0.005000] 2024-04-03 22:00:10,815 - Train: 29.78% [161100/541000] [297.8/1000.0] [batch_t 1.441 (1.431)] [data_t 0.002] [optim_t 1.438] [lr 0.005000] 2024-04-03 22:02:34,098 - Train: 29.80% [161200/541000] [298.0/1000.0] [batch_t 1.431 (1.433)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-03 22:02:59,881 - ==> Total time: 1 day, 21:02:54 Eta: 4 days, 10:07:14 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 22:04:59,512 - Train: 29.82% [161300/541000] [298.2/1000.0] [batch_t 1.450 (1.436)] [data_t 0.003] [optim_t 1.447] [lr 0.005000] 2024-04-03 22:07:22,744 - Train: 29.83% [161400/541000] [298.3/1000.0] [batch_t 1.427 (1.432)] [data_t 0.002] [optim_t 1.425] [lr 0.005000] 2024-04-03 22:09:45,950 - Train: 29.85% [161500/541000] [298.5/1000.0] [batch_t 1.442 (1.432)] [data_t 0.003] [optim_t 1.439] [lr 0.005000] 2024-04-03 22:12:10,969 - Train: 29.87% [161600/541000] [298.7/1000.0] [batch_t 1.457 (1.450)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-03 22:14:36,709 - Train: 29.89% [161700/541000] [298.9/1000.0] [batch_t 1.459 (1.457)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-03 22:16:01,852 - ==> Total time: 1 day, 21:15:56 Eta: 4 days, 10:07:28 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 22:17:05,116 - Train: 29.91% [161800/541000] [299.1/1000.0] [batch_t 1.431 (1.451)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-03 22:19:28,248 - Train: 29.93% [161900/541000] [299.3/1000.0] [batch_t 1.438 (1.431)] [data_t 0.003] [optim_t 1.435] [lr 0.005000] 2024-04-03 22:21:51,295 - Train: 29.94% [162000/541000] [299.4/1000.0] [batch_t 1.426 (1.430)] [data_t 0.003] [optim_t 1.423] [lr 0.005000] 2024-04-03 22:24:14,542 - Train: 29.96% [162100/541000] [299.6/1000.0] [batch_t 1.428 (1.432)] [data_t 0.002] [optim_t 1.425] [lr 0.005000] 2024-04-03 22:26:37,758 - Train: 29.98% [162200/541000] [299.8/1000.0] [batch_t 1.431 (1.432)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-03 22:29:00,923 - Train: 30.00% [162300/541000] [300.0/1000.0] [batch_t 1.432 (1.432)] [data_t 0.003] [optim_t 1.429] [lr 0.005000] 2024-04-03 22:29:23,864 - Test: 36.76% [50/136] [batch_t 0.417 (0.445)] 2024-04-03 22:29:44,889 - Test: 73.53% [100/136] [batch_t 0.426 (0.433)] 2024-04-03 22:29:59,756 - Test: 100.00% [136/136] [batch_t 0.072 (0.427)] 2024-04-03 22:30:48,287 - ==> Metric Time for candle : 0.002 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.657 (mAUROC_px) 8.003 (mAP_px) 0.931 (mF1_max_px) 25.338 (mAUPRO_px) 0.314 (mF1_px_0.2_0.8_0.1) 0.310 (mAcc_px_0.2_0.8_0.1) 0.311 (mIoU_px_0.2_0.8_0.1) 0.908 (mIoU_max_px) 2024-04-03 22:31:25,693 - ==> Metric Time for capsules : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 8.204 (mAUROC_px) 6.282 (mAP_px) 0.721 (mF1_max_px) 20.488 (mAUPRO_px) 0.248 (mF1_px_0.2_0.8_0.1) 0.252 (mAcc_px_0.2_0.8_0.1) 0.254 (mIoU_px_0.2_0.8_0.1) 0.733 (mIoU_max_px) 2024-04-03 22:32:01,456 - ==> Metric Time for cashew : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.744 (mAUROC_px) 5.942 (mAP_px) 0.680 (mF1_max_px) 19.561 (mAUPRO_px) 0.232 (mF1_px_0.2_0.8_0.1) 0.274 (mAcc_px_0.2_0.8_0.1) 0.285 (mIoU_px_0.2_0.8_0.1) 0.834 (mIoU_max_px) 2024-04-03 22:32:37,497 - ==> Metric Time for chewinggum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.686 (mAUROC_px) 5.910 (mAP_px) 0.672 (mF1_max_px) 20.167 (mAUPRO_px) 0.233 (mF1_px_0.2_0.8_0.1) 0.234 (mAcc_px_0.2_0.8_0.1) 0.236 (mIoU_px_0.2_0.8_0.1) 0.676 (mIoU_max_px) 2024-04-03 22:33:14,174 - ==> Metric Time for fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.590 (mAUROC_px) 5.666 (mAP_px) 0.706 (mF1_max_px) 21.110 (mAUPRO_px) 0.230 (mF1_px_0.2_0.8_0.1) 0.227 (mAcc_px_0.2_0.8_0.1) 0.229 (mIoU_px_0.2_0.8_0.1) 0.711 (mIoU_max_px) 2024-04-03 22:34:00,375 - ==> Metric Time for macaroni1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.507 (mAUROC_px) 7.805 (mAP_px) 0.906 (mF1_max_px) 24.879 (mAUPRO_px) 0.311 (mF1_px_0.2_0.8_0.1) 0.311 (mAcc_px_0.2_0.8_0.1) 0.311 (mIoU_px_0.2_0.8_0.1) 0.905 (mIoU_max_px) 2024-04-03 22:34:52,724 - ==> Metric Time for macaroni2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 11.454 (mAUROC_px) 8.521 (mAP_px) 1.099 (mF1_max_px) 28.790 (mAUPRO_px) 0.374 (mF1_px_0.2_0.8_0.1) 0.377 (mAcc_px_0.2_0.8_0.1) 0.374 (mIoU_px_0.2_0.8_0.1) 1.089 (mIoU_max_px) 2024-04-03 22:35:42,956 - ==> Metric Time for pcb1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 11.010 (mAUROC_px) 8.212 (mAP_px) 1.026 (mF1_max_px) 27.671 (mAUPRO_px) 0.338 (mF1_px_0.2_0.8_0.1) 0.343 (mAcc_px_0.2_0.8_0.1) 0.340 (mIoU_px_0.2_0.8_0.1) 0.993 (mIoU_max_px) 2024-04-03 22:36:32,684 - ==> Metric Time for pcb2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.846 (mAUROC_px) 8.309 (mAP_px) 1.015 (mF1_max_px) 27.239 (mAUPRO_px) 0.347 (mF1_px_0.2_0.8_0.1) 0.341 (mAcc_px_0.2_0.8_0.1) 0.341 (mIoU_px_0.2_0.8_0.1) 0.994 (mIoU_max_px) 2024-04-03 22:37:22,009 - ==> Metric Time for pcb3 : 0.001 (mAUROC_sp_max) 0.001 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.789 (mAUROC_px) 8.120 (mAP_px) 1.013 (mF1_max_px) 27.087 (mAUPRO_px) 0.343 (mF1_px_0.2_0.8_0.1) 0.339 (mAcc_px_0.2_0.8_0.1) 0.344 (mIoU_px_0.2_0.8_0.1) 0.991 (mIoU_max_px) 2024-04-03 22:38:13,934 - ==> Metric Time for pcb4 : 0.002 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.818 (mAUROC_px) 8.689 (mAP_px) 0.982 (mF1_max_px) 29.022 (mAUPRO_px) 0.343 (mF1_px_0.2_0.8_0.1) 0.346 (mAcc_px_0.2_0.8_0.1) 0.346 (mIoU_px_0.2_0.8_0.1) 1.085 (mIoU_max_px) 2024-04-03 22:38:51,760 - ==> Metric Time for pipe_fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.979 (mAUROC_px) 5.996 (mAP_px) 0.771 (mF1_max_px) 21.368 (mAUPRO_px) 0.251 (mF1_px_0.2_0.8_0.1) 0.246 (mAcc_px_0.2_0.8_0.1) 0.248 (mIoU_px_0.2_0.8_0.1) 0.740 (mIoU_max_px) 2024-04-03 22:38:51,781 - | Name | mAUROC_sp_max | mAUROC_sp_max (Max) | mAP_sp_max | mAP_sp_max (Max) | mF1_max_sp_max | mF1_max_sp_max (Max) | mAUROC_px | mAUROC_px (Max) | mAP_px | mAP_px (Max) | mF1_max_px | mF1_max_px (Max) | mAUPRO_px | mAUPRO_px (Max) | mF1_px_0.2_0.8_0.1 | mF1_px_0.2_0.8_0.1 (Max) | mAcc_px_0.2_0.8_0.1 | mAcc_px_0.2_0.8_0.1 (Max) | mIoU_px_0.2_0.8_0.1 | mIoU_px_0.2_0.8_0.1 (Max) | mIoU_max_px | mIoU_max_px (Max) | |:----------:|:---------------:|:---------------------:|:------------:|:------------------:|:----------------:|:----------------------:|:-----------:|:------------------:|:--------:|:------------------:|:------------:|:------------------:|:-----------:|:------------------:|:--------------------:|:--------------------------:|:---------------------:|:---------------------------:|:---------------------:|:---------------------------:|:-------------:|:-------------------:| | candle | 96.620 | 97.100 (50 epoch) | 96.883 | 97.346 (50 epoch) | 90.722 | 91.542 (50 epoch) | 99.064 | 99.098 (250 epoch) | 22.913 | 23.463 (150 epoch) | 31.841 | 32.654 (200 epoch) | 96.087 | 96.087 (300 epoch) | 21.022 | 21.022 (300 epoch) | 39.458 | 39.458 (300 epoch) | 12.040 | 12.040 (300 epoch) | 18.935 | 19.513 (200 epoch) | | capsules | 87.750 | 91.750 (100 epoch) | 90.919 | 94.964 (100 epoch) | 86.239 | 88.780 (100 epoch) | 98.932 | 99.123 (100 epoch) | 28.978 | 61.256 (100 epoch) | 36.119 | 59.868 (250 epoch) | 88.586 | 91.811 (100 epoch) | 22.182 | 33.186 (100 epoch) | 60.924 | 62.576 (50 epoch) | 12.924 | 21.701 (100 epoch) | 22.040 | 42.722 (250 epoch) | | cashew | 94.460 | 94.800 (150 epoch) | 97.289 | 97.492 (150 epoch) | 92.386 | 92.386 (200 epoch) | 95.230 | 95.614 (250 epoch) | 49.660 | 50.952 (250 epoch) | 53.620 | 53.875 (250 epoch) | 87.582 | 88.960 (150 epoch) | 27.796 | 28.279 (150 epoch) | 32.629 | 34.522 (150 epoch) | 17.434 | 17.734 (250 epoch) | 36.630 | 36.869 (250 epoch) | | chewinggum | 97.802 | 97.802 (300 epoch) | 98.939 | 98.939 (300 epoch) | 94.416 | 94.416 (300 epoch) | 97.989 | 98.145 (100 epoch) | 52.790 | 59.823 (50 epoch) | 57.432 | 61.088 (50 epoch) | 78.966 | 79.773 (50 epoch) | 29.337 | 35.748 (50 epoch) | 53.709 | 65.611 (50 epoch) | 18.964 | 23.754 (50 epoch) | 40.284 | 43.976 (50 epoch) | | fryum | 95.940 | 95.940 (300 epoch) | 98.018 | 98.018 (300 epoch) | 92.019 | 92.308 (250 epoch) | 97.097 | 97.138 (250 epoch) | 49.250 | 49.934 (250 epoch) | 52.343 | 52.992 (250 epoch) | 91.552 | 91.622 (100 epoch) | 25.373 | 26.312 (50 epoch) | 26.414 | 27.708 (50 epoch) | 15.764 | 16.200 (50 epoch) | 35.449 | 36.047 (250 epoch) | | macaroni1 | 92.084 | 92.084 (300 epoch) | 90.374 | 90.374 (300 epoch) | 82.653 | 82.653 (300 epoch) | 99.533 | 99.533 (300 epoch) | 18.994 | 18.994 (300 epoch) | 28.871 | 29.377 (200 epoch) | 95.397 | 95.899 (250 epoch) | 15.803 | 15.803 (300 epoch) | 37.779 | 40.765 (50 epoch) | 8.828 | 8.828 (300 epoch) | 16.871 | 17.217 (200 epoch) | | macaroni2 | 77.427 | 82.589 (50 epoch) | 70.658 | 81.196 (50 epoch) | 75.000 | 75.622 (50 epoch) | 98.217 | 99.452 (100 epoch) | 4.910 | 11.272 (50 epoch) | 12.307 | 17.568 (50 epoch) | 91.179 | 96.253 (150 epoch) | 4.837 | 10.086 (50 epoch) | 49.365 | 49.562 (50 epoch) | 2.529 | 5.433 (50 epoch) | 6.557 | 9.630 (50 epoch) | | pcb1 | 96.710 | 96.710 (300 epoch) | 95.349 | 95.349 (300 epoch) | 94.737 | 94.737 (300 epoch) | 99.792 | 99.792 (300 epoch) | 80.551 | 82.024 (250 epoch) | 74.247 | 75.268 (250 epoch) | 94.357 | 94.357 (300 epoch) | 46.758 | 47.196 (250 epoch) | 58.807 | 62.362 (50 epoch) | 32.861 | 33.507 (250 epoch) | 59.042 | 60.344 (250 epoch) | | pcb2 | 95.080 | 95.080 (300 epoch) | 94.777 | 94.777 (300 epoch) | 89.796 | 90.000 (150 epoch) | 98.969 | 98.978 (250 epoch) | 12.830 | 14.130 (50 epoch) | 23.882 | 23.977 (250 epoch) | 89.305 | 90.754 (250 epoch) | 15.769 | 16.150 (200 epoch) | 46.211 | 47.820 (150 epoch) | 8.672 | 8.892 (200 epoch) | 13.560 | 13.621 (250 epoch) | | pcb3 | 95.416 | 95.416 (300 epoch) | 95.603 | 95.603 (300 epoch) | 88.889 | 88.991 (250 epoch) | 99.103 | 99.124 (150 epoch) | 18.889 | 20.249 (150 epoch) | 27.451 | 27.912 (150 epoch) | 91.777 | 91.777 (300 epoch) | 19.766 | 20.042 (150 epoch) | 44.693 | 48.341 (50 epoch) | 11.123 | 11.295 (150 epoch) | 15.909 | 16.219 (150 epoch) | | pcb4 | 99.762 | 99.901 (100 epoch) | 99.763 | 99.901 (100 epoch) | 97.537 | 99.000 (50 epoch) | 98.644 | 98.727 (250 epoch) | 47.215 | 47.966 (250 epoch) | 47.048 | 47.709 (250 epoch) | 90.170 | 90.729 (250 epoch) | 25.922 | 26.925 (200 epoch) | 34.795 | 39.716 (50 epoch) | 15.911 | 16.537 (200 epoch) | 30.760 | 31.328 (250 epoch) | | pipe_fryum | 98.880 | 98.880 (300 epoch) | 99.414 | 99.414 (300 epoch) | 97.561 | 97.561 (200 epoch) | 99.095 | 99.096 (250 epoch) | 54.482 | 54.482 (300 epoch) | 58.111 | 58.465 (100 epoch) | 94.440 | 95.525 (150 epoch) | 33.812 | 34.684 (250 epoch) | 39.950 | 43.014 (100 epoch) | 21.867 | 22.399 (250 epoch) | 40.956 | 41.308 (100 epoch) | | Avg | 93.994 | 94.251 (100 epoch) | 93.999 | 94.481 (100 epoch) | 90.163 | 90.163 (300 epoch) | 98.472 | 98.606 (250 epoch) | 36.788 | 39.701 (250 epoch) | 41.939 | 44.125 (250 epoch) | 90.783 | 91.494 (250 epoch) | 24.031 | 25.164 (100 epoch) | 43.728 | 46.179 (50 epoch) | 14.910 | 15.755 (100 epoch) | 28.083 | 29.990 (250 epoch) | 2024-04-03 22:38:52,163 - ==> Total time: 1 day, 21:38:46 Eta: 4 days, 10:30:28 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 22:41:17,789 - Train: 30.02% [162400/541000] [300.2/1000.0] [batch_t 1.439 (1.435)] [data_t 0.003] [optim_t 1.437] [lr 0.005000] 2024-04-03 22:43:41,077 - Train: 30.04% [162500/541000] [300.4/1000.0] [batch_t 1.437 (1.433)] [data_t 0.002] [optim_t 1.435] [lr 0.005000] 2024-04-03 22:46:04,366 - Train: 30.06% [162600/541000] [300.6/1000.0] [batch_t 1.437 (1.433)] [data_t 0.003] [optim_t 1.434] [lr 0.005000] 2024-04-03 22:48:27,706 - Train: 30.07% [162700/541000] [300.7/1000.0] [batch_t 1.431 (1.433)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-03 22:50:51,110 - Train: 30.09% [162800/541000] [300.9/1000.0] [batch_t 1.446 (1.434)] [data_t 0.003] [optim_t 1.443] [lr 0.005000] 2024-04-03 22:51:49,886 - ==> Total time: 1 day, 21:51:44 Eta: 4 days, 10:30:15 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 22:53:16,704 - Train: 30.11% [162900/541000] [301.1/1000.0] [batch_t 1.441 (1.439)] [data_t 0.002] [optim_t 1.438] [lr 0.005000] 2024-04-03 22:55:39,996 - Train: 30.13% [163000/541000] [301.3/1000.0] [batch_t 1.440 (1.433)] [data_t 0.003] [optim_t 1.437] [lr 0.005000] 2024-04-03 22:58:03,262 - Train: 30.15% [163100/541000] [301.5/1000.0] [batch_t 1.431 (1.433)] [data_t 0.002] [optim_t 1.429] [lr 0.005000] 2024-04-03 23:00:26,492 - Train: 30.17% [163200/541000] [301.7/1000.0] [batch_t 1.440 (1.432)] [data_t 0.003] [optim_t 1.437] [lr 0.005000] 2024-04-03 23:02:49,890 - Train: 30.18% [163300/541000] [301.8/1000.0] [batch_t 1.439 (1.434)] [data_t 0.003] [optim_t 1.437] [lr 0.005000] 2024-04-03 23:04:47,410 - ==> Total time: 1 day, 22:04:41 Eta: 4 days, 10:29:55 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 23:05:15,268 - Train: 30.20% [163400/541000] [302.0/1000.0] [batch_t 1.439 (1.454)] [data_t 0.003] [optim_t 1.437] [lr 0.005000] 2024-04-03 23:07:38,466 - Train: 30.22% [163500/541000] [302.2/1000.0] [batch_t 1.432 (1.432)] [data_t 0.003] [optim_t 1.430] [lr 0.005000] 2024-04-03 23:10:01,553 - Train: 30.24% [163600/541000] [302.4/1000.0] [batch_t 1.437 (1.431)] [data_t 0.003] [optim_t 1.434] [lr 0.005000] 2024-04-03 23:12:24,772 - Train: 30.26% [163700/541000] [302.6/1000.0] [batch_t 1.422 (1.432)] [data_t 0.003] [optim_t 1.420] [lr 0.005000] 2024-04-03 23:14:48,108 - Train: 30.28% [163800/541000] [302.8/1000.0] [batch_t 1.425 (1.433)] [data_t 0.003] [optim_t 1.423] [lr 0.005000] 2024-04-03 23:17:11,240 - Train: 30.30% [163900/541000] [303.0/1000.0] [batch_t 1.434 (1.431)] [data_t 0.003] [optim_t 1.431] [lr 0.005000] 2024-04-03 23:17:44,194 - ==> Total time: 1 day, 22:17:38 Eta: 4 days, 10:29:29 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 23:19:36,474 - Train: 30.31% [164000/541000] [303.1/1000.0] [batch_t 1.432 (1.436)] [data_t 0.002] [optim_t 1.430] [lr 0.005000] 2024-04-03 23:21:59,825 - Train: 30.33% [164100/541000] [303.3/1000.0] [batch_t 1.441 (1.433)] [data_t 0.003] [optim_t 1.438] [lr 0.005000] 2024-04-03 23:24:23,136 - Train: 30.35% [164200/541000] [303.5/1000.0] [batch_t 1.441 (1.433)] [data_t 0.003] [optim_t 1.438] [lr 0.005000] 2024-04-03 23:26:46,475 - Train: 30.37% [164300/541000] [303.7/1000.0] [batch_t 1.426 (1.433)] [data_t 0.002] [optim_t 1.424] [lr 0.005000] 2024-04-03 23:29:09,712 - Train: 30.39% [164400/541000] [303.9/1000.0] [batch_t 1.421 (1.432)] [data_t 0.002] [optim_t 1.419] [lr 0.005000] 2024-04-03 23:30:41,361 - ==> Total time: 1 day, 22:30:35 Eta: 4 days, 10:28:59 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 23:31:35,191 - Train: 30.41% [164500/541000] [304.1/1000.0] [batch_t 1.430 (1.447)] [data_t 0.003] [optim_t 1.427] [lr 0.005000] 2024-04-03 23:33:58,442 - Train: 30.43% [164600/541000] [304.3/1000.0] [batch_t 1.433 (1.432)] [data_t 0.002] [optim_t 1.430] [lr 0.005000] 2024-04-03 23:36:21,600 - Train: 30.44% [164700/541000] [304.4/1000.0] [batch_t 1.431 (1.431)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-03 23:38:44,816 - Train: 30.46% [164800/541000] [304.6/1000.0] [batch_t 1.442 (1.432)] [data_t 0.003] [optim_t 1.439] [lr 0.005000] 2024-04-03 23:41:07,998 - Train: 30.48% [164900/541000] [304.8/1000.0] [batch_t 1.442 (1.432)] [data_t 0.003] [optim_t 1.438] [lr 0.005000] 2024-04-03 23:43:31,184 - Train: 30.50% [165000/541000] [305.0/1000.0] [batch_t 1.436 (1.432)] [data_t 0.002] [optim_t 1.433] [lr 0.005000] 2024-04-03 23:43:38,359 - ==> Total time: 1 day, 22:43:32 Eta: 4 days, 10:28:24 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 23:45:57,000 - Train: 30.52% [165100/541000] [305.2/1000.0] [batch_t 1.436 (1.441)] [data_t 0.003] [optim_t 1.434] [lr 0.005000] 2024-04-03 23:48:20,253 - Train: 30.54% [165200/541000] [305.4/1000.0] [batch_t 1.436 (1.432)] [data_t 0.002] [optim_t 1.434] [lr 0.005000] 2024-04-03 23:50:43,430 - Train: 30.55% [165300/541000] [305.5/1000.0] [batch_t 1.427 (1.432)] [data_t 0.003] [optim_t 1.425] [lr 0.005000] 2024-04-03 23:53:06,750 - Train: 30.57% [165400/541000] [305.7/1000.0] [batch_t 1.437 (1.433)] [data_t 0.003] [optim_t 1.434] [lr 0.005000] 2024-04-03 23:55:29,957 - Train: 30.59% [165500/541000] [305.9/1000.0] [batch_t 1.429 (1.432)] [data_t 0.002] [optim_t 1.427] [lr 0.005000] 2024-04-03 23:56:35,813 - ==> Total time: 1 day, 22:56:30 Eta: 4 days, 10:27:45 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-03 23:57:55,281 - Train: 30.61% [165600/541000] [306.1/1000.0] [batch_t 1.436 (1.440)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-04 00:00:18,452 - Train: 30.63% [165700/541000] [306.3/1000.0] [batch_t 1.438 (1.432)] [data_t 0.002] [optim_t 1.436] [lr 0.005000] 2024-04-04 00:02:41,823 - Train: 30.65% [165800/541000] [306.5/1000.0] [batch_t 1.440 (1.434)] [data_t 0.002] [optim_t 1.438] [lr 0.005000] 2024-04-04 00:05:05,081 - Train: 30.67% [165900/541000] [306.7/1000.0] [batch_t 1.427 (1.432)] [data_t 0.003] [optim_t 1.425] [lr 0.005000] 2024-04-04 00:07:28,346 - Train: 30.68% [166000/541000] [306.8/1000.0] [batch_t 1.442 (1.433)] [data_t 0.002] [optim_t 1.440] [lr 0.005000] 2024-04-04 00:09:32,851 - ==> Total time: 1 day, 23:09:27 Eta: 4 days, 10:27:00 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 00:09:53,631 - Train: 30.70% [166100/541000] [307.0/1000.0] [batch_t 1.432 (1.458)] [data_t 0.002] [optim_t 1.430] [lr 0.005000] 2024-04-04 00:12:16,837 - Train: 30.72% [166200/541000] [307.2/1000.0] [batch_t 1.431 (1.432)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-04 00:14:39,939 - Train: 30.74% [166300/541000] [307.4/1000.0] [batch_t 1.428 (1.431)] [data_t 0.003] [optim_t 1.425] [lr 0.005000] 2024-04-04 00:17:03,217 - Train: 30.76% [166400/541000] [307.6/1000.0] [batch_t 1.431 (1.433)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-04 00:19:26,360 - Train: 30.78% [166500/541000] [307.8/1000.0] [batch_t 1.426 (1.431)] [data_t 0.002] [optim_t 1.424] [lr 0.005000] 2024-04-04 00:21:49,559 - Train: 30.79% [166600/541000] [307.9/1000.0] [batch_t 1.437 (1.432)] [data_t 0.002] [optim_t 1.435] [lr 0.005000] 2024-04-04 00:22:29,654 - ==> Total time: 1 day, 23:22:24 Eta: 4 days, 10:26:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 00:24:15,265 - Train: 30.81% [166700/541000] [308.1/1000.0] [batch_t 1.432 (1.439)] [data_t 0.003] [optim_t 1.429] [lr 0.005000] 2024-04-04 00:26:38,487 - Train: 30.83% [166800/541000] [308.3/1000.0] [batch_t 1.444 (1.432)] [data_t 0.003] [optim_t 1.442] [lr 0.005000] 2024-04-04 00:29:01,727 - Train: 30.85% [166900/541000] [308.5/1000.0] [batch_t 1.435 (1.432)] [data_t 0.002] [optim_t 1.433] [lr 0.005000] 2024-04-04 00:31:24,961 - Train: 30.87% [167000/541000] [308.7/1000.0] [batch_t 1.427 (1.432)] [data_t 0.002] [optim_t 1.425] [lr 0.005000] 2024-04-04 00:33:48,197 - Train: 30.89% [167100/541000] [308.9/1000.0] [batch_t 1.432 (1.432)] [data_t 0.003] [optim_t 1.429] [lr 0.005000] 2024-04-04 00:35:26,995 - ==> Total time: 1 day, 23:35:21 Eta: 4 days, 10:25:16 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 00:36:13,457 - Train: 30.91% [167200/541000] [309.1/1000.0] [batch_t 1.443 (1.443)] [data_t 0.002] [optim_t 1.440] [lr 0.005000] 2024-04-04 00:38:36,744 - Train: 30.92% [167300/541000] [309.2/1000.0] [batch_t 1.437 (1.433)] [data_t 0.002] [optim_t 1.434] [lr 0.005000] 2024-04-04 00:40:59,902 - Train: 30.94% [167400/541000] [309.4/1000.0] [batch_t 1.441 (1.431)] [data_t 0.003] [optim_t 1.438] [lr 0.005000] 2024-04-04 00:43:23,141 - Train: 30.96% [167500/541000] [309.6/1000.0] [batch_t 1.437 (1.432)] [data_t 0.003] [optim_t 1.434] [lr 0.005000] 2024-04-04 00:45:46,409 - Train: 30.98% [167600/541000] [309.8/1000.0] [batch_t 1.441 (1.433)] [data_t 0.003] [optim_t 1.438] [lr 0.005000] 2024-04-04 00:48:09,676 - Train: 31.00% [167700/541000] [310.0/1000.0] [batch_t 1.436 (1.433)] [data_t 0.003] [optim_t 1.434] [lr 0.005000] 2024-04-04 00:48:23,985 - ==> Total time: 1 day, 23:48:18 Eta: 4 days, 10:24:17 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 00:50:34,949 - Train: 31.02% [167800/541000] [310.2/1000.0] [batch_t 1.429 (1.436)] [data_t 0.003] [optim_t 1.427] [lr 0.005000] 2024-04-04 00:52:58,297 - Train: 31.04% [167900/541000] [310.4/1000.0] [batch_t 1.427 (1.433)] [data_t 0.002] [optim_t 1.425] [lr 0.005000] 2024-04-04 00:55:21,503 - Train: 31.05% [168000/541000] [310.5/1000.0] [batch_t 1.432 (1.432)] [data_t 0.002] [optim_t 1.430] [lr 0.005000] 2024-04-04 00:57:44,778 - Train: 31.07% [168100/541000] [310.7/1000.0] [batch_t 1.432 (1.433)] [data_t 0.002] [optim_t 1.430] [lr 0.005000] 2024-04-04 01:00:08,070 - Train: 31.09% [168200/541000] [310.9/1000.0] [batch_t 1.445 (1.433)] [data_t 0.003] [optim_t 1.442] [lr 0.005000] 2024-04-04 01:01:21,117 - ==> Total time: 2 days, 0:01:15 Eta: 4 days, 10:23:14 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 01:02:33,332 - Train: 31.11% [168300/541000] [311.1/1000.0] [batch_t 1.429 (1.439)] [data_t 0.003] [optim_t 1.426] [lr 0.005000] 2024-04-04 01:04:56,626 - Train: 31.13% [168400/541000] [311.3/1000.0] [batch_t 1.427 (1.433)] [data_t 0.002] [optim_t 1.424] [lr 0.005000] 2024-04-04 01:07:19,829 - Train: 31.15% [168500/541000] [311.5/1000.0] [batch_t 1.430 (1.432)] [data_t 0.003] [optim_t 1.427] [lr 0.005000] 2024-04-04 01:09:43,195 - Train: 31.16% [168600/541000] [311.6/1000.0] [batch_t 1.430 (1.434)] [data_t 0.002] [optim_t 1.427] [lr 0.005000] 2024-04-04 01:12:06,454 - Train: 31.18% [168700/541000] [311.8/1000.0] [batch_t 1.423 (1.433)] [data_t 0.002] [optim_t 1.421] [lr 0.005000] 2024-04-04 01:14:18,216 - ==> Total time: 2 days, 0:14:12 Eta: 4 days, 10:22:06 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 01:14:31,685 - Train: 31.20% [168800/541000] [312.0/1000.0] [batch_t 1.421 (1.472)] [data_t 0.002] [optim_t 1.419] [lr 0.005000] 2024-04-04 01:16:54,988 - Train: 31.22% [168900/541000] [312.2/1000.0] [batch_t 1.434 (1.433)] [data_t 0.003] [optim_t 1.431] [lr 0.005000] 2024-04-04 01:19:18,263 - Train: 31.24% [169000/541000] [312.4/1000.0] [batch_t 1.430 (1.433)] [data_t 0.003] [optim_t 1.427] [lr 0.005000] 2024-04-04 01:21:41,587 - Train: 31.26% [169100/541000] [312.6/1000.0] [batch_t 1.459 (1.433)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-04 01:24:04,900 - Train: 31.28% [169200/541000] [312.8/1000.0] [batch_t 1.427 (1.433)] [data_t 0.003] [optim_t 1.424] [lr 0.005000] 2024-04-04 01:26:28,122 - Train: 31.29% [169300/541000] [312.9/1000.0] [batch_t 1.425 (1.432)] [data_t 0.003] [optim_t 1.422] [lr 0.005000] 2024-04-04 01:27:15,445 - ==> Total time: 2 days, 0:27:09 Eta: 4 days, 10:20:54 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 01:28:53,462 - Train: 31.31% [169400/541000] [313.1/1000.0] [batch_t 1.436 (1.438)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-04 01:31:16,858 - Train: 31.33% [169500/541000] [313.3/1000.0] [batch_t 1.436 (1.434)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-04 01:33:40,227 - Train: 31.35% [169600/541000] [313.5/1000.0] [batch_t 1.430 (1.434)] [data_t 0.003] [optim_t 1.427] [lr 0.005000] 2024-04-04 01:36:03,528 - Train: 31.37% [169700/541000] [313.7/1000.0] [batch_t 1.446 (1.433)] [data_t 0.002] [optim_t 1.444] [lr 0.005000] 2024-04-04 01:38:26,823 - Train: 31.39% [169800/541000] [313.9/1000.0] [batch_t 1.428 (1.433)] [data_t 0.002] [optim_t 1.425] [lr 0.005000] 2024-04-04 01:40:12,897 - ==> Total time: 2 days, 0:40:07 Eta: 4 days, 10:19:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 01:40:52,386 - Train: 31.40% [169900/541000] [314.0/1000.0] [batch_t 1.427 (1.454)] [data_t 0.002] [optim_t 1.425] [lr 0.005000] 2024-04-04 01:43:15,803 - Train: 31.42% [170000/541000] [314.2/1000.0] [batch_t 1.428 (1.434)] [data_t 0.002] [optim_t 1.426] [lr 0.005000] 2024-04-04 01:45:39,060 - Train: 31.44% [170100/541000] [314.4/1000.0] [batch_t 1.438 (1.432)] [data_t 0.002] [optim_t 1.435] [lr 0.005000] 2024-04-04 01:48:02,458 - Train: 31.46% [170200/541000] [314.6/1000.0] [batch_t 1.426 (1.434)] [data_t 0.003] [optim_t 1.423] [lr 0.005000] 2024-04-04 01:50:25,738 - Train: 31.48% [170300/541000] [314.8/1000.0] [batch_t 1.438 (1.433)] [data_t 0.002] [optim_t 1.436] [lr 0.005000] 2024-04-04 01:52:48,922 - Train: 31.50% [170400/541000] [315.0/1000.0] [batch_t 1.435 (1.432)] [data_t 0.002] [optim_t 1.433] [lr 0.005000] 2024-04-04 01:53:10,472 - ==> Total time: 2 days, 0:53:04 Eta: 4 days, 10:18:17 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 01:55:14,155 - Train: 31.52% [170500/541000] [315.2/1000.0] [batch_t 1.427 (1.435)] [data_t 0.003] [optim_t 1.425] [lr 0.005000] 2024-04-04 01:57:37,495 - Train: 31.53% [170600/541000] [315.3/1000.0] [batch_t 1.437 (1.433)] [data_t 0.003] [optim_t 1.434] [lr 0.005000] 2024-04-04 02:00:03,139 - Train: 31.55% [170700/541000] [315.5/1000.0] [batch_t 1.464 (1.456)] [data_t 0.003] [optim_t 1.461] [lr 0.005000] 2024-04-04 02:02:28,904 - Train: 31.57% [170800/541000] [315.7/1000.0] [batch_t 1.444 (1.458)] [data_t 0.003] [optim_t 1.441] [lr 0.005000] 2024-04-04 02:04:52,192 - Train: 31.59% [170900/541000] [315.9/1000.0] [batch_t 1.447 (1.433)] [data_t 0.003] [optim_t 1.444] [lr 0.005000] 2024-04-04 02:06:12,459 - ==> Total time: 2 days, 1:06:06 Eta: 4 days, 10:17:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 02:07:17,567 - Train: 31.61% [171000/541000] [316.1/1000.0] [batch_t 1.437 (1.441)] [data_t 0.003] [optim_t 1.435] [lr 0.005000] 2024-04-04 02:09:07,346 - Train: 31.63% [171100/541000] [316.3/1000.0] [batch_t 0.607 (1.098)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 02:10:08,061 - Train: 31.65% [171200/541000] [316.5/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 02:11:08,747 - Train: 31.66% [171300/541000] [316.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 02:12:09,411 - Train: 31.68% [171400/541000] [316.8/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 02:13:08,380 - ==> Total time: 2 days, 1:13:02 Eta: 4 days, 10:02:33 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 02:13:12,181 - Train: 31.70% [171500/541000] [317.0/1000.0] [batch_t 0.605 (0.717)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 02:14:12,892 - Train: 31.72% [171600/541000] [317.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 02:15:13,589 - Train: 31.74% [171700/541000] [317.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 02:16:14,277 - Train: 31.76% [171800/541000] [317.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 02:17:14,956 - Train: 31.77% [171900/541000] [317.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 02:18:15,645 - Train: 31.79% [172000/541000] [317.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 02:18:38,857 - ==> Total time: 2 days, 1:18:33 Eta: 4 days, 9:45:04 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 02:19:18,595 - Train: 31.81% [172100/541000] [318.1/1000.0] [batch_t 0.606 (0.614)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 02:20:19,290 - Train: 31.83% [172200/541000] [318.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 02:21:20,009 - Train: 31.85% [172300/541000] [318.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 02:22:20,746 - Train: 31.87% [172400/541000] [318.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 02:23:21,470 - Train: 31.89% [172500/541000] [318.9/1000.0] [batch_t 0.611 (0.607)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-04 02:24:09,402 - ==> Total time: 2 days, 1:24:03 Eta: 4 days, 9:27:40 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 02:24:24,565 - Train: 31.90% [172600/541000] [319.0/1000.0] [batch_t 0.607 (0.638)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 02:25:25,392 - Train: 31.92% [172700/541000] [319.2/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 02:26:26,071 - Train: 31.94% [172800/541000] [319.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 02:27:26,777 - Train: 31.96% [172900/541000] [319.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 02:28:27,493 - Train: 31.98% [173000/541000] [319.8/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 02:29:28,193 - Train: 32.00% [173100/541000] [320.0/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 02:29:40,356 - ==> Total time: 2 days, 1:29:34 Eta: 4 days, 9:10:21 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 02:30:31,272 - Train: 32.01% [173200/541000] [320.1/1000.0] [batch_t 0.607 (0.613)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 02:31:32,156 - Train: 32.03% [173300/541000] [320.3/1000.0] [batch_t 0.606 (0.609)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 02:32:32,851 - Train: 32.05% [173400/541000] [320.5/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 02:33:33,514 - Train: 32.07% [173500/541000] [320.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 02:34:34,174 - Train: 32.09% [173600/541000] [320.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 02:35:11,202 - ==> Total time: 2 days, 1:35:05 Eta: 4 days, 8:53:06 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 02:35:37,020 - Train: 32.11% [173700/541000] [321.1/1000.0] [batch_t 0.606 (0.618)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 02:36:37,712 - Train: 32.13% [173800/541000] [321.3/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 02:37:38,524 - Train: 32.14% [173900/541000] [321.4/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 02:38:39,231 - Train: 32.16% [174000/541000] [321.6/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 02:39:39,938 - Train: 32.18% [174100/541000] [321.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 02:40:40,663 - Train: 32.20% [174200/541000] [322.0/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 02:40:41,881 - ==> Total time: 2 days, 1:40:36 Eta: 4 days, 8:35:55 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 02:41:43,470 - Train: 32.22% [174300/541000] [322.2/1000.0] [batch_t 0.606 (0.611)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 02:42:44,144 - Train: 32.24% [174400/541000] [322.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 02:43:44,975 - Train: 32.26% [174500/541000] [322.6/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 02:44:45,678 - Train: 32.27% [174600/541000] [322.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 02:45:46,419 - Train: 32.29% [174700/541000] [322.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 02:46:12,513 - ==> Total time: 2 days, 1:46:06 Eta: 4 days, 8:18:49 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 02:46:49,181 - Train: 32.31% [174800/541000] [323.1/1000.0] [batch_t 0.605 (0.613)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 02:47:49,872 - Train: 32.33% [174900/541000] [323.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 02:48:50,550 - Train: 32.35% [175000/541000] [323.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 02:49:51,363 - Train: 32.37% [175100/541000] [323.7/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 02:50:52,064 - Train: 32.38% [175200/541000] [323.8/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 02:51:43,069 - ==> Total time: 2 days, 1:51:37 Eta: 4 days, 8:01:47 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 02:51:54,925 - Train: 32.40% [175300/541000] [324.0/1000.0] [batch_t 0.608 (0.631)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 02:52:55,613 - Train: 32.42% [175400/541000] [324.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 02:53:56,267 - Train: 32.44% [175500/541000] [324.4/1000.0] [batch_t 0.607 (0.606)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 02:54:56,954 - Train: 32.46% [175600/541000] [324.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 02:55:57,823 - Train: 32.48% [175700/541000] [324.8/1000.0] [batch_t 0.607 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 02:56:58,502 - Train: 32.50% [175800/541000] [325.0/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 02:57:13,686 - ==> Total time: 2 days, 1:57:08 Eta: 4 days, 7:44:49 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 02:58:01,255 - Train: 32.51% [175900/541000] [325.1/1000.0] [batch_t 0.607 (0.612)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 02:59:01,960 - Train: 32.53% [176000/541000] [325.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 03:00:02,647 - Train: 32.55% [176100/541000] [325.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 03:01:03,345 - Train: 32.57% [176200/541000] [325.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 03:02:04,170 - Train: 32.59% [176300/541000] [325.9/1000.0] [batch_t 0.603 (0.608)] [data_t 0.002] [optim_t 0.601] [lr 0.005000] 2024-04-04 03:02:44,198 - ==> Total time: 2 days, 2:02:38 Eta: 4 days, 7:27:55 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 03:03:07,019 - Train: 32.61% [176400/541000] [326.1/1000.0] [batch_t 0.607 (0.617)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 03:04:07,684 - Train: 32.62% [176500/541000] [326.2/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 03:05:08,334 - Train: 32.64% [176600/541000] [326.4/1000.0] [batch_t 0.607 (0.606)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 03:06:09,006 - Train: 32.66% [176700/541000] [326.6/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 03:07:09,791 - Train: 32.68% [176800/541000] [326.8/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 03:08:10,439 - Train: 32.70% [176900/541000] [327.0/1000.0] [batch_t 0.608 (0.606)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-04 03:08:14,686 - ==> Total time: 2 days, 2:08:09 Eta: 4 days, 7:11:05 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 03:09:13,201 - Train: 32.72% [177000/541000] [327.2/1000.0] [batch_t 0.607 (0.610)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 03:10:13,870 - Train: 32.74% [177100/541000] [327.4/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 03:11:14,543 - Train: 32.75% [177200/541000] [327.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 03:12:15,360 - Train: 32.77% [177300/541000] [327.7/1000.0] [batch_t 0.604 (0.608)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-04 03:13:16,006 - Train: 32.79% [177400/541000] [327.9/1000.0] [batch_t 0.608 (0.606)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 03:13:45,127 - ==> Total time: 2 days, 2:13:39 Eta: 4 days, 6:54:19 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 03:14:18,927 - Train: 32.81% [177500/541000] [328.1/1000.0] [batch_t 0.606 (0.614)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 03:15:19,580 - Train: 32.83% [177600/541000] [328.3/1000.0] [batch_t 0.608 (0.606)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 03:16:20,246 - Train: 32.85% [177700/541000] [328.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 03:17:20,922 - Train: 32.87% [177800/541000] [328.7/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 03:18:21,785 - Train: 32.88% [177900/541000] [328.8/1000.0] [batch_t 0.606 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 03:19:15,813 - ==> Total time: 2 days, 2:19:10 Eta: 4 days, 6:37:38 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 03:19:24,540 - Train: 32.90% [178000/541000] [329.0/1000.0] [batch_t 0.607 (0.641)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 03:20:25,157 - Train: 32.92% [178100/541000] [329.2/1000.0] [batch_t 0.603 (0.606)] [data_t 0.002] [optim_t 0.601] [lr 0.005000] 2024-04-04 03:21:25,809 - Train: 32.94% [178200/541000] [329.4/1000.0] [batch_t 0.606 (0.606)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 03:22:26,468 - Train: 32.96% [178300/541000] [329.6/1000.0] [batch_t 0.606 (0.606)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 03:23:27,224 - Train: 32.98% [178400/541000] [329.8/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 03:24:27,873 - Train: 32.99% [178500/541000] [329.9/1000.0] [batch_t 0.605 (0.606)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 03:24:46,089 - ==> Total time: 2 days, 2:24:40 Eta: 4 days, 6:21:00 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 03:25:30,564 - Train: 33.01% [178600/541000] [330.1/1000.0] [batch_t 0.608 (0.611)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 03:26:31,285 - Train: 33.03% [178700/541000] [330.3/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 03:27:31,997 - Train: 33.05% [178800/541000] [330.5/1000.0] [batch_t 0.612 (0.607)] [data_t 0.002] [optim_t 0.610] [lr 0.005000] 2024-04-04 03:28:32,693 - Train: 33.07% [178900/541000] [330.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 03:29:33,493 - Train: 33.09% [179000/541000] [330.9/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 03:30:16,562 - ==> Total time: 2 days, 2:30:11 Eta: 4 days, 6:04:26 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 03:30:36,349 - Train: 33.11% [179100/541000] [331.1/1000.0] [batch_t 0.606 (0.624)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 03:31:37,020 - Train: 33.12% [179200/541000] [331.2/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 03:32:37,708 - Train: 33.14% [179300/541000] [331.4/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 03:33:38,374 - Train: 33.16% [179400/541000] [331.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 03:34:39,083 - Train: 33.18% [179500/541000] [331.8/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 03:35:39,888 - Train: 33.20% [179600/541000] [332.0/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 03:35:47,170 - ==> Total time: 2 days, 2:35:41 Eta: 4 days, 5:47:57 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 03:36:42,837 - Train: 33.22% [179700/541000] [332.2/1000.0] [batch_t 0.607 (0.613)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 03:37:43,498 - Train: 33.23% [179800/541000] [332.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 03:38:44,180 - Train: 33.25% [179900/541000] [332.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 03:39:44,871 - Train: 33.27% [180000/541000] [332.7/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 03:40:45,582 - Train: 33.29% [180100/541000] [332.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 03:41:17,743 - ==> Total time: 2 days, 2:41:12 Eta: 4 days, 5:31:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 03:41:48,515 - Train: 33.31% [180200/541000] [333.1/1000.0] [batch_t 0.607 (0.618)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 03:42:49,187 - Train: 33.33% [180300/541000] [333.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 03:43:49,836 - Train: 33.35% [180400/541000] [333.5/1000.0] [batch_t 0.607 (0.606)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 03:44:50,657 - Train: 33.36% [180500/541000] [333.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-04 03:45:51,498 - Train: 33.38% [180600/541000] [333.8/1000.0] [batch_t 0.611 (0.608)] [data_t 0.003] [optim_t 0.609] [lr 0.005000] 2024-04-04 03:46:48,589 - ==> Total time: 2 days, 2:46:43 Eta: 4 days, 5:15:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 03:46:54,618 - Train: 33.40% [180700/541000] [334.0/1000.0] [batch_t 0.607 (0.702)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 03:47:55,478 - Train: 33.42% [180800/541000] [334.2/1000.0] [batch_t 0.607 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 03:48:56,232 - Train: 33.44% [180900/541000] [334.4/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 03:49:56,943 - Train: 33.46% [181000/541000] [334.6/1000.0] [batch_t 0.611 (0.607)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-04 03:50:57,678 - Train: 33.48% [181100/541000] [334.8/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 03:51:58,385 - Train: 33.49% [181200/541000] [334.9/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 03:52:19,628 - ==> Total time: 2 days, 2:52:14 Eta: 4 days, 4:58:54 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 03:53:01,167 - Train: 33.51% [181300/541000] [335.1/1000.0] [batch_t 0.608 (0.612)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 03:54:01,987 - Train: 33.53% [181400/541000] [335.3/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 03:55:02,695 - Train: 33.55% [181500/541000] [335.5/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 03:56:03,427 - Train: 33.57% [181600/541000] [335.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 03:57:04,128 - Train: 33.59% [181700/541000] [335.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 03:57:50,274 - ==> Total time: 2 days, 2:57:44 Eta: 4 days, 4:42:41 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 03:58:07,066 - Train: 33.60% [181800/541000] [336.0/1000.0] [batch_t 0.608 (0.621)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 03:59:07,765 - Train: 33.62% [181900/541000] [336.2/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 04:00:08,631 - Train: 33.64% [182000/541000] [336.4/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 04:01:09,332 - Train: 33.66% [182100/541000] [336.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 04:02:10,072 - Train: 33.68% [182200/541000] [336.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 04:03:10,787 - Train: 33.70% [182300/541000] [337.0/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 04:03:21,106 - ==> Total time: 2 days, 3:03:15 Eta: 4 days, 4:26:31 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 04:04:13,665 - Train: 33.72% [182400/541000] [337.2/1000.0] [batch_t 0.605 (0.613)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 04:05:14,418 - Train: 33.73% [182500/541000] [337.3/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 04:06:15,302 - Train: 33.75% [182600/541000] [337.5/1000.0] [batch_t 0.605 (0.609)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 04:07:16,017 - Train: 33.77% [182700/541000] [337.7/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 04:08:16,740 - Train: 33.79% [182800/541000] [337.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 04:08:51,957 - ==> Total time: 2 days, 3:08:46 Eta: 4 days, 4:10:26 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 04:09:19,584 - Train: 33.81% [182900/541000] [338.1/1000.0] [batch_t 0.608 (0.617)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 04:10:20,292 - Train: 33.83% [183000/541000] [338.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 04:11:21,004 - Train: 33.84% [183100/541000] [338.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 04:12:21,888 - Train: 33.86% [183200/541000] [338.6/1000.0] [batch_t 0.606 (0.609)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 04:13:22,594 - Train: 33.88% [183300/541000] [338.8/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 04:14:22,684 - ==> Total time: 2 days, 3:14:17 Eta: 4 days, 3:54:24 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 04:14:25,545 - Train: 33.90% [183400/541000] [339.0/1000.0] [batch_t 1.148 (1.148)] [data_t 0.537] [optim_t 0.611] [lr 0.005000] 2024-04-04 04:15:26,280 - Train: 33.92% [183500/541000] [339.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-04 04:16:26,990 - Train: 33.94% [183600/541000] [339.4/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 04:17:27,899 - Train: 33.96% [183700/541000] [339.6/1000.0] [batch_t 0.607 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 04:18:28,641 - Train: 33.97% [183800/541000] [339.7/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 04:19:29,370 - Train: 33.99% [183900/541000] [339.9/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 04:19:53,649 - ==> Total time: 2 days, 3:19:48 Eta: 4 days, 3:38:26 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 04:20:32,100 - Train: 34.01% [184000/541000] [340.1/1000.0] [batch_t 0.608 (0.614)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 04:21:32,806 - Train: 34.03% [184100/541000] [340.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 04:22:33,513 - Train: 34.05% [184200/541000] [340.5/1000.0] [batch_t 0.609 (0.607)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-04 04:23:34,335 - Train: 34.07% [184300/541000] [340.7/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 04:24:35,045 - Train: 34.09% [184400/541000] [340.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 04:25:24,226 - ==> Total time: 2 days, 3:25:18 Eta: 4 days, 3:22:31 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 04:25:37,918 - Train: 34.10% [184500/541000] [341.0/1000.0] [batch_t 0.610 (0.629)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-04 04:26:38,631 - Train: 34.12% [184600/541000] [341.2/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 04:27:39,372 - Train: 34.14% [184700/541000] [341.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 04:28:40,123 - Train: 34.16% [184800/541000] [341.6/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 04:29:40,871 - Train: 34.18% [184900/541000] [341.8/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 04:30:41,703 - Train: 34.20% [185000/541000] [342.0/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 04:30:55,064 - ==> Total time: 2 days, 3:30:49 Eta: 4 days, 3:06:40 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 04:31:44,576 - Train: 34.21% [185100/541000] [342.1/1000.0] [batch_t 0.608 (0.612)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 04:32:45,313 - Train: 34.23% [185200/541000] [342.3/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 04:33:46,022 - Train: 34.25% [185300/541000] [342.5/1000.0] [batch_t 0.608 (0.607)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-04 04:34:46,731 - Train: 34.27% [185400/541000] [342.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 04:35:47,479 - Train: 34.29% [185500/541000] [342.9/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 04:36:25,860 - ==> Total time: 2 days, 3:36:20 Eta: 4 days, 2:50:53 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 04:36:50,333 - Train: 34.31% [185600/541000] [343.1/1000.0] [batch_t 0.607 (0.616)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 04:37:51,067 - Train: 34.33% [185700/541000] [343.3/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 04:38:51,823 - Train: 34.34% [185800/541000] [343.4/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 04:39:52,563 - Train: 34.36% [185900/541000] [343.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 04:40:53,247 - Train: 34.38% [186000/541000] [343.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 04:41:53,999 - Train: 34.40% [186100/541000] [344.0/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-04 04:41:56,435 - ==> Total time: 2 days, 3:41:50 Eta: 4 days, 2:35:09 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 04:42:56,975 - Train: 34.42% [186200/541000] [344.2/1000.0] [batch_t 0.608 (0.612)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 04:43:57,674 - Train: 34.44% [186300/541000] [344.4/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 04:44:58,388 - Train: 34.45% [186400/541000] [344.5/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 04:45:59,150 - Train: 34.47% [186500/541000] [344.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 04:46:59,889 - Train: 34.49% [186600/541000] [344.9/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 04:47:27,216 - ==> Total time: 2 days, 3:47:21 Eta: 4 days, 2:19:28 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 04:48:02,583 - Train: 34.51% [186700/541000] [345.1/1000.0] [batch_t 0.607 (0.614)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 04:49:03,525 - Train: 34.53% [186800/541000] [345.3/1000.0] [batch_t 0.609 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 04:50:04,231 - Train: 34.55% [186900/541000] [345.5/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 04:51:04,945 - Train: 34.57% [187000/541000] [345.7/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 04:52:05,641 - Train: 34.58% [187100/541000] [345.8/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 04:52:57,845 - ==> Total time: 2 days, 3:52:52 Eta: 4 days, 2:03:52 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 04:53:09,419 - Train: 34.60% [187200/541000] [346.0/1000.0] [batch_t 0.607 (0.632)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 04:54:10,138 - Train: 34.62% [187300/541000] [346.2/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-04 04:55:11,052 - Train: 34.64% [187400/541000] [346.4/1000.0] [batch_t 0.607 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 04:56:11,819 - Train: 34.66% [187500/541000] [346.6/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 04:57:12,561 - Train: 34.68% [187600/541000] [346.8/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 04:58:13,292 - Train: 34.70% [187700/541000] [347.0/1000.0] [batch_t 0.611 (0.607)] [data_t 0.003] [optim_t 0.609] [lr 0.005000] 2024-04-04 04:58:29,703 - ==> Total time: 2 days, 3:58:24 Eta: 4 days, 1:48:20 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 04:59:16,100 - Train: 34.71% [187800/541000] [347.1/1000.0] [batch_t 0.609 (0.612)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 05:00:16,839 - Train: 34.73% [187900/541000] [347.3/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 05:01:17,692 - Train: 34.75% [188000/541000] [347.5/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 05:02:18,464 - Train: 34.77% [188100/541000] [347.7/1000.0] [batch_t 0.612 (0.608)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-04 05:03:19,187 - Train: 34.79% [188200/541000] [347.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 05:04:00,473 - ==> Total time: 2 days, 4:03:54 Eta: 4 days, 1:32:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 05:04:23,405 - Train: 34.81% [188300/541000] [348.1/1000.0] [batch_t 0.606 (0.665)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 05:05:24,156 - Train: 34.82% [188400/541000] [348.2/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 05:06:24,877 - Train: 34.84% [188500/541000] [348.4/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 05:07:25,714 - Train: 34.86% [188600/541000] [348.6/1000.0] [batch_t 0.611 (0.608)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-04 05:08:26,449 - Train: 34.88% [188700/541000] [348.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 05:09:27,131 - Train: 34.90% [188800/541000] [349.0/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 05:09:32,589 - ==> Total time: 2 days, 4:09:27 Eta: 4 days, 1:17:27 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 05:10:29,821 - Train: 34.92% [188900/541000] [349.2/1000.0] [batch_t 0.607 (0.610)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 05:11:30,491 - Train: 34.94% [189000/541000] [349.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 05:12:31,327 - Train: 34.95% [189100/541000] [349.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 05:13:31,995 - Train: 34.97% [189200/541000] [349.7/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 05:14:32,661 - Train: 34.99% [189300/541000] [349.9/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 05:15:16,329 - Test: 36.76% [50/136] [batch_t 0.213 (0.250)] 2024-04-04 05:15:26,990 - Test: 73.53% [100/136] [batch_t 0.219 (0.231)] 2024-04-04 05:15:34,485 - Test: 100.00% [136/136] [batch_t 0.052 (0.225)] 2024-04-04 05:16:22,756 - ==> Metric Time for candle : 0.002 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.533 (mAUROC_px) 8.048 (mAP_px) 0.941 (mF1_max_px) 25.002 (mAUPRO_px) 0.323 (mF1_px_0.2_0.8_0.1) 0.321 (mAcc_px_0.2_0.8_0.1) 0.319 (mIoU_px_0.2_0.8_0.1) 0.934 (mIoU_max_px) 2024-04-04 05:17:00,425 - ==> Metric Time for capsules : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 8.139 (mAUROC_px) 6.221 (mAP_px) 0.809 (mF1_max_px) 20.783 (mAUPRO_px) 0.260 (mF1_px_0.2_0.8_0.1) 0.246 (mAcc_px_0.2_0.8_0.1) 0.257 (mIoU_px_0.2_0.8_0.1) 0.730 (mIoU_max_px) 2024-04-04 05:17:36,224 - ==> Metric Time for cashew : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.658 (mAUROC_px) 5.896 (mAP_px) 0.676 (mF1_max_px) 19.967 (mAUPRO_px) 0.231 (mF1_px_0.2_0.8_0.1) 0.231 (mAcc_px_0.2_0.8_0.1) 0.231 (mIoU_px_0.2_0.8_0.1) 0.702 (mIoU_max_px) 2024-04-04 05:18:12,067 - ==> Metric Time for chewinggum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.599 (mAUROC_px) 5.841 (mAP_px) 0.690 (mF1_max_px) 20.100 (mAUPRO_px) 0.237 (mF1_px_0.2_0.8_0.1) 0.236 (mAcc_px_0.2_0.8_0.1) 0.236 (mIoU_px_0.2_0.8_0.1) 0.691 (mIoU_max_px) 2024-04-04 05:18:48,835 - ==> Metric Time for fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.651 (mAUROC_px) 5.878 (mAP_px) 0.715 (mF1_max_px) 20.898 (mAUPRO_px) 0.239 (mF1_px_0.2_0.8_0.1) 0.242 (mAcc_px_0.2_0.8_0.1) 0.240 (mIoU_px_0.2_0.8_0.1) 0.697 (mIoU_max_px) 2024-04-04 05:19:35,294 - ==> Metric Time for macaroni1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.611 (mAUROC_px) 7.964 (mAP_px) 0.940 (mF1_max_px) 24.776 (mAUPRO_px) 0.319 (mF1_px_0.2_0.8_0.1) 0.321 (mAcc_px_0.2_0.8_0.1) 0.321 (mIoU_px_0.2_0.8_0.1) 0.934 (mIoU_max_px) 2024-04-04 05:20:22,795 - ==> Metric Time for macaroni2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.828 (mAUROC_px) 7.930 (mAP_px) 0.940 (mF1_max_px) 25.638 (mAUPRO_px) 0.321 (mF1_px_0.2_0.8_0.1) 0.317 (mAcc_px_0.2_0.8_0.1) 0.322 (mIoU_px_0.2_0.8_0.1) 0.931 (mIoU_max_px) 2024-04-04 05:21:10,479 - ==> Metric Time for pcb1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.672 (mAUROC_px) 7.986 (mAP_px) 0.929 (mF1_max_px) 25.939 (mAUPRO_px) 0.314 (mF1_px_0.2_0.8_0.1) 0.319 (mAcc_px_0.2_0.8_0.1) 0.321 (mIoU_px_0.2_0.8_0.1) 0.929 (mIoU_max_px) 2024-04-04 05:21:58,509 - ==> Metric Time for pcb2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.624 (mAUROC_px) 8.118 (mAP_px) 0.940 (mF1_max_px) 26.190 (mAUPRO_px) 0.319 (mF1_px_0.2_0.8_0.1) 0.320 (mAcc_px_0.2_0.8_0.1) 0.319 (mIoU_px_0.2_0.8_0.1) 0.926 (mIoU_max_px) 2024-04-04 05:22:45,422 - ==> Metric Time for pcb3 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.551 (mAUROC_px) 7.860 (mAP_px) 0.981 (mF1_max_px) 25.340 (mAUPRO_px) 0.318 (mF1_px_0.2_0.8_0.1) 0.322 (mAcc_px_0.2_0.8_0.1) 0.322 (mIoU_px_0.2_0.8_0.1) 0.941 (mIoU_max_px) 2024-04-04 05:23:34,555 - ==> Metric Time for pcb4 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.840 (mAUROC_px) 8.194 (mAP_px) 0.948 (mF1_max_px) 27.003 (mAUPRO_px) 0.317 (mF1_px_0.2_0.8_0.1) 0.315 (mAcc_px_0.2_0.8_0.1) 0.319 (mIoU_px_0.2_0.8_0.1) 0.919 (mIoU_max_px) 2024-04-04 05:24:10,023 - ==> Metric Time for pipe_fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.567 (mAUROC_px) 5.729 (mAP_px) 0.686 (mF1_max_px) 19.892 (mAUPRO_px) 0.236 (mF1_px_0.2_0.8_0.1) 0.240 (mAcc_px_0.2_0.8_0.1) 0.230 (mIoU_px_0.2_0.8_0.1) 0.677 (mIoU_max_px) 2024-04-04 05:24:10,043 - | Name | mAUROC_sp_max | mAUROC_sp_max (Max) | mAP_sp_max | mAP_sp_max (Max) | mF1_max_sp_max | mF1_max_sp_max (Max) | mAUROC_px | mAUROC_px (Max) | mAP_px | mAP_px (Max) | mF1_max_px | mF1_max_px (Max) | mAUPRO_px | mAUPRO_px (Max) | mF1_px_0.2_0.8_0.1 | mF1_px_0.2_0.8_0.1 (Max) | mAcc_px_0.2_0.8_0.1 | mAcc_px_0.2_0.8_0.1 (Max) | mIoU_px_0.2_0.8_0.1 | mIoU_px_0.2_0.8_0.1 (Max) | mIoU_max_px | mIoU_max_px (Max) | |:----------:|:---------------:|:---------------------:|:------------:|:------------------:|:----------------:|:----------------------:|:-----------:|:------------------:|:--------:|:------------------:|:------------:|:------------------:|:-----------:|:------------------:|:--------------------:|:--------------------------:|:---------------------:|:---------------------------:|:---------------------:|:---------------------------:|:-------------:|:-------------------:| | candle | 96.700 | 97.100 (50 epoch) | 97.007 | 97.346 (50 epoch) | 90.526 | 91.542 (50 epoch) | 99.069 | 99.098 (250 epoch) | 22.553 | 23.463 (150 epoch) | 31.275 | 32.654 (200 epoch) | 95.751 | 96.087 (300 epoch) | 20.420 | 21.022 (300 epoch) | 37.552 | 39.458 (300 epoch) | 11.679 | 12.040 (300 epoch) | 18.536 | 19.513 (200 epoch) | | capsules | 87.783 | 91.750 (100 epoch) | 92.570 | 94.964 (100 epoch) | 86.916 | 88.780 (100 epoch) | 98.857 | 99.123 (100 epoch) | 53.327 | 61.256 (100 epoch) | 56.747 | 59.868 (250 epoch) | 88.647 | 91.811 (100 epoch) | 27.813 | 33.186 (100 epoch) | 60.177 | 62.576 (50 epoch) | 17.568 | 21.701 (100 epoch) | 39.613 | 42.722 (250 epoch) | | cashew | 94.720 | 94.800 (150 epoch) | 97.394 | 97.492 (150 epoch) | 91.866 | 92.386 (200 epoch) | 95.198 | 95.614 (250 epoch) | 49.367 | 50.952 (250 epoch) | 53.352 | 53.875 (250 epoch) | 89.185 | 89.185 (350 epoch) | 28.352 | 28.352 (350 epoch) | 33.442 | 34.522 (150 epoch) | 17.797 | 17.797 (350 epoch) | 36.381 | 36.869 (250 epoch) | | chewinggum | 97.547 | 97.802 (300 epoch) | 98.835 | 98.939 (300 epoch) | 94.118 | 94.416 (300 epoch) | 98.091 | 98.145 (100 epoch) | 53.100 | 59.823 (50 epoch) | 58.082 | 61.088 (50 epoch) | 79.420 | 79.773 (50 epoch) | 30.929 | 35.748 (50 epoch) | 57.148 | 65.611 (50 epoch) | 20.092 | 23.754 (50 epoch) | 40.926 | 43.976 (50 epoch) | | fryum | 95.840 | 95.940 (300 epoch) | 98.004 | 98.018 (300 epoch) | 92.611 | 92.611 (350 epoch) | 97.174 | 97.174 (350 epoch) | 49.933 | 49.934 (250 epoch) | 52.710 | 52.992 (250 epoch) | 92.073 | 92.073 (350 epoch) | 25.638 | 26.312 (50 epoch) | 27.018 | 27.708 (50 epoch) | 15.978 | 16.200 (50 epoch) | 35.787 | 36.047 (250 epoch) | | macaroni1 | 91.639 | 92.084 (300 epoch) | 90.228 | 90.374 (300 epoch) | 82.353 | 82.653 (300 epoch) | 99.557 | 99.557 (350 epoch) | 18.665 | 18.994 (300 epoch) | 28.683 | 29.377 (200 epoch) | 95.872 | 95.899 (250 epoch) | 15.581 | 15.803 (300 epoch) | 38.607 | 40.765 (50 epoch) | 8.694 | 8.828 (300 epoch) | 16.743 | 17.217 (200 epoch) | | macaroni2 | 77.135 | 82.589 (50 epoch) | 67.212 | 81.196 (50 epoch) | 75.439 | 75.622 (50 epoch) | 97.043 | 99.452 (100 epoch) | 1.176 | 11.272 (50 epoch) | 4.242 | 17.568 (50 epoch) | 86.740 | 96.253 (150 epoch) | 1.731 | 10.086 (50 epoch) | 49.644 | 49.644 (350 epoch) | 0.878 | 5.433 (50 epoch) | 2.167 | 9.630 (50 epoch) | | pcb1 | 96.550 | 96.710 (300 epoch) | 95.494 | 95.494 (350 epoch) | 93.596 | 94.737 (300 epoch) | 99.794 | 99.794 (350 epoch) | 80.594 | 82.024 (250 epoch) | 73.986 | 75.268 (250 epoch) | 93.963 | 94.357 (300 epoch) | 46.286 | 47.196 (250 epoch) | 57.819 | 62.362 (50 epoch) | 32.469 | 33.507 (250 epoch) | 58.712 | 60.344 (250 epoch) | | pcb2 | 95.040 | 95.080 (300 epoch) | 94.882 | 94.882 (350 epoch) | 89.899 | 90.000 (150 epoch) | 98.968 | 98.978 (250 epoch) | 12.693 | 14.130 (50 epoch) | 23.823 | 23.977 (250 epoch) | 89.302 | 90.754 (250 epoch) | 15.967 | 16.150 (200 epoch) | 46.536 | 47.820 (150 epoch) | 8.788 | 8.892 (200 epoch) | 13.522 | 13.621 (250 epoch) | | pcb3 | 95.891 | 95.891 (350 epoch) | 96.056 | 96.056 (350 epoch) | 89.000 | 89.000 (350 epoch) | 99.142 | 99.142 (350 epoch) | 18.988 | 20.249 (150 epoch) | 27.928 | 27.928 (350 epoch) | 91.817 | 91.817 (350 epoch) | 19.816 | 20.042 (150 epoch) | 45.222 | 48.341 (50 epoch) | 11.171 | 11.295 (150 epoch) | 16.230 | 16.230 (350 epoch) | | pcb4 | 99.743 | 99.901 (100 epoch) | 99.741 | 99.901 (100 epoch) | 97.561 | 99.000 (50 epoch) | 98.685 | 98.727 (250 epoch) | 46.767 | 47.966 (250 epoch) | 46.566 | 47.709 (250 epoch) | 88.740 | 90.729 (250 epoch) | 25.111 | 26.925 (200 epoch) | 33.244 | 39.716 (50 epoch) | 15.413 | 16.537 (200 epoch) | 30.349 | 31.328 (250 epoch) | | pipe_fryum | 98.900 | 98.900 (350 epoch) | 99.421 | 99.421 (350 epoch) | 97.561 | 97.561 (200 epoch) | 99.091 | 99.096 (250 epoch) | 54.312 | 54.482 (300 epoch) | 57.943 | 58.465 (100 epoch) | 94.124 | 95.525 (150 epoch) | 33.504 | 34.684 (250 epoch) | 39.481 | 43.014 (100 epoch) | 21.651 | 22.399 (250 epoch) | 40.789 | 41.308 (100 epoch) | | Avg | 93.957 | 94.251 (100 epoch) | 93.904 | 94.481 (100 epoch) | 90.120 | 90.163 (300 epoch) | 98.389 | 98.606 (250 epoch) | 38.456 | 39.701 (250 epoch) | 42.945 | 44.125 (250 epoch) | 90.470 | 91.494 (250 epoch) | 24.262 | 25.164 (100 epoch) | 43.824 | 46.179 (50 epoch) | 15.181 | 15.755 (100 epoch) | 29.146 | 29.990 (250 epoch) | 2024-04-04 05:24:10,455 - ==> Total time: 2 days, 4:24:04 Eta: 4 days, 1:19:00 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 05:25:23,464 - Train: 35.01% [189400/541000] [350.1/1000.0] [batch_t 1.410 (1.421)] [data_t 0.002] [optim_t 1.408] [lr 0.005000] 2024-04-04 05:27:44,936 - Train: 35.03% [189500/541000] [350.3/1000.0] [batch_t 1.419 (1.415)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-04 05:30:06,431 - Train: 35.05% [189600/541000] [350.5/1000.0] [batch_t 1.410 (1.415)] [data_t 0.002] [optim_t 1.408] [lr 0.005000] 2024-04-04 05:32:27,951 - Train: 35.06% [189700/541000] [350.6/1000.0] [batch_t 1.420 (1.415)] [data_t 0.002] [optim_t 1.418] [lr 0.005000] 2024-04-04 05:34:49,509 - Train: 35.08% [189800/541000] [350.8/1000.0] [batch_t 1.419 (1.415)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-04 05:36:58,272 - ==> Total time: 2 days, 4:36:52 Eta: 4 days, 1:17:04 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 05:37:13,971 - Train: 35.10% [189900/541000] [351.0/1000.0] [batch_t 1.417 (1.491)] [data_t 0.002] [optim_t 1.415] [lr 0.005000] 2024-04-04 05:39:35,485 - Train: 35.12% [190000/541000] [351.2/1000.0] [batch_t 1.414 (1.415)] [data_t 0.002] [optim_t 1.412] [lr 0.005000] 2024-04-04 05:41:57,006 - Train: 35.14% [190100/541000] [351.4/1000.0] [batch_t 1.419 (1.415)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-04 05:44:18,511 - Train: 35.16% [190200/541000] [351.6/1000.0] [batch_t 1.410 (1.415)] [data_t 0.002] [optim_t 1.408] [lr 0.005000] 2024-04-04 05:46:39,993 - Train: 35.18% [190300/541000] [351.8/1000.0] [batch_t 1.417 (1.415)] [data_t 0.002] [optim_t 1.415] [lr 0.005000] 2024-04-04 05:49:01,462 - Train: 35.19% [190400/541000] [351.9/1000.0] [batch_t 1.412 (1.415)] [data_t 0.002] [optim_t 1.410] [lr 0.005000] 2024-04-04 05:49:46,760 - ==> Total time: 2 days, 4:49:41 Eta: 4 days, 1:15:06 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 05:51:25,101 - Train: 35.21% [190500/541000] [352.1/1000.0] [batch_t 1.418 (1.420)] [data_t 0.002] [optim_t 1.415] [lr 0.005000] 2024-04-04 05:53:46,583 - Train: 35.23% [190600/541000] [352.3/1000.0] [batch_t 1.418 (1.415)] [data_t 0.002] [optim_t 1.415] [lr 0.005000] 2024-04-04 05:56:08,126 - Train: 35.25% [190700/541000] [352.5/1000.0] [batch_t 1.410 (1.415)] [data_t 0.002] [optim_t 1.408] [lr 0.005000] 2024-04-04 05:58:29,600 - Train: 35.27% [190800/541000] [352.7/1000.0] [batch_t 1.417 (1.415)] [data_t 0.002] [optim_t 1.415] [lr 0.005000] 2024-04-04 06:00:51,197 - Train: 35.29% [190900/541000] [352.9/1000.0] [batch_t 1.413 (1.416)] [data_t 0.002] [optim_t 1.411] [lr 0.005000] 2024-04-04 06:02:34,535 - ==> Total time: 2 days, 5:02:28 Eta: 4 days, 1:13:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 06:03:15,096 - Train: 35.30% [191000/541000] [353.0/1000.0] [batch_t 1.424 (1.432)] [data_t 0.002] [optim_t 1.421] [lr 0.005000] 2024-04-04 06:05:36,547 - Train: 35.32% [191100/541000] [353.2/1000.0] [batch_t 1.414 (1.414)] [data_t 0.002] [optim_t 1.412] [lr 0.005000] 2024-04-04 06:07:58,094 - Train: 35.34% [191200/541000] [353.4/1000.0] [batch_t 1.419 (1.415)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-04 06:10:19,687 - Train: 35.36% [191300/541000] [353.6/1000.0] [batch_t 1.417 (1.416)] [data_t 0.002] [optim_t 1.415] [lr 0.005000] 2024-04-04 06:12:44,747 - Train: 35.38% [191400/541000] [353.8/1000.0] [batch_t 1.462 (1.451)] [data_t 0.003] [optim_t 1.459] [lr 0.005000] 2024-04-04 06:15:08,280 - Train: 35.40% [191500/541000] [354.0/1000.0] [batch_t 1.445 (1.435)] [data_t 0.002] [optim_t 1.442] [lr 0.005000] 2024-04-04 06:15:28,277 - ==> Total time: 2 days, 5:15:22 Eta: 4 days, 1:11:06 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 06:17:33,636 - Train: 35.42% [191600/541000] [354.2/1000.0] [batch_t 1.446 (1.438)] [data_t 0.002] [optim_t 1.444] [lr 0.005000] 2024-04-04 06:19:56,897 - Train: 35.43% [191700/541000] [354.3/1000.0] [batch_t 1.445 (1.433)] [data_t 0.002] [optim_t 1.443] [lr 0.005000] 2024-04-04 06:22:20,226 - Train: 35.45% [191800/541000] [354.5/1000.0] [batch_t 1.432 (1.433)] [data_t 0.002] [optim_t 1.430] [lr 0.005000] 2024-04-04 06:24:43,646 - Train: 35.47% [191900/541000] [354.7/1000.0] [batch_t 1.449 (1.434)] [data_t 0.003] [optim_t 1.447] [lr 0.005000] 2024-04-04 06:27:06,959 - Train: 35.49% [192000/541000] [354.9/1000.0] [batch_t 1.432 (1.433)] [data_t 0.003] [optim_t 1.429] [lr 0.005000] 2024-04-04 06:28:25,676 - ==> Total time: 2 days, 5:28:20 Eta: 4 days, 1:09:13 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 06:29:32,328 - Train: 35.51% [192100/541000] [355.1/1000.0] [batch_t 1.427 (1.445)] [data_t 0.003] [optim_t 1.424] [lr 0.005000] 2024-04-04 06:31:55,706 - Train: 35.53% [192200/541000] [355.3/1000.0] [batch_t 1.428 (1.434)] [data_t 0.003] [optim_t 1.425] [lr 0.005000] 2024-04-04 06:34:19,150 - Train: 35.55% [192300/541000] [355.5/1000.0] [batch_t 1.435 (1.434)] [data_t 0.002] [optim_t 1.433] [lr 0.005000] 2024-04-04 06:36:42,678 - Train: 35.56% [192400/541000] [355.6/1000.0] [batch_t 1.439 (1.435)] [data_t 0.003] [optim_t 1.436] [lr 0.005000] 2024-04-04 06:39:06,062 - Train: 35.58% [192500/541000] [355.8/1000.0] [batch_t 1.438 (1.434)] [data_t 0.003] [optim_t 1.435] [lr 0.005000] 2024-04-04 06:41:23,615 - ==> Total time: 2 days, 5:41:18 Eta: 4 days, 1:07:17 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 06:41:31,399 - Train: 35.60% [192600/541000] [356.0/1000.0] [batch_t 1.427 (1.505)] [data_t 0.003] [optim_t 1.424] [lr 0.005000] 2024-04-04 06:43:54,749 - Train: 35.62% [192700/541000] [356.2/1000.0] [batch_t 1.434 (1.433)] [data_t 0.002] [optim_t 1.431] [lr 0.005000] 2024-04-04 06:46:18,197 - Train: 35.64% [192800/541000] [356.4/1000.0] [batch_t 1.446 (1.434)] [data_t 0.002] [optim_t 1.443] [lr 0.005000] 2024-04-04 06:48:41,581 - Train: 35.66% [192900/541000] [356.6/1000.0] [batch_t 1.428 (1.434)] [data_t 0.003] [optim_t 1.426] [lr 0.005000] 2024-04-04 06:51:04,862 - Train: 35.67% [193000/541000] [356.7/1000.0] [batch_t 1.427 (1.433)] [data_t 0.003] [optim_t 1.424] [lr 0.005000] 2024-04-04 06:53:28,209 - Train: 35.69% [193100/541000] [356.9/1000.0] [batch_t 1.437 (1.433)] [data_t 0.003] [optim_t 1.434] [lr 0.005000] 2024-04-04 06:54:21,298 - ==> Total time: 2 days, 5:54:15 Eta: 4 days, 1:05:17 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 06:55:53,598 - Train: 35.71% [193200/541000] [357.1/1000.0] [batch_t 1.434 (1.438)] [data_t 0.003] [optim_t 1.431] [lr 0.005000] 2024-04-04 06:58:16,780 - Train: 35.73% [193300/541000] [357.3/1000.0] [batch_t 1.444 (1.432)] [data_t 0.002] [optim_t 1.442] [lr 0.005000] 2024-04-04 07:00:40,114 - Train: 35.75% [193400/541000] [357.5/1000.0] [batch_t 1.430 (1.433)] [data_t 0.003] [optim_t 1.428] [lr 0.005000] 2024-04-04 07:03:03,393 - Train: 35.77% [193500/541000] [357.7/1000.0] [batch_t 1.450 (1.433)] [data_t 0.003] [optim_t 1.447] [lr 0.005000] 2024-04-04 07:05:26,660 - Train: 35.79% [193600/541000] [357.9/1000.0] [batch_t 1.434 (1.433)] [data_t 0.003] [optim_t 1.431] [lr 0.005000] 2024-04-04 07:07:18,482 - ==> Total time: 2 days, 6:07:12 Eta: 4 days, 1:03:13 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 07:07:52,045 - Train: 35.80% [193700/541000] [358.0/1000.0] [batch_t 1.432 (1.453)] [data_t 0.002] [optim_t 1.430] [lr 0.005000] 2024-04-04 07:10:15,548 - Train: 35.82% [193800/541000] [358.2/1000.0] [batch_t 1.432 (1.435)] [data_t 0.003] [optim_t 1.429] [lr 0.005000] 2024-04-04 07:12:38,878 - Train: 35.84% [193900/541000] [358.4/1000.0] [batch_t 1.427 (1.433)] [data_t 0.002] [optim_t 1.425] [lr 0.005000] 2024-04-04 07:15:02,256 - Train: 35.86% [194000/541000] [358.6/1000.0] [batch_t 1.427 (1.434)] [data_t 0.002] [optim_t 1.425] [lr 0.005000] 2024-04-04 07:17:25,568 - Train: 35.88% [194100/541000] [358.8/1000.0] [batch_t 1.427 (1.433)] [data_t 0.003] [optim_t 1.424] [lr 0.005000] 2024-04-04 07:19:48,955 - Train: 35.90% [194200/541000] [359.0/1000.0] [batch_t 1.428 (1.434)] [data_t 0.003] [optim_t 1.425] [lr 0.005000] 2024-04-04 07:20:16,197 - ==> Total time: 2 days, 6:20:10 Eta: 4 days, 1:01:05 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 07:22:14,363 - Train: 35.91% [194300/541000] [359.1/1000.0] [batch_t 1.445 (1.438)] [data_t 0.003] [optim_t 1.443] [lr 0.005000] 2024-04-04 07:24:37,824 - Train: 35.93% [194400/541000] [359.3/1000.0] [batch_t 1.441 (1.435)] [data_t 0.003] [optim_t 1.438] [lr 0.005000] 2024-04-04 07:27:01,292 - Train: 35.95% [194500/541000] [359.5/1000.0] [batch_t 1.440 (1.435)] [data_t 0.002] [optim_t 1.437] [lr 0.005000] 2024-04-04 07:29:24,571 - Train: 35.97% [194600/541000] [359.7/1000.0] [batch_t 1.445 (1.433)] [data_t 0.002] [optim_t 1.443] [lr 0.005000] 2024-04-04 07:31:47,852 - Train: 35.99% [194700/541000] [359.9/1000.0] [batch_t 1.427 (1.433)] [data_t 0.003] [optim_t 1.425] [lr 0.005000] 2024-04-04 07:33:13,969 - ==> Total time: 2 days, 6:33:08 Eta: 4 days, 0:58:54 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 07:34:13,361 - Train: 36.01% [194800/541000] [360.1/1000.0] [batch_t 1.427 (1.444)] [data_t 0.002] [optim_t 1.425] [lr 0.005000] 2024-04-04 07:36:36,738 - Train: 36.03% [194900/541000] [360.3/1000.0] [batch_t 1.444 (1.434)] [data_t 0.002] [optim_t 1.442] [lr 0.005000] 2024-04-04 07:39:00,125 - Train: 36.04% [195000/541000] [360.4/1000.0] [batch_t 1.428 (1.434)] [data_t 0.003] [optim_t 1.425] [lr 0.005000] 2024-04-04 07:41:23,530 - Train: 36.06% [195100/541000] [360.6/1000.0] [batch_t 1.435 (1.434)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-04 07:43:46,897 - Train: 36.08% [195200/541000] [360.8/1000.0] [batch_t 1.447 (1.434)] [data_t 0.002] [optim_t 1.444] [lr 0.005000] 2024-04-04 07:46:10,369 - Train: 36.10% [195300/541000] [361.0/1000.0] [batch_t 1.434 (1.435)] [data_t 0.002] [optim_t 1.432] [lr 0.005000] 2024-04-04 07:46:11,810 - ==> Total time: 2 days, 6:46:06 Eta: 4 days, 0:56:40 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 07:48:35,611 - Train: 36.12% [195400/541000] [361.2/1000.0] [batch_t 1.439 (1.436)] [data_t 0.002] [optim_t 1.437] [lr 0.005000] 2024-04-04 07:50:59,051 - Train: 36.14% [195500/541000] [361.4/1000.0] [batch_t 1.425 (1.434)] [data_t 0.003] [optim_t 1.422] [lr 0.005000] 2024-04-04 07:53:22,324 - Train: 36.16% [195600/541000] [361.6/1000.0] [batch_t 1.425 (1.433)] [data_t 0.003] [optim_t 1.422] [lr 0.005000] 2024-04-04 07:55:45,668 - Train: 36.17% [195700/541000] [361.7/1000.0] [batch_t 1.441 (1.433)] [data_t 0.003] [optim_t 1.438] [lr 0.005000] 2024-04-04 07:58:09,035 - Train: 36.19% [195800/541000] [361.9/1000.0] [batch_t 1.439 (1.434)] [data_t 0.003] [optim_t 1.436] [lr 0.005000] 2024-04-04 07:59:09,227 - ==> Total time: 2 days, 6:59:03 Eta: 4 days, 0:54:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 08:00:34,693 - Train: 36.21% [195900/541000] [362.1/1000.0] [batch_t 1.437 (1.439)] [data_t 0.003] [optim_t 1.434] [lr 0.005000] 2024-04-04 08:02:58,132 - Train: 36.23% [196000/541000] [362.3/1000.0] [batch_t 1.431 (1.434)] [data_t 0.003] [optim_t 1.429] [lr 0.005000] 2024-04-04 08:05:21,566 - Train: 36.25% [196100/541000] [362.5/1000.0] [batch_t 1.455 (1.434)] [data_t 0.002] [optim_t 1.453] [lr 0.005000] 2024-04-04 08:07:44,915 - Train: 36.27% [196200/541000] [362.7/1000.0] [batch_t 1.439 (1.433)] [data_t 0.003] [optim_t 1.436] [lr 0.005000] 2024-04-04 08:09:31,120 - Train: 36.28% [196300/541000] [362.8/1000.0] [batch_t 0.608 (1.062)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 08:10:21,513 - ==> Total time: 2 days, 7:10:15 Eta: 4 days, 0:48:55 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 08:10:33,942 - Train: 36.30% [196400/541000] [363.0/1000.0] [batch_t 0.610 (0.627)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-04 08:11:34,647 - Train: 36.32% [196500/541000] [363.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 08:12:35,340 - Train: 36.34% [196600/541000] [363.4/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 08:13:36,075 - Train: 36.36% [196700/541000] [363.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 08:14:36,948 - Train: 36.38% [196800/541000] [363.8/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 08:15:37,647 - Train: 36.40% [196900/541000] [364.0/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 08:15:52,215 - ==> Total time: 2 days, 7:15:46 Eta: 4 days, 0:33:30 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 08:16:40,431 - Train: 36.41% [197000/541000] [364.1/1000.0] [batch_t 0.608 (0.612)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 08:17:41,111 - Train: 36.43% [197100/541000] [364.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 08:18:41,791 - Train: 36.45% [197200/541000] [364.5/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 08:19:42,604 - Train: 36.47% [197300/541000] [364.7/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 08:20:43,286 - Train: 36.49% [197400/541000] [364.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 08:21:22,738 - ==> Total time: 2 days, 7:21:17 Eta: 4 days, 0:18:07 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 08:21:45,941 - Train: 36.51% [197500/541000] [365.1/1000.0] [batch_t 0.607 (0.618)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 08:22:46,636 - Train: 36.52% [197600/541000] [365.2/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 08:23:47,340 - Train: 36.54% [197700/541000] [365.4/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 08:24:48,045 - Train: 36.56% [197800/541000] [365.6/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 08:25:48,814 - Train: 36.58% [197900/541000] [365.8/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 08:26:49,511 - Train: 36.60% [198000/541000] [366.0/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 08:26:53,156 - ==> Total time: 2 days, 7:26:47 Eta: 4 days, 0:02:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 08:27:52,288 - Train: 36.62% [198100/541000] [366.2/1000.0] [batch_t 0.607 (0.611)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 08:28:52,971 - Train: 36.64% [198200/541000] [366.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 08:29:53,677 - Train: 36.65% [198300/541000] [366.5/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 08:30:54,516 - Train: 36.67% [198400/541000] [366.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 08:31:55,217 - Train: 36.69% [198500/541000] [366.9/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 08:32:23,739 - ==> Total time: 2 days, 7:32:18 Eta: 3 days, 23:47:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 08:32:58,059 - Train: 36.71% [198600/541000] [367.1/1000.0] [batch_t 0.607 (0.614)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 08:33:58,787 - Train: 36.73% [198700/541000] [367.3/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 08:34:59,506 - Train: 36.75% [198800/541000] [367.5/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 08:36:00,187 - Train: 36.77% [198900/541000] [367.7/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 08:37:00,909 - Train: 36.78% [199000/541000] [367.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 08:37:54,481 - ==> Total time: 2 days, 7:37:48 Eta: 3 days, 23:32:20 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 08:38:03,818 - Train: 36.80% [199100/541000] [368.0/1000.0] [batch_t 0.605 (0.635)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 08:39:04,510 - Train: 36.82% [199200/541000] [368.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 08:40:05,189 - Train: 36.84% [199300/541000] [368.4/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 08:41:05,921 - Train: 36.86% [199400/541000] [368.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 08:42:06,614 - Train: 36.88% [199500/541000] [368.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 08:43:07,458 - Train: 36.89% [199600/541000] [368.9/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 08:43:25,080 - ==> Total time: 2 days, 7:43:19 Eta: 3 days, 23:17:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 08:44:10,155 - Train: 36.91% [199700/541000] [369.1/1000.0] [batch_t 0.608 (0.613)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 08:45:10,862 - Train: 36.93% [199800/541000] [369.3/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 08:46:11,590 - Train: 36.95% [199900/541000] [369.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 08:47:12,336 - Train: 36.97% [200000/541000] [369.7/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 08:48:13,135 - Train: 36.99% [200100/541000] [369.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 08:48:55,622 - ==> Total time: 2 days, 7:48:50 Eta: 3 days, 23:02:04 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 08:49:15,874 - Train: 37.01% [200200/541000] [370.1/1000.0] [batch_t 0.607 (0.618)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 08:50:16,586 - Train: 37.02% [200300/541000] [370.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 08:51:17,275 - Train: 37.04% [200400/541000] [370.4/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 08:52:17,968 - Train: 37.06% [200500/541000] [370.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 08:53:18,694 - Train: 37.08% [200600/541000] [370.8/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 08:54:19,429 - Train: 37.10% [200700/541000] [371.0/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 08:54:26,112 - ==> Total time: 2 days, 7:54:20 Eta: 3 days, 22:47:00 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 08:55:22,653 - Train: 37.12% [200800/541000] [371.2/1000.0] [batch_t 0.609 (0.611)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 08:56:23,350 - Train: 37.13% [200900/541000] [371.3/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 08:57:24,105 - Train: 37.15% [201000/541000] [371.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 08:58:24,928 - Train: 37.17% [201100/541000] [371.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 08:59:25,725 - Train: 37.19% [201200/541000] [371.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 08:59:57,311 - ==> Total time: 2 days, 7:59:51 Eta: 3 days, 22:32:01 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 09:00:28,638 - Train: 37.21% [201300/541000] [372.1/1000.0] [batch_t 0.608 (0.616)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 09:01:29,559 - Train: 37.23% [201400/541000] [372.3/1000.0] [batch_t 0.605 (0.609)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 09:02:30,302 - Train: 37.25% [201500/541000] [372.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 09:03:31,045 - Train: 37.26% [201600/541000] [372.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 09:04:31,809 - Train: 37.28% [201700/541000] [372.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 09:05:28,317 - ==> Total time: 2 days, 8:05:22 Eta: 3 days, 22:17:05 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 09:05:34,860 - Train: 37.30% [201800/541000] [373.0/1000.0] [batch_t 0.610 (0.678)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-04 09:06:35,625 - Train: 37.32% [201900/541000] [373.2/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 09:07:36,479 - Train: 37.34% [202000/541000] [373.4/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 09:08:37,213 - Train: 37.36% [202100/541000] [373.6/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 09:09:37,960 - Train: 37.38% [202200/541000] [373.8/1000.0] [batch_t 0.610 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 09:10:38,749 - Train: 37.39% [202300/541000] [373.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 09:10:59,399 - ==> Total time: 2 days, 8:10:53 Eta: 3 days, 22:02:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 09:11:43,147 - Train: 37.41% [202400/541000] [374.1/1000.0] [batch_t 0.608 (0.633)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 09:12:43,906 - Train: 37.43% [202500/541000] [374.3/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 09:13:44,852 - Train: 37.45% [202600/541000] [374.5/1000.0] [batch_t 0.612 (0.609)] [data_t 0.003] [optim_t 0.609] [lr 0.005000] 2024-04-04 09:14:45,643 - Train: 37.47% [202700/541000] [374.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 09:15:46,373 - Train: 37.49% [202800/541000] [374.9/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 09:16:31,922 - ==> Total time: 2 days, 8:16:26 Eta: 3 days, 21:47:23 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 09:16:49,399 - Train: 37.50% [202900/541000] [375.0/1000.0] [batch_t 0.608 (0.625)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 09:17:50,236 - Train: 37.52% [203000/541000] [375.2/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 09:18:50,997 - Train: 37.54% [203100/541000] [375.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 09:19:51,883 - Train: 37.56% [203200/541000] [375.6/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 09:20:52,653 - Train: 37.58% [203300/541000] [375.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 09:21:53,416 - Train: 37.60% [203400/541000] [376.0/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-04 09:22:03,135 - ==> Total time: 2 days, 8:21:57 Eta: 3 days, 21:32:36 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 09:22:57,159 - Train: 37.62% [203500/541000] [376.2/1000.0] [batch_t 0.611 (0.615)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-04 09:23:57,927 - Train: 37.63% [203600/541000] [376.3/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 09:24:58,693 - Train: 37.65% [203700/541000] [376.5/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 09:25:59,567 - Train: 37.67% [203800/541000] [376.7/1000.0] [batch_t 0.607 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 09:27:00,328 - Train: 37.69% [203900/541000] [376.9/1000.0] [batch_t 0.612 (0.608)] [data_t 0.002] [optim_t 0.610] [lr 0.005000] 2024-04-04 09:27:34,990 - ==> Total time: 2 days, 8:27:29 Eta: 3 days, 21:17:53 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 09:28:03,411 - Train: 37.71% [204000/541000] [377.1/1000.0] [batch_t 0.607 (0.615)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 09:29:04,166 - Train: 37.73% [204100/541000] [377.3/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 09:30:04,911 - Train: 37.74% [204200/541000] [377.4/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 09:31:05,669 - Train: 37.76% [204300/541000] [377.6/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 09:32:06,522 - Train: 37.78% [204400/541000] [377.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 09:33:06,041 - ==> Total time: 2 days, 8:33:00 Eta: 3 days, 21:03:12 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 09:33:09,526 - Train: 37.80% [204500/541000] [378.0/1000.0] [batch_t 0.608 (0.785)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 09:34:10,263 - Train: 37.82% [204600/541000] [378.2/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 09:35:11,025 - Train: 37.84% [204700/541000] [378.4/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 09:36:11,782 - Train: 37.86% [204800/541000] [378.6/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 09:37:12,519 - Train: 37.87% [204900/541000] [378.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 09:38:13,417 - Train: 37.89% [205000/541000] [378.9/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 09:38:37,124 - ==> Total time: 2 days, 8:38:31 Eta: 3 days, 20:48:33 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 09:39:16,394 - Train: 37.91% [205100/541000] [379.1/1000.0] [batch_t 0.608 (0.614)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 09:40:17,169 - Train: 37.93% [205200/541000] [379.3/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 09:41:17,951 - Train: 37.95% [205300/541000] [379.5/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 09:42:18,691 - Train: 37.97% [205400/541000] [379.7/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 09:43:19,586 - Train: 37.99% [205500/541000] [379.9/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 09:44:08,241 - ==> Total time: 2 days, 8:44:02 Eta: 3 days, 20:33:58 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 09:44:22,567 - Train: 38.00% [205600/541000] [380.0/1000.0] [batch_t 0.607 (0.625)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 09:45:23,312 - Train: 38.02% [205700/541000] [380.2/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 09:46:24,049 - Train: 38.04% [205800/541000] [380.4/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 09:47:24,816 - Train: 38.06% [205900/541000] [380.6/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 09:48:25,608 - Train: 38.08% [206000/541000] [380.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 09:49:26,478 - Train: 38.10% [206100/541000] [381.0/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 09:49:39,241 - ==> Total time: 2 days, 8:49:33 Eta: 3 days, 20:19:25 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 09:50:29,538 - Train: 38.11% [206200/541000] [381.1/1000.0] [batch_t 0.608 (0.612)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 09:51:30,284 - Train: 38.13% [206300/541000] [381.3/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 09:52:31,000 - Train: 38.15% [206400/541000] [381.5/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 09:53:31,732 - Train: 38.17% [206500/541000] [381.7/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 09:54:32,611 - Train: 38.19% [206600/541000] [381.9/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 09:55:10,269 - ==> Total time: 2 days, 8:55:04 Eta: 3 days, 20:04:55 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 09:55:35,528 - Train: 38.21% [206700/541000] [382.1/1000.0] [batch_t 0.607 (0.616)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 09:56:36,284 - Train: 38.23% [206800/541000] [382.3/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 09:57:37,055 - Train: 38.24% [206900/541000] [382.4/1000.0] [batch_t 0.611 (0.608)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-04 09:58:37,848 - Train: 38.26% [207000/541000] [382.6/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 09:59:38,587 - Train: 38.28% [207100/541000] [382.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 10:00:39,482 - Train: 38.30% [207200/541000] [383.0/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 10:00:41,306 - ==> Total time: 2 days, 9:00:35 Eta: 3 days, 19:50:27 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 10:01:42,298 - Train: 38.32% [207300/541000] [383.2/1000.0] [batch_t 0.607 (0.612)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 10:02:43,028 - Train: 38.34% [207400/541000] [383.4/1000.0] [batch_t 0.610 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 10:03:43,754 - Train: 38.35% [207500/541000] [383.5/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 10:04:44,493 - Train: 38.37% [207600/541000] [383.7/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 10:05:45,205 - Train: 38.39% [207700/541000] [383.9/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 10:06:11,925 - ==> Total time: 2 days, 9:06:06 Eta: 3 days, 19:36:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 10:06:48,029 - Train: 38.41% [207800/541000] [384.1/1000.0] [batch_t 0.607 (0.617)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 10:07:48,760 - Train: 38.43% [207900/541000] [384.3/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 10:08:49,500 - Train: 38.45% [208000/541000] [384.5/1000.0] [batch_t 0.608 (0.607)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-04 10:09:50,266 - Train: 38.47% [208100/541000] [384.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 10:10:50,999 - Train: 38.48% [208200/541000] [384.8/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 10:11:42,634 - ==> Total time: 2 days, 9:11:37 Eta: 3 days, 19:21:40 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 10:11:53,915 - Train: 38.50% [208300/541000] [385.0/1000.0] [batch_t 0.608 (0.633)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 10:12:54,661 - Train: 38.52% [208400/541000] [385.2/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 10:13:55,503 - Train: 38.54% [208500/541000] [385.4/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 10:14:56,241 - Train: 38.56% [208600/541000] [385.6/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 10:15:56,978 - Train: 38.58% [208700/541000] [385.8/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 10:16:57,739 - Train: 38.60% [208800/541000] [386.0/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-04 10:17:13,547 - ==> Total time: 2 days, 9:17:07 Eta: 3 days, 19:07:21 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 10:18:00,555 - Train: 38.61% [208900/541000] [386.1/1000.0] [batch_t 0.607 (0.613)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 10:19:01,301 - Train: 38.63% [209000/541000] [386.3/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 10:20:02,172 - Train: 38.65% [209100/541000] [386.5/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 10:21:02,920 - Train: 38.67% [209200/541000] [386.7/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 10:22:03,674 - Train: 38.69% [209300/541000] [386.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 10:22:44,375 - ==> Total time: 2 days, 9:22:38 Eta: 3 days, 18:53:04 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 10:23:06,679 - Train: 38.71% [209400/541000] [387.1/1000.0] [batch_t 0.607 (0.621)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 10:24:07,425 - Train: 38.72% [209500/541000] [387.2/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 10:25:08,174 - Train: 38.74% [209600/541000] [387.4/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 10:26:09,054 - Train: 38.76% [209700/541000] [387.6/1000.0] [batch_t 0.609 (0.609)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 10:27:09,789 - Train: 38.78% [209800/541000] [387.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 10:28:10,542 - Train: 38.80% [209900/541000] [388.0/1000.0] [batch_t 0.608 (0.607)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-04 10:28:15,408 - ==> Total time: 2 days, 9:28:09 Eta: 3 days, 18:38:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 10:29:13,246 - Train: 38.82% [210000/541000] [388.2/1000.0] [batch_t 0.607 (0.610)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 10:30:13,970 - Train: 38.84% [210100/541000] [388.4/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 10:31:14,873 - Train: 38.85% [210200/541000] [388.5/1000.0] [batch_t 0.607 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 10:32:15,616 - Train: 38.87% [210300/541000] [388.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 10:33:16,354 - Train: 38.89% [210400/541000] [388.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 10:33:46,105 - ==> Total time: 2 days, 9:33:40 Eta: 3 days, 18:24:40 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 10:34:19,228 - Train: 38.91% [210500/541000] [389.1/1000.0] [batch_t 0.608 (0.616)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 10:35:19,944 - Train: 38.93% [210600/541000] [389.3/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 10:36:20,711 - Train: 38.95% [210700/541000] [389.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 10:37:21,563 - Train: 38.96% [210800/541000] [389.6/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 10:38:22,284 - Train: 38.98% [210900/541000] [389.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 10:39:16,937 - ==> Total time: 2 days, 9:39:11 Eta: 3 days, 18:10:31 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 10:39:24,958 - Train: 39.00% [211000/541000] [390.0/1000.0] [batch_t 0.608 (0.642)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 10:40:25,700 - Train: 39.02% [211100/541000] [390.2/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 10:41:26,429 - Train: 39.04% [211200/541000] [390.4/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 10:42:27,152 - Train: 39.06% [211300/541000] [390.6/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 10:43:28,017 - Train: 39.08% [211400/541000] [390.8/1000.0] [batch_t 0.607 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 10:44:28,788 - Train: 39.09% [211500/541000] [390.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 10:44:47,627 - ==> Total time: 2 days, 9:44:42 Eta: 3 days, 17:56:25 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 10:45:31,577 - Train: 39.11% [211600/541000] [391.1/1000.0] [batch_t 0.607 (0.614)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 10:46:32,344 - Train: 39.13% [211700/541000] [391.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 10:47:33,082 - Train: 39.15% [211800/541000] [391.5/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 10:48:34,001 - Train: 39.17% [211900/541000] [391.7/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 10:49:34,755 - Train: 39.19% [212000/541000] [391.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 10:50:18,499 - ==> Total time: 2 days, 9:50:12 Eta: 3 days, 17:42:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 10:50:37,685 - Train: 39.21% [212100/541000] [392.1/1000.0] [batch_t 0.608 (0.620)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 10:51:38,420 - Train: 39.22% [212200/541000] [392.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 10:52:39,140 - Train: 39.24% [212300/541000] [392.4/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 10:53:39,885 - Train: 39.26% [212400/541000] [392.6/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 10:54:40,760 - Train: 39.28% [212500/541000] [392.8/1000.0] [batch_t 0.607 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 10:55:41,510 - Train: 39.30% [212600/541000] [393.0/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 10:55:49,416 - ==> Total time: 2 days, 9:55:43 Eta: 3 days, 17:28:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 10:56:44,227 - Train: 39.32% [212700/541000] [393.2/1000.0] [batch_t 0.607 (0.612)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 10:57:44,957 - Train: 39.33% [212800/541000] [393.3/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 10:58:45,666 - Train: 39.35% [212900/541000] [393.5/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 10:59:46,517 - Train: 39.37% [213000/541000] [393.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-04 11:00:47,250 - Train: 39.39% [213100/541000] [393.9/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 11:01:20,048 - ==> Total time: 2 days, 10:01:14 Eta: 3 days, 17:14:23 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 11:01:50,003 - Train: 39.41% [213200/541000] [394.1/1000.0] [batch_t 0.609 (0.616)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 11:02:50,747 - Train: 39.43% [213300/541000] [394.3/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 11:03:51,477 - Train: 39.45% [213400/541000] [394.5/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 11:04:52,242 - Train: 39.46% [213500/541000] [394.6/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 11:05:53,090 - Train: 39.48% [213600/541000] [394.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 11:06:50,790 - ==> Total time: 2 days, 10:06:45 Eta: 3 days, 17:00:28 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 11:06:55,722 - Train: 39.50% [213700/541000] [395.0/1000.0] [batch_t 0.608 (0.680)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 11:07:56,468 - Train: 39.52% [213800/541000] [395.2/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 11:08:57,200 - Train: 39.54% [213900/541000] [395.4/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 11:09:57,945 - Train: 39.56% [214000/541000] [395.6/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 11:10:58,819 - Train: 39.57% [214100/541000] [395.7/1000.0] [batch_t 0.609 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 11:11:59,553 - Train: 39.59% [214200/541000] [395.9/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 11:12:21,418 - ==> Total time: 2 days, 10:12:15 Eta: 3 days, 16:46:35 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 11:13:02,262 - Train: 39.61% [214300/541000] [396.1/1000.0] [batch_t 0.607 (0.612)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 11:14:03,002 - Train: 39.63% [214400/541000] [396.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 11:15:03,742 - Train: 39.65% [214500/541000] [396.5/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 11:16:04,476 - Train: 39.67% [214600/541000] [396.7/1000.0] [batch_t 0.611 (0.607)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-04 11:17:05,227 - Train: 39.69% [214700/541000] [396.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 11:17:52,139 - ==> Total time: 2 days, 10:17:46 Eta: 3 days, 16:32:44 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 11:18:08,082 - Train: 39.70% [214800/541000] [397.0/1000.0] [batch_t 0.607 (0.626)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 11:19:08,817 - Train: 39.72% [214900/541000] [397.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-04 11:20:09,565 - Train: 39.74% [215000/541000] [397.4/1000.0] [batch_t 0.610 (0.607)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-04 11:21:10,341 - Train: 39.76% [215100/541000] [397.6/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 11:22:11,129 - Train: 39.78% [215200/541000] [397.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 11:23:11,901 - Train: 39.80% [215300/541000] [398.0/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 11:23:22,845 - ==> Total time: 2 days, 10:23:17 Eta: 3 days, 16:18:56 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 11:24:14,962 - Train: 39.82% [215400/541000] [398.2/1000.0] [batch_t 0.607 (0.614)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 11:25:15,706 - Train: 39.83% [215500/541000] [398.3/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 11:26:16,460 - Train: 39.85% [215600/541000] [398.5/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 11:27:17,225 - Train: 39.87% [215700/541000] [398.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 11:28:17,967 - Train: 39.89% [215800/541000] [398.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 11:28:53,811 - ==> Total time: 2 days, 10:28:48 Eta: 3 days, 16:05:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 11:29:20,739 - Train: 39.91% [215900/541000] [399.1/1000.0] [batch_t 0.606 (0.614)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 11:30:21,524 - Train: 39.93% [216000/541000] [399.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 11:31:22,489 - Train: 39.94% [216100/541000] [399.4/1000.0] [batch_t 0.607 (0.610)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 11:32:23,272 - Train: 39.96% [216200/541000] [399.6/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-04 11:33:24,065 - Train: 39.98% [216300/541000] [399.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 11:34:24,854 - Train: 40.00% [216400/541000] [400.0/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 11:34:37,246 - Test: 36.76% [50/136] [batch_t 0.223 (0.235)] 2024-04-04 11:34:48,017 - Test: 73.53% [100/136] [batch_t 0.212 (0.225)] 2024-04-04 11:34:55,449 - Test: 100.00% [136/136] [batch_t 0.053 (0.220)] 2024-04-04 11:35:43,690 - ==> Metric Time for candle : 0.002 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.423 (mAUROC_px) 8.131 (mAP_px) 0.975 (mF1_max_px) 25.059 (mAUPRO_px) 0.317 (mF1_px_0.2_0.8_0.1) 0.315 (mAcc_px_0.2_0.8_0.1) 0.312 (mIoU_px_0.2_0.8_0.1) 0.918 (mIoU_max_px) 2024-04-04 11:36:21,059 - ==> Metric Time for capsules : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 8.106 (mAUROC_px) 6.229 (mAP_px) 0.727 (mF1_max_px) 20.601 (mAUPRO_px) 0.260 (mF1_px_0.2_0.8_0.1) 0.252 (mAcc_px_0.2_0.8_0.1) 0.254 (mIoU_px_0.2_0.8_0.1) 0.725 (mIoU_max_px) 2024-04-04 11:36:56,582 - ==> Metric Time for cashew : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.755 (mAUROC_px) 5.982 (mAP_px) 0.702 (mF1_max_px) 19.473 (mAUPRO_px) 0.236 (mF1_px_0.2_0.8_0.1) 0.235 (mAcc_px_0.2_0.8_0.1) 0.235 (mIoU_px_0.2_0.8_0.1) 0.695 (mIoU_max_px) 2024-04-04 11:37:32,396 - ==> Metric Time for chewinggum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.679 (mAUROC_px) 5.830 (mAP_px) 0.693 (mF1_max_px) 19.994 (mAUPRO_px) 0.240 (mF1_px_0.2_0.8_0.1) 0.239 (mAcc_px_0.2_0.8_0.1) 0.240 (mIoU_px_0.2_0.8_0.1) 0.686 (mIoU_max_px) 2024-04-04 11:38:08,975 - ==> Metric Time for fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.687 (mAUROC_px) 5.701 (mAP_px) 0.713 (mF1_max_px) 20.857 (mAUPRO_px) 0.235 (mF1_px_0.2_0.8_0.1) 0.237 (mAcc_px_0.2_0.8_0.1) 0.244 (mIoU_px_0.2_0.8_0.1) 0.694 (mIoU_max_px) 2024-04-04 11:38:55,306 - ==> Metric Time for macaroni1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.620 (mAUROC_px) 7.964 (mAP_px) 0.931 (mF1_max_px) 24.660 (mAUPRO_px) 0.318 (mF1_px_0.2_0.8_0.1) 0.318 (mAcc_px_0.2_0.8_0.1) 0.320 (mIoU_px_0.2_0.8_0.1) 0.929 (mIoU_max_px) 2024-04-04 11:39:42,182 - ==> Metric Time for macaroni2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.562 (mAUROC_px) 7.922 (mAP_px) 0.937 (mF1_max_px) 25.283 (mAUPRO_px) 0.320 (mF1_px_0.2_0.8_0.1) 0.320 (mAcc_px_0.2_0.8_0.1) 0.322 (mIoU_px_0.2_0.8_0.1) 0.932 (mIoU_max_px) 2024-04-04 11:40:29,856 - ==> Metric Time for pcb1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.605 (mAUROC_px) 8.029 (mAP_px) 0.965 (mF1_max_px) 25.921 (mAUPRO_px) 0.316 (mF1_px_0.2_0.8_0.1) 0.317 (mAcc_px_0.2_0.8_0.1) 0.319 (mIoU_px_0.2_0.8_0.1) 0.924 (mIoU_max_px) 2024-04-04 11:41:17,276 - ==> Metric Time for pcb2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.508 (mAUROC_px) 8.085 (mAP_px) 0.932 (mF1_max_px) 25.698 (mAUPRO_px) 0.322 (mF1_px_0.2_0.8_0.1) 0.324 (mAcc_px_0.2_0.8_0.1) 0.322 (mIoU_px_0.2_0.8_0.1) 0.946 (mIoU_max_px) 2024-04-04 11:42:04,245 - ==> Metric Time for pcb3 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.591 (mAUROC_px) 7.912 (mAP_px) 0.933 (mF1_max_px) 25.348 (mAUPRO_px) 0.314 (mF1_px_0.2_0.8_0.1) 0.318 (mAcc_px_0.2_0.8_0.1) 0.322 (mIoU_px_0.2_0.8_0.1) 0.949 (mIoU_max_px) 2024-04-04 11:42:53,535 - ==> Metric Time for pcb4 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.780 (mAUROC_px) 8.167 (mAP_px) 0.942 (mF1_max_px) 27.231 (mAUPRO_px) 0.319 (mF1_px_0.2_0.8_0.1) 0.318 (mAcc_px_0.2_0.8_0.1) 0.322 (mIoU_px_0.2_0.8_0.1) 0.928 (mIoU_max_px) 2024-04-04 11:43:29,480 - ==> Metric Time for pipe_fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.752 (mAUROC_px) 5.869 (mAP_px) 0.694 (mF1_max_px) 19.998 (mAUPRO_px) 0.245 (mF1_px_0.2_0.8_0.1) 0.242 (mAcc_px_0.2_0.8_0.1) 0.240 (mIoU_px_0.2_0.8_0.1) 0.689 (mIoU_max_px) 2024-04-04 11:43:29,502 - | Name | mAUROC_sp_max | mAUROC_sp_max (Max) | mAP_sp_max | mAP_sp_max (Max) | mF1_max_sp_max | mF1_max_sp_max (Max) | mAUROC_px | mAUROC_px (Max) | mAP_px | mAP_px (Max) | mF1_max_px | mF1_max_px (Max) | mAUPRO_px | mAUPRO_px (Max) | mF1_px_0.2_0.8_0.1 | mF1_px_0.2_0.8_0.1 (Max) | mAcc_px_0.2_0.8_0.1 | mAcc_px_0.2_0.8_0.1 (Max) | mIoU_px_0.2_0.8_0.1 | mIoU_px_0.2_0.8_0.1 (Max) | mIoU_max_px | mIoU_max_px (Max) | |:----------:|:---------------:|:---------------------:|:------------:|:------------------:|:----------------:|:----------------------:|:-----------:|:------------------:|:--------:|:------------------:|:------------:|:------------------:|:-----------:|:------------------:|:--------------------:|:--------------------------:|:---------------------:|:---------------------------:|:---------------------:|:---------------------------:|:-------------:|:-------------------:| | candle | 96.070 | 97.100 (50 epoch) | 96.436 | 97.346 (50 epoch) | 89.947 | 91.542 (50 epoch) | 99.050 | 99.098 (250 epoch) | 22.430 | 23.463 (150 epoch) | 31.307 | 32.654 (200 epoch) | 95.203 | 96.087 (300 epoch) | 21.069 | 21.069 (400 epoch) | 40.472 | 40.472 (400 epoch) | 12.043 | 12.043 (400 epoch) | 18.559 | 19.513 (200 epoch) | | capsules | 85.817 | 91.750 (100 epoch) | 89.952 | 94.964 (100 epoch) | 85.185 | 88.780 (100 epoch) | 98.050 | 99.123 (100 epoch) | 38.426 | 61.256 (100 epoch) | 44.818 | 59.868 (250 epoch) | 85.229 | 91.811 (100 epoch) | 20.702 | 33.186 (100 epoch) | 57.218 | 62.576 (50 epoch) | 12.275 | 21.701 (100 epoch) | 28.881 | 42.722 (250 epoch) | | cashew | 94.240 | 94.800 (150 epoch) | 97.202 | 97.492 (150 epoch) | 92.079 | 92.386 (200 epoch) | 95.204 | 95.614 (250 epoch) | 50.910 | 50.952 (250 epoch) | 54.438 | 54.438 (400 epoch) | 88.813 | 89.185 (350 epoch) | 27.668 | 28.352 (350 epoch) | 31.734 | 34.522 (150 epoch) | 17.408 | 17.797 (350 epoch) | 37.398 | 37.398 (400 epoch) | | chewinggum | 97.253 | 97.802 (300 epoch) | 98.747 | 98.939 (300 epoch) | 94.241 | 94.416 (300 epoch) | 98.025 | 98.145 (100 epoch) | 50.188 | 59.823 (50 epoch) | 56.737 | 61.088 (50 epoch) | 80.312 | 80.312 (400 epoch) | 28.843 | 35.748 (50 epoch) | 54.353 | 65.611 (50 epoch) | 18.592 | 23.754 (50 epoch) | 39.603 | 43.976 (50 epoch) | | fryum | 95.760 | 95.940 (300 epoch) | 97.936 | 98.018 (300 epoch) | 93.204 | 93.204 (400 epoch) | 97.104 | 97.174 (350 epoch) | 49.791 | 49.934 (250 epoch) | 52.059 | 52.992 (250 epoch) | 92.006 | 92.073 (350 epoch) | 26.238 | 26.312 (50 epoch) | 27.822 | 27.822 (400 epoch) | 16.329 | 16.329 (400 epoch) | 35.189 | 36.047 (250 epoch) | | macaroni1 | 91.892 | 92.084 (300 epoch) | 90.128 | 90.374 (300 epoch) | 82.609 | 82.653 (300 epoch) | 99.546 | 99.557 (350 epoch) | 19.348 | 19.348 (400 epoch) | 29.024 | 29.377 (200 epoch) | 94.907 | 95.899 (250 epoch) | 15.634 | 15.803 (300 epoch) | 39.121 | 40.765 (50 epoch) | 8.747 | 8.828 (300 epoch) | 16.976 | 17.217 (200 epoch) | | macaroni2 | 80.946 | 82.589 (50 epoch) | 72.270 | 81.196 (50 epoch) | 75.983 | 75.983 (400 epoch) | 98.266 | 99.452 (100 epoch) | 2.081 | 11.272 (50 epoch) | 6.116 | 17.568 (50 epoch) | 91.268 | 96.253 (150 epoch) | 2.229 | 10.086 (50 epoch) | 53.747 | 53.747 (400 epoch) | 1.136 | 5.433 (50 epoch) | 3.154 | 9.630 (50 epoch) | | pcb1 | 96.670 | 96.710 (300 epoch) | 95.525 | 95.525 (400 epoch) | 94.231 | 94.737 (300 epoch) | 99.802 | 99.802 (400 epoch) | 82.254 | 82.254 (400 epoch) | 75.028 | 75.268 (250 epoch) | 93.393 | 94.357 (300 epoch) | 47.630 | 47.630 (400 epoch) | 58.855 | 62.362 (50 epoch) | 33.701 | 33.701 (400 epoch) | 60.036 | 60.344 (250 epoch) | | pcb2 | 95.360 | 95.360 (400 epoch) | 95.265 | 95.265 (400 epoch) | 90.196 | 90.196 (400 epoch) | 98.961 | 98.978 (250 epoch) | 12.634 | 14.130 (50 epoch) | 23.820 | 23.977 (250 epoch) | 89.038 | 90.754 (250 epoch) | 15.967 | 16.150 (200 epoch) | 46.471 | 47.820 (150 epoch) | 8.787 | 8.892 (200 epoch) | 13.521 | 13.621 (250 epoch) | | pcb3 | 95.663 | 95.891 (350 epoch) | 96.032 | 96.056 (350 epoch) | 89.855 | 89.855 (400 epoch) | 99.112 | 99.142 (350 epoch) | 18.896 | 20.249 (150 epoch) | 27.715 | 27.928 (350 epoch) | 91.328 | 91.817 (350 epoch) | 19.687 | 20.042 (150 epoch) | 44.810 | 48.341 (50 epoch) | 11.084 | 11.295 (150 epoch) | 16.087 | 16.230 (350 epoch) | | pcb4 | 99.673 | 99.901 (100 epoch) | 99.675 | 99.901 (100 epoch) | 97.087 | 99.000 (50 epoch) | 98.722 | 98.727 (250 epoch) | 47.580 | 47.966 (250 epoch) | 47.400 | 47.709 (250 epoch) | 89.046 | 90.729 (250 epoch) | 25.826 | 26.925 (200 epoch) | 34.219 | 39.716 (50 epoch) | 15.893 | 16.537 (200 epoch) | 31.061 | 31.328 (250 epoch) | | pipe_fryum | 98.720 | 98.900 (350 epoch) | 99.342 | 99.421 (350 epoch) | 97.000 | 97.561 (200 epoch) | 99.076 | 99.096 (250 epoch) | 54.928 | 54.928 (400 epoch) | 58.495 | 58.495 (400 epoch) | 94.771 | 95.525 (150 epoch) | 32.185 | 34.684 (250 epoch) | 36.915 | 43.014 (100 epoch) | 20.827 | 22.399 (250 epoch) | 41.338 | 41.338 (400 epoch) | | Avg | 94.005 | 94.251 (100 epoch) | 94.042 | 94.481 (100 epoch) | 90.135 | 90.163 (300 epoch) | 98.410 | 98.606 (250 epoch) | 37.456 | 39.701 (250 epoch) | 42.246 | 44.125 (250 epoch) | 90.443 | 91.494 (250 epoch) | 23.640 | 25.164 (100 epoch) | 43.811 | 46.179 (50 epoch) | 14.735 | 15.755 (100 epoch) | 28.484 | 29.990 (250 epoch) | 2024-04-04 11:43:29,876 - ==> Total time: 2 days, 10:43:24 Eta: 3 days, 16:05:06 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 11:45:55,672 - Train: 40.02% [216500/541000] [400.2/1000.0] [batch_t 1.444 (1.436)] [data_t 0.002] [optim_t 1.442] [lr 0.005000] 2024-04-04 11:48:19,030 - Train: 40.04% [216600/541000] [400.4/1000.0] [batch_t 1.434 (1.433)] [data_t 0.002] [optim_t 1.431] [lr 0.005000] 2024-04-04 11:50:42,241 - Train: 40.06% [216700/541000] [400.6/1000.0] [batch_t 1.425 (1.432)] [data_t 0.003] [optim_t 1.423] [lr 0.005000] 2024-04-04 11:53:05,570 - Train: 40.07% [216800/541000] [400.7/1000.0] [batch_t 1.435 (1.433)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-04 11:55:28,776 - Train: 40.09% [216900/541000] [400.9/1000.0] [batch_t 1.433 (1.432)] [data_t 0.002] [optim_t 1.430] [lr 0.005000] 2024-04-04 11:56:27,441 - ==> Total time: 2 days, 10:56:21 Eta: 3 days, 16:02:29 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 11:57:53,995 - Train: 40.11% [217000/541000] [401.1/1000.0] [batch_t 1.432 (1.439)] [data_t 0.003] [optim_t 1.429] [lr 0.005000] 2024-04-04 12:00:17,266 - Train: 40.13% [217100/541000] [401.3/1000.0] [batch_t 1.435 (1.433)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-04 12:02:40,728 - Train: 40.15% [217200/541000] [401.5/1000.0] [batch_t 1.430 (1.435)] [data_t 0.002] [optim_t 1.428] [lr 0.005000] 2024-04-04 12:05:03,988 - Train: 40.17% [217300/541000] [401.7/1000.0] [batch_t 1.440 (1.433)] [data_t 0.002] [optim_t 1.438] [lr 0.005000] 2024-04-04 12:07:27,102 - Train: 40.18% [217400/541000] [401.8/1000.0] [batch_t 1.438 (1.431)] [data_t 0.002] [optim_t 1.436] [lr 0.005000] 2024-04-04 12:09:24,661 - ==> Total time: 2 days, 11:09:19 Eta: 3 days, 15:59:49 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 12:09:52,652 - Train: 40.20% [217500/541000] [402.0/1000.0] [batch_t 1.439 (1.455)] [data_t 0.003] [optim_t 1.436] [lr 0.005000] 2024-04-04 12:12:15,824 - Train: 40.22% [217600/541000] [402.2/1000.0] [batch_t 1.429 (1.432)] [data_t 0.003] [optim_t 1.426] [lr 0.005000] 2024-04-04 12:14:39,066 - Train: 40.24% [217700/541000] [402.4/1000.0] [batch_t 1.426 (1.432)] [data_t 0.003] [optim_t 1.423] [lr 0.005000] 2024-04-04 12:17:02,167 - Train: 40.26% [217800/541000] [402.6/1000.0] [batch_t 1.444 (1.431)] [data_t 0.002] [optim_t 1.442] [lr 0.005000] 2024-04-04 12:19:25,389 - Train: 40.28% [217900/541000] [402.8/1000.0] [batch_t 1.441 (1.432)] [data_t 0.003] [optim_t 1.439] [lr 0.005000] 2024-04-04 12:21:48,608 - Train: 40.30% [218000/541000] [403.0/1000.0] [batch_t 1.427 (1.432)] [data_t 0.003] [optim_t 1.424] [lr 0.005000] 2024-04-04 12:22:21,502 - ==> Total time: 2 days, 11:22:15 Eta: 3 days, 15:57:06 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 12:24:13,642 - Train: 40.31% [218100/541000] [403.1/1000.0] [batch_t 1.425 (1.435)] [data_t 0.003] [optim_t 1.422] [lr 0.005000] 2024-04-04 12:26:36,853 - Train: 40.33% [218200/541000] [403.3/1000.0] [batch_t 1.444 (1.432)] [data_t 0.003] [optim_t 1.441] [lr 0.005000] 2024-04-04 12:29:00,112 - Train: 40.35% [218300/541000] [403.5/1000.0] [batch_t 1.436 (1.432)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-04 12:31:23,519 - Train: 40.37% [218400/541000] [403.7/1000.0] [batch_t 1.438 (1.434)] [data_t 0.002] [optim_t 1.436] [lr 0.005000] 2024-04-04 12:33:46,822 - Train: 40.39% [218500/541000] [403.9/1000.0] [batch_t 1.425 (1.433)] [data_t 0.003] [optim_t 1.422] [lr 0.005000] 2024-04-04 12:35:18,551 - ==> Total time: 2 days, 11:35:12 Eta: 3 days, 15:54:19 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 12:36:12,313 - Train: 40.41% [218600/541000] [404.1/1000.0] [batch_t 1.434 (1.448)] [data_t 0.002] [optim_t 1.432] [lr 0.005000] 2024-04-04 12:38:35,579 - Train: 40.43% [218700/541000] [404.3/1000.0] [batch_t 1.410 (1.433)] [data_t 0.003] [optim_t 1.408] [lr 0.005000] 2024-04-04 12:41:00,407 - Train: 40.44% [218800/541000] [404.4/1000.0] [batch_t 1.456 (1.448)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-04 12:43:26,006 - Train: 40.46% [218900/541000] [404.6/1000.0] [batch_t 1.449 (1.456)] [data_t 0.003] [optim_t 1.446] [lr 0.005000] 2024-04-04 12:45:49,967 - Train: 40.48% [219000/541000] [404.8/1000.0] [batch_t 1.428 (1.440)] [data_t 0.002] [optim_t 1.426] [lr 0.005000] 2024-04-04 12:48:13,167 - Train: 40.50% [219100/541000] [405.0/1000.0] [batch_t 1.427 (1.432)] [data_t 0.002] [optim_t 1.425] [lr 0.005000] 2024-04-04 12:48:20,343 - ==> Total time: 2 days, 11:48:14 Eta: 3 days, 15:51:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 12:50:38,517 - Train: 40.52% [219200/541000] [405.2/1000.0] [batch_t 1.434 (1.436)] [data_t 0.003] [optim_t 1.431] [lr 0.005000] 2024-04-04 12:53:01,899 - Train: 40.54% [219300/541000] [405.4/1000.0] [batch_t 1.425 (1.434)] [data_t 0.003] [optim_t 1.422] [lr 0.005000] 2024-04-04 12:55:25,226 - Train: 40.55% [219400/541000] [405.5/1000.0] [batch_t 1.449 (1.433)] [data_t 0.002] [optim_t 1.447] [lr 0.005000] 2024-04-04 12:57:48,505 - Train: 40.57% [219500/541000] [405.7/1000.0] [batch_t 1.436 (1.433)] [data_t 0.002] [optim_t 1.434] [lr 0.005000] 2024-04-04 13:00:11,806 - Train: 40.59% [219600/541000] [405.9/1000.0] [batch_t 1.429 (1.433)] [data_t 0.003] [optim_t 1.426] [lr 0.005000] 2024-04-04 13:01:17,688 - ==> Total time: 2 days, 12:01:12 Eta: 3 days, 15:48:45 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 13:02:37,210 - Train: 40.61% [219700/541000] [406.1/1000.0] [batch_t 1.440 (1.440)] [data_t 0.003] [optim_t 1.437] [lr 0.005000] 2024-04-04 13:05:00,465 - Train: 40.63% [219800/541000] [406.3/1000.0] [batch_t 1.433 (1.432)] [data_t 0.003] [optim_t 1.430] [lr 0.005000] 2024-04-04 13:07:23,832 - Train: 40.65% [219900/541000] [406.5/1000.0] [batch_t 1.435 (1.434)] [data_t 0.002] [optim_t 1.432] [lr 0.005000] 2024-04-04 13:09:47,061 - Train: 40.67% [220000/541000] [406.7/1000.0] [batch_t 1.424 (1.432)] [data_t 0.003] [optim_t 1.422] [lr 0.005000] 2024-04-04 13:12:10,380 - Train: 40.68% [220100/541000] [406.8/1000.0] [batch_t 1.438 (1.433)] [data_t 0.003] [optim_t 1.435] [lr 0.005000] 2024-04-04 13:14:14,892 - ==> Total time: 2 days, 12:14:09 Eta: 3 days, 15:45:50 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 13:14:35,921 - Train: 40.70% [220200/541000] [407.0/1000.0] [batch_t 1.426 (1.466)] [data_t 0.002] [optim_t 1.423] [lr 0.005000] 2024-04-04 13:16:59,250 - Train: 40.72% [220300/541000] [407.2/1000.0] [batch_t 1.437 (1.433)] [data_t 0.003] [optim_t 1.434] [lr 0.005000] 2024-04-04 13:19:22,311 - Train: 40.74% [220400/541000] [407.4/1000.0] [batch_t 1.435 (1.431)] [data_t 0.002] [optim_t 1.432] [lr 0.005000] 2024-04-04 13:21:45,427 - Train: 40.76% [220500/541000] [407.6/1000.0] [batch_t 1.446 (1.431)] [data_t 0.003] [optim_t 1.444] [lr 0.005000] 2024-04-04 13:24:08,613 - Train: 40.78% [220600/541000] [407.8/1000.0] [batch_t 1.430 (1.432)] [data_t 0.003] [optim_t 1.427] [lr 0.005000] 2024-04-04 13:26:31,835 - Train: 40.79% [220700/541000] [407.9/1000.0] [batch_t 1.421 (1.432)] [data_t 0.003] [optim_t 1.418] [lr 0.005000] 2024-04-04 13:27:11,951 - ==> Total time: 2 days, 12:27:06 Eta: 3 days, 15:42:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 13:28:57,338 - Train: 40.81% [220800/541000] [408.1/1000.0] [batch_t 1.421 (1.439)] [data_t 0.003] [optim_t 1.418] [lr 0.005000] 2024-04-04 13:31:20,481 - Train: 40.83% [220900/541000] [408.3/1000.0] [batch_t 1.427 (1.431)] [data_t 0.003] [optim_t 1.424] [lr 0.005000] 2024-04-04 13:33:43,785 - Train: 40.85% [221000/541000] [408.5/1000.0] [batch_t 1.428 (1.433)] [data_t 0.002] [optim_t 1.426] [lr 0.005000] 2024-04-04 13:36:06,968 - Train: 40.87% [221100/541000] [408.7/1000.0] [batch_t 1.434 (1.432)] [data_t 0.002] [optim_t 1.431] [lr 0.005000] 2024-04-04 13:38:30,178 - Train: 40.89% [221200/541000] [408.9/1000.0] [batch_t 1.426 (1.432)] [data_t 0.002] [optim_t 1.424] [lr 0.005000] 2024-04-04 13:40:09,086 - ==> Total time: 2 days, 12:40:03 Eta: 3 days, 15:39:50 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 13:40:55,510 - Train: 40.91% [221300/541000] [409.1/1000.0] [batch_t 1.425 (1.445)] [data_t 0.002] [optim_t 1.422] [lr 0.005000] 2024-04-04 13:43:18,916 - Train: 40.92% [221400/541000] [409.2/1000.0] [batch_t 1.440 (1.434)] [data_t 0.003] [optim_t 1.437] [lr 0.005000] 2024-04-04 13:45:42,202 - Train: 40.94% [221500/541000] [409.4/1000.0] [batch_t 1.430 (1.433)] [data_t 0.002] [optim_t 1.428] [lr 0.005000] 2024-04-04 13:48:05,464 - Train: 40.96% [221600/541000] [409.6/1000.0] [batch_t 1.421 (1.433)] [data_t 0.003] [optim_t 1.418] [lr 0.005000] 2024-04-04 13:50:28,627 - Train: 40.98% [221700/541000] [409.8/1000.0] [batch_t 1.421 (1.432)] [data_t 0.003] [optim_t 1.418] [lr 0.005000] 2024-04-04 13:52:51,957 - Train: 41.00% [221800/541000] [410.0/1000.0] [batch_t 1.424 (1.433)] [data_t 0.003] [optim_t 1.421] [lr 0.005000] 2024-04-04 13:53:06,294 - ==> Total time: 2 days, 12:53:00 Eta: 3 days, 15:36:46 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 13:55:17,234 - Train: 41.02% [221900/541000] [410.2/1000.0] [batch_t 1.424 (1.436)] [data_t 0.003] [optim_t 1.422] [lr 0.005000] 2024-04-04 13:57:40,369 - Train: 41.04% [222000/541000] [410.4/1000.0] [batch_t 1.426 (1.431)] [data_t 0.002] [optim_t 1.423] [lr 0.005000] 2024-04-04 14:00:03,535 - Train: 41.05% [222100/541000] [410.5/1000.0] [batch_t 1.435 (1.432)] [data_t 0.002] [optim_t 1.433] [lr 0.005000] 2024-04-04 14:02:26,612 - Train: 41.07% [222200/541000] [410.7/1000.0] [batch_t 1.429 (1.431)] [data_t 0.002] [optim_t 1.427] [lr 0.005000] 2024-04-04 14:04:49,824 - Train: 41.09% [222300/541000] [410.9/1000.0] [batch_t 1.435 (1.432)] [data_t 0.003] [optim_t 1.432] [lr 0.005000] 2024-04-04 14:06:02,951 - ==> Total time: 2 days, 13:05:57 Eta: 3 days, 15:33:38 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 14:07:17,265 - Train: 41.11% [222400/541000] [411.1/1000.0] [batch_t 1.436 (1.452)] [data_t 0.002] [optim_t 1.433] [lr 0.005000] 2024-04-04 14:09:31,394 - Train: 41.13% [222500/541000] [411.3/1000.0] [batch_t 0.606 (1.341)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 14:10:32,078 - Train: 41.15% [222600/541000] [411.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 14:11:32,776 - Train: 41.16% [222700/541000] [411.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 14:12:33,455 - Train: 41.18% [222800/541000] [411.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 14:13:29,407 - ==> Total time: 2 days, 13:13:23 Eta: 3 days, 15:22:36 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 14:13:36,285 - Train: 41.20% [222900/541000] [412.0/1000.0] [batch_t 0.607 (0.650)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 14:14:36,981 - Train: 41.22% [223000/541000] [412.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 14:15:37,676 - Train: 41.24% [223100/541000] [412.4/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 14:16:38,371 - Train: 41.26% [223200/541000] [412.6/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 14:17:39,083 - Train: 41.28% [223300/541000] [412.8/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 14:18:39,759 - Train: 41.29% [223400/541000] [412.9/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 14:18:59,785 - ==> Total time: 2 days, 13:18:54 Eta: 3 days, 15:08:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 14:19:44,218 - Train: 41.31% [223500/541000] [413.1/1000.0] [batch_t 0.606 (0.639)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 14:20:44,899 - Train: 41.33% [223600/541000] [413.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 14:21:45,588 - Train: 41.35% [223700/541000] [413.5/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 14:22:46,289 - Train: 41.37% [223800/541000] [413.7/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 14:23:46,973 - Train: 41.39% [223900/541000] [413.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 14:24:31,953 - ==> Total time: 2 days, 13:24:26 Eta: 3 days, 14:55:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 14:24:49,795 - Train: 41.40% [224000/541000] [414.0/1000.0] [batch_t 0.607 (0.621)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 14:25:50,494 - Train: 41.42% [224100/541000] [414.2/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 14:26:51,178 - Train: 41.44% [224200/541000] [414.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 14:27:51,896 - Train: 41.46% [224300/541000] [414.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 14:28:52,603 - Train: 41.48% [224400/541000] [414.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 14:29:53,305 - Train: 41.50% [224500/541000] [415.0/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 14:30:02,401 - ==> Total time: 2 days, 13:29:56 Eta: 3 days, 14:41:29 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 14:30:57,631 - Train: 41.52% [224600/541000] [415.2/1000.0] [batch_t 0.606 (0.617)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 14:31:58,312 - Train: 41.53% [224700/541000] [415.3/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 14:32:59,004 - Train: 41.55% [224800/541000] [415.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 14:33:59,672 - Train: 41.57% [224900/541000] [415.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 14:35:00,354 - Train: 41.59% [225000/541000] [415.9/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 14:35:34,344 - ==> Total time: 2 days, 13:35:28 Eta: 3 days, 14:27:53 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 14:36:03,316 - Train: 41.61% [225100/541000] [416.1/1000.0] [batch_t 0.608 (0.622)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 14:37:04,040 - Train: 41.63% [225200/541000] [416.3/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 14:38:04,764 - Train: 41.65% [225300/541000] [416.5/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 14:39:05,526 - Train: 41.66% [225400/541000] [416.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 14:40:06,271 - Train: 41.68% [225500/541000] [416.8/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 14:41:05,163 - ==> Total time: 2 days, 13:40:59 Eta: 3 days, 14:14:17 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 14:41:09,403 - Train: 41.70% [225600/541000] [417.0/1000.0] [batch_t 0.607 (0.724)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 14:42:10,118 - Train: 41.72% [225700/541000] [417.2/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 14:43:10,942 - Train: 41.74% [225800/541000] [417.4/1000.0] [batch_t 0.605 (0.608)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 14:44:11,648 - Train: 41.76% [225900/541000] [417.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 14:45:12,349 - Train: 41.77% [226000/541000] [417.7/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 14:46:13,066 - Train: 41.79% [226100/541000] [417.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 14:46:36,140 - ==> Total time: 2 days, 13:46:30 Eta: 3 days, 14:00:44 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 14:47:15,807 - Train: 41.81% [226200/541000] [418.1/1000.0] [batch_t 0.607 (0.613)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 14:48:16,643 - Train: 41.83% [226300/541000] [418.3/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 14:49:17,358 - Train: 41.85% [226400/541000] [418.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 14:50:18,067 - Train: 41.87% [226500/541000] [418.7/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 14:51:18,774 - Train: 41.89% [226600/541000] [418.9/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 14:52:06,717 - ==> Total time: 2 days, 13:52:01 Eta: 3 days, 13:47:12 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 14:52:21,584 - Train: 41.90% [226700/541000] [419.0/1000.0] [batch_t 0.606 (0.625)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 14:53:22,280 - Train: 41.92% [226800/541000] [419.2/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 14:54:23,127 - Train: 41.94% [226900/541000] [419.4/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 14:55:23,828 - Train: 41.96% [227000/541000] [419.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 14:56:24,520 - Train: 41.98% [227100/541000] [419.8/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 14:57:25,238 - Train: 42.00% [227200/541000] [420.0/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 14:57:37,384 - ==> Total time: 2 days, 13:57:31 Eta: 3 days, 13:33:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 14:58:28,117 - Train: 42.01% [227300/541000] [420.1/1000.0] [batch_t 0.607 (0.612)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 14:59:28,810 - Train: 42.03% [227400/541000] [420.3/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 15:00:29,517 - Train: 42.05% [227500/541000] [420.5/1000.0] [batch_t 0.608 (0.607)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-04 15:01:30,327 - Train: 42.07% [227600/541000] [420.7/1000.0] [batch_t 0.604 (0.608)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-04 15:02:31,024 - Train: 42.09% [227700/541000] [420.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 15:03:08,037 - ==> Total time: 2 days, 14:03:02 Eta: 3 days, 13:20:17 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 15:03:33,732 - Train: 42.11% [227800/541000] [421.1/1000.0] [batch_t 0.608 (0.616)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 15:04:34,424 - Train: 42.13% [227900/541000] [421.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 15:05:35,129 - Train: 42.14% [228000/541000] [421.4/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 15:06:35,966 - Train: 42.16% [228100/541000] [421.6/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 15:07:36,670 - Train: 42.18% [228200/541000] [421.8/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 15:08:37,350 - Train: 42.20% [228300/541000] [422.0/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 15:08:38,568 - ==> Total time: 2 days, 14:08:33 Eta: 3 days, 13:06:52 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 15:09:40,167 - Train: 42.22% [228400/541000] [422.2/1000.0] [batch_t 0.607 (0.611)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 15:10:40,881 - Train: 42.24% [228500/541000] [422.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 15:11:41,577 - Train: 42.26% [228600/541000] [422.6/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 15:12:42,431 - Train: 42.27% [228700/541000] [422.7/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 15:13:43,132 - Train: 42.29% [228800/541000] [422.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 15:14:09,224 - ==> Total time: 2 days, 14:14:03 Eta: 3 days, 12:53:30 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 15:14:45,843 - Train: 42.31% [228900/541000] [423.1/1000.0] [batch_t 0.607 (0.614)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 15:15:46,552 - Train: 42.33% [229000/541000] [423.3/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 15:16:47,290 - Train: 42.35% [229100/541000] [423.5/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 15:17:48,132 - Train: 42.37% [229200/541000] [423.7/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 15:18:48,836 - Train: 42.38% [229300/541000] [423.8/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 15:19:39,861 - ==> Total time: 2 days, 14:19:34 Eta: 3 days, 12:40:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 15:19:51,589 - Train: 42.40% [229400/541000] [424.0/1000.0] [batch_t 0.606 (0.629)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 15:20:52,299 - Train: 42.42% [229500/541000] [424.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 15:21:53,032 - Train: 42.44% [229600/541000] [424.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 15:22:53,775 - Train: 42.46% [229700/541000] [424.6/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 15:23:54,513 - Train: 42.48% [229800/541000] [424.8/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 15:24:55,349 - Train: 42.50% [229900/541000] [425.0/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 15:25:10,528 - ==> Total time: 2 days, 14:25:04 Eta: 3 days, 12:26:52 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 15:25:58,023 - Train: 42.51% [230000/541000] [425.1/1000.0] [batch_t 0.607 (0.611)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 15:26:58,712 - Train: 42.53% [230100/541000] [425.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 15:27:59,410 - Train: 42.55% [230200/541000] [425.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 15:29:00,122 - Train: 42.57% [230300/541000] [425.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 15:30:00,927 - Train: 42.59% [230400/541000] [425.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 15:30:41,006 - ==> Total time: 2 days, 14:30:35 Eta: 3 days, 12:13:36 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 15:31:03,606 - Train: 42.61% [230500/541000] [426.1/1000.0] [batch_t 0.605 (0.617)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 15:32:04,302 - Train: 42.62% [230600/541000] [426.2/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 15:33:05,004 - Train: 42.64% [230700/541000] [426.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 15:34:05,678 - Train: 42.66% [230800/541000] [426.6/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 15:35:06,384 - Train: 42.68% [230900/541000] [426.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 15:36:07,069 - Train: 42.70% [231000/541000] [427.0/1000.0] [batch_t 0.608 (0.607)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-04 15:36:11,321 - ==> Total time: 2 days, 14:36:05 Eta: 3 days, 12:00:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 15:37:10,066 - Train: 42.72% [231100/541000] [427.2/1000.0] [batch_t 0.608 (0.611)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 15:38:10,763 - Train: 42.74% [231200/541000] [427.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 15:39:11,460 - Train: 42.75% [231300/541000] [427.5/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 15:40:12,173 - Train: 42.77% [231400/541000] [427.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 15:41:12,878 - Train: 42.79% [231500/541000] [427.9/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 15:41:42,004 - ==> Total time: 2 days, 14:41:36 Eta: 3 days, 11:47:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 15:42:15,626 - Train: 42.81% [231600/541000] [428.1/1000.0] [batch_t 0.608 (0.614)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 15:43:16,334 - Train: 42.83% [231700/541000] [428.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 15:44:17,156 - Train: 42.85% [231800/541000] [428.5/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 15:45:17,855 - Train: 42.87% [231900/541000] [428.7/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 15:46:18,556 - Train: 42.88% [232000/541000] [428.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 15:47:12,560 - ==> Total time: 2 days, 14:47:07 Eta: 3 days, 11:34:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 15:47:21,335 - Train: 42.90% [232100/541000] [429.0/1000.0] [batch_t 0.606 (0.639)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 15:48:22,050 - Train: 42.92% [232200/541000] [429.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 15:49:22,898 - Train: 42.94% [232300/541000] [429.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 15:50:23,608 - Train: 42.96% [232400/541000] [429.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 15:51:24,298 - Train: 42.98% [232500/541000] [429.8/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 15:52:24,972 - Train: 42.99% [232600/541000] [429.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 15:52:43,174 - ==> Total time: 2 days, 14:52:37 Eta: 3 days, 11:20:55 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 15:53:27,858 - Train: 43.01% [232700/541000] [430.1/1000.0] [batch_t 0.607 (0.613)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 15:54:28,561 - Train: 43.03% [232800/541000] [430.3/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 15:55:29,398 - Train: 43.05% [232900/541000] [430.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 15:56:30,118 - Train: 43.07% [233000/541000] [430.7/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 15:57:30,805 - Train: 43.09% [233100/541000] [430.9/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 15:58:13,915 - ==> Total time: 2 days, 14:58:08 Eta: 3 days, 11:07:50 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 15:58:33,585 - Train: 43.11% [233200/541000] [431.1/1000.0] [batch_t 0.602 (0.619)] [data_t 0.002] [optim_t 0.600] [lr 0.005000] 2024-04-04 15:59:34,290 - Train: 43.12% [233300/541000] [431.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:00:35,022 - Train: 43.14% [233400/541000] [431.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:01:35,727 - Train: 43.16% [233500/541000] [431.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:02:36,552 - Train: 43.18% [233600/541000] [431.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 16:03:37,251 - Train: 43.20% [233700/541000] [432.0/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 16:03:44,534 - ==> Total time: 2 days, 15:03:38 Eta: 3 days, 10:54:47 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 16:04:39,944 - Train: 43.22% [233800/541000] [432.2/1000.0] [batch_t 0.605 (0.611)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 16:05:40,659 - Train: 43.23% [233900/541000] [432.3/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 16:06:41,356 - Train: 43.25% [234000/541000] [432.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:07:42,232 - Train: 43.27% [234100/541000] [432.7/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 16:08:43,021 - Train: 43.29% [234200/541000] [432.9/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 16:09:15,199 - ==> Total time: 2 days, 15:09:09 Eta: 3 days, 10:41:47 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 16:09:45,820 - Train: 43.31% [234300/541000] [433.1/1000.0] [batch_t 0.607 (0.615)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:10:46,511 - Train: 43.33% [234400/541000] [433.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:11:47,207 - Train: 43.35% [234500/541000] [433.5/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:12:48,042 - Train: 43.36% [234600/541000] [433.6/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 16:13:48,756 - Train: 43.38% [234700/541000] [433.8/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 16:14:45,824 - ==> Total time: 2 days, 15:14:40 Eta: 3 days, 10:28:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 16:14:51,909 - Train: 43.40% [234800/541000] [434.0/1000.0] [batch_t 0.608 (0.712)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 16:15:52,627 - Train: 43.42% [234900/541000] [434.2/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 16:16:53,335 - Train: 43.44% [235000/541000] [434.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:17:54,042 - Train: 43.46% [235100/541000] [434.6/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 16:18:54,939 - Train: 43.48% [235200/541000] [434.8/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 16:19:55,669 - Train: 43.49% [235300/541000] [434.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:20:16,932 - ==> Total time: 2 days, 15:20:11 Eta: 3 days, 10:15:52 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 16:20:58,657 - Train: 43.51% [235400/541000] [435.1/1000.0] [batch_t 0.607 (0.614)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:21:59,360 - Train: 43.53% [235500/541000] [435.3/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:23:00,045 - Train: 43.55% [235600/541000] [435.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:24:00,754 - Train: 43.57% [235700/541000] [435.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:25:01,662 - Train: 43.59% [235800/541000] [435.9/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:25:47,876 - ==> Total time: 2 days, 15:25:42 Eta: 3 days, 10:02:58 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 16:26:04,515 - Train: 43.60% [235900/541000] [436.0/1000.0] [batch_t 0.607 (0.623)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:27:05,315 - Train: 43.62% [236000/541000] [436.2/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 16:28:06,101 - Train: 43.64% [236100/541000] [436.4/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:29:06,890 - Train: 43.66% [236200/541000] [436.6/1000.0] [batch_t 0.611 (0.608)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-04 16:30:08,703 - Train: 43.68% [236300/541000] [436.8/1000.0] [batch_t 0.607 (0.618)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:31:09,420 - Train: 43.70% [236400/541000] [437.0/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:31:19,746 - ==> Total time: 2 days, 15:31:14 Eta: 3 days, 9:50:07 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 16:32:12,317 - Train: 43.72% [236500/541000] [437.2/1000.0] [batch_t 0.607 (0.612)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 16:33:13,031 - Train: 43.73% [236600/541000] [437.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:34:13,710 - Train: 43.75% [236700/541000] [437.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:35:14,595 - Train: 43.77% [236800/541000] [437.7/1000.0] [batch_t 0.606 (0.609)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 16:36:15,328 - Train: 43.79% [236900/541000] [437.9/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 16:36:50,550 - ==> Total time: 2 days, 15:36:44 Eta: 3 days, 9:37:17 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 16:37:18,042 - Train: 43.81% [237000/541000] [438.1/1000.0] [batch_t 0.607 (0.616)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 16:38:18,755 - Train: 43.83% [237100/541000] [438.3/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 16:39:19,455 - Train: 43.84% [237200/541000] [438.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:40:20,149 - Train: 43.86% [237300/541000] [438.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:41:20,848 - Train: 43.88% [237400/541000] [438.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:42:21,086 - ==> Total time: 2 days, 15:42:15 Eta: 3 days, 9:24:28 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 16:42:23,914 - Train: 43.90% [237500/541000] [439.0/1000.0] [batch_t 1.032 (1.032)] [data_t 0.421] [optim_t 0.611] [lr 0.005000] 2024-04-04 16:43:24,616 - Train: 43.92% [237600/541000] [439.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:44:25,316 - Train: 43.94% [237700/541000] [439.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:45:25,994 - Train: 43.96% [237800/541000] [439.6/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 16:46:26,716 - Train: 43.97% [237900/541000] [439.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:47:27,407 - Train: 43.99% [238000/541000] [439.9/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 16:47:51,692 - ==> Total time: 2 days, 15:47:46 Eta: 3 days, 9:11:42 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 16:48:30,195 - Train: 44.01% [238100/541000] [440.1/1000.0] [batch_t 0.608 (0.614)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 16:49:30,942 - Train: 44.03% [238200/541000] [440.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:50:31,645 - Train: 44.05% [238300/541000] [440.5/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:51:32,331 - Train: 44.07% [238400/541000] [440.7/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 16:52:33,016 - Train: 44.09% [238500/541000] [440.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:53:22,193 - ==> Total time: 2 days, 15:53:16 Eta: 3 days, 8:58:57 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 16:53:36,260 - Train: 44.10% [238600/541000] [441.0/1000.0] [batch_t 0.606 (0.634)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 16:54:36,961 - Train: 44.12% [238700/541000] [441.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:55:37,646 - Train: 44.14% [238800/541000] [441.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:56:38,336 - Train: 44.16% [238900/541000] [441.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:57:38,997 - Train: 44.18% [239000/541000] [441.8/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 16:58:39,689 - Train: 44.20% [239100/541000] [442.0/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 16:58:53,040 - ==> Total time: 2 days, 15:58:47 Eta: 3 days, 8:46:15 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 16:59:42,650 - Train: 44.21% [239200/541000] [442.1/1000.0] [batch_t 0.607 (0.612)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 17:00:43,472 - Train: 44.23% [239300/541000] [442.3/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 17:01:44,171 - Train: 44.25% [239400/541000] [442.5/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 17:02:44,877 - Train: 44.27% [239500/541000] [442.7/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 17:03:45,571 - Train: 44.29% [239600/541000] [442.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 17:04:23,823 - ==> Total time: 2 days, 16:04:18 Eta: 3 days, 8:33:35 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 17:04:48,636 - Train: 44.31% [239700/541000] [443.1/1000.0] [batch_t 0.606 (0.618)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 17:05:49,489 - Train: 44.33% [239800/541000] [443.3/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 17:06:50,181 - Train: 44.34% [239900/541000] [443.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 17:07:50,901 - Train: 44.36% [240000/541000] [443.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 17:08:51,619 - Train: 44.38% [240100/541000] [443.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 17:09:52,324 - Train: 44.40% [240200/541000] [444.0/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 17:09:54,755 - ==> Total time: 2 days, 16:09:49 Eta: 3 days, 8:20:56 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 17:10:55,026 - Train: 44.42% [240300/541000] [444.2/1000.0] [batch_t 0.606 (0.611)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 17:11:55,681 - Train: 44.44% [240400/541000] [444.4/1000.0] [batch_t 0.607 (0.606)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 17:12:56,460 - Train: 44.45% [240500/541000] [444.5/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 17:13:57,098 - Train: 44.47% [240600/541000] [444.7/1000.0] [batch_t 0.606 (0.606)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 17:14:57,745 - Train: 44.49% [240700/541000] [444.9/1000.0] [batch_t 0.608 (0.606)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 17:15:25,044 - ==> Total time: 2 days, 16:15:19 Eta: 3 days, 8:08:19 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 17:16:00,578 - Train: 44.51% [240800/541000] [445.1/1000.0] [batch_t 0.606 (0.614)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 17:17:01,225 - Train: 44.53% [240900/541000] [445.3/1000.0] [batch_t 0.606 (0.606)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 17:18:01,999 - Train: 44.55% [241000/541000] [445.5/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 17:19:02,663 - Train: 44.57% [241100/541000] [445.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 17:20:03,317 - Train: 44.58% [241200/541000] [445.8/1000.0] [batch_t 0.607 (0.606)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 17:20:55,467 - ==> Total time: 2 days, 16:20:49 Eta: 3 days, 7:55:44 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 17:21:06,181 - Train: 44.60% [241300/541000] [446.0/1000.0] [batch_t 0.607 (0.635)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 17:22:06,833 - Train: 44.62% [241400/541000] [446.2/1000.0] [batch_t 0.606 (0.606)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 17:23:07,470 - Train: 44.64% [241500/541000] [446.4/1000.0] [batch_t 0.604 (0.606)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 17:24:08,095 - Train: 44.66% [241600/541000] [446.6/1000.0] [batch_t 0.606 (0.606)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 17:25:08,856 - Train: 44.68% [241700/541000] [446.8/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 17:26:09,509 - Train: 44.70% [241800/541000] [447.0/1000.0] [batch_t 0.607 (0.606)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 17:26:25,888 - ==> Total time: 2 days, 16:26:20 Eta: 3 days, 7:43:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 17:27:12,257 - Train: 44.71% [241900/541000] [447.1/1000.0] [batch_t 0.606 (0.612)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 17:28:12,896 - Train: 44.73% [242000/541000] [447.3/1000.0] [batch_t 0.606 (0.606)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 17:29:13,517 - Train: 44.75% [242100/541000] [447.5/1000.0] [batch_t 0.603 (0.606)] [data_t 0.002] [optim_t 0.601] [lr 0.005000] 2024-04-04 17:30:14,280 - Train: 44.77% [242200/541000] [447.7/1000.0] [batch_t 0.610 (0.608)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-04 17:31:14,917 - Train: 44.79% [242300/541000] [447.9/1000.0] [batch_t 0.606 (0.606)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 17:31:56,145 - ==> Total time: 2 days, 16:31:50 Eta: 3 days, 7:30:39 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 17:32:17,536 - Train: 44.81% [242400/541000] [448.1/1000.0] [batch_t 0.606 (0.619)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 17:33:18,179 - Train: 44.82% [242500/541000] [448.2/1000.0] [batch_t 0.607 (0.606)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 17:34:18,790 - Train: 44.84% [242600/541000] [448.4/1000.0] [batch_t 0.606 (0.606)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 17:35:19,433 - Train: 44.86% [242700/541000] [448.6/1000.0] [batch_t 0.607 (0.606)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 17:36:20,167 - Train: 44.88% [242800/541000] [448.8/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 17:37:20,807 - Train: 44.90% [242900/541000] [449.0/1000.0] [batch_t 0.606 (0.606)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 17:37:26,270 - ==> Total time: 2 days, 16:37:20 Eta: 3 days, 7:18:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 17:38:23,631 - Train: 44.92% [243000/541000] [449.2/1000.0] [batch_t 0.605 (0.611)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 17:39:24,304 - Train: 44.94% [243100/541000] [449.4/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 17:40:24,963 - Train: 44.95% [243200/541000] [449.5/1000.0] [batch_t 0.606 (0.606)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 17:41:25,798 - Train: 44.97% [243300/541000] [449.7/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 17:42:26,457 - Train: 44.99% [243400/541000] [449.9/1000.0] [batch_t 0.606 (0.606)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 17:43:11,432 - Test: 36.76% [50/136] [batch_t 0.219 (0.281)] 2024-04-04 17:43:22,073 - Test: 73.53% [100/136] [batch_t 0.212 (0.247)] 2024-04-04 17:43:29,535 - Test: 100.00% [136/136] [batch_t 0.053 (0.237)] 2024-04-04 17:44:17,751 - ==> Metric Time for candle : 0.002 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.543 (mAUROC_px) 8.072 (mAP_px) 0.941 (mF1_max_px) 24.980 (mAUPRO_px) 0.323 (mF1_px_0.2_0.8_0.1) 0.320 (mAcc_px_0.2_0.8_0.1) 0.318 (mIoU_px_0.2_0.8_0.1) 0.944 (mIoU_max_px) 2024-04-04 17:44:54,560 - ==> Metric Time for capsules : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 8.161 (mAUROC_px) 6.253 (mAP_px) 0.749 (mF1_max_px) 19.916 (mAUPRO_px) 0.255 (mF1_px_0.2_0.8_0.1) 0.256 (mAcc_px_0.2_0.8_0.1) 0.260 (mIoU_px_0.2_0.8_0.1) 0.748 (mIoU_max_px) 2024-04-04 17:45:30,867 - ==> Metric Time for cashew : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.714 (mAUROC_px) 6.042 (mAP_px) 0.697 (mF1_max_px) 20.237 (mAUPRO_px) 0.242 (mF1_px_0.2_0.8_0.1) 0.237 (mAcc_px_0.2_0.8_0.1) 0.244 (mIoU_px_0.2_0.8_0.1) 0.696 (mIoU_max_px) 2024-04-04 17:46:06,880 - ==> Metric Time for chewinggum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.724 (mAUROC_px) 5.947 (mAP_px) 0.697 (mF1_max_px) 20.024 (mAUPRO_px) 0.241 (mF1_px_0.2_0.8_0.1) 0.239 (mAcc_px_0.2_0.8_0.1) 0.242 (mIoU_px_0.2_0.8_0.1) 0.693 (mIoU_max_px) 2024-04-04 17:46:42,878 - ==> Metric Time for fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.576 (mAUROC_px) 5.608 (mAP_px) 0.698 (mF1_max_px) 20.487 (mAUPRO_px) 0.239 (mF1_px_0.2_0.8_0.1) 0.239 (mAcc_px_0.2_0.8_0.1) 0.238 (mIoU_px_0.2_0.8_0.1) 0.707 (mIoU_max_px) 2024-04-04 17:47:29,085 - ==> Metric Time for macaroni1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.419 (mAUROC_px) 7.831 (mAP_px) 0.938 (mF1_max_px) 24.862 (mAUPRO_px) 0.319 (mF1_px_0.2_0.8_0.1) 0.318 (mAcc_px_0.2_0.8_0.1) 0.324 (mIoU_px_0.2_0.8_0.1) 0.930 (mIoU_max_px) 2024-04-04 17:48:15,591 - ==> Metric Time for macaroni2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.509 (mAUROC_px) 8.019 (mAP_px) 0.944 (mF1_max_px) 24.843 (mAUPRO_px) 0.326 (mF1_px_0.2_0.8_0.1) 0.323 (mAcc_px_0.2_0.8_0.1) 0.326 (mIoU_px_0.2_0.8_0.1) 0.940 (mIoU_max_px) 2024-04-04 17:49:02,893 - ==> Metric Time for pcb1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.559 (mAUROC_px) 7.959 (mAP_px) 0.941 (mF1_max_px) 25.637 (mAUPRO_px) 0.323 (mF1_px_0.2_0.8_0.1) 0.327 (mAcc_px_0.2_0.8_0.1) 0.336 (mIoU_px_0.2_0.8_0.1) 0.943 (mIoU_max_px) 2024-04-04 17:49:50,724 - ==> Metric Time for pcb2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.671 (mAUROC_px) 8.166 (mAP_px) 0.947 (mF1_max_px) 25.868 (mAUPRO_px) 0.323 (mF1_px_0.2_0.8_0.1) 0.322 (mAcc_px_0.2_0.8_0.1) 0.322 (mIoU_px_0.2_0.8_0.1) 0.940 (mIoU_max_px) 2024-04-04 17:50:38,515 - ==> Metric Time for pcb3 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.493 (mAUROC_px) 7.833 (mAP_px) 0.952 (mF1_max_px) 26.333 (mAUPRO_px) 0.319 (mF1_px_0.2_0.8_0.1) 0.323 (mAcc_px_0.2_0.8_0.1) 0.326 (mIoU_px_0.2_0.8_0.1) 0.937 (mIoU_max_px) 2024-04-04 17:51:29,273 - ==> Metric Time for pcb4 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.502 (mAUROC_px) 7.960 (mAP_px) 0.992 (mF1_max_px) 29.052 (mAUPRO_px) 0.363 (mF1_px_0.2_0.8_0.1) 0.323 (mAcc_px_0.2_0.8_0.1) 0.324 (mIoU_px_0.2_0.8_0.1) 0.961 (mIoU_max_px) 2024-04-04 17:52:05,544 - ==> Metric Time for pipe_fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.761 (mAUROC_px) 5.824 (mAP_px) 0.728 (mF1_max_px) 20.344 (mAUPRO_px) 0.238 (mF1_px_0.2_0.8_0.1) 0.240 (mAcc_px_0.2_0.8_0.1) 0.232 (mIoU_px_0.2_0.8_0.1) 0.691 (mIoU_max_px) 2024-04-04 17:52:05,565 - | Name | mAUROC_sp_max | mAUROC_sp_max (Max) | mAP_sp_max | mAP_sp_max (Max) | mF1_max_sp_max | mF1_max_sp_max (Max) | mAUROC_px | mAUROC_px (Max) | mAP_px | mAP_px (Max) | mF1_max_px | mF1_max_px (Max) | mAUPRO_px | mAUPRO_px (Max) | mF1_px_0.2_0.8_0.1 | mF1_px_0.2_0.8_0.1 (Max) | mAcc_px_0.2_0.8_0.1 | mAcc_px_0.2_0.8_0.1 (Max) | mIoU_px_0.2_0.8_0.1 | mIoU_px_0.2_0.8_0.1 (Max) | mIoU_max_px | mIoU_max_px (Max) | |:----------:|:---------------:|:---------------------:|:------------:|:------------------:|:----------------:|:----------------------:|:-----------:|:------------------:|:--------:|:------------------:|:------------:|:------------------:|:-----------:|:------------------:|:--------------------:|:--------------------------:|:---------------------:|:---------------------------:|:---------------------:|:---------------------------:|:-------------:|:-------------------:| | candle | 96.030 | 97.100 (50 epoch) | 96.361 | 97.346 (50 epoch) | 89.005 | 91.542 (50 epoch) | 99.058 | 99.098 (250 epoch) | 22.306 | 23.463 (150 epoch) | 31.112 | 32.654 (200 epoch) | 95.294 | 96.087 (300 epoch) | 21.111 | 21.111 (450 epoch) | 40.661 | 40.661 (450 epoch) | 12.065 | 12.065 (450 epoch) | 18.422 | 19.513 (200 epoch) | | capsules | 84.233 | 91.750 (100 epoch) | 89.812 | 94.964 (100 epoch) | 85.149 | 88.780 (100 epoch) | 97.897 | 99.123 (100 epoch) | 38.474 | 61.256 (100 epoch) | 42.315 | 59.868 (250 epoch) | 83.709 | 91.811 (100 epoch) | 20.163 | 33.186 (100 epoch) | 60.539 | 62.576 (50 epoch) | 11.893 | 21.701 (100 epoch) | 26.835 | 42.722 (250 epoch) | | cashew | 93.980 | 94.800 (150 epoch) | 96.889 | 97.492 (150 epoch) | 92.462 | 92.462 (450 epoch) | 94.953 | 95.614 (250 epoch) | 42.476 | 50.952 (250 epoch) | 43.240 | 54.438 (400 epoch) | 86.449 | 89.185 (350 epoch) | 23.738 | 28.352 (350 epoch) | 29.881 | 34.522 (150 epoch) | 14.337 | 17.797 (350 epoch) | 27.583 | 37.398 (400 epoch) | | chewinggum | 96.782 | 97.802 (300 epoch) | 98.579 | 98.939 (300 epoch) | 94.792 | 94.792 (450 epoch) | 97.998 | 98.145 (100 epoch) | 49.848 | 59.823 (50 epoch) | 56.394 | 61.088 (50 epoch) | 80.174 | 80.312 (400 epoch) | 29.294 | 35.748 (50 epoch) | 57.224 | 65.611 (50 epoch) | 18.851 | 23.754 (50 epoch) | 39.270 | 43.976 (50 epoch) | | fryum | 96.200 | 96.200 (450 epoch) | 98.136 | 98.136 (450 epoch) | 92.823 | 93.204 (400 epoch) | 97.197 | 97.197 (450 epoch) | 50.218 | 50.218 (450 epoch) | 52.681 | 52.992 (250 epoch) | 92.074 | 92.074 (450 epoch) | 26.452 | 26.452 (450 epoch) | 28.594 | 28.594 (450 epoch) | 16.462 | 16.462 (450 epoch) | 35.760 | 36.047 (250 epoch) | | macaroni1 | 91.983 | 92.084 (300 epoch) | 90.144 | 90.374 (300 epoch) | 83.978 | 83.978 (450 epoch) | 99.592 | 99.592 (450 epoch) | 19.436 | 19.436 (450 epoch) | 29.010 | 29.377 (200 epoch) | 96.043 | 96.043 (450 epoch) | 15.503 | 15.803 (300 epoch) | 37.556 | 40.765 (50 epoch) | 8.650 | 8.828 (300 epoch) | 16.966 | 17.217 (200 epoch) | | macaroni2 | 79.514 | 82.589 (50 epoch) | 70.698 | 81.196 (50 epoch) | 74.178 | 75.983 (400 epoch) | 97.469 | 99.452 (100 epoch) | 1.156 | 11.272 (50 epoch) | 4.770 | 17.568 (50 epoch) | 88.757 | 96.253 (150 epoch) | 1.506 | 10.086 (50 epoch) | 54.584 | 54.584 (450 epoch) | 0.763 | 5.433 (50 epoch) | 2.443 | 9.630 (50 epoch) | | pcb1 | 96.510 | 96.710 (300 epoch) | 95.321 | 95.525 (400 epoch) | 94.231 | 94.737 (300 epoch) | 99.805 | 99.805 (450 epoch) | 83.141 | 83.141 (450 epoch) | 75.588 | 75.588 (450 epoch) | 93.678 | 94.357 (300 epoch) | 47.720 | 47.720 (450 epoch) | 58.193 | 62.362 (50 epoch) | 33.816 | 33.816 (450 epoch) | 60.756 | 60.756 (450 epoch) | | pcb2 | 95.210 | 95.360 (400 epoch) | 94.919 | 95.265 (400 epoch) | 90.385 | 90.385 (450 epoch) | 99.003 | 99.003 (450 epoch) | 12.864 | 14.130 (50 epoch) | 23.894 | 23.977 (250 epoch) | 90.136 | 90.754 (250 epoch) | 15.885 | 16.150 (200 epoch) | 46.581 | 47.820 (150 epoch) | 8.737 | 8.892 (200 epoch) | 13.568 | 13.621 (250 epoch) | | pcb3 | 95.406 | 95.891 (350 epoch) | 95.611 | 96.056 (350 epoch) | 88.670 | 89.855 (400 epoch) | 99.121 | 99.142 (350 epoch) | 18.755 | 20.249 (150 epoch) | 28.078 | 28.078 (450 epoch) | 91.724 | 91.817 (350 epoch) | 19.901 | 20.042 (150 epoch) | 45.738 | 48.341 (50 epoch) | 11.227 | 11.295 (150 epoch) | 16.332 | 16.332 (450 epoch) | | pcb4 | 99.733 | 99.901 (100 epoch) | 99.731 | 99.901 (100 epoch) | 97.561 | 99.000 (50 epoch) | 98.686 | 98.727 (250 epoch) | 47.059 | 47.966 (250 epoch) | 47.129 | 47.709 (250 epoch) | 89.055 | 90.729 (250 epoch) | 25.221 | 26.925 (200 epoch) | 33.512 | 39.716 (50 epoch) | 15.476 | 16.537 (200 epoch) | 30.829 | 31.328 (250 epoch) | | pipe_fryum | 98.860 | 98.900 (350 epoch) | 99.415 | 99.421 (350 epoch) | 96.618 | 97.561 (200 epoch) | 99.114 | 99.114 (450 epoch) | 54.998 | 54.998 (450 epoch) | 58.733 | 58.733 (450 epoch) | 94.762 | 95.525 (150 epoch) | 31.913 | 34.684 (250 epoch) | 36.550 | 43.014 (100 epoch) | 20.668 | 22.399 (250 epoch) | 41.576 | 41.576 (450 epoch) | | Avg | 93.703 | 94.251 (100 epoch) | 93.801 | 94.481 (100 epoch) | 89.988 | 90.163 (300 epoch) | 98.324 | 98.606 (250 epoch) | 36.728 | 39.701 (250 epoch) | 41.079 | 44.125 (250 epoch) | 90.155 | 91.494 (250 epoch) | 23.201 | 25.164 (100 epoch) | 44.134 | 46.179 (50 epoch) | 14.412 | 15.755 (100 epoch) | 27.528 | 29.990 (250 epoch) | 2024-04-04 17:52:05,909 - ==> Total time: 2 days, 16:52:00 Eta: 3 days, 7:16:53 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 17:53:20,247 - Train: 45.01% [243500/541000] [450.1/1000.0] [batch_t 1.414 (1.432)] [data_t 0.002] [optim_t 1.412] [lr 0.005000] 2024-04-04 17:55:41,728 - Train: 45.03% [243600/541000] [450.3/1000.0] [batch_t 1.411 (1.415)] [data_t 0.002] [optim_t 1.408] [lr 0.005000] 2024-04-04 17:58:03,196 - Train: 45.05% [243700/541000] [450.5/1000.0] [batch_t 1.410 (1.415)] [data_t 0.002] [optim_t 1.408] [lr 0.005000] 2024-04-04 18:00:24,706 - Train: 45.06% [243800/541000] [450.6/1000.0] [batch_t 1.414 (1.415)] [data_t 0.002] [optim_t 1.412] [lr 0.005000] 2024-04-04 18:02:46,250 - Train: 45.08% [243900/541000] [450.8/1000.0] [batch_t 1.414 (1.415)] [data_t 0.002] [optim_t 1.412] [lr 0.005000] 2024-04-04 18:04:54,998 - ==> Total time: 2 days, 17:04:49 Eta: 3 days, 7:13:19 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 18:05:09,836 - Train: 45.10% [244000/541000] [451.0/1000.0] [batch_t 1.417 (1.462)] [data_t 0.002] [optim_t 1.415] [lr 0.005000] 2024-04-04 18:07:31,209 - Train: 45.12% [244100/541000] [451.2/1000.0] [batch_t 1.414 (1.414)] [data_t 0.004] [optim_t 1.410] [lr 0.005000] 2024-04-04 18:09:52,694 - Train: 45.14% [244200/541000] [451.4/1000.0] [batch_t 1.414 (1.415)] [data_t 0.002] [optim_t 1.411] [lr 0.005000] 2024-04-04 18:12:14,145 - Train: 45.16% [244300/541000] [451.6/1000.0] [batch_t 1.407 (1.414)] [data_t 0.002] [optim_t 1.405] [lr 0.005000] 2024-04-04 18:14:35,600 - Train: 45.18% [244400/541000] [451.8/1000.0] [batch_t 1.417 (1.414)] [data_t 0.002] [optim_t 1.415] [lr 0.005000] 2024-04-04 18:16:57,116 - Train: 45.19% [244500/541000] [451.9/1000.0] [batch_t 1.404 (1.415)] [data_t 0.002] [optim_t 1.402] [lr 0.005000] 2024-04-04 18:17:42,382 - ==> Total time: 2 days, 17:17:36 Eta: 3 days, 7:09:40 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 18:19:20,862 - Train: 45.21% [244600/541000] [452.1/1000.0] [batch_t 1.405 (1.424)] [data_t 0.002] [optim_t 1.402] [lr 0.005000] 2024-04-04 18:21:42,301 - Train: 45.23% [244700/541000] [452.3/1000.0] [batch_t 1.415 (1.414)] [data_t 0.002] [optim_t 1.413] [lr 0.005000] 2024-04-04 18:24:03,729 - Train: 45.25% [244800/541000] [452.5/1000.0] [batch_t 1.404 (1.414)] [data_t 0.002] [optim_t 1.401] [lr 0.005000] 2024-04-04 18:26:25,161 - Train: 45.27% [244900/541000] [452.7/1000.0] [batch_t 1.412 (1.414)] [data_t 0.002] [optim_t 1.409] [lr 0.005000] 2024-04-04 18:28:46,528 - Train: 45.29% [245000/541000] [452.9/1000.0] [batch_t 1.416 (1.414)] [data_t 0.002] [optim_t 1.414] [lr 0.005000] 2024-04-04 18:30:30,912 - ==> Total time: 2 days, 17:30:25 Eta: 3 days, 7:06:00 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 18:31:12,250 - Train: 45.30% [245100/541000] [453.0/1000.0] [batch_t 1.461 (1.468)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-04 18:33:37,910 - Train: 45.32% [245200/541000] [453.2/1000.0] [batch_t 1.455 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-04 18:36:03,565 - Train: 45.34% [245300/541000] [453.4/1000.0] [batch_t 1.460 (1.456)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-04 18:38:29,194 - Train: 45.36% [245400/541000] [453.6/1000.0] [batch_t 1.459 (1.456)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-04 18:40:54,790 - Train: 45.38% [245500/541000] [453.8/1000.0] [batch_t 1.454 (1.456)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-04 18:43:20,410 - Train: 45.40% [245600/541000] [454.0/1000.0] [batch_t 1.459 (1.456)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-04 18:43:40,801 - ==> Total time: 2 days, 17:43:35 Eta: 3 days, 7:02:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 18:45:48,555 - Train: 45.42% [245700/541000] [454.2/1000.0] [batch_t 1.453 (1.461)] [data_t 0.003] [optim_t 1.450] [lr 0.005000] 2024-04-04 18:48:14,193 - Train: 45.43% [245800/541000] [454.3/1000.0] [batch_t 1.460 (1.456)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-04 18:50:39,809 - Train: 45.45% [245900/541000] [454.5/1000.0] [batch_t 1.459 (1.456)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-04 18:53:05,366 - Train: 45.47% [246000/541000] [454.7/1000.0] [batch_t 1.461 (1.455)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-04 18:55:31,069 - Train: 45.49% [246100/541000] [454.9/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-04 18:56:51,150 - ==> Total time: 2 days, 17:56:45 Eta: 3 days, 6:59:24 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 18:57:59,194 - Train: 45.51% [246200/541000] [455.1/1000.0] [batch_t 1.462 (1.474)] [data_t 0.003] [optim_t 1.459] [lr 0.005000] 2024-04-04 19:00:24,906 - Train: 45.53% [246300/541000] [455.3/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-04 19:02:50,636 - Train: 45.55% [246400/541000] [455.5/1000.0] [batch_t 1.457 (1.457)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-04 19:05:16,305 - Train: 45.56% [246500/541000] [455.6/1000.0] [batch_t 1.457 (1.457)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-04 19:07:42,017 - Train: 45.58% [246600/541000] [455.8/1000.0] [batch_t 1.454 (1.457)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-04 19:10:01,860 - ==> Total time: 2 days, 18:09:56 Eta: 3 days, 6:56:04 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 19:10:12,371 - Train: 45.60% [246700/541000] [456.0/1000.0] [batch_t 1.462 (1.568)] [data_t 0.003] [optim_t 1.459] [lr 0.005000] 2024-04-04 19:12:37,963 - Train: 45.62% [246800/541000] [456.2/1000.0] [batch_t 1.455 (1.456)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-04 19:15:03,614 - Train: 45.64% [246900/541000] [456.4/1000.0] [batch_t 1.456 (1.456)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-04 19:17:29,293 - Train: 45.66% [247000/541000] [456.6/1000.0] [batch_t 1.458 (1.457)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-04 19:19:54,981 - Train: 45.67% [247100/541000] [456.7/1000.0] [batch_t 1.453 (1.457)] [data_t 0.003] [optim_t 1.450] [lr 0.005000] 2024-04-04 19:22:20,676 - Train: 45.69% [247200/541000] [456.9/1000.0] [batch_t 1.455 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-04 19:23:14,597 - ==> Total time: 2 days, 18:23:09 Eta: 3 days, 6:52:42 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 19:24:48,426 - Train: 45.71% [247300/541000] [457.1/1000.0] [batch_t 1.456 (1.462)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-04 19:27:14,095 - Train: 45.73% [247400/541000] [457.3/1000.0] [batch_t 1.454 (1.457)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-04 19:29:39,787 - Train: 45.75% [247500/541000] [457.5/1000.0] [batch_t 1.455 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-04 19:32:05,440 - Train: 45.77% [247600/541000] [457.7/1000.0] [batch_t 1.451 (1.456)] [data_t 0.003] [optim_t 1.447] [lr 0.005000] 2024-04-04 19:34:31,094 - Train: 45.79% [247700/541000] [457.9/1000.0] [batch_t 1.455 (1.456)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-04 19:36:24,707 - ==> Total time: 2 days, 18:36:19 Eta: 3 days, 6:49:16 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 19:36:59,339 - Train: 45.80% [247800/541000] [458.0/1000.0] [batch_t 1.460 (1.496)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-04 19:39:25,002 - Train: 45.82% [247900/541000] [458.2/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-04 19:41:50,675 - Train: 45.84% [248000/541000] [458.4/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-04 19:44:15,178 - Train: 45.86% [248100/541000] [458.6/1000.0] [batch_t 1.410 (1.445)] [data_t 0.002] [optim_t 1.407] [lr 0.005000] 2024-04-04 19:46:36,831 - Train: 45.88% [248200/541000] [458.8/1000.0] [batch_t 1.417 (1.416)] [data_t 0.002] [optim_t 1.415] [lr 0.005000] 2024-04-04 19:48:58,445 - Train: 45.90% [248300/541000] [459.0/1000.0] [batch_t 1.418 (1.416)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-04 19:49:25,369 - ==> Total time: 2 days, 18:49:19 Eta: 3 days, 6:45:35 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 19:51:22,230 - Train: 45.91% [248400/541000] [459.1/1000.0] [batch_t 1.417 (1.422)] [data_t 0.002] [optim_t 1.415] [lr 0.005000] 2024-04-04 19:53:43,818 - Train: 45.93% [248500/541000] [459.3/1000.0] [batch_t 1.409 (1.416)] [data_t 0.002] [optim_t 1.407] [lr 0.005000] 2024-04-04 19:56:05,429 - Train: 45.95% [248600/541000] [459.5/1000.0] [batch_t 1.406 (1.416)] [data_t 0.002] [optim_t 1.404] [lr 0.005000] 2024-04-04 19:58:27,140 - Train: 45.97% [248700/541000] [459.7/1000.0] [batch_t 1.419 (1.417)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-04 20:00:48,795 - Train: 45.99% [248800/541000] [459.9/1000.0] [batch_t 1.419 (1.416)] [data_t 0.003] [optim_t 1.417] [lr 0.005000] 2024-04-04 20:02:13,779 - ==> Total time: 2 days, 19:02:08 Eta: 3 days, 6:41:38 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 20:03:12,688 - Train: 46.01% [248900/541000] [460.1/1000.0] [batch_t 1.419 (1.431)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-04 20:05:34,280 - Train: 46.03% [249000/541000] [460.3/1000.0] [batch_t 1.420 (1.416)] [data_t 0.002] [optim_t 1.418] [lr 0.005000] 2024-04-04 20:07:55,871 - Train: 46.04% [249100/541000] [460.4/1000.0] [batch_t 1.414 (1.416)] [data_t 0.002] [optim_t 1.412] [lr 0.005000] 2024-04-04 20:09:53,424 - Train: 46.06% [249200/541000] [460.6/1000.0] [batch_t 0.606 (1.175)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 20:10:54,172 - Train: 46.08% [249300/541000] [460.8/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 20:11:54,870 - Train: 46.10% [249400/541000] [461.0/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 20:11:55,481 - ==> Total time: 2 days, 19:11:49 Eta: 3 days, 6:34:00 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 20:12:57,807 - Train: 46.12% [249500/541000] [461.2/1000.0] [batch_t 0.605 (0.610)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 20:13:58,488 - Train: 46.14% [249600/541000] [461.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 20:14:59,154 - Train: 46.16% [249700/541000] [461.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 20:15:59,813 - Train: 46.17% [249800/541000] [461.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 20:17:00,592 - Train: 46.19% [249900/541000] [461.9/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 20:17:26,062 - ==> Total time: 2 days, 19:17:20 Eta: 3 days, 6:21:29 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 20:18:03,365 - Train: 46.21% [250000/541000] [462.1/1000.0] [batch_t 0.607 (0.613)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 20:19:04,042 - Train: 46.23% [250100/541000] [462.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 20:20:04,706 - Train: 46.25% [250200/541000] [462.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 20:21:05,367 - Train: 46.27% [250300/541000] [462.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 20:22:06,164 - Train: 46.28% [250400/541000] [462.8/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 20:22:56,518 - ==> Total time: 2 days, 19:22:50 Eta: 3 days, 6:09:00 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 20:23:08,818 - Train: 46.30% [250500/541000] [463.0/1000.0] [batch_t 0.606 (0.624)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 20:24:09,489 - Train: 46.32% [250600/541000] [463.2/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 20:25:10,178 - Train: 46.34% [250700/541000] [463.4/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 20:26:10,866 - Train: 46.36% [250800/541000] [463.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 20:27:11,580 - Train: 46.38% [250900/541000] [463.8/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 20:28:12,257 - Train: 46.40% [251000/541000] [464.0/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 20:28:26,819 - ==> Total time: 2 days, 19:28:21 Eta: 3 days, 5:56:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 20:29:15,354 - Train: 46.41% [251100/541000] [464.1/1000.0] [batch_t 0.606 (0.614)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 20:30:16,004 - Train: 46.43% [251200/541000] [464.3/1000.0] [batch_t 0.607 (0.606)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 20:31:16,676 - Train: 46.45% [251300/541000] [464.5/1000.0] [batch_t 0.612 (0.607)] [data_t 0.002] [optim_t 0.610] [lr 0.005000] 2024-04-04 20:32:17,333 - Train: 46.47% [251400/541000] [464.7/1000.0] [batch_t 0.607 (0.606)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 20:33:17,982 - Train: 46.49% [251500/541000] [464.9/1000.0] [batch_t 0.607 (0.606)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 20:33:57,408 - ==> Total time: 2 days, 19:33:51 Eta: 3 days, 5:44:07 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 20:34:20,754 - Train: 46.51% [251600/541000] [465.1/1000.0] [batch_t 0.606 (0.618)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 20:35:21,515 - Train: 46.52% [251700/541000] [465.2/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 20:36:22,158 - Train: 46.54% [251800/541000] [465.4/1000.0] [batch_t 0.606 (0.606)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 20:37:22,841 - Train: 46.56% [251900/541000] [465.6/1000.0] [batch_t 0.612 (0.607)] [data_t 0.002] [optim_t 0.610] [lr 0.005000] 2024-04-04 20:38:23,519 - Train: 46.58% [252000/541000] [465.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 20:39:24,235 - Train: 46.60% [252100/541000] [466.0/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 20:39:27,881 - ==> Total time: 2 days, 19:39:22 Eta: 3 days, 5:31:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 20:40:27,241 - Train: 46.62% [252200/541000] [466.2/1000.0] [batch_t 0.607 (0.611)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 20:41:27,935 - Train: 46.64% [252300/541000] [466.4/1000.0] [batch_t 0.608 (0.607)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-04 20:42:28,666 - Train: 46.65% [252400/541000] [466.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 20:43:29,353 - Train: 46.67% [252500/541000] [466.7/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 20:44:30,030 - Train: 46.69% [252600/541000] [466.9/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 20:44:58,553 - ==> Total time: 2 days, 19:44:52 Eta: 3 days, 5:19:21 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 20:45:33,153 - Train: 46.71% [252700/541000] [467.1/1000.0] [batch_t 0.611 (0.619)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-04 20:46:33,996 - Train: 46.73% [252800/541000] [467.3/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 20:47:34,683 - Train: 46.75% [252900/541000] [467.5/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 20:48:35,335 - Train: 46.77% [253000/541000] [467.7/1000.0] [batch_t 0.607 (0.606)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 20:49:35,987 - Train: 46.78% [253100/541000] [467.8/1000.0] [batch_t 0.607 (0.606)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 20:50:29,359 - ==> Total time: 2 days, 19:50:23 Eta: 3 days, 5:07:01 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 20:50:38,941 - Train: 46.80% [253200/541000] [468.0/1000.0] [batch_t 0.605 (0.645)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 20:51:39,604 - Train: 46.82% [253300/541000] [468.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 20:52:40,338 - Train: 46.84% [253400/541000] [468.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 20:53:41,006 - Train: 46.86% [253500/541000] [468.6/1000.0] [batch_t 0.611 (0.607)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-04 20:54:41,651 - Train: 46.88% [253600/541000] [468.8/1000.0] [batch_t 0.607 (0.606)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 20:55:42,342 - Train: 46.89% [253700/541000] [468.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 20:55:59,944 - ==> Total time: 2 days, 19:55:54 Eta: 3 days, 4:54:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 20:56:46,267 - Train: 46.91% [253800/541000] [469.1/1000.0] [batch_t 0.606 (0.625)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 20:57:46,963 - Train: 46.93% [253900/541000] [469.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 20:58:47,742 - Train: 46.95% [254000/541000] [469.5/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 20:59:48,403 - Train: 46.97% [254100/541000] [469.7/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 21:00:49,075 - Train: 46.99% [254200/541000] [469.9/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 21:01:31,544 - ==> Total time: 2 days, 20:01:25 Eta: 3 days, 4:42:28 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 21:01:51,998 - Train: 47.01% [254300/541000] [470.1/1000.0] [batch_t 0.608 (0.619)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 21:02:52,657 - Train: 47.02% [254400/541000] [470.2/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 21:03:53,471 - Train: 47.04% [254500/541000] [470.4/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 21:04:54,166 - Train: 47.06% [254600/541000] [470.6/1000.0] [batch_t 0.611 (0.607)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-04 21:05:54,881 - Train: 47.08% [254700/541000] [470.8/1000.0] [batch_t 0.611 (0.607)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-04 21:06:55,564 - Train: 47.10% [254800/541000] [471.0/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 21:07:02,246 - ==> Total time: 2 days, 20:06:56 Eta: 3 days, 4:30:13 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 21:08:00,450 - Train: 47.12% [254900/541000] [471.2/1000.0] [batch_t 0.608 (0.614)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 21:09:01,140 - Train: 47.13% [255000/541000] [471.3/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 21:10:01,920 - Train: 47.15% [255100/541000] [471.5/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 21:11:02,613 - Train: 47.17% [255200/541000] [471.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 21:12:03,301 - Train: 47.19% [255300/541000] [471.9/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 21:12:34,854 - ==> Total time: 2 days, 20:12:29 Eta: 3 days, 4:18:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 21:13:06,210 - Train: 47.21% [255400/541000] [472.1/1000.0] [batch_t 0.608 (0.615)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 21:14:06,892 - Train: 47.23% [255500/541000] [472.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 21:15:07,552 - Train: 47.25% [255600/541000] [472.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 21:16:08,318 - Train: 47.26% [255700/541000] [472.6/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-04 21:17:08,981 - Train: 47.28% [255800/541000] [472.8/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 21:18:05,401 - ==> Total time: 2 days, 20:17:59 Eta: 3 days, 4:05:50 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 21:18:12,427 - Train: 47.30% [255900/541000] [473.0/1000.0] [batch_t 0.607 (0.696)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 21:19:25,331 - Train: 47.32% [256000/541000] [473.2/1000.0] [batch_t 0.607 (0.729)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 21:20:26,021 - Train: 47.34% [256100/541000] [473.4/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 21:21:26,707 - Train: 47.36% [256200/541000] [473.6/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 21:22:27,593 - Train: 47.38% [256300/541000] [473.8/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 21:23:28,313 - Train: 47.39% [256400/541000] [473.9/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 21:23:48,963 - ==> Total time: 2 days, 20:23:43 Eta: 3 days, 3:53:55 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 21:24:39,004 - Train: 47.41% [256500/541000] [474.1/1000.0] [batch_t 0.607 (0.731)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 21:25:39,686 - Train: 47.43% [256600/541000] [474.3/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 21:26:40,387 - Train: 47.45% [256700/541000] [474.5/1000.0] [batch_t 0.609 (0.607)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-04 21:27:41,247 - Train: 47.47% [256800/541000] [474.7/1000.0] [batch_t 0.607 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 21:28:41,973 - Train: 47.49% [256900/541000] [474.9/1000.0] [batch_t 0.610 (0.607)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-04 21:29:27,492 - ==> Total time: 2 days, 20:29:21 Eta: 3 days, 3:41:55 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 21:29:45,102 - Train: 47.50% [257000/541000] [475.0/1000.0] [batch_t 0.606 (0.626)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 21:30:45,778 - Train: 47.52% [257100/541000] [475.2/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 21:31:46,476 - Train: 47.54% [257200/541000] [475.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 21:32:47,178 - Train: 47.56% [257300/541000] [475.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 21:33:48,039 - Train: 47.58% [257400/541000] [475.8/1000.0] [batch_t 0.610 (0.609)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-04 21:34:48,717 - Train: 47.60% [257500/541000] [476.0/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 21:34:58,428 - ==> Total time: 2 days, 20:34:52 Eta: 3 days, 3:29:49 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 21:35:51,769 - Train: 47.62% [257600/541000] [476.2/1000.0] [batch_t 0.606 (0.614)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 21:36:52,444 - Train: 47.63% [257700/541000] [476.3/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 21:37:53,169 - Train: 47.65% [257800/541000] [476.5/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 21:38:53,872 - Train: 47.67% [257900/541000] [476.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 21:39:54,716 - Train: 47.69% [258000/541000] [476.9/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 21:40:29,359 - ==> Total time: 2 days, 20:40:23 Eta: 3 days, 3:17:45 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 21:40:57,548 - Train: 47.71% [258100/541000] [477.1/1000.0] [batch_t 0.607 (0.617)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 21:41:58,259 - Train: 47.73% [258200/541000] [477.3/1000.0] [batch_t 0.612 (0.607)] [data_t 0.002] [optim_t 0.610] [lr 0.005000] 2024-04-04 21:42:58,951 - Train: 47.74% [258300/541000] [477.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 21:43:59,688 - Train: 47.76% [258400/541000] [477.6/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 21:45:04,258 - Train: 47.78% [258500/541000] [477.8/1000.0] [batch_t 0.606 (0.646)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 21:46:03,899 - ==> Total time: 2 days, 20:45:58 Eta: 3 days, 3:05:46 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 21:46:08,134 - Train: 47.80% [258600/541000] [478.0/1000.0] [batch_t 0.606 (0.868)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 21:47:08,885 - Train: 47.82% [258700/541000] [478.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 21:48:09,595 - Train: 47.84% [258800/541000] [478.4/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 21:49:10,272 - Train: 47.86% [258900/541000] [478.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 21:50:10,972 - Train: 47.87% [259000/541000] [478.7/1000.0] [batch_t 0.608 (0.607)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-04 21:51:11,856 - Train: 47.89% [259100/541000] [478.9/1000.0] [batch_t 0.611 (0.609)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-04 21:51:35,512 - ==> Total time: 2 days, 20:51:29 Eta: 3 days, 2:53:45 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 21:52:14,961 - Train: 47.91% [259200/541000] [479.1/1000.0] [batch_t 0.607 (0.613)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 21:53:15,657 - Train: 47.93% [259300/541000] [479.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 21:54:16,406 - Train: 47.95% [259400/541000] [479.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-04 21:55:17,113 - Train: 47.97% [259500/541000] [479.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 21:56:17,805 - Train: 47.99% [259600/541000] [479.9/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 21:57:06,361 - ==> Total time: 2 days, 20:57:00 Eta: 3 days, 2:41:45 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 21:57:20,805 - Train: 48.00% [259700/541000] [480.0/1000.0] [batch_t 0.605 (0.627)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 21:58:21,471 - Train: 48.02% [259800/541000] [480.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 21:59:22,194 - Train: 48.04% [259900/541000] [480.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 22:00:22,936 - Train: 48.06% [260000/541000] [480.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 22:01:23,623 - Train: 48.08% [260100/541000] [480.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 22:02:24,308 - Train: 48.10% [260200/541000] [481.0/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 22:02:37,047 - ==> Total time: 2 days, 21:02:31 Eta: 3 days, 2:29:47 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 22:03:27,196 - Train: 48.11% [260300/541000] [481.1/1000.0] [batch_t 0.607 (0.613)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 22:04:28,042 - Train: 48.13% [260400/541000] [481.3/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 22:05:28,738 - Train: 48.15% [260500/541000] [481.5/1000.0] [batch_t 0.611 (0.607)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-04 22:06:29,454 - Train: 48.17% [260600/541000] [481.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 22:07:30,180 - Train: 48.19% [260700/541000] [481.9/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 22:08:07,807 - ==> Total time: 2 days, 21:08:02 Eta: 3 days, 2:17:50 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 22:08:33,140 - Train: 48.21% [260800/541000] [482.1/1000.0] [batch_t 0.604 (0.618)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 22:09:33,801 - Train: 48.23% [260900/541000] [482.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 22:10:34,456 - Train: 48.24% [261000/541000] [482.4/1000.0] [batch_t 0.606 (0.606)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 22:11:35,249 - Train: 48.26% [261100/541000] [482.6/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 22:12:35,938 - Train: 48.28% [261200/541000] [482.8/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 22:13:36,662 - Train: 48.30% [261300/541000] [483.0/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 22:13:38,487 - ==> Total time: 2 days, 21:13:32 Eta: 3 days, 2:05:55 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 22:14:40,117 - Train: 48.32% [261400/541000] [483.2/1000.0] [batch_t 0.607 (0.612)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 22:15:40,849 - Train: 48.34% [261500/541000] [483.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 22:16:41,574 - Train: 48.35% [261600/541000] [483.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-04 22:17:42,416 - Train: 48.37% [261700/541000] [483.7/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 22:18:43,155 - Train: 48.39% [261800/541000] [483.9/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 22:19:09,857 - ==> Total time: 2 days, 21:19:04 Eta: 3 days, 1:54:03 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 22:19:46,149 - Train: 48.41% [261900/541000] [484.1/1000.0] [batch_t 0.606 (0.614)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 22:20:46,864 - Train: 48.43% [262000/541000] [484.3/1000.0] [batch_t 0.611 (0.607)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-04 22:21:47,537 - Train: 48.45% [262100/541000] [484.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 22:22:48,386 - Train: 48.47% [262200/541000] [484.7/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-04 22:23:49,124 - Train: 48.48% [262300/541000] [484.8/1000.0] [batch_t 0.610 (0.607)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-04 22:24:40,723 - ==> Total time: 2 days, 21:24:35 Eta: 3 days, 1:42:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 22:24:51,901 - Train: 48.50% [262400/541000] [485.0/1000.0] [batch_t 0.607 (0.633)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 22:25:52,619 - Train: 48.52% [262500/541000] [485.2/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 22:26:53,293 - Train: 48.54% [262600/541000] [485.4/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 22:27:53,990 - Train: 48.56% [262700/541000] [485.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 22:28:54,685 - Train: 48.58% [262800/541000] [485.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 22:29:55,518 - Train: 48.60% [262900/541000] [486.0/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 22:30:11,309 - ==> Total time: 2 days, 21:30:05 Eta: 3 days, 1:30:20 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 22:30:58,375 - Train: 48.61% [263000/541000] [486.1/1000.0] [batch_t 0.607 (0.612)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 22:31:59,090 - Train: 48.63% [263100/541000] [486.3/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 22:32:59,782 - Train: 48.65% [263200/541000] [486.5/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 22:34:00,482 - Train: 48.67% [263300/541000] [486.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 22:35:01,301 - Train: 48.69% [263400/541000] [486.9/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 22:35:41,957 - ==> Total time: 2 days, 21:35:36 Eta: 3 days, 1:18:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 22:36:04,224 - Train: 48.71% [263500/541000] [487.1/1000.0] [batch_t 0.606 (0.619)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 22:37:04,931 - Train: 48.72% [263600/541000] [487.2/1000.0] [batch_t 0.608 (0.607)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-04 22:38:05,600 - Train: 48.74% [263700/541000] [487.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 22:39:06,302 - Train: 48.76% [263800/541000] [487.6/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 22:40:06,994 - Train: 48.78% [263900/541000] [487.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 22:41:07,841 - Train: 48.80% [264000/541000] [488.0/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 22:41:12,699 - ==> Total time: 2 days, 21:41:07 Eta: 3 days, 1:06:44 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 22:42:10,719 - Train: 48.82% [264100/541000] [488.2/1000.0] [batch_t 0.609 (0.611)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 22:43:11,439 - Train: 48.84% [264200/541000] [488.4/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 22:44:12,139 - Train: 48.85% [264300/541000] [488.5/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 22:45:12,840 - Train: 48.87% [264400/541000] [488.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 22:46:13,519 - Train: 48.89% [264500/541000] [488.9/1000.0] [batch_t 0.608 (0.607)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-04 22:46:43,261 - ==> Total time: 2 days, 21:46:37 Eta: 3 days, 0:54:59 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 22:47:16,710 - Train: 48.91% [264600/541000] [489.1/1000.0] [batch_t 0.606 (0.621)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 22:48:17,396 - Train: 48.93% [264700/541000] [489.3/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 22:49:18,105 - Train: 48.95% [264800/541000] [489.5/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 22:50:18,802 - Train: 48.96% [264900/541000] [489.6/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 22:51:19,508 - Train: 48.98% [265000/541000] [489.8/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 22:52:14,097 - ==> Total time: 2 days, 21:52:08 Eta: 3 days, 0:43:15 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 22:52:22,842 - Train: 49.00% [265100/541000] [490.0/1000.0] [batch_t 0.607 (0.662)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 22:53:23,560 - Train: 49.02% [265200/541000] [490.2/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 22:54:24,245 - Train: 49.04% [265300/541000] [490.4/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 22:55:24,929 - Train: 49.06% [265400/541000] [490.6/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 22:56:25,610 - Train: 49.08% [265500/541000] [490.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 22:57:26,310 - Train: 49.09% [265600/541000] [490.9/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 22:57:45,119 - ==> Total time: 2 days, 21:57:39 Eta: 3 days, 0:31:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 22:58:29,559 - Train: 49.11% [265700/541000] [491.1/1000.0] [batch_t 0.613 (0.615)] [data_t 0.002] [optim_t 0.611] [lr 0.005000] 2024-04-04 22:59:30,268 - Train: 49.13% [265800/541000] [491.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 23:00:30,987 - Train: 49.15% [265900/541000] [491.5/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 23:01:31,659 - Train: 49.17% [266000/541000] [491.7/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 23:02:32,345 - Train: 49.19% [266100/541000] [491.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 23:03:16,032 - ==> Total time: 2 days, 22:03:10 Eta: 3 days, 0:19:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 23:03:35,490 - Train: 49.21% [266200/541000] [492.1/1000.0] [batch_t 0.606 (0.624)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 23:04:36,319 - Train: 49.22% [266300/541000] [492.2/1000.0] [batch_t 0.605 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 23:05:37,062 - Train: 49.24% [266400/541000] [492.4/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 23:06:47,728 - Train: 49.26% [266500/541000] [492.6/1000.0] [batch_t 0.606 (0.707)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 23:07:48,421 - Train: 49.28% [266600/541000] [492.8/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 23:08:49,171 - Train: 49.30% [266700/541000] [493.0/1000.0] [batch_t 0.608 (0.607)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-04 23:08:57,072 - ==> Total time: 2 days, 22:08:51 Eta: 3 days, 0:08:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 23:09:52,147 - Train: 49.32% [266800/541000] [493.2/1000.0] [batch_t 0.606 (0.613)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 23:10:53,042 - Train: 49.33% [266900/541000] [493.3/1000.0] [batch_t 0.610 (0.609)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-04 23:11:53,722 - Train: 49.35% [267000/541000] [493.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 23:12:54,440 - Train: 49.37% [267100/541000] [493.7/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 23:13:55,171 - Train: 49.39% [267200/541000] [493.9/1000.0] [batch_t 0.608 (0.607)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-04 23:14:27,948 - ==> Total time: 2 days, 22:14:22 Eta: 2 days, 23:56:44 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 23:14:58,334 - Train: 49.41% [267300/541000] [494.1/1000.0] [batch_t 0.609 (0.617)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 23:15:59,191 - Train: 49.43% [267400/541000] [494.3/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 23:16:59,869 - Train: 49.45% [267500/541000] [494.5/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 23:18:00,615 - Train: 49.46% [267600/541000] [494.6/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 23:19:01,339 - Train: 49.48% [267700/541000] [494.8/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 23:19:59,025 - ==> Total time: 2 days, 22:19:53 Eta: 2 days, 23:45:08 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 23:20:04,221 - Train: 49.50% [267800/541000] [495.0/1000.0] [batch_t 0.607 (0.688)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 23:21:04,920 - Train: 49.52% [267900/541000] [495.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 23:22:05,723 - Train: 49.54% [268000/541000] [495.4/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 23:23:06,393 - Train: 49.56% [268100/541000] [495.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 23:24:07,077 - Train: 49.57% [268200/541000] [495.7/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-04 23:25:07,853 - Train: 49.59% [268300/541000] [495.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 23:25:29,740 - ==> Total time: 2 days, 22:25:24 Eta: 2 days, 23:33:33 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 23:26:10,636 - Train: 49.61% [268400/541000] [496.1/1000.0] [batch_t 0.607 (0.612)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 23:27:11,503 - Train: 49.63% [268500/541000] [496.3/1000.0] [batch_t 0.606 (0.609)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 23:28:12,217 - Train: 49.65% [268600/541000] [496.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 23:29:12,933 - Train: 49.67% [268700/541000] [496.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 23:30:13,707 - Train: 49.69% [268800/541000] [496.9/1000.0] [batch_t 0.613 (0.608)] [data_t 0.002] [optim_t 0.611] [lr 0.005000] 2024-04-04 23:31:00,504 - ==> Total time: 2 days, 22:30:54 Eta: 2 days, 23:21:59 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 23:31:18,431 - Train: 49.70% [268900/541000] [497.0/1000.0] [batch_t 0.606 (0.705)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 23:32:19,086 - Train: 49.72% [269000/541000] [497.2/1000.0] [batch_t 0.605 (0.606)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 23:33:19,892 - Train: 49.74% [269100/541000] [497.4/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 23:34:20,586 - Train: 49.76% [269200/541000] [497.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 23:35:21,299 - Train: 49.78% [269300/541000] [497.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 23:36:21,976 - Train: 49.80% [269400/541000] [498.0/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 23:36:32,905 - ==> Total time: 2 days, 22:36:27 Eta: 2 days, 23:10:29 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 23:37:24,885 - Train: 49.82% [269500/541000] [498.2/1000.0] [batch_t 0.607 (0.612)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 23:38:25,573 - Train: 49.83% [269600/541000] [498.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 23:39:26,366 - Train: 49.85% [269700/541000] [498.5/1000.0] [batch_t 0.605 (0.608)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-04 23:40:27,120 - Train: 49.87% [269800/541000] [498.7/1000.0] [batch_t 0.608 (0.607)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-04 23:41:27,832 - Train: 49.89% [269900/541000] [498.9/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 23:42:03,691 - ==> Total time: 2 days, 22:41:58 Eta: 2 days, 22:58:58 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 23:42:31,722 - Train: 49.91% [270000/541000] [499.1/1000.0] [batch_t 0.605 (0.617)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-04 23:43:32,480 - Train: 49.93% [270100/541000] [499.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 23:44:33,211 - Train: 49.94% [270200/541000] [499.4/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 23:45:34,059 - Train: 49.96% [270300/541000] [499.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-04 23:46:34,733 - Train: 49.98% [270400/541000] [499.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 23:47:35,463 - Train: 50.00% [270500/541000] [500.0/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-04 23:47:47,256 - Test: 36.76% [50/136] [batch_t 0.213 (0.225)] 2024-04-04 23:47:57,962 - Test: 73.53% [100/136] [batch_t 0.212 (0.220)] 2024-04-04 23:48:08,678 - Test: 100.00% [136/136] [batch_t 0.053 (0.240)] 2024-04-04 23:48:57,408 - ==> Metric Time for candle : 0.002 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.645 (mAUROC_px) 8.143 (mAP_px) 0.945 (mF1_max_px) 25.345 (mAUPRO_px) 0.321 (mF1_px_0.2_0.8_0.1) 0.321 (mAcc_px_0.2_0.8_0.1) 0.326 (mIoU_px_0.2_0.8_0.1) 0.946 (mIoU_max_px) 2024-04-04 23:49:35,833 - ==> Metric Time for capsules : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 8.506 (mAUROC_px) 6.522 (mAP_px) 0.781 (mF1_max_px) 20.859 (mAUPRO_px) 0.261 (mF1_px_0.2_0.8_0.1) 0.259 (mAcc_px_0.2_0.8_0.1) 0.257 (mIoU_px_0.2_0.8_0.1) 0.765 (mIoU_max_px) 2024-04-04 23:50:12,336 - ==> Metric Time for cashew : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 8.053 (mAUROC_px) 6.241 (mAP_px) 0.732 (mF1_max_px) 19.841 (mAUPRO_px) 0.240 (mF1_px_0.2_0.8_0.1) 0.242 (mAcc_px_0.2_0.8_0.1) 0.242 (mIoU_px_0.2_0.8_0.1) 0.700 (mIoU_max_px) 2024-04-04 23:50:48,719 - ==> Metric Time for chewinggum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.882 (mAUROC_px) 5.989 (mAP_px) 0.807 (mF1_max_px) 20.084 (mAUPRO_px) 0.238 (mF1_px_0.2_0.8_0.1) 0.237 (mAcc_px_0.2_0.8_0.1) 0.237 (mIoU_px_0.2_0.8_0.1) 0.696 (mIoU_max_px) 2024-04-04 23:51:26,093 - ==> Metric Time for fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.896 (mAUROC_px) 5.770 (mAP_px) 0.707 (mF1_max_px) 21.351 (mAUPRO_px) 0.240 (mF1_px_0.2_0.8_0.1) 0.244 (mAcc_px_0.2_0.8_0.1) 0.237 (mIoU_px_0.2_0.8_0.1) 0.718 (mIoU_max_px) 2024-04-04 23:52:13,443 - ==> Metric Time for macaroni1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.653 (mAUROC_px) 8.091 (mAP_px) 0.967 (mF1_max_px) 25.400 (mAUPRO_px) 0.325 (mF1_px_0.2_0.8_0.1) 0.354 (mAcc_px_0.2_0.8_0.1) 0.328 (mIoU_px_0.2_0.8_0.1) 0.937 (mIoU_max_px) 2024-04-04 23:53:00,664 - ==> Metric Time for macaroni2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.684 (mAUROC_px) 8.106 (mAP_px) 0.969 (mF1_max_px) 25.240 (mAUPRO_px) 0.344 (mF1_px_0.2_0.8_0.1) 0.330 (mAcc_px_0.2_0.8_0.1) 0.324 (mIoU_px_0.2_0.8_0.1) 0.941 (mIoU_max_px) 2024-04-04 23:53:50,179 - ==> Metric Time for pcb1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.822 (mAUROC_px) 8.288 (mAP_px) 0.935 (mF1_max_px) 26.973 (mAUPRO_px) 0.374 (mF1_px_0.2_0.8_0.1) 0.377 (mAcc_px_0.2_0.8_0.1) 0.377 (mIoU_px_0.2_0.8_0.1) 1.085 (mIoU_max_px) 2024-04-04 23:54:41,053 - ==> Metric Time for pcb2 : 0.002 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 11.032 (mAUROC_px) 8.479 (mAP_px) 1.026 (mF1_max_px) 27.984 (mAUPRO_px) 0.340 (mF1_px_0.2_0.8_0.1) 0.338 (mAcc_px_0.2_0.8_0.1) 0.346 (mIoU_px_0.2_0.8_0.1) 1.025 (mIoU_max_px) 2024-04-04 23:55:31,382 - ==> Metric Time for pcb3 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.858 (mAUROC_px) 8.167 (mAP_px) 1.020 (mF1_max_px) 27.902 (mAUPRO_px) 0.347 (mF1_px_0.2_0.8_0.1) 0.350 (mAcc_px_0.2_0.8_0.1) 0.356 (mIoU_px_0.2_0.8_0.1) 1.017 (mIoU_max_px) 2024-04-04 23:56:22,564 - ==> Metric Time for pcb4 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.889 (mAUROC_px) 8.389 (mAP_px) 1.086 (mF1_max_px) 28.501 (mAUPRO_px) 0.337 (mF1_px_0.2_0.8_0.1) 0.342 (mAcc_px_0.2_0.8_0.1) 0.343 (mIoU_px_0.2_0.8_0.1) 0.994 (mIoU_max_px) 2024-04-04 23:57:00,097 - ==> Metric Time for pipe_fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.835 (mAUROC_px) 5.893 (mAP_px) 0.728 (mF1_max_px) 21.310 (mAUPRO_px) 0.257 (mF1_px_0.2_0.8_0.1) 0.258 (mAcc_px_0.2_0.8_0.1) 0.262 (mIoU_px_0.2_0.8_0.1) 0.766 (mIoU_max_px) 2024-04-04 23:57:00,118 - | Name | mAUROC_sp_max | mAUROC_sp_max (Max) | mAP_sp_max | mAP_sp_max (Max) | mF1_max_sp_max | mF1_max_sp_max (Max) | mAUROC_px | mAUROC_px (Max) | mAP_px | mAP_px (Max) | mF1_max_px | mF1_max_px (Max) | mAUPRO_px | mAUPRO_px (Max) | mF1_px_0.2_0.8_0.1 | mF1_px_0.2_0.8_0.1 (Max) | mAcc_px_0.2_0.8_0.1 | mAcc_px_0.2_0.8_0.1 (Max) | mIoU_px_0.2_0.8_0.1 | mIoU_px_0.2_0.8_0.1 (Max) | mIoU_max_px | mIoU_max_px (Max) | |:----------:|:---------------:|:---------------------:|:------------:|:------------------:|:----------------:|:----------------------:|:-----------:|:------------------:|:--------:|:------------------:|:------------:|:------------------:|:-----------:|:------------------:|:--------------------:|:--------------------------:|:---------------------:|:---------------------------:|:---------------------:|:---------------------------:|:-------------:|:-------------------:| | candle | 95.930 | 97.100 (50 epoch) | 96.350 | 97.346 (50 epoch) | 89.362 | 91.542 (50 epoch) | 99.094 | 99.098 (250 epoch) | 22.257 | 23.463 (150 epoch) | 31.171 | 32.654 (200 epoch) | 95.422 | 96.087 (300 epoch) | 21.331 | 21.331 (500 epoch) | 41.303 | 41.303 (500 epoch) | 12.185 | 12.185 (500 epoch) | 18.463 | 19.513 (200 epoch) | | capsules | 86.167 | 91.750 (100 epoch) | 91.571 | 94.964 (100 epoch) | 85.047 | 88.780 (100 epoch) | 97.747 | 99.123 (100 epoch) | 38.944 | 61.256 (100 epoch) | 44.372 | 59.868 (250 epoch) | 84.644 | 91.811 (100 epoch) | 20.867 | 33.186 (100 epoch) | 62.220 | 62.576 (50 epoch) | 12.390 | 21.701 (100 epoch) | 28.512 | 42.722 (250 epoch) | | cashew | 93.080 | 94.800 (150 epoch) | 95.768 | 97.492 (150 epoch) | 90.909 | 92.462 (450 epoch) | 95.283 | 95.614 (250 epoch) | 41.225 | 50.952 (250 epoch) | 43.450 | 54.438 (400 epoch) | 85.946 | 89.185 (350 epoch) | 26.336 | 28.352 (350 epoch) | 34.283 | 34.522 (150 epoch) | 15.881 | 17.797 (350 epoch) | 27.755 | 37.398 (400 epoch) | | chewinggum | 97.233 | 97.802 (300 epoch) | 98.747 | 98.939 (300 epoch) | 94.301 | 94.792 (450 epoch) | 97.909 | 98.145 (100 epoch) | 46.798 | 59.823 (50 epoch) | 53.393 | 61.088 (50 epoch) | 79.180 | 80.312 (400 epoch) | 26.306 | 35.748 (50 epoch) | 52.503 | 65.611 (50 epoch) | 16.533 | 23.754 (50 epoch) | 36.419 | 43.976 (50 epoch) | | fryum | 96.160 | 96.200 (450 epoch) | 98.096 | 98.136 (450 epoch) | 92.891 | 93.204 (400 epoch) | 97.103 | 97.197 (450 epoch) | 50.043 | 50.218 (450 epoch) | 52.341 | 52.992 (250 epoch) | 91.755 | 92.074 (450 epoch) | 25.819 | 26.452 (450 epoch) | 27.110 | 28.594 (450 epoch) | 16.077 | 16.462 (450 epoch) | 35.447 | 36.047 (250 epoch) | | macaroni1 | 92.104 | 92.104 (500 epoch) | 90.505 | 90.505 (500 epoch) | 81.915 | 83.978 (450 epoch) | 99.594 | 99.594 (500 epoch) | 19.123 | 19.436 (450 epoch) | 29.118 | 29.377 (200 epoch) | 95.859 | 96.043 (450 epoch) | 15.430 | 15.803 (300 epoch) | 36.742 | 40.765 (50 epoch) | 8.593 | 8.828 (300 epoch) | 17.040 | 17.217 (200 epoch) | | macaroni2 | 75.552 | 82.589 (50 epoch) | 61.513 | 81.196 (50 epoch) | 74.439 | 75.983 (400 epoch) | 95.054 | 99.452 (100 epoch) | 0.327 | 11.272 (50 epoch) | 1.939 | 17.568 (50 epoch) | 80.260 | 96.253 (150 epoch) | 0.557 | 10.086 (50 epoch) | 35.839 | 54.584 (450 epoch) | 0.280 | 5.433 (50 epoch) | 0.979 | 9.630 (50 epoch) | | pcb1 | 96.450 | 96.710 (300 epoch) | 95.389 | 95.525 (400 epoch) | 94.118 | 94.737 (300 epoch) | 99.799 | 99.805 (450 epoch) | 82.439 | 83.141 (450 epoch) | 75.342 | 75.588 (450 epoch) | 93.229 | 94.357 (300 epoch) | 46.958 | 47.720 (450 epoch) | 57.301 | 62.362 (50 epoch) | 33.110 | 33.816 (450 epoch) | 60.439 | 60.756 (450 epoch) | | pcb2 | 95.520 | 95.520 (500 epoch) | 95.568 | 95.568 (500 epoch) | 90.099 | 90.385 (450 epoch) | 98.986 | 99.003 (450 epoch) | 12.826 | 14.130 (50 epoch) | 23.793 | 23.977 (250 epoch) | 91.111 | 91.111 (500 epoch) | 15.888 | 16.150 (200 epoch) | 45.114 | 47.820 (150 epoch) | 8.735 | 8.892 (200 epoch) | 13.503 | 13.621 (250 epoch) | | pcb3 | 95.337 | 95.891 (350 epoch) | 95.629 | 96.056 (350 epoch) | 89.552 | 89.855 (400 epoch) | 99.104 | 99.142 (350 epoch) | 18.082 | 20.249 (150 epoch) | 27.231 | 28.078 (450 epoch) | 91.477 | 91.817 (350 epoch) | 19.346 | 20.042 (150 epoch) | 44.721 | 48.341 (50 epoch) | 10.878 | 11.295 (150 epoch) | 15.761 | 16.332 (450 epoch) | | pcb4 | 99.653 | 99.901 (100 epoch) | 99.658 | 99.901 (100 epoch) | 96.970 | 99.000 (50 epoch) | 98.711 | 98.727 (250 epoch) | 46.924 | 47.966 (250 epoch) | 46.875 | 47.709 (250 epoch) | 90.876 | 90.876 (500 epoch) | 24.384 | 26.925 (200 epoch) | 32.051 | 39.716 (50 epoch) | 14.986 | 16.537 (200 epoch) | 30.612 | 31.328 (250 epoch) | | pipe_fryum | 98.920 | 98.920 (500 epoch) | 99.442 | 99.442 (500 epoch) | 97.087 | 97.561 (200 epoch) | 99.034 | 99.114 (450 epoch) | 52.904 | 54.998 (450 epoch) | 56.921 | 58.733 (450 epoch) | 94.773 | 95.525 (150 epoch) | 30.707 | 34.684 (250 epoch) | 35.300 | 43.014 (100 epoch) | 19.760 | 22.399 (250 epoch) | 39.783 | 41.576 (450 epoch) | | Avg | 93.509 | 94.251 (100 epoch) | 93.186 | 94.481 (100 epoch) | 89.724 | 90.163 (300 epoch) | 98.118 | 98.606 (250 epoch) | 35.991 | 39.701 (250 epoch) | 40.495 | 44.125 (250 epoch) | 89.544 | 91.494 (250 epoch) | 22.827 | 25.164 (100 epoch) | 42.041 | 46.179 (50 epoch) | 14.117 | 15.755 (100 epoch) | 27.059 | 29.990 (250 epoch) | 2024-04-04 23:57:00,454 - ==> Total time: 2 days, 22:56:54 Eta: 2 days, 22:56:54 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-04 23:59:27,662 - Train: 50.02% [270600/541000] [500.2/1000.0] [batch_t 1.410 (1.429)] [data_t 0.002] [optim_t 1.408] [lr 0.005000] 2024-04-05 00:01:49,228 - Train: 50.04% [270700/541000] [500.4/1000.0] [batch_t 1.416 (1.416)] [data_t 0.002] [optim_t 1.414] [lr 0.005000] 2024-04-05 00:04:11,140 - Train: 50.06% [270800/541000] [500.6/1000.0] [batch_t 1.444 (1.419)] [data_t 0.003] [optim_t 1.441] [lr 0.005000] 2024-04-05 00:06:35,405 - Train: 50.07% [270900/541000] [500.7/1000.0] [batch_t 1.443 (1.443)] [data_t 0.003] [optim_t 1.439] [lr 0.005000] 2024-04-05 00:08:59,646 - Train: 50.09% [271000/541000] [500.9/1000.0] [batch_t 1.447 (1.442)] [data_t 0.003] [optim_t 1.444] [lr 0.005000] 2024-04-05 00:09:58,810 - ==> Total time: 2 days, 23:09:53 Eta: 2 days, 22:52:50 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 00:11:26,562 - Train: 50.11% [271100/541000] [501.1/1000.0] [batch_t 1.438 (1.451)] [data_t 0.003] [optim_t 1.434] [lr 0.005000] 2024-04-05 00:13:50,828 - Train: 50.13% [271200/541000] [501.3/1000.0] [batch_t 1.436 (1.443)] [data_t 0.003] [optim_t 1.433] [lr 0.005000] 2024-04-05 00:16:15,101 - Train: 50.15% [271300/541000] [501.5/1000.0] [batch_t 1.443 (1.443)] [data_t 0.003] [optim_t 1.440] [lr 0.005000] 2024-04-05 00:18:39,405 - Train: 50.17% [271400/541000] [501.7/1000.0] [batch_t 1.450 (1.443)] [data_t 0.003] [optim_t 1.447] [lr 0.005000] 2024-04-05 00:21:04,335 - Train: 50.18% [271500/541000] [501.8/1000.0] [batch_t 1.456 (1.449)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-05 00:23:03,799 - ==> Total time: 2 days, 23:22:58 Eta: 2 days, 22:48:50 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 00:23:33,179 - Train: 50.20% [271600/541000] [502.0/1000.0] [batch_t 1.466 (1.478)] [data_t 0.003] [optim_t 1.463] [lr 0.005000] 2024-04-05 00:25:58,900 - Train: 50.22% [271700/541000] [502.2/1000.0] [batch_t 1.464 (1.457)] [data_t 0.003] [optim_t 1.461] [lr 0.005000] 2024-04-05 00:28:24,537 - Train: 50.24% [271800/541000] [502.4/1000.0] [batch_t 1.458 (1.456)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 00:30:50,163 - Train: 50.26% [271900/541000] [502.6/1000.0] [batch_t 1.453 (1.456)] [data_t 0.003] [optim_t 1.450] [lr 0.005000] 2024-04-05 00:33:15,768 - Train: 50.28% [272000/541000] [502.8/1000.0] [batch_t 1.455 (1.456)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-05 00:35:41,327 - Train: 50.30% [272100/541000] [503.0/1000.0] [batch_t 1.459 (1.455)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 00:36:14,837 - ==> Total time: 2 days, 23:36:09 Eta: 2 days, 22:44:54 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 00:38:09,229 - Train: 50.31% [272200/541000] [503.1/1000.0] [batch_t 1.456 (1.463)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-05 00:40:34,841 - Train: 50.33% [272300/541000] [503.3/1000.0] [batch_t 1.455 (1.456)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-05 00:43:00,534 - Train: 50.35% [272400/541000] [503.5/1000.0] [batch_t 1.459 (1.457)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-05 00:45:26,400 - Train: 50.37% [272500/541000] [503.7/1000.0] [batch_t 1.455 (1.459)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-05 00:47:52,145 - Train: 50.39% [272600/541000] [503.9/1000.0] [batch_t 1.455 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-05 00:49:25,411 - ==> Total time: 2 days, 23:49:19 Eta: 2 days, 22:40:55 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 00:50:19,940 - Train: 50.41% [272700/541000] [504.1/1000.0] [batch_t 1.459 (1.465)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-05 00:52:45,539 - Train: 50.43% [272800/541000] [504.3/1000.0] [batch_t 1.459 (1.456)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 00:55:11,069 - Train: 50.44% [272900/541000] [504.4/1000.0] [batch_t 1.459 (1.455)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-05 00:57:36,685 - Train: 50.46% [273000/541000] [504.6/1000.0] [batch_t 1.453 (1.456)] [data_t 0.003] [optim_t 1.449] [lr 0.005000] 2024-04-05 01:00:02,315 - Train: 50.48% [273100/541000] [504.8/1000.0] [batch_t 1.456 (1.456)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-05 01:02:27,915 - Train: 50.50% [273200/541000] [505.0/1000.0] [batch_t 1.454 (1.456)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 01:02:35,193 - ==> Total time: 3 days, 0:02:29 Eta: 2 days, 22:36:54 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 01:04:55,727 - Train: 50.52% [273300/541000] [505.2/1000.0] [batch_t 1.458 (1.460)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 01:07:21,404 - Train: 50.54% [273400/541000] [505.4/1000.0] [batch_t 1.454 (1.457)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 01:09:47,023 - Train: 50.55% [273500/541000] [505.5/1000.0] [batch_t 1.459 (1.456)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-05 01:12:12,631 - Train: 50.57% [273600/541000] [505.7/1000.0] [batch_t 1.458 (1.456)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 01:14:38,253 - Train: 50.59% [273700/541000] [505.9/1000.0] [batch_t 1.460 (1.456)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-05 01:15:45,246 - ==> Total time: 3 days, 0:15:39 Eta: 2 days, 22:32:50 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 01:17:06,165 - Train: 50.61% [273800/541000] [506.1/1000.0] [batch_t 1.459 (1.465)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-05 01:19:31,750 - Train: 50.63% [273900/541000] [506.3/1000.0] [batch_t 1.453 (1.456)] [data_t 0.003] [optim_t 1.450] [lr 0.005000] 2024-04-05 01:21:57,356 - Train: 50.65% [274000/541000] [506.5/1000.0] [batch_t 1.454 (1.456)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 01:24:22,937 - Train: 50.67% [274100/541000] [506.7/1000.0] [batch_t 1.451 (1.456)] [data_t 0.003] [optim_t 1.448] [lr 0.005000] 2024-04-05 01:26:48,623 - Train: 50.68% [274200/541000] [506.8/1000.0] [batch_t 1.449 (1.457)] [data_t 0.003] [optim_t 1.446] [lr 0.005000] 2024-04-05 01:28:55,278 - ==> Total time: 3 days, 0:28:49 Eta: 2 days, 22:28:44 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 01:29:16,380 - Train: 50.70% [274300/541000] [507.0/1000.0] [batch_t 1.458 (1.491)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 01:31:42,053 - Train: 50.72% [274400/541000] [507.2/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-05 01:34:07,772 - Train: 50.74% [274500/541000] [507.4/1000.0] [batch_t 1.449 (1.457)] [data_t 0.003] [optim_t 1.446] [lr 0.005000] 2024-04-05 01:36:33,485 - Train: 50.76% [274600/541000] [507.6/1000.0] [batch_t 1.454 (1.457)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 01:38:59,122 - Train: 50.78% [274700/541000] [507.8/1000.0] [batch_t 1.458 (1.456)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 01:41:24,688 - Train: 50.79% [274800/541000] [507.9/1000.0] [batch_t 1.452 (1.456)] [data_t 0.003] [optim_t 1.449] [lr 0.005000] 2024-04-05 01:42:05,452 - ==> Total time: 3 days, 0:41:59 Eta: 2 days, 22:24:36 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 01:43:53,059 - Train: 50.81% [274900/541000] [508.1/1000.0] [batch_t 1.459 (1.462)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-05 01:46:18,746 - Train: 50.83% [275000/541000] [508.3/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-05 01:48:44,412 - Train: 50.85% [275100/541000] [508.5/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-05 01:51:10,017 - Train: 50.87% [275200/541000] [508.7/1000.0] [batch_t 1.458 (1.456)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 01:53:35,593 - Train: 50.89% [275300/541000] [508.9/1000.0] [batch_t 1.459 (1.456)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 01:55:16,105 - ==> Total time: 3 days, 0:55:10 Eta: 2 days, 22:20:27 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 01:56:03,646 - Train: 50.91% [275400/541000] [509.1/1000.0] [batch_t 1.449 (1.473)] [data_t 0.003] [optim_t 1.446] [lr 0.005000] 2024-04-05 01:58:29,287 - Train: 50.92% [275500/541000] [509.2/1000.0] [batch_t 1.456 (1.456)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-05 02:00:54,965 - Train: 50.94% [275600/541000] [509.4/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-05 02:03:20,617 - Train: 50.96% [275700/541000] [509.6/1000.0] [batch_t 1.460 (1.456)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-05 02:05:46,298 - Train: 50.98% [275800/541000] [509.8/1000.0] [batch_t 1.455 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-05 02:08:11,837 - Train: 51.00% [275900/541000] [510.0/1000.0] [batch_t 1.454 (1.455)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 02:08:26,412 - ==> Total time: 3 days, 1:08:20 Eta: 2 days, 22:16:15 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 02:10:39,775 - Train: 51.02% [276000/541000] [510.2/1000.0] [batch_t 1.457 (1.460)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-05 02:13:05,475 - Train: 51.04% [276100/541000] [510.4/1000.0] [batch_t 1.458 (1.457)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-05 02:15:31,093 - Train: 51.05% [276200/541000] [510.5/1000.0] [batch_t 1.455 (1.456)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-05 02:17:56,801 - Train: 51.07% [276300/541000] [510.7/1000.0] [batch_t 1.458 (1.457)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 02:20:22,477 - Train: 51.09% [276400/541000] [510.9/1000.0] [batch_t 1.459 (1.457)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-05 02:21:36,768 - ==> Total time: 3 days, 1:21:31 Eta: 2 days, 22:12:01 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 02:22:50,173 - Train: 51.11% [276500/541000] [511.1/1000.0] [batch_t 1.451 (1.464)] [data_t 0.003] [optim_t 1.448] [lr 0.005000] 2024-04-05 02:25:15,798 - Train: 51.13% [276600/541000] [511.3/1000.0] [batch_t 1.464 (1.456)] [data_t 0.003] [optim_t 1.461] [lr 0.005000] 2024-04-05 02:27:41,426 - Train: 51.15% [276700/541000] [511.5/1000.0] [batch_t 1.458 (1.456)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 02:30:07,077 - Train: 51.16% [276800/541000] [511.6/1000.0] [batch_t 1.456 (1.456)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-05 02:32:32,807 - Train: 51.18% [276900/541000] [511.8/1000.0] [batch_t 1.450 (1.457)] [data_t 0.003] [optim_t 1.447] [lr 0.005000] 2024-04-05 02:34:46,794 - ==> Total time: 3 days, 1:34:41 Eta: 2 days, 22:07:44 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 02:35:00,584 - Train: 51.20% [277000/541000] [512.0/1000.0] [batch_t 1.456 (1.506)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-05 02:37:26,167 - Train: 51.22% [277100/541000] [512.2/1000.0] [batch_t 1.454 (1.456)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 02:39:51,744 - Train: 51.24% [277200/541000] [512.4/1000.0] [batch_t 1.458 (1.456)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 02:42:17,352 - Train: 51.26% [277300/541000] [512.6/1000.0] [batch_t 1.458 (1.456)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 02:44:43,041 - Train: 51.28% [277400/541000] [512.8/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-05 02:47:08,714 - Train: 51.29% [277500/541000] [512.9/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-05 02:47:56,790 - ==> Total time: 3 days, 1:47:51 Eta: 2 days, 22:03:26 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 02:49:36,594 - Train: 51.31% [277600/541000] [513.1/1000.0] [batch_t 1.461 (1.463)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-05 02:52:02,249 - Train: 51.33% [277700/541000] [513.3/1000.0] [batch_t 1.458 (1.456)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 02:54:27,945 - Train: 51.35% [277800/541000] [513.5/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-05 02:56:53,645 - Train: 51.37% [277900/541000] [513.7/1000.0] [batch_t 1.449 (1.457)] [data_t 0.003] [optim_t 1.446] [lr 0.005000] 2024-04-05 02:59:19,330 - Train: 51.39% [278000/541000] [513.9/1000.0] [batch_t 1.455 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-05 03:01:07,176 - ==> Total time: 3 days, 2:01:01 Eta: 2 days, 21:59:06 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 03:01:47,241 - Train: 51.40% [278100/541000] [514.0/1000.0] [batch_t 1.464 (1.469)] [data_t 0.003] [optim_t 1.461] [lr 0.005000] 2024-04-05 03:04:12,980 - Train: 51.42% [278200/541000] [514.2/1000.0] [batch_t 1.454 (1.457)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 03:06:38,632 - Train: 51.44% [278300/541000] [514.4/1000.0] [batch_t 1.459 (1.456)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 03:09:04,245 - Train: 51.46% [278400/541000] [514.6/1000.0] [batch_t 1.455 (1.456)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-05 03:11:29,834 - Train: 51.48% [278500/541000] [514.8/1000.0] [batch_t 1.454 (1.456)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 03:13:55,429 - Train: 51.50% [278600/541000] [515.0/1000.0] [batch_t 1.454 (1.456)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 03:14:17,287 - ==> Total time: 3 days, 2:14:11 Eta: 2 days, 21:54:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 03:16:23,410 - Train: 51.52% [278700/541000] [515.2/1000.0] [batch_t 1.458 (1.463)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 03:18:49,061 - Train: 51.53% [278800/541000] [515.3/1000.0] [batch_t 1.455 (1.456)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-05 03:21:14,752 - Train: 51.55% [278900/541000] [515.5/1000.0] [batch_t 1.450 (1.457)] [data_t 0.003] [optim_t 1.447] [lr 0.005000] 2024-04-05 03:23:40,395 - Train: 51.57% [279000/541000] [515.7/1000.0] [batch_t 1.456 (1.456)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-05 03:26:06,082 - Train: 51.59% [279100/541000] [515.9/1000.0] [batch_t 1.459 (1.457)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 03:27:27,617 - ==> Total time: 3 days, 2:27:22 Eta: 2 days, 21:50:19 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 03:28:33,874 - Train: 51.61% [279200/541000] [516.1/1000.0] [batch_t 1.454 (1.464)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 03:30:59,513 - Train: 51.63% [279300/541000] [516.3/1000.0] [batch_t 1.459 (1.456)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 03:33:25,156 - Train: 51.65% [279400/541000] [516.5/1000.0] [batch_t 1.454 (1.456)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 03:35:50,867 - Train: 51.66% [279500/541000] [516.6/1000.0] [batch_t 1.459 (1.457)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-05 03:38:16,584 - Train: 51.68% [279600/541000] [516.8/1000.0] [batch_t 1.459 (1.457)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-05 03:40:37,870 - ==> Total time: 3 days, 2:40:32 Eta: 2 days, 21:45:52 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 03:40:44,367 - Train: 51.70% [279700/541000] [517.0/1000.0] [batch_t 1.455 (1.581)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-05 03:43:09,940 - Train: 51.72% [279800/541000] [517.2/1000.0] [batch_t 1.458 (1.456)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 03:45:35,540 - Train: 51.74% [279900/541000] [517.4/1000.0] [batch_t 1.455 (1.456)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-05 03:48:01,178 - Train: 51.76% [280000/541000] [517.6/1000.0] [batch_t 1.460 (1.456)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-05 03:50:26,875 - Train: 51.77% [280100/541000] [517.7/1000.0] [batch_t 1.454 (1.457)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 03:52:52,568 - Train: 51.79% [280200/541000] [517.9/1000.0] [batch_t 1.458 (1.457)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 03:53:47,966 - ==> Total time: 3 days, 2:53:42 Eta: 2 days, 21:41:24 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 03:55:20,691 - Train: 51.81% [280300/541000] [518.1/1000.0] [batch_t 1.456 (1.463)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-05 03:57:46,414 - Train: 51.83% [280400/541000] [518.3/1000.0] [batch_t 1.459 (1.457)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-05 04:00:11,991 - Train: 51.85% [280500/541000] [518.5/1000.0] [batch_t 1.459 (1.456)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-05 04:02:37,618 - Train: 51.87% [280600/541000] [518.7/1000.0] [batch_t 1.460 (1.456)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-05 04:05:03,276 - Train: 51.89% [280700/541000] [518.9/1000.0] [batch_t 1.455 (1.456)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-05 04:06:58,263 - ==> Total time: 3 days, 3:06:52 Eta: 2 days, 21:36:53 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 04:07:30,965 - Train: 51.90% [280800/541000] [519.0/1000.0] [batch_t 1.451 (1.473)] [data_t 0.003] [optim_t 1.448] [lr 0.005000] 2024-04-05 04:09:56,569 - Train: 51.92% [280900/541000] [519.2/1000.0] [batch_t 1.450 (1.456)] [data_t 0.003] [optim_t 1.447] [lr 0.005000] 2024-04-05 04:12:22,239 - Train: 51.94% [281000/541000] [519.4/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-05 04:14:48,017 - Train: 51.96% [281100/541000] [519.6/1000.0] [batch_t 1.454 (1.458)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 04:17:13,685 - Train: 51.98% [281200/541000] [519.8/1000.0] [batch_t 1.454 (1.457)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 04:19:39,311 - Train: 52.00% [281300/541000] [520.0/1000.0] [batch_t 1.454 (1.456)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-05 04:20:08,461 - ==> Total time: 3 days, 3:20:02 Eta: 2 days, 21:32:21 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 04:22:07,177 - Train: 52.01% [281400/541000] [520.1/1000.0] [batch_t 1.454 (1.462)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 04:24:32,837 - Train: 52.03% [281500/541000] [520.3/1000.0] [batch_t 1.455 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-05 04:26:58,503 - Train: 52.05% [281600/541000] [520.5/1000.0] [batch_t 1.458 (1.457)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 04:29:24,280 - Train: 52.07% [281700/541000] [520.7/1000.0] [batch_t 1.458 (1.458)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 04:31:49,842 - Train: 52.09% [281800/541000] [520.9/1000.0] [batch_t 1.455 (1.456)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 04:33:18,623 - ==> Total time: 3 days, 3:33:13 Eta: 2 days, 21:27:46 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 04:34:18,198 - Train: 52.11% [281900/541000] [521.1/1000.0] [batch_t 1.460 (1.480)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-05 04:36:43,808 - Train: 52.13% [282000/541000] [521.3/1000.0] [batch_t 1.453 (1.456)] [data_t 0.003] [optim_t 1.450] [lr 0.005000] 2024-04-05 04:39:09,394 - Train: 52.14% [282100/541000] [521.4/1000.0] [batch_t 1.451 (1.456)] [data_t 0.003] [optim_t 1.448] [lr 0.005000] 2024-04-05 04:41:35,025 - Train: 52.16% [282200/541000] [521.6/1000.0] [batch_t 1.460 (1.456)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-05 04:44:00,754 - Train: 52.18% [282300/541000] [521.8/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-05 04:46:26,389 - Train: 52.20% [282400/541000] [522.0/1000.0] [batch_t 1.450 (1.456)] [data_t 0.003] [optim_t 1.447] [lr 0.005000] 2024-04-05 04:46:29,305 - ==> Total time: 3 days, 3:46:23 Eta: 2 days, 21:23:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 04:48:54,280 - Train: 52.22% [282500/541000] [522.2/1000.0] [batch_t 1.449 (1.461)] [data_t 0.003] [optim_t 1.447] [lr 0.005000] 2024-04-05 04:51:19,984 - Train: 52.24% [282600/541000] [522.4/1000.0] [batch_t 1.454 (1.457)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 04:53:45,658 - Train: 52.26% [282700/541000] [522.6/1000.0] [batch_t 1.454 (1.457)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 04:56:11,325 - Train: 52.27% [282800/541000] [522.7/1000.0] [batch_t 1.464 (1.457)] [data_t 0.003] [optim_t 1.461] [lr 0.005000] 2024-04-05 04:58:36,992 - Train: 52.29% [282900/541000] [522.9/1000.0] [batch_t 1.459 (1.457)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-05 04:59:39,582 - ==> Total time: 3 days, 3:59:34 Eta: 2 days, 21:18:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 05:01:05,019 - Train: 52.31% [283000/541000] [523.1/1000.0] [batch_t 1.459 (1.464)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-05 05:03:30,744 - Train: 52.33% [283100/541000] [523.3/1000.0] [batch_t 1.449 (1.457)] [data_t 0.003] [optim_t 1.446] [lr 0.005000] 2024-04-05 05:05:56,378 - Train: 52.35% [283200/541000] [523.5/1000.0] [batch_t 1.457 (1.456)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-05 05:08:22,022 - Train: 52.37% [283300/541000] [523.7/1000.0] [batch_t 1.454 (1.456)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 05:10:47,692 - Train: 52.38% [283400/541000] [523.8/1000.0] [batch_t 1.459 (1.457)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-05 05:12:49,987 - ==> Total time: 3 days, 4:12:44 Eta: 2 days, 21:13:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 05:13:15,483 - Train: 52.40% [283500/541000] [524.0/1000.0] [batch_t 1.454 (1.480)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 05:15:41,120 - Train: 52.42% [283600/541000] [524.2/1000.0] [batch_t 1.452 (1.456)] [data_t 0.003] [optim_t 1.449] [lr 0.005000] 2024-04-05 05:18:06,852 - Train: 52.44% [283700/541000] [524.4/1000.0] [batch_t 1.459 (1.457)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 05:20:32,459 - Train: 52.46% [283800/541000] [524.6/1000.0] [batch_t 1.450 (1.456)] [data_t 0.003] [optim_t 1.447] [lr 0.005000] 2024-04-05 05:22:58,147 - Train: 52.48% [283900/541000] [524.8/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-05 05:25:23,770 - Train: 52.50% [284000/541000] [525.0/1000.0] [batch_t 1.459 (1.456)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-05 05:26:00,199 - ==> Total time: 3 days, 4:25:54 Eta: 2 days, 21:09:09 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 05:27:53,534 - Train: 52.51% [284100/541000] [525.1/1000.0] [batch_t 1.459 (1.486)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-05 05:30:19,317 - Train: 52.53% [284200/541000] [525.3/1000.0] [batch_t 1.459 (1.458)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 05:32:45,112 - Train: 52.55% [284300/541000] [525.5/1000.0] [batch_t 1.454 (1.458)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 05:35:10,886 - Train: 52.57% [284400/541000] [525.7/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-05 05:37:36,640 - Train: 52.59% [284500/541000] [525.9/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-05 05:39:12,767 - ==> Total time: 3 days, 4:39:07 Eta: 2 days, 21:04:27 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 05:40:04,404 - Train: 52.61% [284600/541000] [526.1/1000.0] [batch_t 1.458 (1.466)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-05 05:42:30,064 - Train: 52.62% [284700/541000] [526.2/1000.0] [batch_t 1.455 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-05 05:44:55,702 - Train: 52.64% [284800/541000] [526.4/1000.0] [batch_t 1.455 (1.456)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-05 05:47:21,269 - Train: 52.66% [284900/541000] [526.6/1000.0] [batch_t 1.451 (1.456)] [data_t 0.003] [optim_t 1.448] [lr 0.005000] 2024-04-05 05:49:46,947 - Train: 52.68% [285000/541000] [526.8/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-05 05:52:12,539 - Train: 52.70% [285100/541000] [527.0/1000.0] [batch_t 1.462 (1.456)] [data_t 0.004] [optim_t 1.458] [lr 0.005000] 2024-04-05 05:52:22,746 - ==> Total time: 3 days, 4:52:17 Eta: 2 days, 20:59:40 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 05:54:40,222 - Train: 52.72% [285200/541000] [527.2/1000.0] [batch_t 1.460 (1.460)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-05 05:57:05,938 - Train: 52.74% [285300/541000] [527.4/1000.0] [batch_t 1.455 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-05 05:59:31,653 - Train: 52.75% [285400/541000] [527.5/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-05 06:01:57,417 - Train: 52.77% [285500/541000] [527.7/1000.0] [batch_t 1.449 (1.458)] [data_t 0.003] [optim_t 1.446] [lr 0.005000] 2024-04-05 06:04:23,134 - Train: 52.79% [285600/541000] [527.9/1000.0] [batch_t 1.449 (1.457)] [data_t 0.003] [optim_t 1.445] [lr 0.005000] 2024-04-05 06:05:33,135 - ==> Total time: 3 days, 5:05:27 Eta: 2 days, 20:54:52 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 06:06:51,140 - Train: 52.81% [285700/541000] [528.1/1000.0] [batch_t 1.453 (1.467)] [data_t 0.003] [optim_t 1.450] [lr 0.005000] 2024-04-05 06:09:16,687 - Train: 52.83% [285800/541000] [528.3/1000.0] [batch_t 1.459 (1.455)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-05 06:11:42,305 - Train: 52.85% [285900/541000] [528.5/1000.0] [batch_t 1.451 (1.456)] [data_t 0.003] [optim_t 1.448] [lr 0.005000] 2024-04-05 06:14:07,934 - Train: 52.87% [286000/541000] [528.7/1000.0] [batch_t 1.459 (1.456)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-05 06:16:33,652 - Train: 52.88% [286100/541000] [528.8/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-05 06:18:43,288 - ==> Total time: 3 days, 5:18:37 Eta: 2 days, 20:50:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 06:19:01,534 - Train: 52.90% [286200/541000] [529.0/1000.0] [batch_t 1.454 (1.498)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 06:21:27,140 - Train: 52.92% [286300/541000] [529.2/1000.0] [batch_t 1.454 (1.456)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 06:23:52,741 - Train: 52.94% [286400/541000] [529.4/1000.0] [batch_t 1.454 (1.456)] [data_t 0.003] [optim_t 1.450] [lr 0.005000] 2024-04-05 06:26:18,379 - Train: 52.96% [286500/541000] [529.6/1000.0] [batch_t 1.458 (1.456)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 06:28:44,006 - Train: 52.98% [286600/541000] [529.8/1000.0] [batch_t 1.456 (1.456)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-05 06:31:09,711 - Train: 52.99% [286700/541000] [529.9/1000.0] [batch_t 1.459 (1.457)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-05 06:31:53,429 - ==> Total time: 3 days, 5:31:47 Eta: 2 days, 20:45:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 06:33:37,620 - Train: 53.01% [286800/541000] [530.1/1000.0] [batch_t 1.454 (1.462)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 06:36:03,348 - Train: 53.03% [286900/541000] [530.3/1000.0] [batch_t 1.457 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-05 06:38:29,025 - Train: 53.05% [287000/541000] [530.5/1000.0] [batch_t 1.454 (1.457)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 06:40:54,786 - Train: 53.07% [287100/541000] [530.7/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-05 06:43:20,484 - Train: 53.09% [287200/541000] [530.9/1000.0] [batch_t 1.454 (1.457)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 06:45:03,932 - ==> Total time: 3 days, 5:44:58 Eta: 2 days, 20:40:17 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 06:45:49,261 - Train: 53.11% [287300/541000] [531.1/1000.0] [batch_t 1.452 (1.484)] [data_t 0.003] [optim_t 1.449] [lr 0.005000] 2024-04-05 06:48:14,885 - Train: 53.12% [287400/541000] [531.2/1000.0] [batch_t 1.460 (1.456)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-05 06:50:40,582 - Train: 53.14% [287500/541000] [531.4/1000.0] [batch_t 1.451 (1.457)] [data_t 0.003] [optim_t 1.448] [lr 0.005000] 2024-04-05 06:53:06,236 - Train: 53.16% [287600/541000] [531.6/1000.0] [batch_t 1.459 (1.456)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 06:55:31,909 - Train: 53.18% [287700/541000] [531.8/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-05 06:57:57,531 - Train: 53.20% [287800/541000] [532.0/1000.0] [batch_t 1.456 (1.456)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-05 06:58:15,037 - ==> Total time: 3 days, 5:58:09 Eta: 2 days, 20:35:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 07:00:25,484 - Train: 53.22% [287900/541000] [532.2/1000.0] [batch_t 1.459 (1.462)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-05 07:02:51,170 - Train: 53.23% [288000/541000] [532.3/1000.0] [batch_t 1.465 (1.457)] [data_t 0.003] [optim_t 1.462] [lr 0.005000] 2024-04-05 07:05:16,907 - Train: 53.25% [288100/541000] [532.5/1000.0] [batch_t 1.459 (1.457)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-05 07:07:42,637 - Train: 53.27% [288200/541000] [532.7/1000.0] [batch_t 1.464 (1.457)] [data_t 0.003] [optim_t 1.461] [lr 0.005000] 2024-04-05 07:10:08,368 - Train: 53.29% [288300/541000] [532.9/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-05 07:11:25,592 - ==> Total time: 3 days, 6:11:20 Eta: 2 days, 20:30:25 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 07:12:36,441 - Train: 53.31% [288400/541000] [533.1/1000.0] [batch_t 1.462 (1.466)] [data_t 0.003] [optim_t 1.459] [lr 0.005000] 2024-04-05 07:15:02,199 - Train: 53.33% [288500/541000] [533.3/1000.0] [batch_t 1.455 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-05 07:17:27,893 - Train: 53.35% [288600/541000] [533.5/1000.0] [batch_t 1.458 (1.457)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 07:19:53,573 - Train: 53.36% [288700/541000] [533.6/1000.0] [batch_t 1.458 (1.457)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 07:22:19,254 - Train: 53.38% [288800/541000] [533.8/1000.0] [batch_t 1.458 (1.457)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 07:24:36,207 - ==> Total time: 3 days, 6:24:30 Eta: 2 days, 20:25:26 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 07:24:47,315 - Train: 53.40% [288900/541000] [534.0/1000.0] [batch_t 1.461 (1.513)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-05 07:27:13,063 - Train: 53.42% [289000/541000] [534.2/1000.0] [batch_t 1.459 (1.457)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-05 07:29:38,783 - Train: 53.44% [289100/541000] [534.4/1000.0] [batch_t 1.459 (1.457)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 07:32:04,502 - Train: 53.46% [289200/541000] [534.6/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-05 07:34:30,254 - Train: 53.48% [289300/541000] [534.8/1000.0] [batch_t 1.464 (1.457)] [data_t 0.003] [optim_t 1.461] [lr 0.005000] 2024-04-05 07:36:55,928 - Train: 53.49% [289400/541000] [534.9/1000.0] [batch_t 1.457 (1.457)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-05 07:37:46,925 - ==> Total time: 3 days, 6:37:41 Eta: 2 days, 20:20:25 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 07:39:24,026 - Train: 53.51% [289500/541000] [535.1/1000.0] [batch_t 1.454 (1.465)] [data_t 0.003] [optim_t 1.450] [lr 0.005000] 2024-04-05 07:41:49,561 - Train: 53.53% [289600/541000] [535.3/1000.0] [batch_t 1.454 (1.455)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 07:44:15,209 - Train: 53.55% [289700/541000] [535.5/1000.0] [batch_t 1.458 (1.456)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 07:46:40,842 - Train: 53.57% [289800/541000] [535.7/1000.0] [batch_t 1.458 (1.456)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 07:49:06,543 - Train: 53.59% [289900/541000] [535.9/1000.0] [batch_t 1.465 (1.457)] [data_t 0.003] [optim_t 1.462] [lr 0.005000] 2024-04-05 07:50:57,253 - ==> Total time: 3 days, 6:50:51 Eta: 2 days, 20:15:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 07:51:34,376 - Train: 53.60% [290000/541000] [536.0/1000.0] [batch_t 1.453 (1.475)] [data_t 0.003] [optim_t 1.450] [lr 0.005000] 2024-04-05 07:54:00,034 - Train: 53.62% [290100/541000] [536.2/1000.0] [batch_t 1.454 (1.456)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 07:56:25,764 - Train: 53.64% [290200/541000] [536.4/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-05 07:58:51,467 - Train: 53.66% [290300/541000] [536.6/1000.0] [batch_t 1.455 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-05 08:01:17,180 - Train: 53.68% [290400/541000] [536.8/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-05 08:03:42,925 - Train: 53.70% [290500/541000] [537.0/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-05 08:04:07,691 - ==> Total time: 3 days, 7:04:02 Eta: 2 days, 20:10:17 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 08:06:10,718 - Train: 53.72% [290600/541000] [537.2/1000.0] [batch_t 1.460 (1.461)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-05 08:08:36,322 - Train: 53.73% [290700/541000] [537.3/1000.0] [batch_t 1.454 (1.456)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 08:10:26,202 - Train: 53.75% [290800/541000] [537.5/1000.0] [batch_t 0.611 (1.099)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-05 08:11:26,913 - Train: 53.77% [290900/541000] [537.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 08:12:27,839 - Train: 53.79% [291000/541000] [537.9/1000.0] [batch_t 0.607 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 08:13:03,130 - ==> Total time: 3 days, 7:12:57 Eta: 2 days, 20:01:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 08:13:32,290 - Train: 53.81% [291100/541000] [538.1/1000.0] [batch_t 0.608 (0.625)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 08:14:33,134 - Train: 53.83% [291200/541000] [538.3/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 08:15:33,962 - Train: 53.84% [291300/541000] [538.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 08:16:34,804 - Train: 53.86% [291400/541000] [538.6/1000.0] [batch_t 0.610 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-05 08:17:35,680 - Train: 53.88% [291500/541000] [538.8/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 08:18:35,940 - ==> Total time: 3 days, 7:18:30 Eta: 2 days, 19:49:53 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 08:18:38,875 - Train: 53.90% [291600/541000] [539.0/1000.0] [batch_t 0.954 (0.954)] [data_t 0.345] [optim_t 0.608] [lr 0.005000] 2024-04-05 08:19:39,725 - Train: 53.92% [291700/541000] [539.2/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 08:20:40,600 - Train: 53.94% [291800/541000] [539.4/1000.0] [batch_t 0.611 (0.609)] [data_t 0.003] [optim_t 0.608] [lr 0.005000] 2024-04-05 08:21:41,489 - Train: 53.96% [291900/541000] [539.6/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 08:22:42,363 - Train: 53.97% [292000/541000] [539.7/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 08:23:43,258 - Train: 53.99% [292100/541000] [539.9/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 08:24:07,614 - ==> Total time: 3 days, 7:24:02 Eta: 2 days, 19:38:15 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 08:24:46,189 - Train: 54.01% [292200/541000] [540.1/1000.0] [batch_t 0.609 (0.614)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 08:25:47,168 - Train: 54.03% [292300/541000] [540.3/1000.0] [batch_t 0.609 (0.610)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 08:26:48,055 - Train: 54.05% [292400/541000] [540.5/1000.0] [batch_t 0.610 (0.609)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-05 08:27:48,917 - Train: 54.07% [292500/541000] [540.7/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 08:28:49,741 - Train: 54.09% [292600/541000] [540.9/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 08:29:39,015 - ==> Total time: 3 days, 7:29:33 Eta: 2 days, 19:26:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 08:29:53,104 - Train: 54.10% [292700/541000] [541.0/1000.0] [batch_t 0.608 (0.632)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 08:30:53,942 - Train: 54.12% [292800/541000] [541.2/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 08:31:54,908 - Train: 54.14% [292900/541000] [541.4/1000.0] [batch_t 0.609 (0.610)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-05 08:32:55,738 - Train: 54.16% [293000/541000] [541.6/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 08:33:56,573 - Train: 54.18% [293100/541000] [541.8/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 08:34:57,392 - Train: 54.20% [293200/541000] [542.0/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-05 08:35:10,782 - ==> Total time: 3 days, 7:35:05 Eta: 2 days, 19:15:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 08:36:00,408 - Train: 54.21% [293300/541000] [542.1/1000.0] [batch_t 0.608 (0.613)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 08:37:01,230 - Train: 54.23% [293400/541000] [542.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 08:38:02,190 - Train: 54.25% [293500/541000] [542.5/1000.0] [batch_t 0.608 (0.610)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 08:39:03,000 - Train: 54.27% [293600/541000] [542.7/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 08:40:03,814 - Train: 54.29% [293700/541000] [542.9/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 08:40:42,149 - ==> Total time: 3 days, 7:40:36 Eta: 2 days, 19:03:27 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 08:41:06,782 - Train: 54.31% [293800/541000] [543.1/1000.0] [batch_t 0.609 (0.619)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 08:42:07,674 - Train: 54.33% [293900/541000] [543.3/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 08:43:08,531 - Train: 54.34% [294000/541000] [543.4/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-05 08:44:09,508 - Train: 54.36% [294100/541000] [543.6/1000.0] [batch_t 0.609 (0.610)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 08:45:10,336 - Train: 54.38% [294200/541000] [543.8/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 08:46:11,185 - Train: 54.40% [294300/541000] [544.0/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 08:46:13,619 - ==> Total time: 3 days, 7:46:08 Eta: 2 days, 18:51:54 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 08:47:14,180 - Train: 54.42% [294400/541000] [544.2/1000.0] [batch_t 0.608 (0.613)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 08:48:15,021 - Train: 54.44% [294500/541000] [544.4/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 08:49:15,978 - Train: 54.45% [294600/541000] [544.5/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 08:50:16,790 - Train: 54.47% [294700/541000] [544.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 08:51:17,637 - Train: 54.49% [294800/541000] [544.9/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 08:51:45,014 - ==> Total time: 3 days, 7:51:39 Eta: 2 days, 18:40:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 08:52:20,633 - Train: 54.51% [294900/541000] [545.1/1000.0] [batch_t 0.609 (0.614)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 08:53:21,456 - Train: 54.53% [295000/541000] [545.3/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 08:54:22,404 - Train: 54.55% [295100/541000] [545.5/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 08:55:23,218 - Train: 54.57% [295200/541000] [545.7/1000.0] [batch_t 0.610 (0.608)] [data_t 0.003] [optim_t 0.608] [lr 0.005000] 2024-04-05 08:56:24,100 - Train: 54.58% [295300/541000] [545.8/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 08:57:16,475 - ==> Total time: 3 days, 7:57:10 Eta: 2 days, 18:28:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 08:57:27,060 - Train: 54.60% [295400/541000] [546.0/1000.0] [batch_t 0.608 (0.633)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 08:58:27,908 - Train: 54.62% [295500/541000] [546.2/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-05 08:59:28,756 - Train: 54.64% [295600/541000] [546.4/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-05 09:00:29,603 - Train: 54.66% [295700/541000] [546.6/1000.0] [batch_t 0.611 (0.608)] [data_t 0.003] [optim_t 0.609] [lr 0.005000] 2024-04-05 09:01:30,640 - Train: 54.68% [295800/541000] [546.8/1000.0] [batch_t 0.609 (0.610)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 09:02:31,492 - Train: 54.70% [295900/541000] [547.0/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 09:02:47,917 - ==> Total time: 3 days, 8:02:42 Eta: 2 days, 18:17:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 09:03:34,784 - Train: 54.71% [296000/541000] [547.1/1000.0] [batch_t 0.609 (0.614)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 09:04:35,654 - Train: 54.73% [296100/541000] [547.3/1000.0] [batch_t 0.609 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 09:05:36,500 - Train: 54.75% [296200/541000] [547.5/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 09:06:37,500 - Train: 54.77% [296300/541000] [547.7/1000.0] [batch_t 0.609 (0.610)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 09:07:38,359 - Train: 54.79% [296400/541000] [547.9/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 09:08:19,736 - ==> Total time: 3 days, 8:08:14 Eta: 2 days, 18:05:55 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 09:08:41,662 - Train: 54.81% [296500/541000] [548.1/1000.0] [batch_t 0.608 (0.622)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 09:09:42,488 - Train: 54.82% [296600/541000] [548.2/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 09:10:43,318 - Train: 54.84% [296700/541000] [548.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 09:11:44,145 - Train: 54.86% [296800/541000] [548.6/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-05 09:12:44,988 - Train: 54.88% [296900/541000] [548.8/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 09:13:45,941 - Train: 54.90% [297000/541000] [549.0/1000.0] [batch_t 0.611 (0.609)] [data_t 0.003] [optim_t 0.609] [lr 0.005000] 2024-04-05 09:13:51,426 - ==> Total time: 3 days, 8:13:45 Eta: 2 days, 17:54:28 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 09:14:49,271 - Train: 54.92% [297100/541000] [549.2/1000.0] [batch_t 0.608 (0.614)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 09:15:50,137 - Train: 54.94% [297200/541000] [549.4/1000.0] [batch_t 0.607 (0.609)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-05 09:16:50,993 - Train: 54.95% [297300/541000] [549.5/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 09:17:51,869 - Train: 54.97% [297400/541000] [549.7/1000.0] [batch_t 0.609 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 09:18:52,916 - Train: 54.99% [297500/541000] [549.9/1000.0] [batch_t 0.608 (0.610)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 09:19:35,538 - Test: 36.76% [50/136] [batch_t 0.214 (0.231)] 2024-04-05 09:19:46,208 - Test: 73.53% [100/136] [batch_t 0.211 (0.222)] 2024-04-05 09:19:53,677 - Test: 100.00% [136/136] [batch_t 0.054 (0.218)] 2024-04-05 09:20:42,461 - ==> Metric Time for candle : 0.002 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.765 (mAUROC_px) 8.169 (mAP_px) 0.925 (mF1_max_px) 25.272 (mAUPRO_px) 0.317 (mF1_px_0.2_0.8_0.1) 0.315 (mAcc_px_0.2_0.8_0.1) 0.317 (mIoU_px_0.2_0.8_0.1) 0.919 (mIoU_max_px) 2024-04-05 09:21:21,724 - ==> Metric Time for capsules : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 9.430 (mAUROC_px) 6.824 (mAP_px) 0.760 (mF1_max_px) 20.498 (mAUPRO_px) 0.256 (mF1_px_0.2_0.8_0.1) 0.255 (mAcc_px_0.2_0.8_0.1) 0.257 (mIoU_px_0.2_0.8_0.1) 0.745 (mIoU_max_px) 2024-04-05 09:21:57,472 - ==> Metric Time for cashew : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.802 (mAUROC_px) 5.987 (mAP_px) 0.705 (mF1_max_px) 19.628 (mAUPRO_px) 0.239 (mF1_px_0.2_0.8_0.1) 0.237 (mAcc_px_0.2_0.8_0.1) 0.242 (mIoU_px_0.2_0.8_0.1) 0.696 (mIoU_max_px) 2024-04-05 09:22:33,504 - ==> Metric Time for chewinggum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.783 (mAUROC_px) 5.826 (mAP_px) 0.697 (mF1_max_px) 20.095 (mAUPRO_px) 0.243 (mF1_px_0.2_0.8_0.1) 0.240 (mAcc_px_0.2_0.8_0.1) 0.239 (mIoU_px_0.2_0.8_0.1) 0.701 (mIoU_max_px) 2024-04-05 09:23:09,878 - ==> Metric Time for fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.594 (mAUROC_px) 5.681 (mAP_px) 0.704 (mF1_max_px) 20.783 (mAUPRO_px) 0.238 (mF1_px_0.2_0.8_0.1) 0.236 (mAcc_px_0.2_0.8_0.1) 0.237 (mIoU_px_0.2_0.8_0.1) 0.693 (mIoU_max_px) 2024-04-05 09:23:57,043 - ==> Metric Time for macaroni1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.621 (mAUROC_px) 7.980 (mAP_px) 0.923 (mF1_max_px) 25.373 (mAUPRO_px) 0.336 (mF1_px_0.2_0.8_0.1) 0.338 (mAcc_px_0.2_0.8_0.1) 0.335 (mIoU_px_0.2_0.8_0.1) 0.980 (mIoU_max_px) 2024-04-05 09:24:43,795 - ==> Metric Time for macaroni2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.986 (mAUROC_px) 7.864 (mAP_px) 0.924 (mF1_max_px) 24.779 (mAUPRO_px) 0.313 (mF1_px_0.2_0.8_0.1) 0.314 (mAcc_px_0.2_0.8_0.1) 0.343 (mIoU_px_0.2_0.8_0.1) 0.941 (mIoU_max_px) 2024-04-05 09:25:31,868 - ==> Metric Time for pcb1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.617 (mAUROC_px) 8.050 (mAP_px) 0.952 (mF1_max_px) 26.153 (mAUPRO_px) 0.351 (mF1_px_0.2_0.8_0.1) 0.355 (mAcc_px_0.2_0.8_0.1) 0.377 (mIoU_px_0.2_0.8_0.1) 0.939 (mIoU_max_px) 2024-04-05 09:26:20,493 - ==> Metric Time for pcb2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.660 (mAUROC_px) 8.237 (mAP_px) 0.959 (mF1_max_px) 26.596 (mAUPRO_px) 0.318 (mF1_px_0.2_0.8_0.1) 0.323 (mAcc_px_0.2_0.8_0.1) 0.322 (mIoU_px_0.2_0.8_0.1) 0.933 (mIoU_max_px) 2024-04-05 09:27:09,672 - ==> Metric Time for pcb3 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.884 (mAUROC_px) 7.999 (mAP_px) 0.951 (mF1_max_px) 26.700 (mAUPRO_px) 0.407 (mF1_px_0.2_0.8_0.1) 0.410 (mAcc_px_0.2_0.8_0.1) 0.393 (mIoU_px_0.2_0.8_0.1) 1.153 (mIoU_max_px) 2024-04-05 09:27:59,991 - ==> Metric Time for pcb4 : 0.002 (mAUROC_sp_max) 0.001 (mAP_sp_max) 0.001 (mF1_max_sp_max) 11.117 (mAUROC_px) 8.136 (mAP_px) 0.969 (mF1_max_px) 27.870 (mAUPRO_px) 0.323 (mF1_px_0.2_0.8_0.1) 0.323 (mAcc_px_0.2_0.8_0.1) 0.324 (mIoU_px_0.2_0.8_0.1) 0.932 (mIoU_max_px) 2024-04-05 09:28:36,326 - ==> Metric Time for pipe_fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.904 (mAUROC_px) 5.810 (mAP_px) 0.699 (mF1_max_px) 20.271 (mAUPRO_px) 0.240 (mF1_px_0.2_0.8_0.1) 0.240 (mAcc_px_0.2_0.8_0.1) 0.240 (mIoU_px_0.2_0.8_0.1) 0.716 (mIoU_max_px) 2024-04-05 09:28:36,348 - | Name | mAUROC_sp_max | mAUROC_sp_max (Max) | mAP_sp_max | mAP_sp_max (Max) | mF1_max_sp_max | mF1_max_sp_max (Max) | mAUROC_px | mAUROC_px (Max) | mAP_px | mAP_px (Max) | mF1_max_px | mF1_max_px (Max) | mAUPRO_px | mAUPRO_px (Max) | mF1_px_0.2_0.8_0.1 | mF1_px_0.2_0.8_0.1 (Max) | mAcc_px_0.2_0.8_0.1 | mAcc_px_0.2_0.8_0.1 (Max) | mIoU_px_0.2_0.8_0.1 | mIoU_px_0.2_0.8_0.1 (Max) | mIoU_max_px | mIoU_max_px (Max) | |:----------:|:---------------:|:---------------------:|:------------:|:------------------:|:----------------:|:----------------------:|:-----------:|:------------------:|:--------:|:------------------:|:------------:|:------------------:|:-----------:|:------------------:|:--------------------:|:--------------------------:|:---------------------:|:---------------------------:|:---------------------:|:---------------------------:|:-------------:|:-------------------:| | candle | 96.030 | 97.100 (50 epoch) | 96.357 | 97.346 (50 epoch) | 89.423 | 91.542 (50 epoch) | 99.052 | 99.098 (250 epoch) | 21.632 | 23.463 (150 epoch) | 31.075 | 32.654 (200 epoch) | 95.925 | 96.087 (300 epoch) | 20.172 | 21.331 (500 epoch) | 37.879 | 41.303 (500 epoch) | 11.489 | 12.185 (500 epoch) | 18.396 | 19.513 (200 epoch) | | capsules | 81.650 | 91.750 (100 epoch) | 84.268 | 94.964 (100 epoch) | 83.249 | 88.780 (100 epoch) | 96.278 | 99.123 (100 epoch) | 20.536 | 61.256 (100 epoch) | 31.284 | 59.868 (250 epoch) | 74.911 | 91.811 (100 epoch) | 13.129 | 33.186 (100 epoch) | 51.682 | 62.576 (50 epoch) | 7.335 | 21.701 (100 epoch) | 18.543 | 42.722 (250 epoch) | | cashew | 93.200 | 94.800 (150 epoch) | 96.236 | 97.492 (150 epoch) | 92.157 | 92.462 (450 epoch) | 94.681 | 95.614 (250 epoch) | 33.313 | 50.952 (250 epoch) | 37.090 | 54.438 (400 epoch) | 85.608 | 89.185 (350 epoch) | 20.785 | 28.352 (350 epoch) | 28.501 | 34.522 (150 epoch) | 12.188 | 17.797 (350 epoch) | 22.767 | 37.398 (400 epoch) | | chewinggum | 96.958 | 97.802 (300 epoch) | 98.584 | 98.939 (300 epoch) | 94.180 | 94.792 (450 epoch) | 98.012 | 98.145 (100 epoch) | 49.060 | 59.823 (50 epoch) | 55.837 | 61.088 (50 epoch) | 79.389 | 80.312 (400 epoch) | 31.448 | 35.748 (50 epoch) | 62.749 | 65.611 (50 epoch) | 20.210 | 23.754 (50 epoch) | 38.732 | 43.976 (50 epoch) | | fryum | 96.440 | 96.440 (550 epoch) | 98.261 | 98.261 (550 epoch) | 93.839 | 93.839 (550 epoch) | 97.210 | 97.210 (550 epoch) | 50.318 | 50.318 (550 epoch) | 52.619 | 52.992 (250 epoch) | 92.340 | 92.340 (550 epoch) | 26.511 | 26.511 (550 epoch) | 28.320 | 28.594 (450 epoch) | 16.529 | 16.529 (550 epoch) | 35.703 | 36.047 (250 epoch) | | macaroni1 | 91.649 | 92.104 (500 epoch) | 89.862 | 90.505 (500 epoch) | 82.464 | 83.978 (450 epoch) | 99.599 | 99.599 (550 epoch) | 19.697 | 19.697 (550 epoch) | 28.921 | 29.377 (200 epoch) | 95.852 | 96.043 (450 epoch) | 15.685 | 15.803 (300 epoch) | 37.653 | 40.765 (50 epoch) | 8.769 | 8.828 (300 epoch) | 16.905 | 17.217 (200 epoch) | | macaroni2 | 79.746 | 82.589 (50 epoch) | 70.118 | 81.196 (50 epoch) | 76.279 | 76.279 (550 epoch) | 98.585 | 99.452 (100 epoch) | 1.934 | 11.272 (50 epoch) | 4.321 | 17.568 (50 epoch) | 93.513 | 96.253 (150 epoch) | 2.269 | 10.086 (50 epoch) | 52.236 | 54.584 (450 epoch) | 1.152 | 5.433 (50 epoch) | 2.208 | 9.630 (50 epoch) | | pcb1 | 96.200 | 96.710 (300 epoch) | 94.801 | 95.525 (400 epoch) | 93.269 | 94.737 (300 epoch) | 99.803 | 99.805 (450 epoch) | 81.661 | 83.141 (450 epoch) | 74.089 | 75.588 (450 epoch) | 94.051 | 94.357 (300 epoch) | 47.001 | 47.720 (450 epoch) | 56.642 | 62.362 (50 epoch) | 32.912 | 33.816 (450 epoch) | 58.843 | 60.756 (450 epoch) | | pcb2 | 95.670 | 95.670 (550 epoch) | 95.692 | 95.692 (550 epoch) | 90.355 | 90.385 (450 epoch) | 98.993 | 99.003 (450 epoch) | 12.768 | 14.130 (50 epoch) | 23.880 | 23.977 (250 epoch) | 90.155 | 91.111 (500 epoch) | 15.672 | 16.150 (200 epoch) | 43.757 | 47.820 (150 epoch) | 8.615 | 8.892 (200 epoch) | 13.559 | 13.621 (250 epoch) | | pcb3 | 95.624 | 95.891 (350 epoch) | 95.857 | 96.056 (350 epoch) | 89.524 | 89.855 (400 epoch) | 99.141 | 99.142 (350 epoch) | 17.259 | 20.249 (150 epoch) | 27.939 | 28.078 (450 epoch) | 91.996 | 91.996 (550 epoch) | 19.464 | 20.042 (150 epoch) | 45.480 | 48.341 (50 epoch) | 10.961 | 11.295 (150 epoch) | 16.238 | 16.332 (450 epoch) | | pcb4 | 99.683 | 99.901 (100 epoch) | 99.682 | 99.901 (100 epoch) | 97.537 | 99.000 (50 epoch) | 98.714 | 98.727 (250 epoch) | 47.028 | 47.966 (250 epoch) | 47.187 | 47.709 (250 epoch) | 90.759 | 90.876 (500 epoch) | 24.690 | 26.925 (200 epoch) | 32.725 | 39.716 (50 epoch) | 15.156 | 16.537 (200 epoch) | 30.879 | 31.328 (250 epoch) | | pipe_fryum | 98.760 | 98.920 (500 epoch) | 99.355 | 99.442 (500 epoch) | 97.000 | 97.561 (200 epoch) | 99.010 | 99.114 (450 epoch) | 52.991 | 54.998 (450 epoch) | 55.927 | 58.733 (450 epoch) | 93.399 | 95.525 (150 epoch) | 31.786 | 34.684 (250 epoch) | 37.242 | 43.014 (100 epoch) | 20.357 | 22.399 (250 epoch) | 38.818 | 41.576 (450 epoch) | | Avg | 93.468 | 94.251 (100 epoch) | 93.256 | 94.481 (100 epoch) | 89.940 | 90.163 (300 epoch) | 98.256 | 98.606 (250 epoch) | 34.016 | 39.701 (250 epoch) | 39.181 | 44.125 (250 epoch) | 89.825 | 91.494 (250 epoch) | 22.384 | 25.164 (100 epoch) | 42.905 | 46.179 (50 epoch) | 13.806 | 15.755 (100 epoch) | 25.966 | 29.990 (250 epoch) | 2024-04-05 09:28:36,687 - ==> Total time: 3 days, 8:28:31 Eta: 2 days, 17:50:36 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 09:29:52,119 - Train: 55.01% [297600/541000] [550.1/1000.0] [batch_t 1.462 (1.463)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-05 09:32:17,858 - Train: 55.03% [297700/541000] [550.3/1000.0] [batch_t 1.455 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-05 09:34:43,529 - Train: 55.05% [297800/541000] [550.5/1000.0] [batch_t 1.457 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-05 09:37:09,209 - Train: 55.06% [297900/541000] [550.6/1000.0] [batch_t 1.455 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-05 09:39:34,981 - Train: 55.08% [298000/541000] [550.8/1000.0] [batch_t 1.457 (1.458)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-05 09:41:47,570 - ==> Total time: 3 days, 8:41:42 Eta: 2 days, 17:45:24 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 09:42:02,980 - Train: 55.10% [298100/541000] [551.0/1000.0] [batch_t 1.461 (1.510)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-05 09:44:28,715 - Train: 55.12% [298200/541000] [551.2/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-05 09:46:54,489 - Train: 55.14% [298300/541000] [551.4/1000.0] [batch_t 1.454 (1.458)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 09:49:20,250 - Train: 55.16% [298400/541000] [551.6/1000.0] [batch_t 1.462 (1.458)] [data_t 0.003] [optim_t 1.459] [lr 0.005000] 2024-04-05 09:51:45,971 - Train: 55.18% [298500/541000] [551.8/1000.0] [batch_t 1.450 (1.457)] [data_t 0.003] [optim_t 1.447] [lr 0.005000] 2024-04-05 09:54:11,690 - Train: 55.19% [298600/541000] [551.9/1000.0] [batch_t 1.458 (1.457)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-05 09:54:58,335 - ==> Total time: 3 days, 8:54:52 Eta: 2 days, 17:40:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 09:56:39,549 - Train: 55.21% [298700/541000] [552.1/1000.0] [batch_t 1.457 (1.462)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-05 09:59:05,303 - Train: 55.23% [298800/541000] [552.3/1000.0] [batch_t 1.447 (1.457)] [data_t 0.002] [optim_t 1.444] [lr 0.005000] 2024-04-05 10:01:31,053 - Train: 55.25% [298900/541000] [552.5/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-05 10:03:56,876 - Train: 55.27% [299000/541000] [552.7/1000.0] [batch_t 1.457 (1.458)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-05 10:06:22,631 - Train: 55.29% [299100/541000] [552.9/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-05 10:08:08,992 - ==> Total time: 3 days, 9:08:03 Eta: 2 days, 17:34:56 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 10:08:50,564 - Train: 55.30% [299200/541000] [553.0/1000.0] [batch_t 1.462 (1.471)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-05 10:11:16,222 - Train: 55.32% [299300/541000] [553.2/1000.0] [batch_t 1.459 (1.456)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-05 10:13:42,008 - Train: 55.34% [299400/541000] [553.4/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-05 10:16:07,716 - Train: 55.36% [299500/541000] [553.6/1000.0] [batch_t 1.454 (1.457)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 10:18:33,517 - Train: 55.38% [299600/541000] [553.8/1000.0] [batch_t 1.456 (1.458)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-05 10:20:59,244 - Train: 55.40% [299700/541000] [554.0/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-05 10:21:19,614 - ==> Total time: 3 days, 9:21:14 Eta: 2 days, 17:29:39 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 10:23:27,209 - Train: 55.42% [299800/541000] [554.2/1000.0] [batch_t 1.460 (1.462)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-05 10:25:52,968 - Train: 55.43% [299900/541000] [554.3/1000.0] [batch_t 1.457 (1.457)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-05 10:28:18,687 - Train: 55.45% [300000/541000] [554.5/1000.0] [batch_t 1.462 (1.457)] [data_t 0.003] [optim_t 1.459] [lr 0.005000] 2024-04-05 10:30:44,440 - Train: 55.47% [300100/541000] [554.7/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-05 10:33:10,216 - Train: 55.49% [300200/541000] [554.9/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-05 10:34:30,439 - ==> Total time: 3 days, 9:34:24 Eta: 2 days, 17:24:21 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 10:35:38,462 - Train: 55.51% [300300/541000] [555.1/1000.0] [batch_t 1.457 (1.465)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-05 10:38:04,218 - Train: 55.53% [300400/541000] [555.3/1000.0] [batch_t 1.455 (1.457)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 10:40:29,923 - Train: 55.55% [300500/541000] [555.5/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-05 10:42:55,646 - Train: 55.56% [300600/541000] [555.6/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-05 10:45:21,384 - Train: 55.58% [300700/541000] [555.8/1000.0] [batch_t 1.459 (1.457)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-05 10:47:41,372 - ==> Total time: 3 days, 9:47:35 Eta: 2 days, 17:19:00 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 10:47:49,310 - Train: 55.60% [300800/541000] [556.0/1000.0] [batch_t 1.457 (1.553)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-05 10:50:15,087 - Train: 55.62% [300900/541000] [556.2/1000.0] [batch_t 1.452 (1.458)] [data_t 0.003] [optim_t 1.449] [lr 0.005000] 2024-04-05 10:52:40,846 - Train: 55.64% [301000/541000] [556.4/1000.0] [batch_t 1.457 (1.457)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-05 10:55:06,568 - Train: 55.66% [301100/541000] [556.6/1000.0] [batch_t 1.462 (1.457)] [data_t 0.003] [optim_t 1.459] [lr 0.005000] 2024-04-05 10:57:32,309 - Train: 55.67% [301200/541000] [556.7/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-05 10:59:58,084 - Train: 55.69% [301300/541000] [556.9/1000.0] [batch_t 1.457 (1.458)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-05 11:00:52,036 - ==> Total time: 3 days, 10:00:46 Eta: 2 days, 17:13:39 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 11:02:25,923 - Train: 55.71% [301400/541000] [557.1/1000.0] [batch_t 1.454 (1.463)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-05 11:04:51,633 - Train: 55.73% [301500/541000] [557.3/1000.0] [batch_t 1.455 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-05 11:07:17,357 - Train: 55.75% [301600/541000] [557.5/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-05 11:09:43,161 - Train: 55.77% [301700/541000] [557.7/1000.0] [batch_t 1.462 (1.458)] [data_t 0.003] [optim_t 1.459] [lr 0.005000] 2024-04-05 11:11:01,666 - Train: 55.79% [301800/541000] [557.9/1000.0] [batch_t 0.607 (0.785)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 11:11:49,102 - ==> Total time: 3 days, 10:11:43 Eta: 2 days, 17:06:29 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 11:12:04,821 - Train: 55.80% [301900/541000] [558.0/1000.0] [batch_t 0.609 (0.628)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 11:13:05,619 - Train: 55.82% [302000/541000] [558.2/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 11:14:06,410 - Train: 55.84% [302100/541000] [558.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 11:15:07,206 - Train: 55.86% [302200/541000] [558.6/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 11:16:08,028 - Train: 55.88% [302300/541000] [558.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 11:17:08,833 - Train: 55.90% [302400/541000] [559.0/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 11:17:20,389 - ==> Total time: 3 days, 10:17:14 Eta: 2 days, 16:55:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 11:18:11,740 - Train: 55.91% [302500/541000] [559.1/1000.0] [batch_t 0.607 (0.613)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 11:19:12,697 - Train: 55.93% [302600/541000] [559.3/1000.0] [batch_t 0.607 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 11:20:13,570 - Train: 55.95% [302700/541000] [559.5/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 11:21:14,388 - Train: 55.97% [302800/541000] [559.7/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 11:22:15,172 - Train: 55.99% [302900/541000] [559.9/1000.0] [batch_t 0.606 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-05 11:22:51,654 - ==> Total time: 3 days, 10:22:46 Eta: 2 days, 16:43:36 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 11:23:18,087 - Train: 56.01% [303000/541000] [560.1/1000.0] [batch_t 0.609 (0.618)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 11:24:19,039 - Train: 56.03% [303100/541000] [560.3/1000.0] [batch_t 0.606 (0.609)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-05 11:25:19,851 - Train: 56.04% [303200/541000] [560.4/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-05 11:26:20,645 - Train: 56.06% [303300/541000] [560.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 11:27:21,482 - Train: 56.08% [303400/541000] [560.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 11:28:22,278 - Train: 56.10% [303500/541000] [561.0/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 11:28:22,889 - ==> Total time: 3 days, 10:28:17 Eta: 2 days, 16:32:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 11:29:25,609 - Train: 56.12% [303600/541000] [561.2/1000.0] [batch_t 0.608 (0.616)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 11:30:26,529 - Train: 56.14% [303700/541000] [561.4/1000.0] [batch_t 0.609 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 11:31:27,320 - Train: 56.16% [303800/541000] [561.6/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 11:32:28,125 - Train: 56.17% [303900/541000] [561.7/1000.0] [batch_t 0.611 (0.608)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-05 11:33:28,922 - Train: 56.19% [304000/541000] [561.9/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 11:33:54,455 - ==> Total time: 3 days, 10:33:48 Eta: 2 days, 16:20:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 11:34:32,115 - Train: 56.21% [304100/541000] [562.1/1000.0] [batch_t 0.726 (0.616)] [data_t 0.003] [optim_t 0.724] [lr 0.005000] 2024-04-05 11:35:32,912 - Train: 56.23% [304200/541000] [562.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 11:36:33,711 - Train: 56.25% [304300/541000] [562.5/1000.0] [batch_t 0.610 (0.608)] [data_t 0.003] [optim_t 0.608] [lr 0.005000] 2024-04-05 11:37:34,546 - Train: 56.27% [304400/541000] [562.7/1000.0] [batch_t 0.606 (0.608)] [data_t 0.003] [optim_t 0.603] [lr 0.005000] 2024-04-05 11:38:35,348 - Train: 56.28% [304500/541000] [562.8/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 11:39:25,815 - ==> Total time: 3 days, 10:39:20 Eta: 2 days, 16:09:25 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 11:39:38,710 - Train: 56.30% [304600/541000] [563.0/1000.0] [batch_t 0.606 (0.651)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-05 11:40:39,496 - Train: 56.32% [304700/541000] [563.2/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 11:41:40,310 - Train: 56.34% [304800/541000] [563.4/1000.0] [batch_t 0.611 (0.608)] [data_t 0.003] [optim_t 0.608] [lr 0.005000] 2024-04-05 11:42:41,116 - Train: 56.36% [304900/541000] [563.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 11:43:41,924 - Train: 56.38% [305000/541000] [563.8/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 11:44:42,709 - Train: 56.40% [305100/541000] [564.0/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-05 11:44:57,429 - ==> Total time: 3 days, 10:44:51 Eta: 2 days, 15:58:05 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 11:45:45,795 - Train: 56.41% [305200/541000] [564.1/1000.0] [batch_t 0.608 (0.612)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 11:46:46,585 - Train: 56.43% [305300/541000] [564.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 11:47:47,390 - Train: 56.45% [305400/541000] [564.5/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 11:48:48,164 - Train: 56.47% [305500/541000] [564.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 11:49:48,968 - Train: 56.49% [305600/541000] [564.9/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 11:50:28,514 - ==> Total time: 3 days, 10:50:22 Eta: 2 days, 15:46:45 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 11:50:52,072 - Train: 56.51% [305700/541000] [565.1/1000.0] [batch_t 0.609 (0.617)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 11:51:52,853 - Train: 56.52% [305800/541000] [565.2/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 11:52:53,625 - Train: 56.54% [305900/541000] [565.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 11:53:54,426 - Train: 56.56% [306000/541000] [565.6/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 11:54:55,208 - Train: 56.58% [306100/541000] [565.8/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 11:55:55,990 - Train: 56.60% [306200/541000] [566.0/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 11:55:59,640 - ==> Total time: 3 days, 10:55:54 Eta: 2 days, 15:35:26 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 11:57:00,865 - Train: 56.62% [306300/541000] [566.2/1000.0] [batch_t 0.608 (0.632)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 11:58:01,634 - Train: 56.64% [306400/541000] [566.4/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 11:59:02,395 - Train: 56.65% [306500/541000] [566.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 12:00:03,220 - Train: 56.67% [306600/541000] [566.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 12:01:04,033 - Train: 56.69% [306700/541000] [566.9/1000.0] [batch_t 0.612 (0.608)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-05 12:01:32,629 - ==> Total time: 3 days, 11:01:27 Eta: 2 days, 15:24:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 12:02:06,897 - Train: 56.71% [306800/541000] [567.1/1000.0] [batch_t 0.608 (0.614)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 12:03:07,814 - Train: 56.73% [306900/541000] [567.3/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 12:04:08,659 - Train: 56.75% [307000/541000] [567.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 12:05:09,443 - Train: 56.77% [307100/541000] [567.7/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-05 12:06:10,217 - Train: 56.78% [307200/541000] [567.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 12:07:03,692 - ==> Total time: 3 days, 11:06:58 Eta: 2 days, 15:12:54 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 12:07:14,590 - Train: 56.80% [307300/541000] [568.0/1000.0] [batch_t 0.608 (0.651)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 12:08:15,508 - Train: 56.82% [307400/541000] [568.2/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 12:09:16,298 - Train: 56.84% [307500/541000] [568.4/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 12:10:17,117 - Train: 56.86% [307600/541000] [568.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 12:11:17,878 - Train: 56.88% [307700/541000] [568.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 12:12:18,651 - Train: 56.89% [307800/541000] [568.9/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 12:12:36,273 - ==> Total time: 3 days, 11:12:30 Eta: 2 days, 15:01:40 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 12:13:21,577 - Train: 56.91% [307900/541000] [569.1/1000.0] [batch_t 0.608 (0.612)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 12:14:22,520 - Train: 56.93% [308000/541000] [569.3/1000.0] [batch_t 0.607 (0.609)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-05 12:15:23,342 - Train: 56.95% [308100/541000] [569.5/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 12:16:24,118 - Train: 56.97% [308200/541000] [569.7/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 12:17:24,822 - Train: 56.99% [308300/541000] [569.9/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 12:18:07,331 - ==> Total time: 3 days, 11:18:01 Eta: 2 days, 14:50:26 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 12:18:27,640 - Train: 57.01% [308400/541000] [570.1/1000.0] [batch_t 0.608 (0.619)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 12:19:28,333 - Train: 57.02% [308500/541000] [570.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 12:20:29,132 - Train: 57.04% [308600/541000] [570.4/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 12:21:29,876 - Train: 57.06% [308700/541000] [570.6/1000.0] [batch_t 0.609 (0.607)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 12:22:30,647 - Train: 57.08% [308800/541000] [570.8/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-05 12:23:31,460 - Train: 57.10% [308900/541000] [571.0/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 12:23:38,145 - ==> Total time: 3 days, 11:23:32 Eta: 2 days, 14:39:13 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 12:24:34,641 - Train: 57.12% [309000/541000] [571.2/1000.0] [batch_t 0.609 (0.615)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 12:25:35,441 - Train: 57.13% [309100/541000] [571.3/1000.0] [batch_t 0.605 (0.608)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-05 12:26:36,353 - Train: 57.15% [309200/541000] [571.5/1000.0] [batch_t 0.612 (0.609)] [data_t 0.003] [optim_t 0.609] [lr 0.005000] 2024-04-05 12:27:37,195 - Train: 57.17% [309300/541000] [571.7/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 12:28:37,997 - Train: 57.19% [309400/541000] [571.9/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-05 12:29:09,633 - ==> Total time: 3 days, 11:29:04 Eta: 2 days, 14:28:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 12:29:41,176 - Train: 57.21% [309500/541000] [572.1/1000.0] [batch_t 0.609 (0.616)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 12:30:41,963 - Train: 57.23% [309600/541000] [572.3/1000.0] [batch_t 0.613 (0.608)] [data_t 0.003] [optim_t 0.610] [lr 0.005000] 2024-04-05 12:31:42,754 - Train: 57.25% [309700/541000] [572.5/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 12:32:43,695 - Train: 57.26% [309800/541000] [572.6/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 12:33:44,488 - Train: 57.28% [309900/541000] [572.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 12:34:41,013 - ==> Total time: 3 days, 11:34:35 Eta: 2 days, 14:16:52 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 12:34:47,546 - Train: 57.30% [310000/541000] [573.0/1000.0] [batch_t 0.606 (0.665)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-05 12:35:48,339 - Train: 57.32% [310100/541000] [573.2/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 12:36:49,117 - Train: 57.34% [310200/541000] [573.4/1000.0] [batch_t 0.610 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-05 12:37:49,929 - Train: 57.36% [310300/541000] [573.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 12:38:50,890 - Train: 57.38% [310400/541000] [573.8/1000.0] [batch_t 0.608 (0.610)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 12:39:51,701 - Train: 57.39% [310500/541000] [573.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 12:40:12,381 - ==> Total time: 3 days, 11:40:06 Eta: 2 days, 14:05:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 12:40:54,479 - Train: 57.41% [310600/541000] [574.1/1000.0] [batch_t 0.608 (0.613)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 12:41:55,356 - Train: 57.43% [310700/541000] [574.3/1000.0] [batch_t 0.606 (0.609)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-05 12:42:56,191 - Train: 57.45% [310800/541000] [574.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 12:43:57,144 - Train: 57.47% [310900/541000] [574.7/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 12:44:57,831 - Train: 57.49% [311000/541000] [574.9/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 12:45:43,361 - ==> Total time: 3 days, 11:45:37 Eta: 2 days, 13:54:35 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 12:46:00,744 - Train: 57.50% [311100/541000] [575.0/1000.0] [batch_t 0.606 (0.623)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 12:47:01,424 - Train: 57.52% [311200/541000] [575.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 12:48:02,150 - Train: 57.54% [311300/541000] [575.4/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 12:49:02,878 - Train: 57.56% [311400/541000] [575.6/1000.0] [batch_t 0.612 (0.607)] [data_t 0.002] [optim_t 0.610] [lr 0.005000] 2024-04-05 12:50:03,599 - Train: 57.58% [311500/541000] [575.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 12:51:04,403 - Train: 57.60% [311600/541000] [576.0/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 12:51:14,115 - ==> Total time: 3 days, 11:51:08 Eta: 2 days, 13:43:28 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 12:52:07,263 - Train: 57.62% [311700/541000] [576.2/1000.0] [batch_t 0.607 (0.612)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 12:53:07,946 - Train: 57.63% [311800/541000] [576.3/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-05 12:54:08,661 - Train: 57.65% [311900/541000] [576.5/1000.0] [batch_t 0.611 (0.607)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-05 12:55:09,334 - Train: 57.67% [312000/541000] [576.7/1000.0] [batch_t 0.612 (0.607)] [data_t 0.002] [optim_t 0.610] [lr 0.005000] 2024-04-05 12:56:10,085 - Train: 57.69% [312100/541000] [576.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 12:56:44,831 - ==> Total time: 3 days, 11:56:39 Eta: 2 days, 13:32:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 12:57:13,139 - Train: 57.71% [312200/541000] [577.1/1000.0] [batch_t 0.607 (0.616)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 12:58:13,819 - Train: 57.73% [312300/541000] [577.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 12:59:14,489 - Train: 57.74% [312400/541000] [577.4/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-05 13:00:15,239 - Train: 57.76% [312500/541000] [577.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 13:01:15,950 - Train: 57.78% [312600/541000] [577.8/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 13:02:15,409 - ==> Total time: 3 days, 12:02:09 Eta: 2 days, 13:21:18 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 13:02:18,868 - Train: 57.80% [312700/541000] [578.0/1000.0] [batch_t 0.607 (0.792)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 13:03:19,772 - Train: 57.82% [312800/541000] [578.2/1000.0] [batch_t 0.614 (0.609)] [data_t 0.002] [optim_t 0.612] [lr 0.005000] 2024-04-05 13:04:20,463 - Train: 57.84% [312900/541000] [578.4/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 13:05:21,158 - Train: 57.86% [313000/541000] [578.6/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-05 13:06:21,846 - Train: 57.87% [313100/541000] [578.7/1000.0] [batch_t 0.611 (0.607)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-05 13:07:22,534 - Train: 57.89% [313200/541000] [578.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 13:07:46,211 - ==> Total time: 3 days, 12:07:40 Eta: 2 days, 13:10:14 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 13:08:25,501 - Train: 57.91% [313300/541000] [579.1/1000.0] [batch_t 0.606 (0.615)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 13:09:26,202 - Train: 57.93% [313400/541000] [579.3/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 13:10:27,033 - Train: 57.95% [313500/541000] [579.5/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 13:11:27,727 - Train: 57.97% [313600/541000] [579.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 13:12:28,476 - Train: 57.99% [313700/541000] [579.9/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 13:13:17,044 - ==> Total time: 3 days, 12:13:11 Eta: 2 days, 12:59:12 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 13:13:31,502 - Train: 58.00% [313800/541000] [580.0/1000.0] [batch_t 0.602 (0.632)] [data_t 0.002] [optim_t 0.600] [lr 0.005000] 2024-04-05 13:14:32,203 - Train: 58.02% [313900/541000] [580.2/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-05 13:15:33,079 - Train: 58.04% [314000/541000] [580.4/1000.0] [batch_t 0.610 (0.609)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-05 13:16:33,762 - Train: 58.06% [314100/541000] [580.6/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-05 13:17:34,436 - Train: 58.08% [314200/541000] [580.8/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-05 13:18:35,126 - Train: 58.10% [314300/541000] [581.0/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 13:18:47,879 - ==> Total time: 3 days, 12:18:42 Eta: 2 days, 12:48:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 13:19:38,043 - Train: 58.11% [314400/541000] [581.1/1000.0] [batch_t 0.607 (0.613)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 13:20:38,735 - Train: 58.13% [314500/541000] [581.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 13:21:39,584 - Train: 58.15% [314600/541000] [581.5/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 13:22:40,282 - Train: 58.17% [314700/541000] [581.7/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 13:23:40,989 - Train: 58.19% [314800/541000] [581.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 13:24:18,622 - ==> Total time: 3 days, 12:24:13 Eta: 2 days, 12:37:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 13:24:44,073 - Train: 58.21% [314900/541000] [582.1/1000.0] [batch_t 0.607 (0.617)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 13:25:44,754 - Train: 58.23% [315000/541000] [582.3/1000.0] [batch_t 0.610 (0.607)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-05 13:26:45,421 - Train: 58.24% [315100/541000] [582.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 13:27:46,203 - Train: 58.26% [315200/541000] [582.6/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 13:28:46,879 - Train: 58.28% [315300/541000] [582.8/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 13:29:47,554 - Train: 58.30% [315400/541000] [583.0/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 13:29:49,377 - ==> Total time: 3 days, 12:29:43 Eta: 2 days, 12:26:12 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 13:30:50,388 - Train: 58.32% [315500/541000] [583.2/1000.0] [batch_t 0.608 (0.610)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 13:31:51,052 - Train: 58.34% [315600/541000] [583.4/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 13:32:51,920 - Train: 58.35% [315700/541000] [583.5/1000.0] [batch_t 0.609 (0.609)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-05 13:33:52,601 - Train: 58.37% [315800/541000] [583.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 13:34:53,304 - Train: 58.39% [315900/541000] [583.9/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 13:35:19,998 - ==> Total time: 3 days, 12:35:14 Eta: 2 days, 12:15:14 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 13:35:56,384 - Train: 58.41% [316000/541000] [584.1/1000.0] [batch_t 0.606 (0.614)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 13:36:57,089 - Train: 58.43% [316100/541000] [584.3/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 13:37:57,821 - Train: 58.45% [316200/541000] [584.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 13:38:58,574 - Train: 58.47% [316300/541000] [584.7/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-05 13:39:59,364 - Train: 58.48% [316400/541000] [584.8/1000.0] [batch_t 0.605 (0.608)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-05 13:40:50,929 - ==> Total time: 3 days, 12:40:45 Eta: 2 days, 12:04:17 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 13:41:02,748 - Train: 58.50% [316500/541000] [585.0/1000.0] [batch_t 0.606 (0.652)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 13:42:03,435 - Train: 58.52% [316600/541000] [585.2/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 13:43:04,108 - Train: 58.54% [316700/541000] [585.4/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 13:44:04,827 - Train: 58.56% [316800/541000] [585.6/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 13:45:05,642 - Train: 58.58% [316900/541000] [585.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 13:46:06,334 - Train: 58.60% [317000/541000] [586.0/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 13:46:22,114 - ==> Total time: 3 days, 12:46:16 Eta: 2 days, 11:53:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 13:47:09,280 - Train: 58.61% [317100/541000] [586.1/1000.0] [batch_t 0.611 (0.613)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-05 13:48:09,986 - Train: 58.63% [317200/541000] [586.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 13:49:10,695 - Train: 58.65% [317300/541000] [586.5/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 13:50:11,406 - Train: 58.67% [317400/541000] [586.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 13:51:12,262 - Train: 58.69% [317500/541000] [586.9/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 13:51:52,897 - ==> Total time: 3 days, 12:51:47 Eta: 2 days, 11:42:28 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 13:52:15,294 - Train: 58.71% [317600/541000] [587.1/1000.0] [batch_t 0.606 (0.621)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 13:53:15,934 - Train: 58.72% [317700/541000] [587.2/1000.0] [batch_t 0.606 (0.606)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 13:54:16,594 - Train: 58.74% [317800/541000] [587.4/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 13:55:17,270 - Train: 58.76% [317900/541000] [587.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 13:56:17,923 - Train: 58.78% [318000/541000] [587.8/1000.0] [batch_t 0.607 (0.606)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 13:57:18,685 - Train: 58.80% [318100/541000] [588.0/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 13:57:23,541 - ==> Total time: 3 days, 12:57:17 Eta: 2 days, 11:31:34 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 13:58:21,710 - Train: 58.82% [318200/541000] [588.2/1000.0] [batch_t 0.604 (0.611)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-05 13:59:22,350 - Train: 58.84% [318300/541000] [588.4/1000.0] [batch_t 0.607 (0.606)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 14:00:23,013 - Train: 58.85% [318400/541000] [588.5/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-05 14:01:23,690 - Train: 58.87% [318500/541000] [588.7/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 14:02:24,339 - Train: 58.89% [318600/541000] [588.9/1000.0] [batch_t 0.606 (0.606)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 14:02:54,173 - ==> Total time: 3 days, 13:02:48 Eta: 2 days, 11:20:42 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 14:03:27,602 - Train: 58.91% [318700/541000] [589.1/1000.0] [batch_t 0.606 (0.617)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 14:04:28,252 - Train: 58.93% [318800/541000] [589.3/1000.0] [batch_t 0.607 (0.606)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 14:05:28,918 - Train: 58.95% [318900/541000] [589.5/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 14:06:29,578 - Train: 58.96% [319000/541000] [589.6/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-05 14:07:30,249 - Train: 58.98% [319100/541000] [589.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 14:08:24,998 - ==> Total time: 3 days, 13:08:19 Eta: 2 days, 11:09:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 14:08:33,438 - Train: 59.00% [319200/541000] [590.0/1000.0] [batch_t 0.607 (0.642)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 14:09:34,067 - Train: 59.02% [319300/541000] [590.2/1000.0] [batch_t 0.607 (0.606)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 14:10:34,706 - Train: 59.04% [319400/541000] [590.4/1000.0] [batch_t 0.607 (0.606)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 14:11:35,389 - Train: 59.06% [319500/541000] [590.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 14:12:36,097 - Train: 59.08% [319600/541000] [590.8/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 14:13:36,977 - Train: 59.09% [319700/541000] [590.9/1000.0] [batch_t 0.609 (0.609)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-05 14:13:55,809 - ==> Total time: 3 days, 13:13:50 Eta: 2 days, 10:59:01 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 14:14:40,470 - Train: 59.11% [319800/541000] [591.1/1000.0] [batch_t 0.608 (0.612)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 14:15:41,152 - Train: 59.13% [319900/541000] [591.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 14:16:41,829 - Train: 59.15% [320000/541000] [591.5/1000.0] [batch_t 0.608 (0.607)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 14:17:42,514 - Train: 59.17% [320100/541000] [591.7/1000.0] [batch_t 0.613 (0.607)] [data_t 0.002] [optim_t 0.611] [lr 0.005000] 2024-04-05 14:18:43,296 - Train: 59.19% [320200/541000] [591.9/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 14:19:26,971 - ==> Total time: 3 days, 13:19:21 Eta: 2 days, 10:48:12 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 14:19:47,155 - Train: 59.21% [320300/541000] [592.1/1000.0] [batch_t 0.607 (0.652)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 14:20:47,797 - Train: 59.22% [320400/541000] [592.2/1000.0] [batch_t 0.607 (0.606)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 14:21:48,460 - Train: 59.24% [320500/541000] [592.4/1000.0] [batch_t 0.611 (0.607)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-05 14:22:49,150 - Train: 59.26% [320600/541000] [592.6/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 14:23:49,895 - Train: 59.28% [320700/541000] [592.8/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 14:24:50,579 - Train: 59.30% [320800/541000] [593.0/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-05 14:24:58,466 - ==> Total time: 3 days, 13:24:52 Eta: 2 days, 10:37:24 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 14:25:53,814 - Train: 59.32% [320900/541000] [593.2/1000.0] [batch_t 0.607 (0.612)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 14:26:54,483 - Train: 59.33% [321000/541000] [593.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 14:27:55,145 - Train: 59.35% [321100/541000] [593.5/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 14:28:55,937 - Train: 59.37% [321200/541000] [593.7/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 14:29:56,633 - Train: 59.39% [321300/541000] [593.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 14:30:29,390 - ==> Total time: 3 days, 13:30:23 Eta: 2 days, 10:26:38 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 14:30:59,686 - Train: 59.41% [321400/541000] [594.1/1000.0] [batch_t 0.607 (0.615)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 14:32:00,372 - Train: 59.43% [321500/541000] [594.3/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 14:33:01,083 - Train: 59.45% [321600/541000] [594.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 14:34:01,950 - Train: 59.46% [321700/541000] [594.6/1000.0] [batch_t 0.606 (0.609)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 14:35:02,613 - Train: 59.48% [321800/541000] [594.8/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 14:36:00,250 - ==> Total time: 3 days, 13:35:54 Eta: 2 days, 10:15:52 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 14:36:07,127 - Train: 59.50% [321900/541000] [595.0/1000.0] [batch_t 0.606 (1.014)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 14:37:07,799 - Train: 59.52% [322000/541000] [595.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 14:38:08,477 - Train: 59.54% [322100/541000] [595.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 14:39:09,146 - Train: 59.56% [322200/541000] [595.6/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-05 14:40:09,953 - Train: 59.57% [322300/541000] [595.7/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 14:41:10,593 - Train: 59.59% [322400/541000] [595.9/1000.0] [batch_t 0.607 (0.606)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 14:41:32,451 - ==> Total time: 3 days, 13:41:26 Eta: 2 days, 10:05:08 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 14:42:13,842 - Train: 59.61% [322500/541000] [596.1/1000.0] [batch_t 0.606 (0.614)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 14:43:14,524 - Train: 59.63% [322600/541000] [596.3/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-05 14:44:15,199 - Train: 59.65% [322700/541000] [596.5/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 14:45:15,892 - Train: 59.67% [322800/541000] [596.7/1000.0] [batch_t 0.610 (0.607)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-05 14:46:16,824 - Train: 59.69% [322900/541000] [596.9/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 14:47:03,617 - ==> Total time: 3 days, 13:46:58 Eta: 2 days, 9:54:25 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 14:47:21,295 - Train: 59.70% [323000/541000] [597.0/1000.0] [batch_t 0.608 (0.645)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 14:48:22,045 - Train: 59.72% [323100/541000] [597.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 14:49:22,811 - Train: 59.74% [323200/541000] [597.4/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 14:50:23,593 - Train: 59.76% [323300/541000] [597.6/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-05 14:51:24,392 - Train: 59.78% [323400/541000] [597.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 14:52:25,320 - Train: 59.80% [323500/541000] [598.0/1000.0] [batch_t 0.607 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 14:52:36,277 - ==> Total time: 3 days, 13:52:30 Eta: 2 days, 9:43:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 14:53:28,412 - Train: 59.82% [323600/541000] [598.2/1000.0] [batch_t 0.609 (0.612)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-05 14:54:29,176 - Train: 59.83% [323700/541000] [598.3/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 14:55:29,963 - Train: 59.85% [323800/541000] [598.5/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-05 14:56:30,724 - Train: 59.87% [323900/541000] [598.7/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 14:57:31,496 - Train: 59.89% [324000/541000] [598.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 14:58:07,358 - ==> Total time: 3 days, 13:58:01 Eta: 2 days, 9:33:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 14:58:34,598 - Train: 59.91% [324100/541000] [599.1/1000.0] [batch_t 0.606 (0.617)] [data_t 0.003] [optim_t 0.603] [lr 0.005000] 2024-04-05 14:59:35,387 - Train: 59.93% [324200/541000] [599.3/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-05 15:00:36,160 - Train: 59.94% [324300/541000] [599.4/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 15:01:36,932 - Train: 59.96% [324400/541000] [599.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-05 15:02:37,727 - Train: 59.98% [324500/541000] [599.8/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 15:03:38,633 - Train: 60.00% [324600/541000] [600.0/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 15:03:50,640 - Test: 36.76% [50/136] [batch_t 0.214 (0.229)] 2024-04-05 15:04:01,397 - Test: 73.53% [100/136] [batch_t 0.223 (0.222)] 2024-04-05 15:04:10,719 - Test: 100.00% [136/136] [batch_t 0.052 (0.232)] 2024-04-05 15:04:59,957 - ==> Metric Time for candle : 0.002 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.847 (mAUROC_px) 8.186 (mAP_px) 0.938 (mF1_max_px) 25.506 (mAUPRO_px) 0.318 (mF1_px_0.2_0.8_0.1) 0.320 (mAcc_px_0.2_0.8_0.1) 0.316 (mIoU_px_0.2_0.8_0.1) 0.928 (mIoU_max_px) 2024-04-05 15:05:37,447 - ==> Metric Time for capsules : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 8.301 (mAUROC_px) 6.337 (mAP_px) 0.765 (mF1_max_px) 20.363 (mAUPRO_px) 0.252 (mF1_px_0.2_0.8_0.1) 0.251 (mAcc_px_0.2_0.8_0.1) 0.256 (mIoU_px_0.2_0.8_0.1) 0.737 (mIoU_max_px) 2024-04-05 15:06:14,167 - ==> Metric Time for cashew : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.941 (mAUROC_px) 6.208 (mAP_px) 0.706 (mF1_max_px) 20.222 (mAUPRO_px) 0.244 (mF1_px_0.2_0.8_0.1) 0.242 (mAcc_px_0.2_0.8_0.1) 0.244 (mIoU_px_0.2_0.8_0.1) 0.702 (mIoU_max_px) 2024-04-05 15:06:51,347 - ==> Metric Time for chewinggum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.978 (mAUROC_px) 6.077 (mAP_px) 0.698 (mF1_max_px) 20.773 (mAUPRO_px) 0.244 (mF1_px_0.2_0.8_0.1) 0.246 (mAcc_px_0.2_0.8_0.1) 0.242 (mIoU_px_0.2_0.8_0.1) 0.706 (mIoU_max_px) 2024-04-05 15:07:29,189 - ==> Metric Time for fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.916 (mAUROC_px) 5.862 (mAP_px) 0.741 (mF1_max_px) 21.622 (mAUPRO_px) 0.247 (mF1_px_0.2_0.8_0.1) 0.278 (mAcc_px_0.2_0.8_0.1) 0.249 (mIoU_px_0.2_0.8_0.1) 0.711 (mIoU_max_px) 2024-04-05 15:08:17,816 - ==> Metric Time for macaroni1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.918 (mAUROC_px) 8.198 (mAP_px) 0.955 (mF1_max_px) 26.296 (mAUPRO_px) 0.370 (mF1_px_0.2_0.8_0.1) 0.325 (mAcc_px_0.2_0.8_0.1) 0.329 (mIoU_px_0.2_0.8_0.1) 0.952 (mIoU_max_px) 2024-04-05 15:09:05,864 - ==> Metric Time for macaroni2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.001 (mF1_max_sp_max) 10.839 (mAUROC_px) 8.159 (mAP_px) 0.953 (mF1_max_px) 25.803 (mAUPRO_px) 0.335 (mF1_px_0.2_0.8_0.1) 0.342 (mAcc_px_0.2_0.8_0.1) 0.368 (mIoU_px_0.2_0.8_0.1) 0.964 (mIoU_max_px) 2024-04-05 15:09:54,864 - ==> Metric Time for pcb1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.956 (mAUROC_px) 8.411 (mAP_px) 0.951 (mF1_max_px) 26.512 (mAUPRO_px) 0.320 (mF1_px_0.2_0.8_0.1) 0.320 (mAcc_px_0.2_0.8_0.1) 0.322 (mIoU_px_0.2_0.8_0.1) 0.925 (mIoU_max_px) 2024-04-05 15:10:42,990 - ==> Metric Time for pcb2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.708 (mAUROC_px) 8.219 (mAP_px) 0.931 (mF1_max_px) 26.091 (mAUPRO_px) 0.320 (mF1_px_0.2_0.8_0.1) 0.320 (mAcc_px_0.2_0.8_0.1) 0.323 (mIoU_px_0.2_0.8_0.1) 0.932 (mIoU_max_px) 2024-04-05 15:11:30,555 - ==> Metric Time for pcb3 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.616 (mAUROC_px) 7.918 (mAP_px) 0.943 (mF1_max_px) 25.911 (mAUPRO_px) 0.319 (mF1_px_0.2_0.8_0.1) 0.321 (mAcc_px_0.2_0.8_0.1) 0.323 (mIoU_px_0.2_0.8_0.1) 0.932 (mIoU_max_px) 2024-04-05 15:12:19,931 - ==> Metric Time for pcb4 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.725 (mAUROC_px) 8.071 (mAP_px) 0.945 (mF1_max_px) 27.459 (mAUPRO_px) 0.320 (mF1_px_0.2_0.8_0.1) 0.318 (mAcc_px_0.2_0.8_0.1) 0.326 (mIoU_px_0.2_0.8_0.1) 0.931 (mIoU_max_px) 2024-04-05 15:12:56,170 - ==> Metric Time for pipe_fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.808 (mAUROC_px) 5.822 (mAP_px) 0.708 (mF1_max_px) 20.289 (mAUPRO_px) 0.236 (mF1_px_0.2_0.8_0.1) 0.235 (mAcc_px_0.2_0.8_0.1) 0.235 (mIoU_px_0.2_0.8_0.1) 0.688 (mIoU_max_px) 2024-04-05 15:12:56,191 - | Name | mAUROC_sp_max | mAUROC_sp_max (Max) | mAP_sp_max | mAP_sp_max (Max) | mF1_max_sp_max | mF1_max_sp_max (Max) | mAUROC_px | mAUROC_px (Max) | mAP_px | mAP_px (Max) | mF1_max_px | mF1_max_px (Max) | mAUPRO_px | mAUPRO_px (Max) | mF1_px_0.2_0.8_0.1 | mF1_px_0.2_0.8_0.1 (Max) | mAcc_px_0.2_0.8_0.1 | mAcc_px_0.2_0.8_0.1 (Max) | mIoU_px_0.2_0.8_0.1 | mIoU_px_0.2_0.8_0.1 (Max) | mIoU_max_px | mIoU_max_px (Max) | |:----------:|:---------------:|:---------------------:|:------------:|:------------------:|:----------------:|:----------------------:|:-----------:|:------------------:|:--------:|:------------------:|:------------:|:------------------:|:-----------:|:------------------:|:--------------------:|:--------------------------:|:---------------------:|:---------------------------:|:---------------------:|:---------------------------:|:-------------:|:-------------------:| | candle | 95.980 | 97.100 (50 epoch) | 96.403 | 97.346 (50 epoch) | 89.100 | 91.542 (50 epoch) | 99.050 | 99.098 (250 epoch) | 21.926 | 23.463 (150 epoch) | 30.963 | 32.654 (200 epoch) | 95.380 | 96.087 (300 epoch) | 21.044 | 21.331 (500 epoch) | 41.351 | 41.351 (600 epoch) | 12.007 | 12.185 (500 epoch) | 18.317 | 19.513 (200 epoch) | | capsules | 74.167 | 91.750 (100 epoch) | 76.298 | 94.964 (100 epoch) | 82.192 | 88.780 (100 epoch) | 95.053 | 99.123 (100 epoch) | 11.695 | 61.256 (100 epoch) | 22.434 | 59.868 (250 epoch) | 66.099 | 91.811 (100 epoch) | 9.679 | 33.186 (100 epoch) | 52.304 | 62.576 (50 epoch) | 5.224 | 21.701 (100 epoch) | 12.634 | 42.722 (250 epoch) | | cashew | 92.760 | 94.800 (150 epoch) | 95.715 | 97.492 (150 epoch) | 91.089 | 92.462 (450 epoch) | 95.179 | 95.614 (250 epoch) | 37.718 | 50.952 (250 epoch) | 43.740 | 54.438 (400 epoch) | 85.810 | 89.185 (350 epoch) | 22.034 | 28.352 (350 epoch) | 27.464 | 34.522 (150 epoch) | 13.218 | 17.797 (350 epoch) | 27.992 | 37.398 (400 epoch) | | chewinggum | 95.722 | 97.802 (300 epoch) | 98.117 | 98.939 (300 epoch) | 93.617 | 94.792 (450 epoch) | 97.666 | 98.145 (100 epoch) | 38.401 | 59.823 (50 epoch) | 44.375 | 61.088 (50 epoch) | 78.302 | 80.312 (400 epoch) | 24.554 | 35.748 (50 epoch) | 57.118 | 65.611 (50 epoch) | 14.892 | 23.754 (50 epoch) | 28.514 | 43.976 (50 epoch) | | fryum | 96.060 | 96.440 (550 epoch) | 98.063 | 98.261 (550 epoch) | 92.381 | 93.839 (550 epoch) | 97.175 | 97.210 (550 epoch) | 50.672 | 50.672 (600 epoch) | 53.189 | 53.189 (600 epoch) | 91.685 | 92.340 (550 epoch) | 24.971 | 26.511 (550 epoch) | 25.794 | 28.594 (450 epoch) | 15.595 | 16.529 (550 epoch) | 36.230 | 36.230 (600 epoch) | | macaroni1 | 91.700 | 92.104 (500 epoch) | 89.758 | 90.505 (500 epoch) | 82.178 | 83.978 (450 epoch) | 99.576 | 99.599 (550 epoch) | 19.056 | 19.697 (550 epoch) | 28.031 | 29.377 (200 epoch) | 95.867 | 96.043 (450 epoch) | 15.791 | 15.803 (300 epoch) | 38.113 | 40.765 (50 epoch) | 8.796 | 8.828 (300 epoch) | 16.300 | 17.217 (200 epoch) | | macaroni2 | 78.042 | 82.589 (50 epoch) | 66.399 | 81.196 (50 epoch) | 76.712 | 76.712 (600 epoch) | 95.773 | 99.452 (100 epoch) | 0.373 | 11.272 (50 epoch) | 1.566 | 17.568 (50 epoch) | 82.649 | 96.253 (150 epoch) | 0.833 | 10.086 (50 epoch) | 36.053 | 54.584 (450 epoch) | 0.419 | 5.433 (50 epoch) | 0.789 | 9.630 (50 epoch) | | pcb1 | 96.250 | 96.710 (300 epoch) | 95.024 | 95.525 (400 epoch) | 93.596 | 94.737 (300 epoch) | 99.801 | 99.805 (450 epoch) | 81.101 | 83.141 (450 epoch) | 73.878 | 75.588 (450 epoch) | 94.861 | 94.861 (600 epoch) | 46.274 | 47.720 (450 epoch) | 54.576 | 62.362 (50 epoch) | 32.265 | 33.816 (450 epoch) | 58.576 | 60.756 (450 epoch) | | pcb2 | 95.550 | 95.670 (550 epoch) | 95.499 | 95.692 (550 epoch) | 90.385 | 90.385 (450 epoch) | 99.008 | 99.008 (600 epoch) | 12.671 | 14.130 (50 epoch) | 23.773 | 23.977 (250 epoch) | 91.011 | 91.111 (500 epoch) | 15.584 | 16.150 (200 epoch) | 43.866 | 47.820 (150 epoch) | 8.562 | 8.892 (200 epoch) | 13.490 | 13.621 (250 epoch) | | pcb3 | 95.515 | 95.891 (350 epoch) | 95.805 | 96.056 (350 epoch) | 89.000 | 89.855 (400 epoch) | 99.112 | 99.142 (350 epoch) | 16.482 | 20.249 (150 epoch) | 27.627 | 28.078 (450 epoch) | 91.901 | 91.996 (550 epoch) | 18.873 | 20.042 (150 epoch) | 45.239 | 48.341 (50 epoch) | 10.601 | 11.295 (150 epoch) | 16.028 | 16.332 (450 epoch) | | pcb4 | 99.663 | 99.901 (100 epoch) | 99.663 | 99.901 (100 epoch) | 97.537 | 99.000 (50 epoch) | 98.740 | 98.740 (600 epoch) | 47.613 | 47.966 (250 epoch) | 47.029 | 47.709 (250 epoch) | 90.669 | 90.876 (500 epoch) | 24.736 | 26.925 (200 epoch) | 32.243 | 39.716 (50 epoch) | 15.218 | 16.537 (200 epoch) | 30.744 | 31.328 (250 epoch) | | pipe_fryum | 98.880 | 98.920 (500 epoch) | 99.438 | 99.442 (500 epoch) | 97.059 | 97.561 (200 epoch) | 98.969 | 99.114 (450 epoch) | 52.332 | 54.998 (450 epoch) | 55.016 | 58.733 (450 epoch) | 92.833 | 95.525 (150 epoch) | 28.634 | 34.684 (250 epoch) | 32.534 | 43.014 (100 epoch) | 18.277 | 22.399 (250 epoch) | 37.947 | 41.576 (450 epoch) | | Avg | 92.524 | 94.251 (100 epoch) | 92.182 | 94.481 (100 epoch) | 89.570 | 90.163 (300 epoch) | 97.925 | 98.606 (250 epoch) | 32.503 | 39.701 (250 epoch) | 37.635 | 44.125 (250 epoch) | 88.089 | 91.494 (250 epoch) | 21.084 | 25.164 (100 epoch) | 40.555 | 46.179 (50 epoch) | 12.923 | 15.755 (100 epoch) | 24.797 | 29.990 (250 epoch) | 2024-04-05 15:12:56,574 - ==> Total time: 3 days, 14:12:51 Eta: 2 days, 9:28:34 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 15:15:25,289 - Train: 60.02% [324700/541000] [600.2/1000.0] [batch_t 1.449 (1.460)] [data_t 0.003] [optim_t 1.446] [lr 0.005000] 2024-04-05 15:17:50,989 - Train: 60.04% [324800/541000] [600.4/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-05 15:20:16,684 - Train: 60.06% [324900/541000] [600.6/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-05 15:22:40,395 - Train: 60.07% [325000/541000] [600.7/1000.0] [batch_t 1.419 (1.437)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 15:25:02,075 - Train: 60.09% [325100/541000] [600.9/1000.0] [batch_t 1.416 (1.417)] [data_t 0.002] [optim_t 1.414] [lr 0.005000] 2024-04-05 15:26:00,187 - ==> Total time: 3 days, 14:25:54 Eta: 2 days, 9:22:53 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 15:27:27,735 - Train: 60.11% [325200/541000] [601.1/1000.0] [batch_t 1.414 (1.446)] [data_t 0.002] [optim_t 1.412] [lr 0.005000] 2024-04-05 15:29:49,381 - Train: 60.13% [325300/541000] [601.3/1000.0] [batch_t 1.420 (1.416)] [data_t 0.002] [optim_t 1.418] [lr 0.005000] 2024-04-05 15:32:11,079 - Train: 60.15% [325400/541000] [601.5/1000.0] [batch_t 1.413 (1.417)] [data_t 0.002] [optim_t 1.411] [lr 0.005000] 2024-04-05 15:34:32,739 - Train: 60.17% [325500/541000] [601.7/1000.0] [batch_t 1.414 (1.417)] [data_t 0.002] [optim_t 1.412] [lr 0.005000] 2024-04-05 15:36:54,489 - Train: 60.18% [325600/541000] [601.8/1000.0] [batch_t 1.418 (1.417)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-05 15:38:50,678 - ==> Total time: 3 days, 14:38:45 Eta: 2 days, 9:17:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 15:39:18,772 - Train: 60.20% [325700/541000] [602.0/1000.0] [batch_t 1.418 (1.447)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-05 15:41:40,453 - Train: 60.22% [325800/541000] [602.2/1000.0] [batch_t 1.413 (1.417)] [data_t 0.002] [optim_t 1.411] [lr 0.005000] 2024-04-05 15:44:02,247 - Train: 60.24% [325900/541000] [602.4/1000.0] [batch_t 1.420 (1.418)] [data_t 0.002] [optim_t 1.418] [lr 0.005000] 2024-04-05 15:46:23,891 - Train: 60.26% [326000/541000] [602.6/1000.0] [batch_t 1.420 (1.416)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 15:48:45,633 - Train: 60.28% [326100/541000] [602.8/1000.0] [batch_t 1.420 (1.417)] [data_t 0.002] [optim_t 1.418] [lr 0.005000] 2024-04-05 15:51:07,323 - Train: 60.30% [326200/541000] [603.0/1000.0] [batch_t 1.413 (1.417)] [data_t 0.002] [optim_t 1.411] [lr 0.005000] 2024-04-05 15:51:39,894 - ==> Total time: 3 days, 14:51:34 Eta: 2 days, 9:11:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 15:53:31,589 - Train: 60.31% [326300/541000] [603.1/1000.0] [batch_t 1.415 (1.422)] [data_t 0.002] [optim_t 1.413] [lr 0.005000] 2024-04-05 15:55:53,257 - Train: 60.33% [326400/541000] [603.3/1000.0] [batch_t 1.422 (1.417)] [data_t 0.002] [optim_t 1.420] [lr 0.005000] 2024-04-05 15:58:15,045 - Train: 60.35% [326500/541000] [603.5/1000.0] [batch_t 1.420 (1.418)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 16:00:36,817 - Train: 60.37% [326600/541000] [603.7/1000.0] [batch_t 1.418 (1.418)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-05 16:02:58,515 - Train: 60.39% [326700/541000] [603.9/1000.0] [batch_t 1.420 (1.417)] [data_t 0.002] [optim_t 1.418] [lr 0.005000] 2024-04-05 16:04:29,233 - ==> Total time: 3 days, 15:04:23 Eta: 2 days, 9:05:15 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 16:05:22,694 - Train: 60.41% [326800/541000] [604.1/1000.0] [batch_t 1.414 (1.430)] [data_t 0.002] [optim_t 1.412] [lr 0.005000] 2024-04-05 16:07:44,402 - Train: 60.43% [326900/541000] [604.3/1000.0] [batch_t 1.419 (1.417)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 16:10:06,118 - Train: 60.44% [327000/541000] [604.4/1000.0] [batch_t 1.418 (1.417)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-05 16:12:27,761 - Train: 60.46% [327100/541000] [604.6/1000.0] [batch_t 1.420 (1.416)] [data_t 0.002] [optim_t 1.418] [lr 0.005000] 2024-04-05 16:14:49,415 - Train: 60.48% [327200/541000] [604.8/1000.0] [batch_t 1.411 (1.416)] [data_t 0.002] [optim_t 1.409] [lr 0.005000] 2024-04-05 16:17:11,154 - Train: 60.50% [327300/541000] [605.0/1000.0] [batch_t 1.409 (1.417)] [data_t 0.002] [optim_t 1.407] [lr 0.005000] 2024-04-05 16:17:18,234 - ==> Total time: 3 days, 15:17:12 Eta: 2 days, 8:59:20 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 16:19:35,338 - Train: 60.52% [327400/541000] [605.2/1000.0] [batch_t 1.419 (1.420)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 16:21:57,032 - Train: 60.54% [327500/541000] [605.4/1000.0] [batch_t 1.413 (1.417)] [data_t 0.002] [optim_t 1.411] [lr 0.005000] 2024-04-05 16:24:18,781 - Train: 60.55% [327600/541000] [605.5/1000.0] [batch_t 1.419 (1.417)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 16:26:40,486 - Train: 60.57% [327700/541000] [605.7/1000.0] [batch_t 1.410 (1.417)] [data_t 0.002] [optim_t 1.408] [lr 0.005000] 2024-04-05 16:29:02,164 - Train: 60.59% [327800/541000] [605.9/1000.0] [batch_t 1.418 (1.417)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-05 16:30:07,351 - ==> Total time: 3 days, 15:30:01 Eta: 2 days, 8:53:23 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 16:31:26,317 - Train: 60.61% [327900/541000] [606.1/1000.0] [batch_t 1.419 (1.428)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 16:33:47,996 - Train: 60.63% [328000/541000] [606.3/1000.0] [batch_t 1.417 (1.417)] [data_t 0.002] [optim_t 1.414] [lr 0.005000] 2024-04-05 16:36:09,608 - Train: 60.65% [328100/541000] [606.5/1000.0] [batch_t 1.418 (1.416)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-05 16:38:31,258 - Train: 60.67% [328200/541000] [606.7/1000.0] [batch_t 1.415 (1.416)] [data_t 0.002] [optim_t 1.413] [lr 0.005000] 2024-04-05 16:40:52,968 - Train: 60.68% [328300/541000] [606.8/1000.0] [batch_t 1.417 (1.417)] [data_t 0.002] [optim_t 1.415] [lr 0.005000] 2024-04-05 16:42:56,261 - ==> Total time: 3 days, 15:42:50 Eta: 2 days, 8:47:24 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 16:43:16,961 - Train: 60.70% [328400/541000] [607.0/1000.0] [batch_t 1.415 (1.451)] [data_t 0.002] [optim_t 1.412] [lr 0.005000] 2024-04-05 16:45:38,699 - Train: 60.72% [328500/541000] [607.2/1000.0] [batch_t 1.410 (1.417)] [data_t 0.002] [optim_t 1.408] [lr 0.005000] 2024-04-05 16:48:00,424 - Train: 60.74% [328600/541000] [607.4/1000.0] [batch_t 1.413 (1.417)] [data_t 0.002] [optim_t 1.410] [lr 0.005000] 2024-04-05 16:50:22,096 - Train: 60.76% [328700/541000] [607.6/1000.0] [batch_t 1.418 (1.417)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-05 16:52:43,789 - Train: 60.78% [328800/541000] [607.8/1000.0] [batch_t 1.420 (1.417)] [data_t 0.002] [optim_t 1.418] [lr 0.005000] 2024-04-05 16:55:05,471 - Train: 60.79% [328900/541000] [607.9/1000.0] [batch_t 1.410 (1.417)] [data_t 0.002] [optim_t 1.407] [lr 0.005000] 2024-04-05 16:55:45,151 - ==> Total time: 3 days, 15:55:39 Eta: 2 days, 8:41:24 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 16:57:29,683 - Train: 60.81% [329000/541000] [608.1/1000.0] [batch_t 1.420 (1.422)] [data_t 0.002] [optim_t 1.418] [lr 0.005000] 2024-04-05 16:59:51,308 - Train: 60.83% [329100/541000] [608.3/1000.0] [batch_t 1.411 (1.416)] [data_t 0.002] [optim_t 1.409] [lr 0.005000] 2024-04-05 17:02:12,954 - Train: 60.85% [329200/541000] [608.5/1000.0] [batch_t 1.419 (1.416)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 17:04:34,671 - Train: 60.87% [329300/541000] [608.7/1000.0] [batch_t 1.419 (1.417)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 17:06:56,375 - Train: 60.89% [329400/541000] [608.9/1000.0] [batch_t 1.413 (1.417)] [data_t 0.002] [optim_t 1.411] [lr 0.005000] 2024-04-05 17:08:34,118 - ==> Total time: 3 days, 16:08:28 Eta: 2 days, 8:35:23 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 17:09:20,184 - Train: 60.91% [329500/541000] [609.1/1000.0] [batch_t 1.418 (1.431)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-05 17:11:41,919 - Train: 60.92% [329600/541000] [609.2/1000.0] [batch_t 1.413 (1.417)] [data_t 0.002] [optim_t 1.411] [lr 0.005000] 2024-04-05 17:14:03,607 - Train: 60.94% [329700/541000] [609.4/1000.0] [batch_t 1.414 (1.417)] [data_t 0.002] [optim_t 1.412] [lr 0.005000] 2024-04-05 17:16:25,226 - Train: 60.96% [329800/541000] [609.6/1000.0] [batch_t 1.418 (1.416)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-05 17:18:46,829 - Train: 60.98% [329900/541000] [609.8/1000.0] [batch_t 1.415 (1.416)] [data_t 0.004] [optim_t 1.411] [lr 0.005000] 2024-04-05 17:21:08,479 - Train: 61.00% [330000/541000] [610.0/1000.0] [batch_t 1.409 (1.416)] [data_t 0.002] [optim_t 1.407] [lr 0.005000] 2024-04-05 17:21:22,648 - ==> Total time: 3 days, 16:21:17 Eta: 2 days, 8:29:20 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 17:23:32,337 - Train: 61.02% [330100/541000] [610.2/1000.0] [batch_t 1.413 (1.419)] [data_t 0.002] [optim_t 1.411] [lr 0.005000] 2024-04-05 17:25:53,975 - Train: 61.04% [330200/541000] [610.4/1000.0] [batch_t 1.415 (1.416)] [data_t 0.002] [optim_t 1.412] [lr 0.005000] 2024-04-05 17:28:15,604 - Train: 61.05% [330300/541000] [610.5/1000.0] [batch_t 1.416 (1.416)] [data_t 0.002] [optim_t 1.414] [lr 0.005000] 2024-04-05 17:30:37,365 - Train: 61.07% [330400/541000] [610.7/1000.0] [batch_t 1.415 (1.418)] [data_t 0.002] [optim_t 1.413] [lr 0.005000] 2024-04-05 17:32:59,074 - Train: 61.09% [330500/541000] [610.9/1000.0] [batch_t 1.420 (1.417)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 17:34:11,360 - ==> Total time: 3 days, 16:34:05 Eta: 2 days, 8:23:16 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 17:35:22,950 - Train: 61.11% [330600/541000] [611.1/1000.0] [batch_t 1.415 (1.425)] [data_t 0.002] [optim_t 1.413] [lr 0.005000] 2024-04-05 17:37:44,585 - Train: 61.13% [330700/541000] [611.3/1000.0] [batch_t 1.420 (1.416)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 17:40:06,233 - Train: 61.15% [330800/541000] [611.5/1000.0] [batch_t 1.418 (1.416)] [data_t 0.002] [optim_t 1.415] [lr 0.005000] 2024-04-05 17:42:27,936 - Train: 61.16% [330900/541000] [611.6/1000.0] [batch_t 1.421 (1.417)] [data_t 0.002] [optim_t 1.419] [lr 0.005000] 2024-04-05 17:44:49,601 - Train: 61.18% [331000/541000] [611.8/1000.0] [batch_t 1.413 (1.417)] [data_t 0.002] [optim_t 1.411] [lr 0.005000] 2024-04-05 17:46:59,939 - ==> Total time: 3 days, 16:46:54 Eta: 2 days, 8:17:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 17:47:15,068 - Train: 61.20% [331100/541000] [612.0/1000.0] [batch_t 1.420 (1.634)] [data_t 0.002] [optim_t 1.418] [lr 0.005000] 2024-04-05 17:49:36,713 - Train: 61.22% [331200/541000] [612.2/1000.0] [batch_t 1.418 (1.416)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-05 17:51:58,325 - Train: 61.24% [331300/541000] [612.4/1000.0] [batch_t 1.410 (1.416)] [data_t 0.002] [optim_t 1.408] [lr 0.005000] 2024-04-05 17:54:19,925 - Train: 61.26% [331400/541000] [612.6/1000.0] [batch_t 1.419 (1.416)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-05 17:56:41,542 - Train: 61.28% [331500/541000] [612.8/1000.0] [batch_t 1.418 (1.416)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-05 17:59:03,248 - Train: 61.29% [331600/541000] [612.9/1000.0] [batch_t 1.412 (1.417)] [data_t 0.002] [optim_t 1.410] [lr 0.005000] 2024-04-05 17:59:50,054 - ==> Total time: 3 days, 16:59:44 Eta: 2 days, 8:11:05 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 18:01:27,316 - Train: 61.31% [331700/541000] [613.1/1000.0] [batch_t 1.414 (1.422)] [data_t 0.002] [optim_t 1.412] [lr 0.005000] 2024-04-05 18:03:48,927 - Train: 61.33% [331800/541000] [613.3/1000.0] [batch_t 1.407 (1.416)] [data_t 0.002] [optim_t 1.404] [lr 0.005000] 2024-04-05 18:06:10,573 - Train: 61.35% [331900/541000] [613.5/1000.0] [batch_t 1.418 (1.416)] [data_t 0.002] [optim_t 1.415] [lr 0.005000] 2024-04-05 18:08:32,242 - Train: 61.37% [332000/541000] [613.7/1000.0] [batch_t 1.414 (1.417)] [data_t 0.002] [optim_t 1.412] [lr 0.005000] 2024-04-05 18:10:53,960 - Train: 61.39% [332100/541000] [613.9/1000.0] [batch_t 1.417 (1.417)] [data_t 0.002] [optim_t 1.415] [lr 0.005000] 2024-04-05 18:12:38,852 - ==> Total time: 3 days, 17:12:33 Eta: 2 days, 8:04:57 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 18:13:18,233 - Train: 61.40% [332200/541000] [614.0/1000.0] [batch_t 1.418 (1.435)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-05 18:15:39,824 - Train: 61.42% [332300/541000] [614.2/1000.0] [batch_t 1.415 (1.416)] [data_t 0.002] [optim_t 1.413] [lr 0.005000] 2024-04-05 18:18:01,509 - Train: 61.44% [332400/541000] [614.4/1000.0] [batch_t 1.420 (1.417)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 18:20:23,181 - Train: 61.46% [332500/541000] [614.6/1000.0] [batch_t 1.415 (1.417)] [data_t 0.002] [optim_t 1.413] [lr 0.005000] 2024-04-05 18:22:44,790 - Train: 61.48% [332600/541000] [614.8/1000.0] [batch_t 1.415 (1.416)] [data_t 0.002] [optim_t 1.412] [lr 0.005000] 2024-04-05 18:25:06,316 - Train: 61.50% [332700/541000] [615.0/1000.0] [batch_t 1.408 (1.415)] [data_t 0.002] [optim_t 1.406] [lr 0.005000] 2024-04-05 18:25:27,534 - ==> Total time: 3 days, 17:25:21 Eta: 2 days, 7:58:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 18:27:33,183 - Train: 61.52% [332800/541000] [615.2/1000.0] [batch_t 1.418 (1.439)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-05 18:29:54,797 - Train: 61.53% [332900/541000] [615.3/1000.0] [batch_t 1.419 (1.416)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 18:32:16,462 - Train: 61.55% [333000/541000] [615.5/1000.0] [batch_t 1.411 (1.417)] [data_t 0.002] [optim_t 1.409] [lr 0.005000] 2024-04-05 18:34:38,102 - Train: 61.57% [333100/541000] [615.7/1000.0] [batch_t 1.418 (1.416)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-05 18:36:59,751 - Train: 61.59% [333200/541000] [615.9/1000.0] [batch_t 1.418 (1.416)] [data_t 0.002] [optim_t 1.415] [lr 0.005000] 2024-04-05 18:38:19,097 - ==> Total time: 3 days, 17:38:13 Eta: 2 days, 7:52:39 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 18:39:25,804 - Train: 61.61% [333300/541000] [616.1/1000.0] [batch_t 1.420 (1.441)] [data_t 0.002] [optim_t 1.418] [lr 0.005000] 2024-04-05 18:41:47,486 - Train: 61.63% [333400/541000] [616.3/1000.0] [batch_t 1.413 (1.417)] [data_t 0.002] [optim_t 1.411] [lr 0.005000] 2024-04-05 18:44:09,157 - Train: 61.65% [333500/541000] [616.5/1000.0] [batch_t 1.415 (1.417)] [data_t 0.002] [optim_t 1.413] [lr 0.005000] 2024-04-05 18:46:30,769 - Train: 61.66% [333600/541000] [616.6/1000.0] [batch_t 1.413 (1.416)] [data_t 0.002] [optim_t 1.411] [lr 0.005000] 2024-04-05 18:48:52,510 - Train: 61.68% [333700/541000] [616.8/1000.0] [batch_t 1.419 (1.417)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 18:51:09,941 - ==> Total time: 3 days, 17:51:04 Eta: 2 days, 7:46:29 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 18:51:16,839 - Train: 61.70% [333800/541000] [617.0/1000.0] [batch_t 1.415 (1.622)] [data_t 0.002] [optim_t 1.412] [lr 0.005000] 2024-04-05 18:53:38,475 - Train: 61.72% [333900/541000] [617.2/1000.0] [batch_t 1.419 (1.416)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 18:56:00,056 - Train: 61.74% [334000/541000] [617.4/1000.0] [batch_t 1.417 (1.416)] [data_t 0.002] [optim_t 1.415] [lr 0.005000] 2024-04-05 18:58:21,787 - Train: 61.76% [334100/541000] [617.6/1000.0] [batch_t 1.420 (1.417)] [data_t 0.002] [optim_t 1.418] [lr 0.005000] 2024-04-05 19:00:43,444 - Train: 61.77% [334200/541000] [617.7/1000.0] [batch_t 1.416 (1.416)] [data_t 0.002] [optim_t 1.413] [lr 0.005000] 2024-04-05 19:03:05,024 - Train: 61.79% [334300/541000] [617.9/1000.0] [batch_t 1.420 (1.416)] [data_t 0.002] [optim_t 1.418] [lr 0.005000] 2024-04-05 19:03:58,822 - ==> Total time: 3 days, 18:03:53 Eta: 2 days, 7:40:16 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 19:05:32,799 - Train: 61.81% [334400/541000] [618.1/1000.0] [batch_t 1.410 (1.456)] [data_t 0.002] [optim_t 1.408] [lr 0.005000] 2024-04-05 19:07:54,437 - Train: 61.83% [334500/541000] [618.3/1000.0] [batch_t 1.419 (1.416)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-05 19:10:16,056 - Train: 61.85% [334600/541000] [618.5/1000.0] [batch_t 1.419 (1.416)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 19:12:37,611 - Train: 61.87% [334700/541000] [618.7/1000.0] [batch_t 1.418 (1.415)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-05 19:14:59,197 - Train: 61.89% [334800/541000] [618.9/1000.0] [batch_t 1.413 (1.416)] [data_t 0.002] [optim_t 1.411] [lr 0.005000] 2024-04-05 19:16:51,035 - ==> Total time: 3 days, 18:16:45 Eta: 2 days, 7:34:03 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 19:17:24,754 - Train: 61.90% [334900/541000] [619.0/1000.0] [batch_t 1.418 (1.463)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-05 19:19:46,370 - Train: 61.92% [335000/541000] [619.2/1000.0] [batch_t 1.414 (1.416)] [data_t 0.002] [optim_t 1.412] [lr 0.005000] 2024-04-05 19:22:08,036 - Train: 61.94% [335100/541000] [619.4/1000.0] [batch_t 1.415 (1.417)] [data_t 0.002] [optim_t 1.412] [lr 0.005000] 2024-04-05 19:24:29,592 - Train: 61.96% [335200/541000] [619.6/1000.0] [batch_t 1.419 (1.415)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 19:26:51,092 - Train: 61.98% [335300/541000] [619.8/1000.0] [batch_t 1.419 (1.415)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 19:29:12,731 - Train: 62.00% [335400/541000] [620.0/1000.0] [batch_t 1.417 (1.416)] [data_t 0.002] [optim_t 1.415] [lr 0.005000] 2024-04-05 19:29:41,027 - ==> Total time: 3 days, 18:29:35 Eta: 2 days, 7:27:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 19:31:40,708 - Train: 62.01% [335500/541000] [620.1/1000.0] [batch_t 1.412 (1.451)] [data_t 0.002] [optim_t 1.410] [lr 0.005000] 2024-04-05 19:34:02,291 - Train: 62.03% [335600/541000] [620.3/1000.0] [batch_t 1.418 (1.416)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-05 19:36:23,867 - Train: 62.05% [335700/541000] [620.5/1000.0] [batch_t 1.417 (1.416)] [data_t 0.002] [optim_t 1.414] [lr 0.005000] 2024-04-05 19:38:45,438 - Train: 62.07% [335800/541000] [620.7/1000.0] [batch_t 1.412 (1.416)] [data_t 0.002] [optim_t 1.410] [lr 0.005000] 2024-04-05 19:41:06,955 - Train: 62.09% [335900/541000] [620.9/1000.0] [batch_t 1.419 (1.415)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 19:42:33,284 - ==> Total time: 3 days, 18:42:27 Eta: 2 days, 7:21:34 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 19:43:32,459 - Train: 62.11% [336000/541000] [621.1/1000.0] [batch_t 1.418 (1.450)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-05 19:45:54,125 - Train: 62.13% [336100/541000] [621.3/1000.0] [batch_t 1.415 (1.417)] [data_t 0.002] [optim_t 1.413] [lr 0.005000] 2024-04-05 19:48:15,756 - Train: 62.14% [336200/541000] [621.4/1000.0] [batch_t 1.419 (1.416)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 19:50:37,473 - Train: 62.16% [336300/541000] [621.6/1000.0] [batch_t 1.423 (1.417)] [data_t 0.004] [optim_t 1.419] [lr 0.005000] 2024-04-05 19:52:59,073 - Train: 62.18% [336400/541000] [621.8/1000.0] [batch_t 1.409 (1.416)] [data_t 0.002] [optim_t 1.407] [lr 0.005000] 2024-04-05 19:55:20,668 - Train: 62.20% [336500/541000] [622.0/1000.0] [batch_t 1.414 (1.416)] [data_t 0.002] [optim_t 1.412] [lr 0.005000] 2024-04-05 19:55:23,502 - ==> Total time: 3 days, 18:55:17 Eta: 2 days, 7:15:16 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 19:57:45,875 - Train: 62.22% [336600/541000] [622.2/1000.0] [batch_t 1.414 (1.426)] [data_t 0.002] [optim_t 1.412] [lr 0.005000] 2024-04-05 20:00:07,573 - Train: 62.24% [336700/541000] [622.4/1000.0] [batch_t 1.417 (1.417)] [data_t 0.002] [optim_t 1.415] [lr 0.005000] 2024-04-05 20:02:29,250 - Train: 62.26% [336800/541000] [622.6/1000.0] [batch_t 1.420 (1.417)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 20:04:50,931 - Train: 62.27% [336900/541000] [622.7/1000.0] [batch_t 1.411 (1.417)] [data_t 0.002] [optim_t 1.409] [lr 0.005000] 2024-04-05 20:07:12,561 - Train: 62.29% [337000/541000] [622.9/1000.0] [batch_t 1.419 (1.416)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-05 20:08:13,488 - ==> Total time: 3 days, 19:08:07 Eta: 2 days, 7:08:57 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 20:09:37,533 - Train: 62.31% [337100/541000] [623.1/1000.0] [batch_t 1.419 (1.429)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 20:11:59,146 - Train: 62.33% [337200/541000] [623.3/1000.0] [batch_t 1.415 (1.416)] [data_t 0.002] [optim_t 1.412] [lr 0.005000] 2024-04-05 20:14:20,711 - Train: 62.35% [337300/541000] [623.5/1000.0] [batch_t 1.417 (1.416)] [data_t 0.002] [optim_t 1.415] [lr 0.005000] 2024-04-05 20:16:42,457 - Train: 62.37% [337400/541000] [623.7/1000.0] [batch_t 1.420 (1.417)] [data_t 0.002] [optim_t 1.418] [lr 0.005000] 2024-04-05 20:19:04,065 - Train: 62.38% [337500/541000] [623.8/1000.0] [batch_t 1.419 (1.416)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-05 20:21:03,093 - ==> Total time: 3 days, 19:20:57 Eta: 2 days, 7:02:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 20:21:29,556 - Train: 62.40% [337600/541000] [624.0/1000.0] [batch_t 1.415 (1.461)] [data_t 0.002] [optim_t 1.412] [lr 0.005000] 2024-04-05 20:23:51,241 - Train: 62.42% [337700/541000] [624.2/1000.0] [batch_t 1.419 (1.417)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 20:26:12,939 - Train: 62.44% [337800/541000] [624.4/1000.0] [batch_t 1.415 (1.417)] [data_t 0.002] [optim_t 1.412] [lr 0.005000] 2024-04-05 20:28:34,611 - Train: 62.46% [337900/541000] [624.6/1000.0] [batch_t 1.415 (1.417)] [data_t 0.002] [optim_t 1.413] [lr 0.005000] 2024-04-05 20:30:56,277 - Train: 62.48% [338000/541000] [624.8/1000.0] [batch_t 1.418 (1.417)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-05 20:33:17,912 - Train: 62.50% [338100/541000] [625.0/1000.0] [batch_t 1.418 (1.416)] [data_t 0.002] [optim_t 1.415] [lr 0.005000] 2024-04-05 20:33:53,371 - ==> Total time: 3 days, 19:33:47 Eta: 2 days, 6:56:16 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 20:35:41,870 - Train: 62.51% [338200/541000] [625.1/1000.0] [batch_t 1.414 (1.422)] [data_t 0.002] [optim_t 1.412] [lr 0.005000] 2024-04-05 20:38:03,500 - Train: 62.53% [338300/541000] [625.3/1000.0] [batch_t 1.417 (1.416)] [data_t 0.002] [optim_t 1.415] [lr 0.005000] 2024-04-05 20:40:25,194 - Train: 62.55% [338400/541000] [625.5/1000.0] [batch_t 1.420 (1.417)] [data_t 0.002] [optim_t 1.418] [lr 0.005000] 2024-04-05 20:42:46,915 - Train: 62.57% [338500/541000] [625.7/1000.0] [batch_t 1.422 (1.417)] [data_t 0.002] [optim_t 1.419] [lr 0.005000] 2024-04-05 20:45:08,642 - Train: 62.59% [338600/541000] [625.9/1000.0] [batch_t 1.419 (1.417)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-05 20:46:42,205 - ==> Total time: 3 days, 19:46:36 Eta: 2 days, 6:49:53 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 20:47:33,062 - Train: 62.61% [338700/541000] [626.1/1000.0] [batch_t 1.416 (1.436)] [data_t 0.002] [optim_t 1.413] [lr 0.005000] 2024-04-05 20:49:54,767 - Train: 62.62% [338800/541000] [626.2/1000.0] [batch_t 1.419 (1.417)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 20:52:16,354 - Train: 62.64% [338900/541000] [626.4/1000.0] [batch_t 1.413 (1.416)] [data_t 0.002] [optim_t 1.410] [lr 0.005000] 2024-04-05 20:54:37,997 - Train: 62.66% [339000/541000] [626.6/1000.0] [batch_t 1.415 (1.416)] [data_t 0.002] [optim_t 1.413] [lr 0.005000] 2024-04-05 20:56:59,714 - Train: 62.68% [339100/541000] [626.8/1000.0] [batch_t 1.419 (1.417)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 20:59:21,320 - Train: 62.70% [339200/541000] [627.0/1000.0] [batch_t 1.415 (1.416)] [data_t 0.003] [optim_t 1.412] [lr 0.005000] 2024-04-05 20:59:31,232 - ==> Total time: 3 days, 19:59:25 Eta: 2 days, 6:43:29 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 21:01:45,197 - Train: 62.72% [339300/541000] [627.2/1000.0] [batch_t 1.420 (1.419)] [data_t 0.002] [optim_t 1.418] [lr 0.005000] 2024-04-05 21:04:06,864 - Train: 62.74% [339400/541000] [627.4/1000.0] [batch_t 1.405 (1.417)] [data_t 0.002] [optim_t 1.402] [lr 0.005000] 2024-04-05 21:06:28,511 - Train: 62.75% [339500/541000] [627.5/1000.0] [batch_t 1.414 (1.416)] [data_t 0.002] [optim_t 1.412] [lr 0.005000] 2024-04-05 21:08:50,163 - Train: 62.77% [339600/541000] [627.7/1000.0] [batch_t 1.418 (1.416)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-05 21:11:11,898 - Train: 62.79% [339700/541000] [627.9/1000.0] [batch_t 1.421 (1.417)] [data_t 0.002] [optim_t 1.419] [lr 0.005000] 2024-04-05 21:12:19,923 - ==> Total time: 3 days, 20:12:14 Eta: 2 days, 6:37:03 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 21:13:35,859 - Train: 62.81% [339800/541000] [628.1/1000.0] [batch_t 1.419 (1.426)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 21:15:57,546 - Train: 62.83% [339900/541000] [628.3/1000.0] [batch_t 1.415 (1.417)] [data_t 0.002] [optim_t 1.413] [lr 0.005000] 2024-04-05 21:18:19,141 - Train: 62.85% [340000/541000] [628.5/1000.0] [batch_t 1.416 (1.416)] [data_t 0.002] [optim_t 1.413] [lr 0.005000] 2024-04-05 21:20:40,867 - Train: 62.87% [340100/541000] [628.7/1000.0] [batch_t 1.416 (1.417)] [data_t 0.002] [optim_t 1.414] [lr 0.005000] 2024-04-05 21:23:02,551 - Train: 62.88% [340200/541000] [628.8/1000.0] [batch_t 1.414 (1.417)] [data_t 0.002] [optim_t 1.412] [lr 0.005000] 2024-04-05 21:25:08,606 - ==> Total time: 3 days, 20:25:03 Eta: 2 days, 6:30:36 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 21:25:26,960 - Train: 62.90% [340300/541000] [629.0/1000.0] [batch_t 1.423 (1.474)] [data_t 0.002] [optim_t 1.420] [lr 0.005000] 2024-04-05 21:27:48,669 - Train: 62.92% [340400/541000] [629.2/1000.0] [batch_t 1.416 (1.417)] [data_t 0.002] [optim_t 1.414] [lr 0.005000] 2024-04-05 21:30:10,315 - Train: 62.94% [340500/541000] [629.4/1000.0] [batch_t 1.421 (1.416)] [data_t 0.002] [optim_t 1.418] [lr 0.005000] 2024-04-05 21:32:31,981 - Train: 62.96% [340600/541000] [629.6/1000.0] [batch_t 1.418 (1.417)] [data_t 0.002] [optim_t 1.415] [lr 0.005000] 2024-04-05 21:34:53,761 - Train: 62.98% [340700/541000] [629.8/1000.0] [batch_t 1.420 (1.418)] [data_t 0.002] [optim_t 1.418] [lr 0.005000] 2024-04-05 21:37:15,363 - Train: 62.99% [340800/541000] [629.9/1000.0] [batch_t 1.420 (1.416)] [data_t 0.002] [optim_t 1.418] [lr 0.005000] 2024-04-05 21:37:57,880 - ==> Total time: 3 days, 20:37:52 Eta: 2 days, 6:24:08 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 21:39:39,279 - Train: 63.01% [340900/541000] [630.1/1000.0] [batch_t 1.420 (1.421)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 21:42:00,903 - Train: 63.03% [341000/541000] [630.3/1000.0] [batch_t 1.420 (1.416)] [data_t 0.002] [optim_t 1.418] [lr 0.005000] 2024-04-05 21:44:22,558 - Train: 63.05% [341100/541000] [630.5/1000.0] [batch_t 1.409 (1.416)] [data_t 0.002] [optim_t 1.406] [lr 0.005000] 2024-04-05 21:46:44,259 - Train: 63.07% [341200/541000] [630.7/1000.0] [batch_t 1.410 (1.417)] [data_t 0.002] [optim_t 1.408] [lr 0.005000] 2024-04-05 21:49:06,014 - Train: 63.09% [341300/541000] [630.9/1000.0] [batch_t 1.420 (1.417)] [data_t 0.002] [optim_t 1.418] [lr 0.005000] 2024-04-05 21:50:46,609 - ==> Total time: 3 days, 20:50:41 Eta: 2 days, 6:17:39 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 21:51:30,030 - Train: 63.11% [341400/541000] [631.1/1000.0] [batch_t 1.414 (1.431)] [data_t 0.002] [optim_t 1.412] [lr 0.005000] 2024-04-05 21:53:51,686 - Train: 63.12% [341500/541000] [631.2/1000.0] [batch_t 1.415 (1.416)] [data_t 0.002] [optim_t 1.413] [lr 0.005000] 2024-04-05 21:56:13,350 - Train: 63.14% [341600/541000] [631.4/1000.0] [batch_t 1.418 (1.417)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-05 21:58:35,089 - Train: 63.16% [341700/541000] [631.6/1000.0] [batch_t 1.415 (1.417)] [data_t 0.002] [optim_t 1.412] [lr 0.005000] 2024-04-05 22:00:56,791 - Train: 63.18% [341800/541000] [631.8/1000.0] [batch_t 1.418 (1.417)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-05 22:03:18,440 - Train: 63.20% [341900/541000] [632.0/1000.0] [batch_t 1.412 (1.416)] [data_t 0.002] [optim_t 1.410] [lr 0.005000] 2024-04-05 22:03:35,449 - ==> Total time: 3 days, 21:03:29 Eta: 2 days, 6:11:09 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 22:05:42,556 - Train: 63.22% [342000/541000] [632.2/1000.0] [batch_t 1.413 (1.422)] [data_t 0.002] [optim_t 1.411] [lr 0.005000] 2024-04-05 22:08:04,185 - Train: 63.23% [342100/541000] [632.3/1000.0] [batch_t 1.420 (1.416)] [data_t 0.002] [optim_t 1.418] [lr 0.005000] 2024-04-05 22:10:25,865 - Train: 63.25% [342200/541000] [632.5/1000.0] [batch_t 1.415 (1.417)] [data_t 0.002] [optim_t 1.413] [lr 0.005000] 2024-04-05 22:12:47,628 - Train: 63.27% [342300/541000] [632.7/1000.0] [batch_t 1.419 (1.418)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 22:15:09,293 - Train: 63.29% [342400/541000] [632.9/1000.0] [batch_t 1.416 (1.417)] [data_t 0.002] [optim_t 1.414] [lr 0.005000] 2024-04-05 22:16:24,348 - ==> Total time: 3 days, 21:16:18 Eta: 2 days, 6:04:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 22:17:33,308 - Train: 63.31% [342500/541000] [633.1/1000.0] [batch_t 1.420 (1.423)] [data_t 0.002] [optim_t 1.418] [lr 0.005000] 2024-04-05 22:19:54,966 - Train: 63.33% [342600/541000] [633.3/1000.0] [batch_t 1.410 (1.416)] [data_t 0.002] [optim_t 1.408] [lr 0.005000] 2024-04-05 22:22:16,659 - Train: 63.35% [342700/541000] [633.5/1000.0] [batch_t 1.415 (1.417)] [data_t 0.002] [optim_t 1.412] [lr 0.005000] 2024-04-05 22:24:38,312 - Train: 63.36% [342800/541000] [633.6/1000.0] [batch_t 1.419 (1.416)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 22:26:59,945 - Train: 63.38% [342900/541000] [633.8/1000.0] [batch_t 1.410 (1.416)] [data_t 0.002] [optim_t 1.408] [lr 0.005000] 2024-04-05 22:29:13,067 - ==> Total time: 3 days, 21:29:07 Eta: 2 days, 5:58:04 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 22:29:23,799 - Train: 63.40% [343000/541000] [634.0/1000.0] [batch_t 1.418 (1.476)] [data_t 0.002] [optim_t 1.415] [lr 0.005000] 2024-04-05 22:31:45,330 - Train: 63.42% [343100/541000] [634.2/1000.0] [batch_t 1.417 (1.415)] [data_t 0.002] [optim_t 1.415] [lr 0.005000] 2024-04-05 22:34:06,904 - Train: 63.44% [343200/541000] [634.4/1000.0] [batch_t 1.420 (1.416)] [data_t 0.002] [optim_t 1.418] [lr 0.005000] 2024-04-05 22:36:28,523 - Train: 63.46% [343300/541000] [634.6/1000.0] [batch_t 1.409 (1.416)] [data_t 0.002] [optim_t 1.407] [lr 0.005000] 2024-04-05 22:38:50,139 - Train: 63.48% [343400/541000] [634.8/1000.0] [batch_t 1.420 (1.416)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 22:41:11,793 - Train: 63.49% [343500/541000] [634.9/1000.0] [batch_t 1.417 (1.416)] [data_t 0.002] [optim_t 1.415] [lr 0.005000] 2024-04-05 22:42:01,353 - ==> Total time: 3 days, 21:41:55 Eta: 2 days, 5:51:30 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 22:43:38,222 - Train: 63.51% [343600/541000] [635.1/1000.0] [batch_t 1.412 (1.431)] [data_t 0.002] [optim_t 1.410] [lr 0.005000] 2024-04-05 22:45:59,891 - Train: 63.53% [343700/541000] [635.3/1000.0] [batch_t 1.416 (1.417)] [data_t 0.002] [optim_t 1.414] [lr 0.005000] 2024-04-05 22:48:21,582 - Train: 63.55% [343800/541000] [635.5/1000.0] [batch_t 1.419 (1.417)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 22:50:43,318 - Train: 63.57% [343900/541000] [635.7/1000.0] [batch_t 1.420 (1.417)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 22:53:04,976 - Train: 63.59% [344000/541000] [635.9/1000.0] [batch_t 1.418 (1.416)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-05 22:54:52,645 - ==> Total time: 3 days, 21:54:47 Eta: 2 days, 5:44:56 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 22:55:29,293 - Train: 63.60% [344100/541000] [636.0/1000.0] [batch_t 1.419 (1.435)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 22:57:50,993 - Train: 63.62% [344200/541000] [636.2/1000.0] [batch_t 1.419 (1.417)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-05 23:00:12,583 - Train: 63.64% [344300/541000] [636.4/1000.0] [batch_t 1.420 (1.416)] [data_t 0.002] [optim_t 1.418] [lr 0.005000] 2024-04-05 23:02:34,262 - Train: 63.66% [344400/541000] [636.6/1000.0] [batch_t 1.419 (1.417)] [data_t 0.002] [optim_t 1.417] [lr 0.005000] 2024-04-05 23:04:55,969 - Train: 63.68% [344500/541000] [636.8/1000.0] [batch_t 1.417 (1.417)] [data_t 0.002] [optim_t 1.415] [lr 0.005000] 2024-04-05 23:07:17,618 - Train: 63.70% [344600/541000] [637.0/1000.0] [batch_t 1.418 (1.416)] [data_t 0.002] [optim_t 1.416] [lr 0.005000] 2024-04-05 23:07:41,687 - ==> Total time: 3 days, 22:07:36 Eta: 2 days, 5:38:20 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 23:09:41,705 - Train: 63.72% [344700/541000] [637.2/1000.0] [batch_t 1.416 (1.424)] [data_t 0.002] [optim_t 1.413] [lr 0.005000] 2024-04-05 23:11:25,452 - Train: 63.73% [344800/541000] [637.3/1000.0] [batch_t 0.607 (1.037)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 23:12:26,177 - Train: 63.75% [344900/541000] [637.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 23:13:27,061 - Train: 63.77% [345000/541000] [637.7/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 23:14:27,781 - Train: 63.79% [345100/541000] [637.9/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 23:15:03,007 - ==> Total time: 3 days, 22:14:57 Eta: 2 days, 5:28:36 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 23:15:32,123 - Train: 63.81% [345200/541000] [638.1/1000.0] [batch_t 0.609 (0.622)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-05 23:16:32,833 - Train: 63.83% [345300/541000] [638.3/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 23:17:33,551 - Train: 63.84% [345400/541000] [638.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 23:18:34,250 - Train: 63.86% [345500/541000] [638.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 23:19:35,070 - Train: 63.88% [345600/541000] [638.8/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 23:20:35,171 - ==> Total time: 3 days, 22:20:29 Eta: 2 days, 5:17:52 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 23:20:37,893 - Train: 63.90% [345700/541000] [639.0/1000.0] [batch_t 0.916 (0.916)] [data_t 0.302] [optim_t 0.614] [lr 0.005000] 2024-04-05 23:21:38,607 - Train: 63.92% [345800/541000] [639.2/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 23:22:39,298 - Train: 63.94% [345900/541000] [639.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 23:23:39,990 - Train: 63.96% [346000/541000] [639.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 23:24:40,687 - Train: 63.97% [346100/541000] [639.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 23:25:41,547 - Train: 63.99% [346200/541000] [639.9/1000.0] [batch_t 0.609 (0.609)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-05 23:26:05,832 - ==> Total time: 3 days, 22:26:00 Eta: 2 days, 5:07:07 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 23:26:44,655 - Train: 64.01% [346300/541000] [640.1/1000.0] [batch_t 0.607 (0.614)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 23:27:45,361 - Train: 64.03% [346400/541000] [640.3/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 23:28:46,088 - Train: 64.05% [346500/541000] [640.5/1000.0] [batch_t 0.610 (0.607)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-05 23:29:46,797 - Train: 64.07% [346600/541000] [640.7/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 23:30:47,566 - Train: 64.09% [346700/541000] [640.9/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 23:31:36,741 - ==> Total time: 3 days, 22:31:31 Eta: 2 days, 4:56:24 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 23:31:50,826 - Train: 64.10% [346800/541000] [641.0/1000.0] [batch_t 0.606 (0.643)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 23:32:51,537 - Train: 64.12% [346900/541000] [641.2/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 23:33:52,297 - Train: 64.14% [347000/541000] [641.4/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 23:34:52,990 - Train: 64.16% [347100/541000] [641.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 23:35:53,728 - Train: 64.18% [347200/541000] [641.8/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 23:36:54,439 - Train: 64.20% [347300/541000] [642.0/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 23:37:07,797 - ==> Total time: 3 days, 22:37:02 Eta: 2 days, 4:45:41 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 23:37:57,540 - Train: 64.21% [347400/541000] [642.1/1000.0] [batch_t 0.608 (0.615)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 23:38:58,270 - Train: 64.23% [347500/541000] [642.3/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 23:39:58,959 - Train: 64.25% [347600/541000] [642.5/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 23:40:59,680 - Train: 64.27% [347700/541000] [642.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 23:42:00,378 - Train: 64.29% [347800/541000] [642.9/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 23:42:38,617 - ==> Total time: 3 days, 22:42:33 Eta: 2 days, 4:35:00 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 23:43:03,387 - Train: 64.31% [347900/541000] [643.1/1000.0] [batch_t 0.604 (0.619)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-05 23:44:04,081 - Train: 64.33% [348000/541000] [643.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 23:45:04,796 - Train: 64.34% [348100/541000] [643.4/1000.0] [batch_t 0.612 (0.607)] [data_t 0.002] [optim_t 0.610] [lr 0.005000] 2024-04-05 23:46:05,506 - Train: 64.36% [348200/541000] [643.6/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 23:47:06,204 - Train: 64.38% [348300/541000] [643.8/1000.0] [batch_t 0.604 (0.607)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-05 23:48:06,925 - Train: 64.40% [348400/541000] [644.0/1000.0] [batch_t 0.604 (0.607)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-05 23:48:09,360 - ==> Total time: 3 days, 22:48:03 Eta: 2 days, 4:24:19 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 23:49:10,096 - Train: 64.42% [348500/541000] [644.2/1000.0] [batch_t 0.607 (0.613)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 23:50:10,815 - Train: 64.44% [348600/541000] [644.4/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 23:51:11,533 - Train: 64.45% [348700/541000] [644.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 23:52:12,193 - Train: 64.47% [348800/541000] [644.7/1000.0] [batch_t 0.606 (0.606)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 23:53:12,951 - Train: 64.49% [348900/541000] [644.9/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 23:53:40,286 - ==> Total time: 3 days, 22:53:34 Eta: 2 days, 4:13:40 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 23:54:15,949 - Train: 64.51% [349000/541000] [645.1/1000.0] [batch_t 0.608 (0.614)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 23:55:16,818 - Train: 64.53% [349100/541000] [645.3/1000.0] [batch_t 0.606 (0.609)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-05 23:56:17,518 - Train: 64.55% [349200/541000] [645.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 23:57:18,220 - Train: 64.57% [349300/541000] [645.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-05 23:58:18,898 - Train: 64.58% [349400/541000] [645.8/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-05 23:59:11,138 - ==> Total time: 3 days, 22:59:05 Eta: 2 days, 4:03:01 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-05 23:59:21,732 - Train: 64.60% [349500/541000] [646.0/1000.0] [batch_t 0.607 (0.638)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 00:00:22,417 - Train: 64.62% [349600/541000] [646.2/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 00:01:23,323 - Train: 64.64% [349700/541000] [646.4/1000.0] [batch_t 0.612 (0.609)] [data_t 0.002] [optim_t 0.610] [lr 0.005000] 2024-04-06 00:02:24,057 - Train: 64.66% [349800/541000] [646.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 00:03:24,807 - Train: 64.68% [349900/541000] [646.8/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-06 00:04:25,546 - Train: 64.70% [350000/541000] [647.0/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 00:04:41,951 - ==> Total time: 3 days, 23:04:36 Eta: 2 days, 3:52:24 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 00:05:28,516 - Train: 64.71% [350100/541000] [647.1/1000.0] [batch_t 0.607 (0.612)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 00:06:29,367 - Train: 64.73% [350200/541000] [647.3/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 00:07:30,087 - Train: 64.75% [350300/541000] [647.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 00:08:30,788 - Train: 64.77% [350400/541000] [647.7/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-06 00:09:31,515 - Train: 64.79% [350500/541000] [647.9/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-06 00:10:12,823 - ==> Total time: 3 days, 23:10:07 Eta: 2 days, 3:41:47 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 00:10:34,514 - Train: 64.81% [350600/541000] [648.1/1000.0] [batch_t 0.606 (0.619)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-06 00:11:35,378 - Train: 64.82% [350700/541000] [648.2/1000.0] [batch_t 0.607 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 00:12:36,085 - Train: 64.84% [350800/541000] [648.4/1000.0] [batch_t 0.605 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-06 00:13:36,823 - Train: 64.86% [350900/541000] [648.6/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 00:14:37,602 - Train: 64.88% [351000/541000] [648.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 00:15:38,407 - Train: 64.90% [351100/541000] [649.0/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 00:15:43,880 - ==> Total time: 3 days, 23:15:38 Eta: 2 days, 3:31:12 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 00:16:41,726 - Train: 64.92% [351200/541000] [649.2/1000.0] [batch_t 0.608 (0.613)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 00:17:42,745 - Train: 64.94% [351300/541000] [649.4/1000.0] [batch_t 0.610 (0.610)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-06 00:18:43,542 - Train: 64.95% [351400/541000] [649.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 00:19:44,353 - Train: 64.97% [351500/541000] [649.7/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 00:20:45,186 - Train: 64.99% [351600/541000] [649.9/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 00:21:27,528 - Test: 36.76% [50/136] [batch_t 0.215 (0.229)] 2024-04-06 00:21:38,230 - Test: 73.53% [100/136] [batch_t 0.217 (0.221)] 2024-04-06 00:21:45,786 - Test: 100.00% [136/136] [batch_t 0.053 (0.218)] 2024-04-06 00:22:34,191 - ==> Metric Time for candle : 0.002 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.511 (mAUROC_px) 8.110 (mAP_px) 0.943 (mF1_max_px) 25.115 (mAUPRO_px) 0.322 (mF1_px_0.2_0.8_0.1) 0.320 (mAcc_px_0.2_0.8_0.1) 0.319 (mIoU_px_0.2_0.8_0.1) 0.931 (mIoU_max_px) 2024-04-06 00:23:12,001 - ==> Metric Time for capsules : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 8.308 (mAUROC_px) 6.286 (mAP_px) 0.737 (mF1_max_px) 20.750 (mAUPRO_px) 0.253 (mF1_px_0.2_0.8_0.1) 0.251 (mAcc_px_0.2_0.8_0.1) 0.256 (mIoU_px_0.2_0.8_0.1) 0.741 (mIoU_max_px) 2024-04-06 00:23:47,896 - ==> Metric Time for cashew : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.822 (mAUROC_px) 6.072 (mAP_px) 0.692 (mF1_max_px) 19.537 (mAUPRO_px) 0.238 (mF1_px_0.2_0.8_0.1) 0.239 (mAcc_px_0.2_0.8_0.1) 0.248 (mIoU_px_0.2_0.8_0.1) 0.837 (mIoU_max_px) 2024-04-06 00:24:26,143 - ==> Metric Time for chewinggum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 8.320 (mAUROC_px) 6.406 (mAP_px) 0.760 (mF1_max_px) 20.996 (mAUPRO_px) 0.257 (mF1_px_0.2_0.8_0.1) 0.259 (mAcc_px_0.2_0.8_0.1) 0.261 (mIoU_px_0.2_0.8_0.1) 0.755 (mIoU_max_px) 2024-04-06 00:25:04,758 - ==> Metric Time for fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 8.219 (mAUROC_px) 5.966 (mAP_px) 0.718 (mF1_max_px) 22.075 (mAUPRO_px) 0.239 (mF1_px_0.2_0.8_0.1) 0.240 (mAcc_px_0.2_0.8_0.1) 0.243 (mIoU_px_0.2_0.8_0.1) 0.697 (mIoU_max_px) 2024-04-06 00:25:55,075 - ==> Metric Time for macaroni1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 12.015 (mAUROC_px) 8.450 (mAP_px) 1.075 (mF1_max_px) 26.392 (mAUPRO_px) 0.353 (mF1_px_0.2_0.8_0.1) 0.352 (mAcc_px_0.2_0.8_0.1) 0.361 (mIoU_px_0.2_0.8_0.1) 1.017 (mIoU_max_px) 2024-04-06 00:26:43,373 - ==> Metric Time for macaroni2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 11.248 (mAUROC_px) 8.083 (mAP_px) 0.963 (mF1_max_px) 25.812 (mAUPRO_px) 0.322 (mF1_px_0.2_0.8_0.1) 0.322 (mAcc_px_0.2_0.8_0.1) 0.322 (mIoU_px_0.2_0.8_0.1) 0.934 (mIoU_max_px) 2024-04-06 00:27:32,728 - ==> Metric Time for pcb1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.860 (mAUROC_px) 8.491 (mAP_px) 0.949 (mF1_max_px) 26.858 (mAUPRO_px) 0.323 (mF1_px_0.2_0.8_0.1) 0.320 (mAcc_px_0.2_0.8_0.1) 0.322 (mIoU_px_0.2_0.8_0.1) 0.940 (mIoU_max_px) 2024-04-06 00:28:24,198 - ==> Metric Time for pcb2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 11.737 (mAUROC_px) 8.616 (mAP_px) 1.027 (mF1_max_px) 27.659 (mAUPRO_px) 0.347 (mF1_px_0.2_0.8_0.1) 0.349 (mAcc_px_0.2_0.8_0.1) 0.349 (mIoU_px_0.2_0.8_0.1) 1.021 (mIoU_max_px) 2024-04-06 00:29:13,162 - ==> Metric Time for pcb3 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.850 (mAUROC_px) 8.441 (mAP_px) 0.951 (mF1_max_px) 26.517 (mAUPRO_px) 0.324 (mF1_px_0.2_0.8_0.1) 0.322 (mAcc_px_0.2_0.8_0.1) 0.326 (mIoU_px_0.2_0.8_0.1) 0.946 (mIoU_max_px) 2024-04-06 00:30:02,843 - ==> Metric Time for pcb4 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.793 (mAUROC_px) 8.101 (mAP_px) 0.960 (mF1_max_px) 27.567 (mAUPRO_px) 0.328 (mF1_px_0.2_0.8_0.1) 0.329 (mAcc_px_0.2_0.8_0.1) 0.328 (mIoU_px_0.2_0.8_0.1) 0.981 (mIoU_max_px) 2024-04-06 00:30:39,398 - ==> Metric Time for pipe_fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.828 (mAUROC_px) 5.892 (mAP_px) 0.754 (mF1_max_px) 20.419 (mAUPRO_px) 0.243 (mF1_px_0.2_0.8_0.1) 0.240 (mAcc_px_0.2_0.8_0.1) 0.244 (mIoU_px_0.2_0.8_0.1) 0.706 (mIoU_max_px) 2024-04-06 00:30:39,419 - | Name | mAUROC_sp_max | mAUROC_sp_max (Max) | mAP_sp_max | mAP_sp_max (Max) | mF1_max_sp_max | mF1_max_sp_max (Max) | mAUROC_px | mAUROC_px (Max) | mAP_px | mAP_px (Max) | mF1_max_px | mF1_max_px (Max) | mAUPRO_px | mAUPRO_px (Max) | mF1_px_0.2_0.8_0.1 | mF1_px_0.2_0.8_0.1 (Max) | mAcc_px_0.2_0.8_0.1 | mAcc_px_0.2_0.8_0.1 (Max) | mIoU_px_0.2_0.8_0.1 | mIoU_px_0.2_0.8_0.1 (Max) | mIoU_max_px | mIoU_max_px (Max) | |:----------:|:---------------:|:---------------------:|:------------:|:------------------:|:----------------:|:----------------------:|:-----------:|:------------------:|:--------:|:------------------:|:------------:|:------------------:|:-----------:|:------------------:|:--------------------:|:--------------------------:|:---------------------:|:---------------------------:|:---------------------:|:---------------------------:|:-------------:|:-------------------:| | candle | 95.890 | 97.100 (50 epoch) | 96.197 | 97.346 (50 epoch) | 89.655 | 91.542 (50 epoch) | 99.047 | 99.098 (250 epoch) | 21.556 | 23.463 (150 epoch) | 31.076 | 32.654 (200 epoch) | 95.649 | 96.087 (300 epoch) | 21.071 | 21.331 (500 epoch) | 41.526 | 41.526 (650 epoch) | 12.021 | 12.185 (500 epoch) | 18.397 | 19.513 (200 epoch) | | capsules | 86.217 | 91.750 (100 epoch) | 90.355 | 94.964 (100 epoch) | 84.762 | 88.780 (100 epoch) | 98.488 | 99.123 (100 epoch) | 29.544 | 61.256 (100 epoch) | 40.119 | 59.868 (250 epoch) | 87.805 | 91.811 (100 epoch) | 15.774 | 33.186 (100 epoch) | 36.307 | 62.576 (50 epoch) | 9.265 | 21.701 (100 epoch) | 25.093 | 42.722 (250 epoch) | | cashew | 93.380 | 94.800 (150 epoch) | 95.612 | 97.492 (150 epoch) | 91.919 | 92.462 (450 epoch) | 95.247 | 95.614 (250 epoch) | 37.917 | 50.952 (250 epoch) | 42.355 | 54.438 (400 epoch) | 84.029 | 89.185 (350 epoch) | 22.862 | 28.352 (350 epoch) | 29.451 | 34.522 (150 epoch) | 13.656 | 17.797 (350 epoch) | 26.867 | 37.398 (400 epoch) | | chewinggum | 96.939 | 97.802 (300 epoch) | 98.516 | 98.939 (300 epoch) | 94.180 | 94.792 (450 epoch) | 98.028 | 98.145 (100 epoch) | 48.186 | 59.823 (50 epoch) | 55.347 | 61.088 (50 epoch) | 79.919 | 80.312 (400 epoch) | 30.915 | 35.748 (50 epoch) | 62.555 | 65.611 (50 epoch) | 19.844 | 23.754 (50 epoch) | 38.262 | 43.976 (50 epoch) | | fryum | 96.560 | 96.560 (650 epoch) | 98.273 | 98.273 (650 epoch) | 93.897 | 93.897 (650 epoch) | 97.149 | 97.210 (550 epoch) | 50.247 | 50.672 (600 epoch) | 52.759 | 53.189 (600 epoch) | 92.137 | 92.340 (550 epoch) | 25.355 | 26.511 (550 epoch) | 26.321 | 28.594 (450 epoch) | 15.805 | 16.529 (550 epoch) | 35.832 | 36.230 (600 epoch) | | macaroni1 | 91.396 | 92.104 (500 epoch) | 89.423 | 90.505 (500 epoch) | 82.126 | 83.978 (450 epoch) | 99.604 | 99.604 (650 epoch) | 19.707 | 19.707 (650 epoch) | 29.513 | 29.513 (650 epoch) | 96.221 | 96.221 (650 epoch) | 15.586 | 15.803 (300 epoch) | 37.139 | 40.765 (50 epoch) | 8.702 | 8.828 (300 epoch) | 17.311 | 17.311 (650 epoch) | | macaroni2 | 81.107 | 82.589 (50 epoch) | 73.626 | 81.196 (50 epoch) | 76.415 | 76.712 (600 epoch) | 98.693 | 99.452 (100 epoch) | 2.076 | 11.272 (50 epoch) | 4.720 | 17.568 (50 epoch) | 93.251 | 96.253 (150 epoch) | 2.542 | 10.086 (50 epoch) | 49.339 | 54.584 (450 epoch) | 1.292 | 5.433 (50 epoch) | 2.417 | 9.630 (50 epoch) | | pcb1 | 96.790 | 96.790 (650 epoch) | 95.443 | 95.525 (400 epoch) | 94.059 | 94.737 (300 epoch) | 99.802 | 99.805 (450 epoch) | 81.382 | 83.141 (450 epoch) | 74.057 | 75.588 (450 epoch) | 94.355 | 94.861 (600 epoch) | 47.619 | 47.720 (450 epoch) | 57.180 | 62.362 (50 epoch) | 33.404 | 33.816 (450 epoch) | 58.802 | 60.756 (450 epoch) | | pcb2 | 95.530 | 95.670 (550 epoch) | 95.627 | 95.692 (550 epoch) | 90.452 | 90.452 (650 epoch) | 99.001 | 99.008 (600 epoch) | 12.570 | 14.130 (50 epoch) | 23.868 | 23.977 (250 epoch) | 90.985 | 91.111 (500 epoch) | 15.715 | 16.150 (200 epoch) | 44.318 | 47.820 (150 epoch) | 8.637 | 8.892 (200 epoch) | 13.551 | 13.621 (250 epoch) | | pcb3 | 95.317 | 95.891 (350 epoch) | 95.564 | 96.056 (350 epoch) | 89.552 | 89.855 (400 epoch) | 99.142 | 99.142 (350 epoch) | 17.831 | 20.249 (150 epoch) | 27.774 | 28.078 (450 epoch) | 92.048 | 92.048 (650 epoch) | 19.529 | 20.042 (150 epoch) | 45.238 | 48.341 (50 epoch) | 10.985 | 11.295 (150 epoch) | 16.127 | 16.332 (450 epoch) | | pcb4 | 99.693 | 99.901 (100 epoch) | 99.689 | 99.901 (100 epoch) | 98.039 | 99.000 (50 epoch) | 98.746 | 98.746 (650 epoch) | 47.445 | 47.966 (250 epoch) | 47.429 | 47.709 (250 epoch) | 90.332 | 90.876 (500 epoch) | 24.575 | 26.925 (200 epoch) | 32.445 | 39.716 (50 epoch) | 15.104 | 16.537 (200 epoch) | 31.086 | 31.328 (250 epoch) | | pipe_fryum | 99.000 | 99.000 (650 epoch) | 99.489 | 99.489 (650 epoch) | 97.059 | 97.561 (200 epoch) | 98.911 | 99.114 (450 epoch) | 51.636 | 54.998 (450 epoch) | 53.449 | 58.733 (450 epoch) | 93.631 | 95.525 (150 epoch) | 30.426 | 34.684 (250 epoch) | 35.985 | 43.014 (100 epoch) | 19.346 | 22.399 (250 epoch) | 36.471 | 41.576 (450 epoch) | | Avg | 93.985 | 94.251 (100 epoch) | 93.985 | 94.481 (100 epoch) | 90.176 | 90.176 (650 epoch) | 98.488 | 98.606 (250 epoch) | 35.008 | 39.701 (250 epoch) | 40.206 | 44.125 (250 epoch) | 90.863 | 91.494 (250 epoch) | 22.664 | 25.164 (100 epoch) | 41.484 | 46.179 (50 epoch) | 14.005 | 15.755 (100 epoch) | 26.685 | 29.990 (250 epoch) | 2024-04-06 00:30:39,841 - ==> Total time: 3 days, 23:30:34 Eta: 2 days, 3:25:41 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 00:31:55,724 - Train: 65.01% [351700/541000] [650.1/1000.0] [batch_t 1.466 (1.470)] [data_t 0.003] [optim_t 1.462] [lr 0.005000] 2024-04-06 00:34:21,600 - Train: 65.03% [351800/541000] [650.3/1000.0] [batch_t 1.460 (1.459)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 00:36:47,444 - Train: 65.05% [351900/541000] [650.5/1000.0] [batch_t 1.457 (1.458)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-06 00:39:13,265 - Train: 65.06% [352000/541000] [650.6/1000.0] [batch_t 1.450 (1.458)] [data_t 0.003] [optim_t 1.448] [lr 0.005000] 2024-04-06 00:41:39,127 - Train: 65.08% [352100/541000] [650.8/1000.0] [batch_t 1.455 (1.459)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 00:43:51,893 - ==> Total time: 3 days, 23:43:46 Eta: 2 days, 3:19:13 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 00:44:07,364 - Train: 65.10% [352200/541000] [651.0/1000.0] [batch_t 1.455 (1.501)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 00:46:33,241 - Train: 65.12% [352300/541000] [651.2/1000.0] [batch_t 1.455 (1.459)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 00:48:59,053 - Train: 65.14% [352400/541000] [651.4/1000.0] [batch_t 1.458 (1.458)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-06 00:51:24,895 - Train: 65.16% [352500/541000] [651.6/1000.0] [batch_t 1.459 (1.458)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-06 00:53:50,635 - Train: 65.18% [352600/541000] [651.8/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 00:56:16,454 - Train: 65.19% [352700/541000] [651.9/1000.0] [batch_t 1.459 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 00:57:03,091 - ==> Total time: 3 days, 23:56:57 Eta: 2 days, 3:12:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 00:58:45,476 - Train: 65.21% [352800/541000] [652.1/1000.0] [batch_t 1.456 (1.467)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 01:01:11,408 - Train: 65.23% [352900/541000] [652.3/1000.0] [batch_t 1.457 (1.459)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-06 01:03:37,192 - Train: 65.25% [353000/541000] [652.5/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 01:06:03,038 - Train: 65.27% [353100/541000] [652.7/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 01:08:28,848 - Train: 65.29% [353200/541000] [652.9/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 01:10:15,285 - ==> Total time: 4 days, 0:10:09 Eta: 2 days, 3:06:13 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 01:10:56,938 - Train: 65.30% [353300/541000] [653.0/1000.0] [batch_t 1.457 (1.472)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-06 01:13:22,689 - Train: 65.32% [353400/541000] [653.2/1000.0] [batch_t 1.450 (1.457)] [data_t 0.003] [optim_t 1.446] [lr 0.005000] 2024-04-06 01:15:48,526 - Train: 65.34% [353500/541000] [653.4/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 01:18:14,403 - Train: 65.36% [353600/541000] [653.6/1000.0] [batch_t 1.456 (1.459)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 01:20:40,235 - Train: 65.38% [353700/541000] [653.8/1000.0] [batch_t 1.457 (1.458)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-06 01:23:06,166 - Train: 65.40% [353800/541000] [654.0/1000.0] [batch_t 1.462 (1.459)] [data_t 0.003] [optim_t 1.459] [lr 0.005000] 2024-04-06 01:23:26,571 - ==> Total time: 4 days, 0:23:21 Eta: 2 days, 2:59:41 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 01:25:34,079 - Train: 65.42% [353900/541000] [654.2/1000.0] [batch_t 1.455 (1.462)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 01:27:59,934 - Train: 65.43% [354000/541000] [654.3/1000.0] [batch_t 1.458 (1.458)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-06 01:30:25,736 - Train: 65.45% [354100/541000] [654.5/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 01:32:51,611 - Train: 65.47% [354200/541000] [654.7/1000.0] [batch_t 1.460 (1.459)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 01:35:17,414 - Train: 65.49% [354300/541000] [654.9/1000.0] [batch_t 1.456 (1.458)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 01:36:37,625 - ==> Total time: 4 days, 0:36:32 Eta: 2 days, 2:53:08 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 01:37:45,746 - Train: 65.51% [354400/541000] [655.1/1000.0] [batch_t 1.462 (1.470)] [data_t 0.003] [optim_t 1.460] [lr 0.005000] 2024-04-06 01:40:11,559 - Train: 65.53% [354500/541000] [655.3/1000.0] [batch_t 1.455 (1.458)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-06 01:42:37,503 - Train: 65.55% [354600/541000] [655.5/1000.0] [batch_t 1.460 (1.459)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 01:45:03,308 - Train: 65.56% [354700/541000] [655.6/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 01:47:29,123 - Train: 65.58% [354800/541000] [655.8/1000.0] [batch_t 1.454 (1.458)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-06 01:49:49,114 - ==> Total time: 4 days, 0:49:43 Eta: 2 days, 2:46:33 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 01:49:57,365 - Train: 65.60% [354900/541000] [656.0/1000.0] [batch_t 1.450 (1.554)] [data_t 0.003] [optim_t 1.447] [lr 0.005000] 2024-04-06 01:52:23,135 - Train: 65.62% [355000/541000] [656.2/1000.0] [batch_t 1.454 (1.458)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-06 01:54:48,934 - Train: 65.64% [355100/541000] [656.4/1000.0] [batch_t 1.455 (1.458)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 01:57:14,736 - Train: 65.66% [355200/541000] [656.6/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-06 01:59:40,513 - Train: 65.67% [355300/541000] [656.7/1000.0] [batch_t 1.457 (1.458)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 02:02:06,401 - Train: 65.69% [355400/541000] [656.9/1000.0] [batch_t 1.461 (1.459)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 02:03:00,384 - ==> Total time: 4 days, 1:02:54 Eta: 2 days, 2:39:58 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 02:04:36,496 - Train: 65.71% [355500/541000] [657.1/1000.0] [batch_t 1.456 (1.495)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 02:07:02,363 - Train: 65.73% [355600/541000] [657.3/1000.0] [batch_t 1.455 (1.459)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 02:09:28,217 - Train: 65.75% [355700/541000] [657.5/1000.0] [batch_t 1.462 (1.458)] [data_t 0.003] [optim_t 1.459] [lr 0.005000] 2024-04-06 02:11:26,748 - Train: 65.77% [355800/541000] [657.7/1000.0] [batch_t 0.606 (1.185)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-06 02:12:27,466 - Train: 65.79% [355900/541000] [657.9/1000.0] [batch_t 0.604 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-06 02:13:14,817 - ==> Total time: 4 days, 1:13:09 Eta: 2 days, 2:31:49 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 02:13:30,366 - Train: 65.80% [356000/541000] [658.0/1000.0] [batch_t 0.608 (0.626)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 02:14:31,186 - Train: 65.82% [356100/541000] [658.2/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 02:15:32,051 - Train: 65.84% [356200/541000] [658.4/1000.0] [batch_t 0.606 (0.609)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 02:16:32,997 - Train: 65.86% [356300/541000] [658.6/1000.0] [batch_t 0.609 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 02:17:33,790 - Train: 65.88% [356400/541000] [658.8/1000.0] [batch_t 0.606 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 02:18:34,634 - Train: 65.90% [356500/541000] [659.0/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 02:18:46,195 - ==> Total time: 4 days, 1:18:40 Eta: 2 days, 2:21:13 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 02:19:37,758 - Train: 65.91% [356600/541000] [659.1/1000.0] [batch_t 0.609 (0.616)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 02:20:38,610 - Train: 65.93% [356700/541000] [659.3/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 02:21:39,437 - Train: 65.95% [356800/541000] [659.5/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 02:22:40,304 - Train: 65.97% [356900/541000] [659.7/1000.0] [batch_t 0.609 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 02:23:41,318 - Train: 65.99% [357000/541000] [659.9/1000.0] [batch_t 0.609 (0.610)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 02:24:17,835 - ==> Total time: 4 days, 1:24:12 Eta: 2 days, 2:10:39 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 02:24:44,361 - Train: 66.01% [357100/541000] [660.1/1000.0] [batch_t 0.605 (0.618)] [data_t 0.003] [optim_t 0.603] [lr 0.005000] 2024-04-06 02:25:45,175 - Train: 66.03% [357200/541000] [660.3/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 02:26:46,011 - Train: 66.04% [357300/541000] [660.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 02:27:46,814 - Train: 66.06% [357400/541000] [660.6/1000.0] [batch_t 0.606 (0.608)] [data_t 0.003] [optim_t 0.603] [lr 0.005000] 2024-04-06 02:28:47,899 - Train: 66.08% [357500/541000] [660.8/1000.0] [batch_t 0.607 (0.611)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 02:29:48,747 - Train: 66.10% [357600/541000] [661.0/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-06 02:29:49,360 - ==> Total time: 4 days, 1:29:43 Eta: 2 days, 2:00:05 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 02:30:51,890 - Train: 66.12% [357700/541000] [661.2/1000.0] [batch_t 0.607 (0.612)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 02:31:52,750 - Train: 66.14% [357800/541000] [661.4/1000.0] [batch_t 0.610 (0.609)] [data_t 0.003] [optim_t 0.608] [lr 0.005000] 2024-04-06 02:32:53,607 - Train: 66.16% [357900/541000] [661.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 02:33:54,614 - Train: 66.17% [358000/541000] [661.7/1000.0] [batch_t 0.606 (0.610)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 02:34:55,448 - Train: 66.19% [358100/541000] [661.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 02:35:20,991 - ==> Total time: 4 days, 1:35:15 Eta: 2 days, 1:49:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 02:35:58,388 - Train: 66.21% [358200/541000] [662.1/1000.0] [batch_t 0.608 (0.615)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 02:36:59,215 - Train: 66.23% [358300/541000] [662.3/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-06 02:38:00,035 - Train: 66.25% [358400/541000] [662.5/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 02:39:00,882 - Train: 66.27% [358500/541000] [662.7/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 02:40:01,876 - Train: 66.28% [358600/541000] [662.8/1000.0] [batch_t 0.608 (0.610)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 02:40:52,363 - ==> Total time: 4 days, 1:40:46 Eta: 2 days, 1:39:00 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 02:41:04,968 - Train: 66.30% [358700/541000] [663.0/1000.0] [batch_t 0.609 (0.636)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 02:42:05,797 - Train: 66.32% [358800/541000] [663.2/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 02:43:06,625 - Train: 66.34% [358900/541000] [663.4/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-06 02:44:07,454 - Train: 66.36% [359000/541000] [663.6/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 02:45:08,320 - Train: 66.38% [359100/541000] [663.8/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 02:46:09,352 - Train: 66.40% [359200/541000] [664.0/1000.0] [batch_t 0.608 (0.610)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 02:46:23,954 - ==> Total time: 4 days, 1:46:18 Eta: 2 days, 1:28:29 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 02:47:12,547 - Train: 66.41% [359300/541000] [664.1/1000.0] [batch_t 0.609 (0.616)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-06 02:48:13,430 - Train: 66.43% [359400/541000] [664.3/1000.0] [batch_t 0.609 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 02:49:14,296 - Train: 66.45% [359500/541000] [664.5/1000.0] [batch_t 0.609 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 02:50:15,191 - Train: 66.47% [359600/541000] [664.7/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 02:51:16,214 - Train: 66.49% [359700/541000] [664.9/1000.0] [batch_t 0.607 (0.610)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 02:51:55,773 - ==> Total time: 4 days, 1:51:50 Eta: 2 days, 1:17:59 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 02:52:19,604 - Train: 66.51% [359800/541000] [665.1/1000.0] [batch_t 0.607 (0.618)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 02:53:20,477 - Train: 66.52% [359900/541000] [665.2/1000.0] [batch_t 0.605 (0.609)] [data_t 0.003] [optim_t 0.603] [lr 0.005000] 2024-04-06 02:54:21,329 - Train: 66.54% [360000/541000] [665.4/1000.0] [batch_t 0.611 (0.608)] [data_t 0.003] [optim_t 0.609] [lr 0.005000] 2024-04-06 02:55:22,175 - Train: 66.56% [360100/541000] [665.6/1000.0] [batch_t 0.610 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-06 02:56:23,002 - Train: 66.58% [360200/541000] [665.8/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 02:57:23,858 - Train: 66.60% [360300/541000] [666.0/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 02:57:27,513 - ==> Total time: 4 days, 1:57:21 Eta: 2 days, 1:07:30 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 02:58:27,125 - Train: 66.62% [360400/541000] [666.2/1000.0] [batch_t 0.609 (0.613)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 02:59:27,952 - Train: 66.64% [360500/541000] [666.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 03:00:28,853 - Train: 66.65% [360600/541000] [666.5/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 03:01:29,681 - Train: 66.67% [360700/541000] [666.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 03:02:30,566 - Train: 66.69% [360800/541000] [666.9/1000.0] [batch_t 0.610 (0.609)] [data_t 0.003] [optim_t 0.608] [lr 0.005000] 2024-04-06 03:02:59,182 - ==> Total time: 4 days, 2:02:53 Eta: 2 days, 0:57:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 03:03:34,180 - Train: 66.71% [360900/541000] [667.1/1000.0] [batch_t 0.609 (0.620)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 03:04:34,989 - Train: 66.73% [361000/541000] [667.3/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-06 03:05:35,843 - Train: 66.75% [361100/541000] [667.5/1000.0] [batch_t 0.613 (0.608)] [data_t 0.003] [optim_t 0.611] [lr 0.005000] 2024-04-06 03:06:36,673 - Train: 66.77% [361200/541000] [667.7/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 03:07:37,540 - Train: 66.78% [361300/541000] [667.8/1000.0] [batch_t 0.612 (0.609)] [data_t 0.003] [optim_t 0.610] [lr 0.005000] 2024-04-06 03:08:31,109 - ==> Total time: 4 days, 2:08:25 Eta: 2 days, 0:46:34 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 03:08:40,681 - Train: 66.80% [361400/541000] [668.0/1000.0] [batch_t 0.608 (0.634)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 03:09:41,494 - Train: 66.82% [361500/541000] [668.2/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 03:10:42,453 - Train: 66.84% [361600/541000] [668.4/1000.0] [batch_t 0.607 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 03:11:43,361 - Train: 66.86% [361700/541000] [668.6/1000.0] [batch_t 0.613 (0.609)] [data_t 0.003] [optim_t 0.610] [lr 0.005000] 2024-04-06 03:12:44,184 - Train: 66.88% [361800/541000] [668.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 03:13:45,050 - Train: 66.89% [361900/541000] [668.9/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 03:14:02,704 - ==> Total time: 4 days, 2:13:57 Eta: 2 days, 0:36:08 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 03:14:49,903 - Train: 66.91% [362000/541000] [669.1/1000.0] [batch_t 0.612 (0.621)] [data_t 0.003] [optim_t 0.610] [lr 0.005000] 2024-04-06 03:15:50,685 - Train: 66.93% [362100/541000] [669.3/1000.0] [batch_t 0.606 (0.608)] [data_t 0.003] [optim_t 0.603] [lr 0.005000] 2024-04-06 03:16:51,598 - Train: 66.95% [362200/541000] [669.5/1000.0] [batch_t 0.609 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 03:17:52,410 - Train: 66.97% [362300/541000] [669.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 03:18:53,246 - Train: 66.99% [362400/541000] [669.9/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-06 03:19:35,846 - ==> Total time: 4 days, 2:19:30 Eta: 2 days, 0:25:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 03:19:56,347 - Train: 67.01% [362500/541000] [670.1/1000.0] [batch_t 0.608 (0.621)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 03:20:57,193 - Train: 67.02% [362600/541000] [670.2/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 03:21:58,027 - Train: 67.04% [362700/541000] [670.4/1000.0] [batch_t 0.610 (0.608)] [data_t 0.003] [optim_t 0.608] [lr 0.005000] 2024-04-06 03:22:58,988 - Train: 67.06% [362800/541000] [670.6/1000.0] [batch_t 0.608 (0.610)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 03:23:59,813 - Train: 67.08% [362900/541000] [670.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 03:25:00,663 - Train: 67.10% [363000/541000] [671.0/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 03:25:07,358 - ==> Total time: 4 days, 2:25:01 Eta: 2 days, 0:15:18 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 03:26:03,673 - Train: 67.12% [363100/541000] [671.2/1000.0] [batch_t 0.609 (0.612)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 03:27:04,556 - Train: 67.13% [363200/541000] [671.3/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 03:28:05,377 - Train: 67.15% [363300/541000] [671.5/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-06 03:29:06,354 - Train: 67.17% [363400/541000] [671.7/1000.0] [batch_t 0.608 (0.610)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 03:30:07,200 - Train: 67.19% [363500/541000] [671.9/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 03:30:38,858 - ==> Total time: 4 days, 2:30:33 Eta: 2 days, 0:04:54 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 03:31:10,240 - Train: 67.21% [363600/541000] [672.1/1000.0] [batch_t 0.607 (0.616)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 03:32:11,034 - Train: 67.23% [363700/541000] [672.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 03:33:11,848 - Train: 67.25% [363800/541000] [672.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 03:34:12,666 - Train: 67.26% [363900/541000] [672.6/1000.0] [batch_t 0.612 (0.608)] [data_t 0.003] [optim_t 0.609] [lr 0.005000] 2024-04-06 03:35:13,599 - Train: 67.28% [364000/541000] [672.8/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 03:36:10,169 - ==> Total time: 4 days, 2:36:04 Eta: 1 day, 23:54:31 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 03:36:16,441 - Train: 67.30% [364100/541000] [673.0/1000.0] [batch_t 0.608 (0.664)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 03:37:17,303 - Train: 67.32% [364200/541000] [673.2/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 03:38:18,208 - Train: 67.34% [364300/541000] [673.4/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 03:39:19,064 - Train: 67.36% [364400/541000] [673.6/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-06 03:40:20,074 - Train: 67.38% [364500/541000] [673.8/1000.0] [batch_t 0.606 (0.610)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 03:41:20,926 - Train: 67.39% [364600/541000] [673.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 03:41:41,607 - ==> Total time: 4 days, 2:41:36 Eta: 1 day, 23:44:09 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 03:42:24,099 - Train: 67.41% [364700/541000] [674.1/1000.0] [batch_t 0.608 (0.614)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 03:43:24,933 - Train: 67.43% [364800/541000] [674.3/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 03:44:25,791 - Train: 67.45% [364900/541000] [674.5/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 03:45:26,650 - Train: 67.47% [365000/541000] [674.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 03:46:27,692 - Train: 67.49% [365100/541000] [674.9/1000.0] [batch_t 0.607 (0.610)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 03:47:13,325 - ==> Total time: 4 days, 2:47:07 Eta: 1 day, 23:33:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 03:47:30,748 - Train: 67.50% [365200/541000] [675.0/1000.0] [batch_t 0.608 (0.624)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 03:48:31,589 - Train: 67.52% [365300/541000] [675.2/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 03:49:32,424 - Train: 67.54% [365400/541000] [675.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 03:50:33,293 - Train: 67.56% [365500/541000] [675.6/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 03:51:34,164 - Train: 67.58% [365600/541000] [675.8/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 03:52:35,167 - Train: 67.60% [365700/541000] [676.0/1000.0] [batch_t 0.607 (0.610)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 03:52:44,903 - ==> Total time: 4 days, 2:52:39 Eta: 1 day, 23:23:27 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 03:53:38,184 - Train: 67.62% [365800/541000] [676.2/1000.0] [batch_t 0.610 (0.613)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-06 03:54:39,052 - Train: 67.63% [365900/541000] [676.3/1000.0] [batch_t 0.609 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 03:55:39,911 - Train: 67.65% [366000/541000] [676.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 03:56:40,755 - Train: 67.67% [366100/541000] [676.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 03:57:41,724 - Train: 67.69% [366200/541000] [676.9/1000.0] [batch_t 0.608 (0.610)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 03:58:16,387 - ==> Total time: 4 days, 2:58:10 Eta: 1 day, 23:13:08 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 03:58:44,767 - Train: 67.71% [366300/541000] [677.1/1000.0] [batch_t 0.612 (0.617)] [data_t 0.003] [optim_t 0.609] [lr 0.005000] 2024-04-06 03:59:45,657 - Train: 67.73% [366400/541000] [677.3/1000.0] [batch_t 0.609 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 04:00:46,512 - Train: 67.74% [366500/541000] [677.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 04:01:47,364 - Train: 67.76% [366600/541000] [677.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 04:02:48,227 - Train: 67.78% [366700/541000] [677.8/1000.0] [batch_t 0.607 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 04:03:47,885 - ==> Total time: 4 days, 3:03:42 Eta: 1 day, 23:02:49 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 04:03:51,822 - Train: 67.80% [366800/541000] [678.0/1000.0] [batch_t 0.607 (0.792)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 04:04:52,657 - Train: 67.82% [366900/541000] [678.2/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 04:05:53,491 - Train: 67.84% [367000/541000] [678.4/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-06 04:06:54,328 - Train: 67.86% [367100/541000] [678.6/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 04:07:55,177 - Train: 67.87% [367200/541000] [678.7/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 04:08:56,036 - Train: 67.89% [367300/541000] [678.9/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 04:09:19,769 - ==> Total time: 4 days, 3:09:14 Eta: 1 day, 22:52:31 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 04:09:59,248 - Train: 67.91% [367400/541000] [679.1/1000.0] [batch_t 0.608 (0.614)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 04:11:00,114 - Train: 67.93% [367500/541000] [679.3/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 04:12:00,960 - Train: 67.95% [367600/541000] [679.5/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 04:13:01,779 - Train: 67.97% [367700/541000] [679.7/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 04:14:02,611 - Train: 67.99% [367800/541000] [679.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 04:14:51,429 - ==> Total time: 4 days, 3:14:45 Eta: 1 day, 22:42:14 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 04:15:05,649 - Train: 68.00% [367900/541000] [680.0/1000.0] [batch_t 0.608 (0.625)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 04:16:06,481 - Train: 68.02% [368000/541000] [680.2/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 04:17:07,341 - Train: 68.04% [368100/541000] [680.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 04:18:08,204 - Train: 68.06% [368200/541000] [680.6/1000.0] [batch_t 0.609 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 04:19:09,048 - Train: 68.08% [368300/541000] [680.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 04:20:09,886 - Train: 68.10% [368400/541000] [681.0/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-06 04:20:22,660 - ==> Total time: 4 days, 3:20:17 Eta: 1 day, 22:31:58 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 04:21:13,317 - Train: 68.11% [368500/541000] [681.1/1000.0] [batch_t 0.613 (0.614)] [data_t 0.003] [optim_t 0.610] [lr 0.005000] 2024-04-06 04:22:14,174 - Train: 68.13% [368600/541000] [681.3/1000.0] [batch_t 0.612 (0.608)] [data_t 0.002] [optim_t 0.610] [lr 0.005000] 2024-04-06 04:23:15,038 - Train: 68.15% [368700/541000] [681.5/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 04:24:15,901 - Train: 68.17% [368800/541000] [681.7/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 04:25:16,763 - Train: 68.19% [368900/541000] [681.9/1000.0] [batch_t 0.609 (0.609)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-06 04:25:54,468 - ==> Total time: 4 days, 3:25:48 Eta: 1 day, 22:21:42 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 04:26:19,739 - Train: 68.21% [369000/541000] [682.1/1000.0] [batch_t 0.606 (0.617)] [data_t 0.003] [optim_t 0.603] [lr 0.005000] 2024-04-06 04:27:20,736 - Train: 68.23% [369100/541000] [682.3/1000.0] [batch_t 0.607 (0.610)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 04:28:21,583 - Train: 68.24% [369200/541000] [682.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 04:29:22,420 - Train: 68.26% [369300/541000] [682.6/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 04:30:23,258 - Train: 68.28% [369400/541000] [682.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 04:31:24,106 - Train: 68.30% [369500/541000] [683.0/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 04:31:25,936 - ==> Total time: 4 days, 3:31:20 Eta: 1 day, 22:11:28 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 04:32:27,085 - Train: 68.32% [369600/541000] [683.2/1000.0] [batch_t 0.609 (0.613)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 04:33:27,928 - Train: 68.34% [369700/541000] [683.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 04:34:28,947 - Train: 68.35% [369800/541000] [683.5/1000.0] [batch_t 0.609 (0.610)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 04:35:29,807 - Train: 68.37% [369900/541000] [683.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 04:36:30,661 - Train: 68.39% [370000/541000] [683.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 04:36:57,434 - ==> Total time: 4 days, 3:36:51 Eta: 1 day, 22:01:14 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 04:37:33,784 - Train: 68.41% [370100/541000] [684.1/1000.0] [batch_t 0.608 (0.615)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 04:38:34,625 - Train: 68.43% [370200/541000] [684.3/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 04:39:35,607 - Train: 68.45% [370300/541000] [684.5/1000.0] [batch_t 0.609 (0.610)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 04:40:36,455 - Train: 68.47% [370400/541000] [684.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 04:41:37,313 - Train: 68.48% [370500/541000] [684.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 04:42:29,060 - ==> Total time: 4 days, 3:42:23 Eta: 1 day, 21:51:01 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 04:42:40,515 - Train: 68.50% [370600/541000] [685.0/1000.0] [batch_t 0.609 (0.629)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 04:43:41,359 - Train: 68.52% [370700/541000] [685.2/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 04:44:42,195 - Train: 68.54% [370800/541000] [685.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 04:45:43,227 - Train: 68.56% [370900/541000] [685.6/1000.0] [batch_t 0.607 (0.610)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 04:46:44,102 - Train: 68.58% [371000/541000] [685.8/1000.0] [batch_t 0.607 (0.609)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 04:47:44,951 - Train: 68.60% [371100/541000] [686.0/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-06 04:48:00,760 - ==> Total time: 4 days, 3:47:55 Eta: 1 day, 21:40:49 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 04:48:49,497 - Train: 68.61% [371200/541000] [686.1/1000.0] [batch_t 0.609 (0.632)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 04:49:50,380 - Train: 68.63% [371300/541000] [686.3/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 04:50:51,229 - Train: 68.65% [371400/541000] [686.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 04:51:52,238 - Train: 68.67% [371500/541000] [686.7/1000.0] [batch_t 0.608 (0.610)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 04:52:53,118 - Train: 68.69% [371600/541000] [686.9/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 04:53:33,907 - ==> Total time: 4 days, 3:53:28 Eta: 1 day, 21:30:39 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 04:53:56,359 - Train: 68.71% [371700/541000] [687.1/1000.0] [batch_t 0.608 (0.623)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 04:54:57,188 - Train: 68.72% [371800/541000] [687.2/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 04:55:58,038 - Train: 68.74% [371900/541000] [687.4/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 04:56:59,036 - Train: 68.76% [372000/541000] [687.6/1000.0] [batch_t 0.608 (0.610)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 04:57:59,893 - Train: 68.78% [372100/541000] [687.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 04:59:00,728 - Train: 68.80% [372200/541000] [688.0/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-06 04:59:05,597 - ==> Total time: 4 days, 3:59:00 Eta: 1 day, 21:20:28 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 05:00:04,900 - Train: 68.82% [372300/541000] [688.2/1000.0] [batch_t 0.608 (0.616)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 05:01:05,738 - Train: 68.84% [372400/541000] [688.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 05:02:06,589 - Train: 68.85% [372500/541000] [688.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 05:03:07,580 - Train: 68.87% [372600/541000] [688.7/1000.0] [batch_t 0.608 (0.610)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 05:04:08,407 - Train: 68.89% [372700/541000] [688.9/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-06 05:04:38,201 - ==> Total time: 4 days, 4:04:32 Eta: 1 day, 21:10:19 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 05:05:11,284 - Train: 68.91% [372800/541000] [689.1/1000.0] [batch_t 0.608 (0.615)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 05:06:12,191 - Train: 68.93% [372900/541000] [689.3/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 05:07:13,014 - Train: 68.95% [373000/541000] [689.5/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 05:08:13,860 - Train: 68.96% [373100/541000] [689.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 05:09:14,829 - Train: 68.98% [373200/541000] [689.8/1000.0] [batch_t 0.609 (0.610)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-06 05:10:09,581 - ==> Total time: 4 days, 4:10:04 Eta: 1 day, 21:00:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 05:10:17,869 - Train: 69.00% [373300/541000] [690.0/1000.0] [batch_t 0.606 (0.649)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 05:11:18,734 - Train: 69.02% [373400/541000] [690.2/1000.0] [batch_t 0.609 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 05:12:19,445 - Train: 69.04% [373500/541000] [690.4/1000.0] [batch_t 0.604 (0.607)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-06 05:13:20,297 - Train: 69.06% [373600/541000] [690.6/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 05:14:21,066 - Train: 69.08% [373700/541000] [690.8/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 05:15:21,909 - Train: 69.09% [373800/541000] [690.9/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 05:15:40,765 - ==> Total time: 4 days, 4:15:35 Eta: 1 day, 20:50:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 05:16:24,896 - Train: 69.11% [373900/541000] [691.1/1000.0] [batch_t 0.608 (0.614)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 05:17:25,734 - Train: 69.13% [374000/541000] [691.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 05:18:26,580 - Train: 69.15% [374100/541000] [691.5/1000.0] [batch_t 0.611 (0.608)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-06 05:19:27,410 - Train: 69.17% [374200/541000] [691.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 05:20:28,373 - Train: 69.19% [374300/541000] [691.9/1000.0] [batch_t 0.607 (0.610)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 05:21:12,197 - ==> Total time: 4 days, 4:21:06 Eta: 1 day, 20:39:54 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 05:21:31,317 - Train: 69.21% [374400/541000] [692.1/1000.0] [batch_t 0.607 (0.622)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 05:22:32,149 - Train: 69.22% [374500/541000] [692.2/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 05:23:32,962 - Train: 69.24% [374600/541000] [692.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 05:24:33,833 - Train: 69.26% [374700/541000] [692.6/1000.0] [batch_t 0.607 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 05:25:34,691 - Train: 69.28% [374800/541000] [692.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 05:26:35,560 - Train: 69.30% [374900/541000] [693.0/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 05:26:43,470 - ==> Total time: 4 days, 4:26:37 Eta: 1 day, 20:29:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 05:27:38,907 - Train: 69.32% [375000/541000] [693.2/1000.0] [batch_t 0.613 (0.612)] [data_t 0.003] [optim_t 0.610] [lr 0.005000] 2024-04-06 05:28:39,778 - Train: 69.33% [375100/541000] [693.3/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 05:29:40,643 - Train: 69.35% [375200/541000] [693.5/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 05:30:41,548 - Train: 69.37% [375300/541000] [693.7/1000.0] [batch_t 0.607 (0.609)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 05:31:42,400 - Train: 69.39% [375400/541000] [693.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 05:32:15,261 - ==> Total time: 4 days, 4:32:09 Eta: 1 day, 20:19:42 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 05:32:45,680 - Train: 69.41% [375500/541000] [694.1/1000.0] [batch_t 0.608 (0.619)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 05:33:46,516 - Train: 69.43% [375600/541000] [694.3/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 05:34:47,368 - Train: 69.45% [375700/541000] [694.5/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-06 05:35:48,245 - Train: 69.46% [375800/541000] [694.6/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 05:36:49,068 - Train: 69.48% [375900/541000] [694.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 05:37:46,872 - ==> Total time: 4 days, 4:37:41 Eta: 1 day, 20:09:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 05:37:52,031 - Train: 69.50% [376000/541000] [695.0/1000.0] [batch_t 0.609 (0.690)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 05:38:52,863 - Train: 69.52% [376100/541000] [695.2/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 05:39:53,809 - Train: 69.54% [376200/541000] [695.4/1000.0] [batch_t 0.609 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 05:40:54,638 - Train: 69.56% [376300/541000] [695.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 05:41:55,494 - Train: 69.57% [376400/541000] [695.7/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 05:42:56,311 - Train: 69.59% [376500/541000] [695.9/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-06 05:43:18,210 - ==> Total time: 4 days, 4:43:12 Eta: 1 day, 19:59:33 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 05:43:59,661 - Train: 69.61% [376600/541000] [696.1/1000.0] [batch_t 0.609 (0.617)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 05:45:00,514 - Train: 69.63% [376700/541000] [696.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 05:46:01,346 - Train: 69.65% [376800/541000] [696.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 05:47:02,186 - Train: 69.67% [376900/541000] [696.7/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 05:48:03,040 - Train: 69.69% [377000/541000] [696.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 05:48:49,892 - ==> Total time: 4 days, 4:48:44 Eta: 1 day, 19:49:30 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 05:49:06,168 - Train: 69.70% [377100/541000] [697.0/1000.0] [batch_t 0.609 (0.623)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 05:50:07,008 - Train: 69.72% [377200/541000] [697.2/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 05:51:07,989 - Train: 69.74% [377300/541000] [697.4/1000.0] [batch_t 0.608 (0.610)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 05:52:08,811 - Train: 69.76% [377400/541000] [697.6/1000.0] [batch_t 0.606 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 05:53:09,675 - Train: 69.78% [377500/541000] [697.8/1000.0] [batch_t 0.607 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 05:54:10,528 - Train: 69.80% [377600/541000] [698.0/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-06 05:54:21,489 - ==> Total time: 4 days, 4:54:15 Eta: 1 day, 19:39:28 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 05:55:13,436 - Train: 69.82% [377700/541000] [698.2/1000.0] [batch_t 0.609 (0.612)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-06 05:56:14,427 - Train: 69.83% [377800/541000] [698.3/1000.0] [batch_t 0.609 (0.610)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 05:57:15,323 - Train: 69.85% [377900/541000] [698.5/1000.0] [batch_t 0.609 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 05:58:16,170 - Train: 69.87% [378000/541000] [698.7/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 05:59:17,023 - Train: 69.89% [378100/541000] [698.9/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 05:59:52,932 - ==> Total time: 4 days, 4:59:47 Eta: 1 day, 19:29:26 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 06:00:20,127 - Train: 69.91% [378200/541000] [699.1/1000.0] [batch_t 0.609 (0.617)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 06:01:20,995 - Train: 69.93% [378300/541000] [699.3/1000.0] [batch_t 0.609 (0.609)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-06 06:02:21,843 - Train: 69.94% [378400/541000] [699.4/1000.0] [batch_t 0.611 (0.608)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-06 06:03:22,815 - Train: 69.96% [378500/541000] [699.6/1000.0] [batch_t 0.609 (0.610)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-06 06:04:23,671 - Train: 69.98% [378600/541000] [699.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 06:05:24,513 - Train: 70.00% [378700/541000] [700.0/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-06 06:05:36,401 - Test: 36.76% [50/136] [batch_t 0.217 (0.228)] 2024-04-06 06:05:47,181 - Test: 73.53% [100/136] [batch_t 0.213 (0.222)] 2024-04-06 06:05:54,772 - Test: 100.00% [136/136] [batch_t 0.053 (0.219)] 2024-04-06 06:06:42,772 - ==> Metric Time for candle : 0.002 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.577 (mAUROC_px) 8.146 (mAP_px) 0.929 (mF1_max_px) 24.702 (mAUPRO_px) 0.317 (mF1_px_0.2_0.8_0.1) 0.318 (mAcc_px_0.2_0.8_0.1) 0.318 (mIoU_px_0.2_0.8_0.1) 0.926 (mIoU_max_px) 2024-04-06 06:07:19,522 - ==> Metric Time for capsules : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 8.199 (mAUROC_px) 6.259 (mAP_px) 0.741 (mF1_max_px) 19.839 (mAUPRO_px) 0.255 (mF1_px_0.2_0.8_0.1) 0.254 (mAcc_px_0.2_0.8_0.1) 0.252 (mIoU_px_0.2_0.8_0.1) 0.739 (mIoU_max_px) 2024-04-06 06:07:55,115 - ==> Metric Time for cashew : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.773 (mAUROC_px) 6.139 (mAP_px) 0.691 (mF1_max_px) 19.401 (mAUPRO_px) 0.230 (mF1_px_0.2_0.8_0.1) 0.236 (mAcc_px_0.2_0.8_0.1) 0.238 (mIoU_px_0.2_0.8_0.1) 0.686 (mIoU_max_px) 2024-04-06 06:08:31,405 - ==> Metric Time for chewinggum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.734 (mAUROC_px) 5.896 (mAP_px) 0.683 (mF1_max_px) 20.360 (mAUPRO_px) 0.239 (mF1_px_0.2_0.8_0.1) 0.239 (mAcc_px_0.2_0.8_0.1) 0.239 (mIoU_px_0.2_0.8_0.1) 0.694 (mIoU_max_px) 2024-04-06 06:09:07,735 - ==> Metric Time for fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.671 (mAUROC_px) 5.719 (mAP_px) 0.703 (mF1_max_px) 20.620 (mAUPRO_px) 0.241 (mF1_px_0.2_0.8_0.1) 0.238 (mAcc_px_0.2_0.8_0.1) 0.239 (mIoU_px_0.2_0.8_0.1) 0.692 (mIoU_max_px) 2024-04-06 06:09:56,605 - ==> Metric Time for macaroni1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.970 (mAUROC_px) 9.447 (mAP_px) 1.140 (mF1_max_px) 25.161 (mAUPRO_px) 0.318 (mF1_px_0.2_0.8_0.1) 0.319 (mAcc_px_0.2_0.8_0.1) 0.318 (mIoU_px_0.2_0.8_0.1) 0.926 (mIoU_max_px) 2024-04-06 06:10:42,870 - ==> Metric Time for macaroni2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.499 (mAUROC_px) 7.866 (mAP_px) 0.940 (mF1_max_px) 24.803 (mAUPRO_px) 0.317 (mF1_px_0.2_0.8_0.1) 0.318 (mAcc_px_0.2_0.8_0.1) 0.317 (mIoU_px_0.2_0.8_0.1) 0.929 (mIoU_max_px) 2024-04-06 06:11:30,273 - ==> Metric Time for pcb1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.677 (mAUROC_px) 7.995 (mAP_px) 0.943 (mF1_max_px) 25.630 (mAUPRO_px) 0.319 (mF1_px_0.2_0.8_0.1) 0.318 (mAcc_px_0.2_0.8_0.1) 0.318 (mIoU_px_0.2_0.8_0.1) 0.926 (mIoU_max_px) 2024-04-06 06:12:17,626 - ==> Metric Time for pcb2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.676 (mAUROC_px) 8.114 (mAP_px) 0.990 (mF1_max_px) 25.414 (mAUPRO_px) 0.317 (mF1_px_0.2_0.8_0.1) 0.316 (mAcc_px_0.2_0.8_0.1) 0.319 (mIoU_px_0.2_0.8_0.1) 0.929 (mIoU_max_px) 2024-04-06 06:13:05,260 - ==> Metric Time for pcb3 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.559 (mAUROC_px) 8.021 (mAP_px) 0.986 (mF1_max_px) 25.900 (mAUPRO_px) 0.317 (mF1_px_0.2_0.8_0.1) 0.317 (mAcc_px_0.2_0.8_0.1) 0.321 (mIoU_px_0.2_0.8_0.1) 0.932 (mIoU_max_px) 2024-04-06 06:13:54,822 - ==> Metric Time for pcb4 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.702 (mAUROC_px) 8.143 (mAP_px) 0.948 (mF1_max_px) 27.590 (mAUPRO_px) 0.324 (mF1_px_0.2_0.8_0.1) 0.323 (mAcc_px_0.2_0.8_0.1) 0.322 (mIoU_px_0.2_0.8_0.1) 0.933 (mIoU_max_px) 2024-04-06 06:14:30,932 - ==> Metric Time for pipe_fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.872 (mAUROC_px) 5.862 (mAP_px) 0.693 (mF1_max_px) 20.058 (mAUPRO_px) 0.236 (mF1_px_0.2_0.8_0.1) 0.235 (mAcc_px_0.2_0.8_0.1) 0.240 (mIoU_px_0.2_0.8_0.1) 0.695 (mIoU_max_px) 2024-04-06 06:14:30,952 - | Name | mAUROC_sp_max | mAUROC_sp_max (Max) | mAP_sp_max | mAP_sp_max (Max) | mF1_max_sp_max | mF1_max_sp_max (Max) | mAUROC_px | mAUROC_px (Max) | mAP_px | mAP_px (Max) | mF1_max_px | mF1_max_px (Max) | mAUPRO_px | mAUPRO_px (Max) | mF1_px_0.2_0.8_0.1 | mF1_px_0.2_0.8_0.1 (Max) | mAcc_px_0.2_0.8_0.1 | mAcc_px_0.2_0.8_0.1 (Max) | mIoU_px_0.2_0.8_0.1 | mIoU_px_0.2_0.8_0.1 (Max) | mIoU_max_px | mIoU_max_px (Max) | |:----------:|:---------------:|:---------------------:|:------------:|:------------------:|:----------------:|:----------------------:|:-----------:|:------------------:|:--------:|:------------------:|:------------:|:------------------:|:-----------:|:------------------:|:--------------------:|:--------------------------:|:---------------------:|:---------------------------:|:---------------------:|:---------------------------:|:-------------:|:-------------------:| | candle | 96.140 | 97.100 (50 epoch) | 96.423 | 97.346 (50 epoch) | 89.952 | 91.542 (50 epoch) | 99.019 | 99.098 (250 epoch) | 21.563 | 23.463 (150 epoch) | 31.242 | 32.654 (200 epoch) | 95.484 | 96.087 (300 epoch) | 20.951 | 21.331 (500 epoch) | 41.485 | 41.526 (650 epoch) | 11.949 | 12.185 (500 epoch) | 18.513 | 19.513 (200 epoch) | | capsules | 87.217 | 91.750 (100 epoch) | 89.806 | 94.964 (100 epoch) | 86.432 | 88.780 (100 epoch) | 99.193 | 99.193 (700 epoch) | 52.072 | 61.256 (100 epoch) | 55.625 | 59.868 (250 epoch) | 89.616 | 91.811 (100 epoch) | 27.929 | 33.186 (100 epoch) | 53.485 | 62.576 (50 epoch) | 17.711 | 21.701 (100 epoch) | 38.529 | 42.722 (250 epoch) | | cashew | 94.200 | 94.800 (150 epoch) | 97.151 | 97.492 (150 epoch) | 92.929 | 92.929 (700 epoch) | 96.185 | 96.185 (700 epoch) | 40.665 | 50.952 (250 epoch) | 40.981 | 54.438 (400 epoch) | 83.646 | 89.185 (350 epoch) | 25.635 | 28.352 (350 epoch) | 32.194 | 34.522 (150 epoch) | 15.400 | 17.797 (350 epoch) | 25.771 | 37.398 (400 epoch) | | chewinggum | 96.998 | 97.802 (300 epoch) | 98.510 | 98.939 (300 epoch) | 94.737 | 94.792 (450 epoch) | 97.962 | 98.145 (100 epoch) | 46.910 | 59.823 (50 epoch) | 55.554 | 61.088 (50 epoch) | 80.302 | 80.312 (400 epoch) | 31.526 | 35.748 (50 epoch) | 65.299 | 65.611 (50 epoch) | 20.231 | 23.754 (50 epoch) | 38.460 | 43.976 (50 epoch) | | fryum | 96.160 | 96.560 (650 epoch) | 98.105 | 98.273 (650 epoch) | 92.754 | 93.897 (650 epoch) | 97.175 | 97.210 (550 epoch) | 50.821 | 50.821 (700 epoch) | 52.744 | 53.189 (600 epoch) | 91.498 | 92.340 (550 epoch) | 25.731 | 26.511 (550 epoch) | 26.609 | 28.594 (450 epoch) | 16.049 | 16.529 (550 epoch) | 35.818 | 36.230 (600 epoch) | | macaroni1 | 91.922 | 92.104 (500 epoch) | 89.884 | 90.505 (500 epoch) | 83.810 | 83.978 (450 epoch) | 99.603 | 99.604 (650 epoch) | 20.435 | 20.435 (700 epoch) | 29.431 | 29.513 (650 epoch) | 95.856 | 96.221 (650 epoch) | 16.200 | 16.200 (700 epoch) | 37.336 | 40.765 (50 epoch) | 9.078 | 9.078 (700 epoch) | 17.255 | 17.311 (650 epoch) | | macaroni2 | 80.371 | 82.589 (50 epoch) | 74.330 | 81.196 (50 epoch) | 75.648 | 76.712 (600 epoch) | 98.776 | 99.452 (100 epoch) | 3.071 | 11.272 (50 epoch) | 5.041 | 17.568 (50 epoch) | 93.689 | 96.253 (150 epoch) | 2.922 | 10.086 (50 epoch) | 46.029 | 54.584 (450 epoch) | 1.489 | 5.433 (50 epoch) | 2.586 | 9.630 (50 epoch) | | pcb1 | 96.800 | 96.800 (700 epoch) | 95.568 | 95.568 (700 epoch) | 94.286 | 94.737 (300 epoch) | 99.806 | 99.806 (700 epoch) | 82.407 | 83.141 (450 epoch) | 74.854 | 75.588 (450 epoch) | 94.657 | 94.861 (600 epoch) | 48.256 | 48.256 (700 epoch) | 57.524 | 62.362 (50 epoch) | 34.013 | 34.013 (700 epoch) | 59.813 | 60.756 (450 epoch) | | pcb2 | 95.320 | 95.670 (550 epoch) | 95.543 | 95.692 (550 epoch) | 90.196 | 90.452 (650 epoch) | 99.023 | 99.023 (700 epoch) | 12.772 | 14.130 (50 epoch) | 23.946 | 23.977 (250 epoch) | 91.576 | 91.576 (700 epoch) | 15.727 | 16.150 (200 epoch) | 44.301 | 47.820 (150 epoch) | 8.653 | 8.892 (200 epoch) | 13.602 | 13.621 (250 epoch) | | pcb3 | 95.743 | 95.891 (350 epoch) | 95.910 | 96.056 (350 epoch) | 89.756 | 89.855 (400 epoch) | 99.133 | 99.142 (350 epoch) | 17.396 | 20.249 (150 epoch) | 27.636 | 28.078 (450 epoch) | 91.367 | 92.048 (650 epoch) | 19.410 | 20.042 (150 epoch) | 44.497 | 48.341 (50 epoch) | 10.924 | 11.295 (150 epoch) | 16.034 | 16.332 (450 epoch) | | pcb4 | 99.782 | 99.901 (100 epoch) | 99.782 | 99.901 (100 epoch) | 97.561 | 99.000 (50 epoch) | 98.695 | 98.746 (650 epoch) | 47.392 | 47.966 (250 epoch) | 47.394 | 47.709 (250 epoch) | 90.824 | 90.876 (500 epoch) | 24.630 | 26.925 (200 epoch) | 32.803 | 39.716 (50 epoch) | 15.138 | 16.537 (200 epoch) | 31.057 | 31.328 (250 epoch) | | pipe_fryum | 98.900 | 99.000 (650 epoch) | 99.424 | 99.489 (650 epoch) | 97.561 | 97.561 (200 epoch) | 99.108 | 99.114 (450 epoch) | 56.529 | 56.529 (700 epoch) | 58.682 | 58.733 (450 epoch) | 95.154 | 95.525 (150 epoch) | 32.535 | 34.684 (250 epoch) | 36.732 | 43.014 (100 epoch) | 21.143 | 22.399 (250 epoch) | 41.525 | 41.576 (450 epoch) | | Avg | 94.129 | 94.251 (100 epoch) | 94.203 | 94.481 (100 epoch) | 90.468 | 90.468 (700 epoch) | 98.640 | 98.640 (700 epoch) | 37.670 | 39.701 (250 epoch) | 41.928 | 44.125 (250 epoch) | 91.139 | 91.494 (250 epoch) | 24.288 | 25.164 (100 epoch) | 43.191 | 46.179 (50 epoch) | 15.148 | 15.755 (100 epoch) | 28.247 | 29.990 (250 epoch) | 2024-04-06 06:14:31,371 - ==> Total time: 4 days, 5:14:25 Eta: 1 day, 19:23:19 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 06:16:59,700 - Train: 70.02% [378800/541000] [700.2/1000.0] [batch_t 1.454 (1.459)] [data_t 0.003] [optim_t 1.450] [lr 0.005000] 2024-04-06 06:19:25,486 - Train: 70.04% [378900/541000] [700.4/1000.0] [batch_t 1.456 (1.458)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 06:21:51,217 - Train: 70.06% [379000/541000] [700.6/1000.0] [batch_t 1.458 (1.457)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-06 06:24:16,953 - Train: 70.07% [379100/541000] [700.7/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 06:26:42,699 - Train: 70.09% [379200/541000] [700.9/1000.0] [batch_t 1.453 (1.457)] [data_t 0.003] [optim_t 1.450] [lr 0.005000] 2024-04-06 06:27:42,467 - ==> Total time: 4 days, 5:27:36 Eta: 1 day, 19:16:34 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 06:29:10,672 - Train: 70.11% [379300/541000] [701.1/1000.0] [batch_t 1.447 (1.463)] [data_t 0.003] [optim_t 1.444] [lr 0.005000] 2024-04-06 06:31:36,403 - Train: 70.13% [379400/541000] [701.3/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 06:34:02,144 - Train: 70.15% [379500/541000] [701.5/1000.0] [batch_t 1.455 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 06:36:27,904 - Train: 70.17% [379600/541000] [701.7/1000.0] [batch_t 1.456 (1.458)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 06:38:53,650 - Train: 70.18% [379700/541000] [701.8/1000.0] [batch_t 1.459 (1.457)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-06 06:40:53,163 - ==> Total time: 4 days, 5:40:47 Eta: 1 day, 19:09:47 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 06:41:21,647 - Train: 70.20% [379800/541000] [702.0/1000.0] [batch_t 1.460 (1.475)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 06:43:47,385 - Train: 70.22% [379900/541000] [702.2/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 06:46:13,130 - Train: 70.24% [380000/541000] [702.4/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 06:48:38,875 - Train: 70.26% [380100/541000] [702.6/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 06:51:04,634 - Train: 70.28% [380200/541000] [702.8/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 06:53:30,279 - Train: 70.30% [380300/541000] [703.0/1000.0] [batch_t 1.455 (1.456)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 06:54:03,803 - ==> Total time: 4 days, 5:53:58 Eta: 1 day, 19:03:00 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 06:55:58,797 - Train: 70.31% [380400/541000] [703.1/1000.0] [batch_t 1.456 (1.461)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 06:58:24,614 - Train: 70.33% [380500/541000] [703.3/1000.0] [batch_t 1.465 (1.458)] [data_t 0.003] [optim_t 1.461] [lr 0.005000] 2024-04-06 07:00:50,379 - Train: 70.35% [380600/541000] [703.5/1000.0] [batch_t 1.450 (1.458)] [data_t 0.003] [optim_t 1.447] [lr 0.005000] 2024-04-06 07:03:16,129 - Train: 70.37% [380700/541000] [703.7/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 07:05:41,875 - Train: 70.39% [380800/541000] [703.9/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-06 07:07:15,204 - ==> Total time: 4 days, 6:07:09 Eta: 1 day, 18:56:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 07:08:09,877 - Train: 70.41% [380900/541000] [704.1/1000.0] [batch_t 1.456 (1.467)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 07:10:35,596 - Train: 70.43% [381000/541000] [704.3/1000.0] [batch_t 1.454 (1.457)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-06 07:13:01,308 - Train: 70.44% [381100/541000] [704.4/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 07:15:27,042 - Train: 70.46% [381200/541000] [704.6/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 07:17:52,784 - Train: 70.48% [381300/541000] [704.8/1000.0] [batch_t 1.455 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 07:20:18,505 - Train: 70.50% [381400/541000] [705.0/1000.0] [batch_t 1.461 (1.457)] [data_t 0.002] [optim_t 1.458] [lr 0.005000] 2024-04-06 07:20:25,781 - ==> Total time: 4 days, 6:20:20 Eta: 1 day, 18:49:21 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 07:22:46,580 - Train: 70.52% [381500/541000] [705.2/1000.0] [batch_t 1.455 (1.462)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 07:25:12,406 - Train: 70.54% [381600/541000] [705.4/1000.0] [batch_t 1.454 (1.458)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-06 07:27:38,126 - Train: 70.55% [381700/541000] [705.5/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 07:30:03,930 - Train: 70.57% [381800/541000] [705.7/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 07:32:29,726 - Train: 70.59% [381900/541000] [705.9/1000.0] [batch_t 1.456 (1.458)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 07:33:36,770 - ==> Total time: 4 days, 6:33:31 Eta: 1 day, 18:42:30 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 07:34:57,559 - Train: 70.61% [382000/541000] [706.1/1000.0] [batch_t 1.460 (1.463)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 07:37:23,316 - Train: 70.63% [382100/541000] [706.3/1000.0] [batch_t 1.462 (1.457)] [data_t 0.003] [optim_t 1.459] [lr 0.005000] 2024-04-06 07:39:49,082 - Train: 70.65% [382200/541000] [706.5/1000.0] [batch_t 1.455 (1.458)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-06 07:42:14,846 - Train: 70.67% [382300/541000] [706.7/1000.0] [batch_t 1.460 (1.458)] [data_t 0.004] [optim_t 1.457] [lr 0.005000] 2024-04-06 07:44:40,565 - Train: 70.68% [382400/541000] [706.8/1000.0] [batch_t 1.455 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 07:46:47,349 - ==> Total time: 4 days, 6:46:41 Eta: 1 day, 18:35:38 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 07:47:08,762 - Train: 70.70% [382500/541000] [707.0/1000.0] [batch_t 1.458 (1.502)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-06 07:49:34,521 - Train: 70.72% [382600/541000] [707.2/1000.0] [batch_t 1.455 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 07:52:00,262 - Train: 70.74% [382700/541000] [707.4/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 07:54:26,028 - Train: 70.76% [382800/541000] [707.6/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 07:56:51,832 - Train: 70.78% [382900/541000] [707.8/1000.0] [batch_t 1.457 (1.458)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 07:59:17,635 - Train: 70.79% [383000/541000] [707.9/1000.0] [batch_t 1.457 (1.458)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-06 07:59:58,465 - ==> Total time: 4 days, 6:59:52 Eta: 1 day, 18:28:45 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 08:01:45,536 - Train: 70.81% [383100/541000] [708.1/1000.0] [batch_t 1.450 (1.464)] [data_t 0.003] [optim_t 1.447] [lr 0.005000] 2024-04-06 08:04:11,272 - Train: 70.83% [383200/541000] [708.3/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 08:06:37,045 - Train: 70.85% [383300/541000] [708.5/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 08:09:02,759 - Train: 70.87% [383400/541000] [708.7/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 08:11:28,504 - Train: 70.89% [383500/541000] [708.9/1000.0] [batch_t 1.459 (1.457)] [data_t 0.004] [optim_t 1.456] [lr 0.005000] 2024-04-06 08:13:09,033 - ==> Total time: 4 days, 7:13:03 Eta: 1 day, 18:21:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 08:13:56,517 - Train: 70.91% [383600/541000] [709.1/1000.0] [batch_t 1.461 (1.469)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 08:16:22,315 - Train: 70.92% [383700/541000] [709.2/1000.0] [batch_t 1.462 (1.458)] [data_t 0.003] [optim_t 1.459] [lr 0.005000] 2024-04-06 08:18:48,129 - Train: 70.94% [383800/541000] [709.4/1000.0] [batch_t 1.454 (1.458)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-06 08:21:13,841 - Train: 70.96% [383900/541000] [709.6/1000.0] [batch_t 1.465 (1.457)] [data_t 0.003] [optim_t 1.462] [lr 0.005000] 2024-04-06 08:23:39,617 - Train: 70.98% [384000/541000] [709.8/1000.0] [batch_t 1.456 (1.458)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 08:26:05,358 - Train: 71.00% [384100/541000] [710.0/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 08:26:19,923 - ==> Total time: 4 days, 7:26:14 Eta: 1 day, 18:14:56 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 08:28:33,544 - Train: 71.02% [384200/541000] [710.2/1000.0] [batch_t 1.455 (1.462)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-06 08:30:59,272 - Train: 71.04% [384300/541000] [710.4/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 08:33:24,951 - Train: 71.05% [384400/541000] [710.5/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 08:35:50,706 - Train: 71.07% [384500/541000] [710.7/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 08:38:16,495 - Train: 71.09% [384600/541000] [710.9/1000.0] [batch_t 1.455 (1.458)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 08:39:30,865 - ==> Total time: 4 days, 7:39:25 Eta: 1 day, 18:08:00 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 08:40:44,796 - Train: 71.11% [384700/541000] [711.1/1000.0] [batch_t 1.459 (1.465)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-06 08:43:10,517 - Train: 71.13% [384800/541000] [711.3/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 08:45:36,293 - Train: 71.15% [384900/541000] [711.5/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 08:48:02,085 - Train: 71.16% [385000/541000] [711.6/1000.0] [batch_t 1.463 (1.458)] [data_t 0.003] [optim_t 1.459] [lr 0.005000] 2024-04-06 08:50:27,956 - Train: 71.18% [385100/541000] [711.8/1000.0] [batch_t 1.460 (1.459)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 08:52:42,007 - ==> Total time: 4 days, 7:52:36 Eta: 1 day, 18:01:03 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 08:52:56,058 - Train: 71.20% [385200/541000] [712.0/1000.0] [batch_t 1.462 (1.523)] [data_t 0.003] [optim_t 1.459] [lr 0.005000] 2024-04-06 08:55:21,834 - Train: 71.22% [385300/541000] [712.2/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 08:57:47,552 - Train: 71.24% [385400/541000] [712.4/1000.0] [batch_t 1.454 (1.457)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-06 09:00:13,331 - Train: 71.26% [385500/541000] [712.6/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 09:02:39,011 - Train: 71.28% [385600/541000] [712.8/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 09:05:04,784 - Train: 71.29% [385700/541000] [712.9/1000.0] [batch_t 1.455 (1.458)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 09:05:52,880 - ==> Total time: 4 days, 8:05:47 Eta: 1 day, 17:54:04 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 09:07:32,804 - Train: 71.31% [385800/541000] [713.1/1000.0] [batch_t 1.457 (1.464)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-06 09:09:58,604 - Train: 71.33% [385900/541000] [713.3/1000.0] [batch_t 1.455 (1.458)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 09:12:24,407 - Train: 71.35% [386000/541000] [713.5/1000.0] [batch_t 1.453 (1.458)] [data_t 0.003] [optim_t 1.450] [lr 0.005000] 2024-04-06 09:14:50,160 - Train: 71.37% [386100/541000] [713.7/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 09:17:15,889 - Train: 71.39% [386200/541000] [713.9/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 09:19:03,717 - ==> Total time: 4 days, 8:18:58 Eta: 1 day, 17:47:05 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 09:19:44,858 - Train: 71.40% [386300/541000] [714.0/1000.0] [batch_t 1.463 (1.485)] [data_t 0.004] [optim_t 1.460] [lr 0.005000] 2024-04-06 09:22:10,558 - Train: 71.42% [386400/541000] [714.2/1000.0] [batch_t 1.454 (1.457)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-06 09:24:36,352 - Train: 71.44% [386500/541000] [714.4/1000.0] [batch_t 1.457 (1.458)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-06 09:27:02,175 - Train: 71.46% [386600/541000] [714.6/1000.0] [batch_t 1.454 (1.458)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-06 09:29:27,903 - Train: 71.48% [386700/541000] [714.8/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 09:31:53,596 - Train: 71.50% [386800/541000] [715.0/1000.0] [batch_t 1.457 (1.457)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-06 09:32:15,471 - ==> Total time: 4 days, 8:32:09 Eta: 1 day, 17:40:05 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 09:34:21,676 - Train: 71.52% [386900/541000] [715.2/1000.0] [batch_t 1.461 (1.461)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 09:36:47,391 - Train: 71.53% [387000/541000] [715.3/1000.0] [batch_t 1.452 (1.457)] [data_t 0.003] [optim_t 1.449] [lr 0.005000] 2024-04-06 09:39:13,134 - Train: 71.55% [387100/541000] [715.5/1000.0] [batch_t 1.449 (1.457)] [data_t 0.003] [optim_t 1.446] [lr 0.005000] 2024-04-06 09:41:38,963 - Train: 71.57% [387200/541000] [715.7/1000.0] [batch_t 1.455 (1.458)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 09:44:04,710 - Train: 71.59% [387300/541000] [715.9/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 09:45:26,382 - ==> Total time: 4 days, 8:45:20 Eta: 1 day, 17:33:04 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 09:46:32,753 - Train: 71.61% [387400/541000] [716.1/1000.0] [batch_t 1.461 (1.465)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 09:48:58,499 - Train: 71.63% [387500/541000] [716.3/1000.0] [batch_t 1.458 (1.457)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-06 09:51:24,296 - Train: 71.65% [387600/541000] [716.5/1000.0] [batch_t 1.459 (1.458)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-06 09:53:50,054 - Train: 71.66% [387700/541000] [716.6/1000.0] [batch_t 1.458 (1.457)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-06 09:56:15,803 - Train: 71.68% [387800/541000] [716.8/1000.0] [batch_t 1.460 (1.457)] [data_t 0.004] [optim_t 1.457] [lr 0.005000] 2024-04-06 09:58:37,228 - ==> Total time: 4 days, 8:58:31 Eta: 1 day, 17:26:01 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 09:58:44,083 - Train: 71.70% [387900/541000] [717.0/1000.0] [batch_t 1.457 (1.659)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-06 10:01:09,862 - Train: 71.72% [388000/541000] [717.2/1000.0] [batch_t 1.456 (1.458)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 10:03:35,647 - Train: 71.74% [388100/541000] [717.4/1000.0] [batch_t 1.450 (1.458)] [data_t 0.003] [optim_t 1.447] [lr 0.005000] 2024-04-06 10:06:01,388 - Train: 71.76% [388200/541000] [717.6/1000.0] [batch_t 1.457 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 10:08:27,149 - Train: 71.77% [388300/541000] [717.7/1000.0] [batch_t 1.456 (1.458)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 10:10:52,907 - Train: 71.79% [388400/541000] [717.9/1000.0] [batch_t 1.457 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 10:11:48,321 - ==> Total time: 4 days, 9:11:42 Eta: 1 day, 17:18:58 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 10:13:21,113 - Train: 71.81% [388500/541000] [718.1/1000.0] [batch_t 1.460 (1.466)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 10:15:46,813 - Train: 71.83% [388600/541000] [718.3/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 10:18:12,542 - Train: 71.85% [388700/541000] [718.5/1000.0] [batch_t 1.452 (1.457)] [data_t 0.003] [optim_t 1.449] [lr 0.005000] 2024-04-06 10:20:38,347 - Train: 71.87% [388800/541000] [718.7/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 10:23:04,145 - Train: 71.89% [388900/541000] [718.9/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 10:24:59,275 - ==> Total time: 4 days, 9:24:53 Eta: 1 day, 17:11:53 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 10:25:33,677 - Train: 71.90% [389000/541000] [719.0/1000.0] [batch_t 1.456 (1.547)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 10:27:59,350 - Train: 71.92% [389100/541000] [719.2/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 10:30:25,112 - Train: 71.94% [389200/541000] [719.4/1000.0] [batch_t 1.455 (1.458)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 10:32:50,874 - Train: 71.96% [389300/541000] [719.6/1000.0] [batch_t 1.462 (1.458)] [data_t 0.003] [optim_t 1.459] [lr 0.005000] 2024-04-06 10:35:16,629 - Train: 71.98% [389400/541000] [719.8/1000.0] [batch_t 1.465 (1.457)] [data_t 0.003] [optim_t 1.462] [lr 0.005000] 2024-04-06 10:37:42,430 - Train: 72.00% [389500/541000] [720.0/1000.0] [batch_t 1.455 (1.458)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 10:38:11,594 - ==> Total time: 4 days, 9:38:06 Eta: 1 day, 17:04:49 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 10:40:10,557 - Train: 72.01% [389600/541000] [720.1/1000.0] [batch_t 1.462 (1.464)] [data_t 0.003] [optim_t 1.459] [lr 0.005000] 2024-04-06 10:42:36,343 - Train: 72.03% [389700/541000] [720.3/1000.0] [batch_t 1.455 (1.458)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-06 10:45:02,116 - Train: 72.05% [389800/541000] [720.5/1000.0] [batch_t 1.456 (1.458)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 10:47:27,832 - Train: 72.07% [389900/541000] [720.7/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 10:49:53,640 - Train: 72.09% [390000/541000] [720.9/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-06 10:51:22,544 - ==> Total time: 4 days, 9:51:16 Eta: 1 day, 16:57:42 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 10:52:21,685 - Train: 72.11% [390100/541000] [721.1/1000.0] [batch_t 1.455 (1.467)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 10:54:47,367 - Train: 72.13% [390200/541000] [721.3/1000.0] [batch_t 1.454 (1.457)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-06 10:57:13,081 - Train: 72.14% [390300/541000] [721.4/1000.0] [batch_t 1.455 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 10:59:38,863 - Train: 72.16% [390400/541000] [721.6/1000.0] [batch_t 1.454 (1.458)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-06 11:02:04,644 - Train: 72.18% [390500/541000] [721.8/1000.0] [batch_t 1.457 (1.458)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-06 11:04:30,450 - Train: 72.20% [390600/541000] [722.0/1000.0] [batch_t 1.456 (1.458)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 11:04:33,373 - ==> Total time: 4 days, 10:04:27 Eta: 1 day, 16:50:34 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 11:06:58,807 - Train: 72.22% [390700/541000] [722.2/1000.0] [batch_t 1.466 (1.463)] [data_t 0.003] [optim_t 1.463] [lr 0.005000] 2024-04-06 11:09:24,549 - Train: 72.24% [390800/541000] [722.4/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 11:11:50,293 - Train: 72.26% [390900/541000] [722.6/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 11:14:16,046 - Train: 72.27% [391000/541000] [722.7/1000.0] [batch_t 1.454 (1.457)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-06 11:16:41,806 - Train: 72.29% [391100/541000] [722.9/1000.0] [batch_t 1.458 (1.458)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-06 11:17:44,536 - ==> Total time: 4 days, 10:17:38 Eta: 1 day, 16:43:26 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 11:19:09,983 - Train: 72.31% [391200/541000] [723.1/1000.0] [batch_t 1.456 (1.468)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 11:21:35,746 - Train: 72.33% [391300/541000] [723.3/1000.0] [batch_t 1.451 (1.458)] [data_t 0.003] [optim_t 1.448] [lr 0.005000] 2024-04-06 11:24:01,510 - Train: 72.35% [391400/541000] [723.5/1000.0] [batch_t 1.452 (1.458)] [data_t 0.003] [optim_t 1.449] [lr 0.005000] 2024-04-06 11:26:27,309 - Train: 72.37% [391500/541000] [723.7/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 11:28:53,070 - Train: 72.38% [391600/541000] [723.8/1000.0] [batch_t 1.458 (1.458)] [data_t 0.003] [optim_t 1.455] [lr 0.005000] 2024-04-06 11:30:55,608 - ==> Total time: 4 days, 10:30:50 Eta: 1 day, 16:36:17 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 11:31:21,184 - Train: 72.40% [391700/541000] [724.0/1000.0] [batch_t 1.462 (1.482)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 11:33:47,062 - Train: 72.42% [391800/541000] [724.2/1000.0] [batch_t 1.461 (1.459)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 11:36:12,836 - Train: 72.44% [391900/541000] [724.4/1000.0] [batch_t 1.462 (1.458)] [data_t 0.003] [optim_t 1.459] [lr 0.005000] 2024-04-06 11:38:38,615 - Train: 72.46% [392000/541000] [724.6/1000.0] [batch_t 1.452 (1.458)] [data_t 0.003] [optim_t 1.449] [lr 0.005000] 2024-04-06 11:41:04,472 - Train: 72.48% [392100/541000] [724.8/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 11:43:30,295 - Train: 72.50% [392200/541000] [725.0/1000.0] [batch_t 1.456 (1.458)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 11:44:06,754 - ==> Total time: 4 days, 10:44:01 Eta: 1 day, 16:29:06 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 11:45:58,513 - Train: 72.51% [392300/541000] [725.1/1000.0] [batch_t 1.461 (1.463)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 11:48:24,284 - Train: 72.53% [392400/541000] [725.3/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 11:50:50,145 - Train: 72.55% [392500/541000] [725.5/1000.0] [batch_t 1.461 (1.459)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 11:53:16,032 - Train: 72.57% [392600/541000] [725.7/1000.0] [batch_t 1.460 (1.459)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 11:55:41,854 - Train: 72.59% [392700/541000] [725.9/1000.0] [batch_t 1.459 (1.458)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-06 11:57:18,102 - ==> Total time: 4 days, 10:57:12 Eta: 1 day, 16:21:55 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 11:58:10,540 - Train: 72.61% [392800/541000] [726.1/1000.0] [batch_t 1.455 (1.470)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-06 12:00:36,314 - Train: 72.62% [392900/541000] [726.2/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 12:03:02,069 - Train: 72.64% [393000/541000] [726.4/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 12:05:27,865 - Train: 72.66% [393100/541000] [726.6/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 12:07:53,661 - Train: 72.68% [393200/541000] [726.8/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 12:10:19,462 - Train: 72.70% [393300/541000] [727.0/1000.0] [batch_t 1.461 (1.458)] [data_t 0.004] [optim_t 1.457] [lr 0.005000] 2024-04-06 12:10:29,686 - ==> Total time: 4 days, 11:10:24 Eta: 1 day, 16:14:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 12:12:47,727 - Train: 72.72% [393400/541000] [727.2/1000.0] [batch_t 1.460 (1.462)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 12:15:13,494 - Train: 72.74% [393500/541000] [727.4/1000.0] [batch_t 1.459 (1.458)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-06 12:17:39,288 - Train: 72.75% [393600/541000] [727.5/1000.0] [batch_t 1.459 (1.458)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-06 12:20:05,048 - Train: 72.77% [393700/541000] [727.7/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 12:22:30,884 - Train: 72.79% [393800/541000] [727.9/1000.0] [batch_t 1.451 (1.458)] [data_t 0.003] [optim_t 1.448] [lr 0.005000] 2024-04-06 12:23:40,892 - ==> Total time: 4 days, 11:23:35 Eta: 1 day, 16:07:29 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 12:24:59,875 - Train: 72.81% [393900/541000] [728.1/1000.0] [batch_t 1.461 (1.475)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 12:27:25,614 - Train: 72.83% [394000/541000] [728.3/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-06 12:29:51,433 - Train: 72.85% [394100/541000] [728.5/1000.0] [batch_t 1.467 (1.458)] [data_t 0.003] [optim_t 1.464] [lr 0.005000] 2024-04-06 12:32:17,168 - Train: 72.87% [394200/541000] [728.7/1000.0] [batch_t 1.454 (1.457)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-06 12:34:42,947 - Train: 72.88% [394300/541000] [728.8/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 12:36:52,694 - ==> Total time: 4 days, 11:36:47 Eta: 1 day, 16:00:15 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 12:37:12,418 - Train: 72.90% [394400/541000] [729.0/1000.0] [batch_t 1.455 (1.545)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-06 12:39:38,213 - Train: 72.92% [394500/541000] [729.2/1000.0] [batch_t 1.459 (1.458)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-06 12:42:04,009 - Train: 72.94% [394600/541000] [729.4/1000.0] [batch_t 1.457 (1.458)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 12:44:29,782 - Train: 72.96% [394700/541000] [729.6/1000.0] [batch_t 1.462 (1.458)] [data_t 0.003] [optim_t 1.459] [lr 0.005000] 2024-04-06 12:46:55,547 - Train: 72.98% [394800/541000] [729.8/1000.0] [batch_t 1.455 (1.458)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 12:49:21,384 - Train: 72.99% [394900/541000] [729.9/1000.0] [batch_t 1.462 (1.458)] [data_t 0.003] [optim_t 1.459] [lr 0.005000] 2024-04-06 12:50:05,090 - ==> Total time: 4 days, 11:49:59 Eta: 1 day, 15:53:00 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 12:51:49,606 - Train: 73.01% [395000/541000] [730.1/1000.0] [batch_t 1.461 (1.465)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 12:54:15,383 - Train: 73.03% [395100/541000] [730.3/1000.0] [batch_t 1.459 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 12:56:41,180 - Train: 73.05% [395200/541000] [730.5/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 12:59:06,955 - Train: 73.07% [395300/541000] [730.7/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 13:01:32,781 - Train: 73.09% [395400/541000] [730.9/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 13:03:16,316 - ==> Total time: 4 days, 12:03:10 Eta: 1 day, 15:45:44 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 13:04:00,698 - Train: 73.11% [395500/541000] [731.1/1000.0] [batch_t 1.460 (1.473)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 13:06:26,516 - Train: 73.12% [395600/541000] [731.2/1000.0] [batch_t 1.458 (1.458)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-06 13:08:52,343 - Train: 73.14% [395700/541000] [731.4/1000.0] [batch_t 1.462 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 13:11:18,210 - Train: 73.16% [395800/541000] [731.6/1000.0] [batch_t 1.461 (1.459)] [data_t 0.002] [optim_t 1.458] [lr 0.005000] 2024-04-06 13:13:44,023 - Train: 73.18% [395900/541000] [731.8/1000.0] [batch_t 1.457 (1.458)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-06 13:16:09,795 - Train: 73.20% [396000/541000] [732.0/1000.0] [batch_t 1.457 (1.458)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-06 13:16:27,300 - ==> Total time: 4 days, 12:16:21 Eta: 1 day, 15:38:26 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 13:18:37,756 - Train: 73.22% [396100/541000] [732.2/1000.0] [batch_t 1.460 (1.463)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 13:21:03,520 - Train: 73.23% [396200/541000] [732.3/1000.0] [batch_t 1.457 (1.458)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-06 13:23:29,251 - Train: 73.25% [396300/541000] [732.5/1000.0] [batch_t 1.458 (1.457)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-06 13:25:55,040 - Train: 73.27% [396400/541000] [732.7/1000.0] [batch_t 1.456 (1.458)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 13:28:20,787 - Train: 73.29% [396500/541000] [732.9/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 13:29:38,062 - ==> Total time: 4 days, 12:29:32 Eta: 1 day, 15:31:08 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 13:30:49,102 - Train: 73.31% [396600/541000] [733.1/1000.0] [batch_t 1.456 (1.468)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 13:33:14,907 - Train: 73.33% [396700/541000] [733.3/1000.0] [batch_t 1.456 (1.458)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 13:35:40,749 - Train: 73.35% [396800/541000] [733.5/1000.0] [batch_t 1.459 (1.458)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-06 13:38:06,587 - Train: 73.36% [396900/541000] [733.6/1000.0] [batch_t 1.459 (1.458)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-06 13:40:32,399 - Train: 73.38% [397000/541000] [733.8/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 13:42:49,468 - ==> Total time: 4 days, 12:42:43 Eta: 1 day, 15:23:49 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 13:43:00,691 - Train: 73.40% [397100/541000] [734.0/1000.0] [batch_t 1.457 (1.530)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-06 13:45:26,521 - Train: 73.42% [397200/541000] [734.2/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 13:47:52,353 - Train: 73.44% [397300/541000] [734.4/1000.0] [batch_t 1.456 (1.458)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 13:50:18,137 - Train: 73.46% [397400/541000] [734.6/1000.0] [batch_t 1.455 (1.458)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 13:52:43,840 - Train: 73.48% [397500/541000] [734.8/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 13:55:09,602 - Train: 73.49% [397600/541000] [734.9/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-06 13:56:00,637 - ==> Total time: 4 days, 12:55:55 Eta: 1 day, 15:16:29 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 13:57:38,720 - Train: 73.51% [397700/541000] [735.1/1000.0] [batch_t 1.453 (1.478)] [data_t 0.003] [optim_t 1.450] [lr 0.005000] 2024-04-06 14:00:04,527 - Train: 73.53% [397800/541000] [735.3/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 14:02:30,316 - Train: 73.55% [397900/541000] [735.5/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 14:04:56,151 - Train: 73.57% [398000/541000] [735.7/1000.0] [batch_t 1.459 (1.458)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-06 14:07:21,995 - Train: 73.59% [398100/541000] [735.9/1000.0] [batch_t 1.452 (1.458)] [data_t 0.003] [optim_t 1.448] [lr 0.005000] 2024-04-06 14:09:12,806 - ==> Total time: 4 days, 13:09:07 Eta: 1 day, 15:09:08 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 14:09:50,190 - Train: 73.60% [398200/541000] [736.0/1000.0] [batch_t 1.456 (1.471)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 14:11:58,517 - Train: 73.62% [398300/541000] [736.2/1000.0] [batch_t 0.607 (1.283)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 14:12:59,199 - Train: 73.64% [398400/541000] [736.4/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-06 14:13:59,881 - Train: 73.66% [398500/541000] [736.6/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 14:15:00,637 - Train: 73.68% [398600/541000] [736.8/1000.0] [batch_t 0.609 (0.607)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-06 14:16:01,612 - Train: 73.70% [398700/541000] [737.0/1000.0] [batch_t 0.609 (0.610)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 14:16:11,951 - ==> Total time: 4 days, 13:16:06 Eta: 1 day, 14:59:33 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 14:17:04,760 - Train: 73.72% [398800/541000] [737.2/1000.0] [batch_t 0.607 (0.614)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 14:18:05,540 - Train: 73.73% [398900/541000] [737.3/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-06 14:19:06,360 - Train: 73.75% [399000/541000] [737.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 14:20:07,162 - Train: 73.77% [399100/541000] [737.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 14:21:07,967 - Train: 73.79% [399200/541000] [737.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 14:21:43,254 - ==> Total time: 4 days, 13:21:37 Eta: 1 day, 14:49:28 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 14:22:11,136 - Train: 73.81% [399300/541000] [738.1/1000.0] [batch_t 0.606 (0.617)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 14:23:11,974 - Train: 73.83% [399400/541000] [738.3/1000.0] [batch_t 0.610 (0.608)] [data_t 0.003] [optim_t 0.608] [lr 0.005000] 2024-04-06 14:24:12,799 - Train: 73.84% [399500/541000] [738.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 14:25:13,590 - Train: 73.86% [399600/541000] [738.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 14:26:14,395 - Train: 73.88% [399700/541000] [738.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 14:27:14,599 - ==> Total time: 4 days, 13:27:09 Eta: 1 day, 14:39:23 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 14:27:17,451 - Train: 73.90% [399800/541000] [739.0/1000.0] [batch_t 0.983 (0.983)] [data_t 0.367] [optim_t 0.616] [lr 0.005000] 2024-04-06 14:28:18,374 - Train: 73.92% [399900/541000] [739.2/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 14:29:19,167 - Train: 73.94% [400000/541000] [739.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 14:30:19,946 - Train: 73.96% [400100/541000] [739.6/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 14:31:20,728 - Train: 73.97% [400200/541000] [739.7/1000.0] [batch_t 0.610 (0.608)] [data_t 0.002] [optim_t 0.608] [lr 0.005000] 2024-04-06 14:32:21,521 - Train: 73.99% [400300/541000] [739.9/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 14:32:45,847 - ==> Total time: 4 days, 13:32:40 Eta: 1 day, 14:29:19 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 14:33:24,626 - Train: 74.01% [400400/541000] [740.1/1000.0] [batch_t 0.609 (0.615)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 14:34:25,473 - Train: 74.03% [400500/541000] [740.3/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 14:35:26,456 - Train: 74.05% [400600/541000] [740.5/1000.0] [batch_t 0.608 (0.610)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 14:36:27,274 - Train: 74.07% [400700/541000] [740.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 14:37:28,083 - Train: 74.09% [400800/541000] [740.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 14:38:17,350 - ==> Total time: 4 days, 13:38:11 Eta: 1 day, 14:19:15 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 14:38:31,277 - Train: 74.10% [400900/541000] [741.0/1000.0] [batch_t 0.607 (0.631)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 14:39:32,107 - Train: 74.12% [401000/541000] [741.2/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 14:40:32,962 - Train: 74.14% [401100/541000] [741.4/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-06 14:41:33,939 - Train: 74.16% [401200/541000] [741.6/1000.0] [batch_t 0.608 (0.610)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 14:42:34,760 - Train: 74.18% [401300/541000] [741.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 14:43:35,586 - Train: 74.20% [401400/541000] [742.0/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 14:43:48,970 - ==> Total time: 4 days, 13:43:43 Eta: 1 day, 14:09:13 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 14:44:38,459 - Train: 74.21% [401500/541000] [742.1/1000.0] [batch_t 0.608 (0.612)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 14:45:39,261 - Train: 74.23% [401600/541000] [742.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 14:46:40,173 - Train: 74.25% [401700/541000] [742.5/1000.0] [batch_t 0.607 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 14:47:40,973 - Train: 74.27% [401800/541000] [742.7/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 14:48:41,760 - Train: 74.29% [401900/541000] [742.9/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 14:49:20,101 - ==> Total time: 4 days, 13:49:14 Eta: 1 day, 13:59:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 14:49:44,875 - Train: 74.31% [402000/541000] [743.1/1000.0] [batch_t 0.607 (0.618)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 14:50:45,655 - Train: 74.33% [402100/541000] [743.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 14:51:46,408 - Train: 74.34% [402200/541000] [743.4/1000.0] [batch_t 0.607 (0.607)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 14:52:47,163 - Train: 74.36% [402300/541000] [743.6/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 14:53:48,080 - Train: 74.38% [402400/541000] [743.8/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 14:54:48,875 - Train: 74.40% [402500/541000] [744.0/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 14:54:51,312 - ==> Total time: 4 days, 13:54:45 Eta: 1 day, 13:49:09 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 14:55:51,711 - Train: 74.42% [402600/541000] [744.2/1000.0] [batch_t 0.608 (0.612)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 14:56:52,549 - Train: 74.44% [402700/541000] [744.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 14:57:53,367 - Train: 74.45% [402800/541000] [744.5/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 14:58:54,149 - Train: 74.47% [402900/541000] [744.7/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 14:59:55,091 - Train: 74.49% [403000/541000] [744.9/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 15:00:22,464 - ==> Total time: 4 days, 14:00:16 Eta: 1 day, 13:39:09 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 15:00:57,916 - Train: 74.51% [403100/541000] [745.1/1000.0] [batch_t 0.617 (0.614)] [data_t 0.003] [optim_t 0.614] [lr 0.005000] 2024-04-06 15:01:58,673 - Train: 74.53% [403200/541000] [745.3/1000.0] [batch_t 0.609 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 15:02:59,461 - Train: 74.55% [403300/541000] [745.5/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 15:04:00,255 - Train: 74.57% [403400/541000] [745.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 15:05:01,039 - Train: 74.58% [403500/541000] [745.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 15:05:53,310 - ==> Total time: 4 days, 14:05:47 Eta: 1 day, 13:29:09 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 15:06:04,070 - Train: 74.60% [403600/541000] [746.0/1000.0] [batch_t 0.611 (0.637)] [data_t 0.003] [optim_t 0.609] [lr 0.005000] 2024-04-06 15:07:04,850 - Train: 74.62% [403700/541000] [746.2/1000.0] [batch_t 0.610 (0.608)] [data_t 0.003] [optim_t 0.608] [lr 0.005000] 2024-04-06 15:08:05,624 - Train: 74.64% [403800/541000] [746.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 15:09:06,420 - Train: 74.66% [403900/541000] [746.6/1000.0] [batch_t 0.611 (0.608)] [data_t 0.003] [optim_t 0.608] [lr 0.005000] 2024-04-06 15:10:07,210 - Train: 74.68% [404000/541000] [746.8/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-06 15:11:08,426 - Train: 74.70% [404100/541000] [747.0/1000.0] [batch_t 0.608 (0.612)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 15:11:24,840 - ==> Total time: 4 days, 14:11:19 Eta: 1 day, 13:19:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 15:12:14,418 - Train: 74.71% [404200/541000] [747.1/1000.0] [batch_t 0.608 (0.626)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 15:13:15,385 - Train: 74.73% [404300/541000] [747.3/1000.0] [batch_t 0.608 (0.610)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 15:14:16,217 - Train: 74.75% [404400/541000] [747.5/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-06 15:15:17,024 - Train: 74.77% [404500/541000] [747.7/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 15:16:24,132 - Train: 74.79% [404600/541000] [747.9/1000.0] [batch_t 0.608 (0.671)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 15:17:05,478 - ==> Total time: 4 days, 14:16:59 Eta: 1 day, 13:09:15 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 15:17:28,463 - Train: 74.81% [404700/541000] [748.1/1000.0] [batch_t 0.607 (0.653)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 15:18:29,332 - Train: 74.82% [404800/541000] [748.2/1000.0] [batch_t 0.609 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 15:19:30,195 - Train: 74.84% [404900/541000] [748.4/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 15:20:31,172 - Train: 74.86% [405000/541000] [748.6/1000.0] [batch_t 0.609 (0.610)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 15:21:32,013 - Train: 74.88% [405100/541000] [748.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 15:22:32,854 - Train: 74.90% [405200/541000] [749.0/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 15:22:38,335 - ==> Total time: 4 days, 14:22:32 Eta: 1 day, 12:59:18 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 15:23:37,362 - Train: 74.92% [405300/541000] [749.2/1000.0] [batch_t 0.608 (0.619)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 15:24:38,179 - Train: 74.94% [405400/541000] [749.4/1000.0] [batch_t 0.611 (0.608)] [data_t 0.002] [optim_t 0.609] [lr 0.005000] 2024-04-06 15:25:38,896 - Train: 74.95% [405500/541000] [749.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 15:26:39,741 - Train: 74.97% [405600/541000] [749.7/1000.0] [batch_t 0.605 (0.608)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-06 15:27:40,544 - Train: 74.99% [405700/541000] [749.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 15:28:42,722 - Test: 36.76% [50/136] [batch_t 0.219 (0.611)] 2024-04-06 15:29:07,356 - Test: 73.53% [100/136] [batch_t 0.210 (0.552)] 2024-04-06 15:29:26,572 - Test: 100.00% [136/136] [batch_t 0.055 (0.547)] 2024-04-06 15:30:15,497 - ==> Metric Time for candle : 0.002 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.660 (mAUROC_px) 8.133 (mAP_px) 0.934 (mF1_max_px) 25.499 (mAUPRO_px) 0.312 (mF1_px_0.2_0.8_0.1) 0.314 (mAcc_px_0.2_0.8_0.1) 0.321 (mIoU_px_0.2_0.8_0.1) 0.917 (mIoU_max_px) 2024-04-06 15:30:52,978 - ==> Metric Time for capsules : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 8.303 (mAUROC_px) 6.380 (mAP_px) 0.772 (mF1_max_px) 20.325 (mAUPRO_px) 0.251 (mF1_px_0.2_0.8_0.1) 0.261 (mAcc_px_0.2_0.8_0.1) 0.249 (mIoU_px_0.2_0.8_0.1) 0.750 (mIoU_max_px) 2024-04-06 15:31:28,669 - ==> Metric Time for cashew : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.809 (mAUROC_px) 6.060 (mAP_px) 0.691 (mF1_max_px) 19.453 (mAUPRO_px) 0.233 (mF1_px_0.2_0.8_0.1) 0.239 (mAcc_px_0.2_0.8_0.1) 0.237 (mIoU_px_0.2_0.8_0.1) 0.790 (mIoU_max_px) 2024-04-06 15:32:06,358 - ==> Metric Time for chewinggum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.738 (mAUROC_px) 5.893 (mAP_px) 0.688 (mF1_max_px) 21.779 (mAUPRO_px) 0.235 (mF1_px_0.2_0.8_0.1) 0.240 (mAcc_px_0.2_0.8_0.1) 0.246 (mIoU_px_0.2_0.8_0.1) 0.691 (mIoU_max_px) 2024-04-06 15:32:43,268 - ==> Metric Time for fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.679 (mAUROC_px) 5.662 (mAP_px) 0.694 (mF1_max_px) 21.298 (mAUPRO_px) 0.237 (mF1_px_0.2_0.8_0.1) 0.235 (mAcc_px_0.2_0.8_0.1) 0.233 (mIoU_px_0.2_0.8_0.1) 0.694 (mIoU_max_px) 2024-04-06 15:33:29,283 - ==> Metric Time for macaroni1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.342 (mAUROC_px) 7.800 (mAP_px) 0.916 (mF1_max_px) 24.841 (mAUPRO_px) 0.315 (mF1_px_0.2_0.8_0.1) 0.312 (mAcc_px_0.2_0.8_0.1) 0.313 (mIoU_px_0.2_0.8_0.1) 0.917 (mIoU_max_px) 2024-04-06 15:34:16,173 - ==> Metric Time for macaroni2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.486 (mAUROC_px) 7.887 (mAP_px) 0.921 (mF1_max_px) 25.467 (mAUPRO_px) 0.311 (mF1_px_0.2_0.8_0.1) 0.319 (mAcc_px_0.2_0.8_0.1) 0.315 (mIoU_px_0.2_0.8_0.1) 0.907 (mIoU_max_px) 2024-04-06 15:35:04,006 - ==> Metric Time for pcb1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.576 (mAUROC_px) 8.045 (mAP_px) 0.921 (mF1_max_px) 26.115 (mAUPRO_px) 0.329 (mF1_px_0.2_0.8_0.1) 0.328 (mAcc_px_0.2_0.8_0.1) 0.313 (mIoU_px_0.2_0.8_0.1) 0.918 (mIoU_max_px) 2024-04-06 15:35:51,893 - ==> Metric Time for pcb2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.686 (mAUROC_px) 8.099 (mAP_px) 0.921 (mF1_max_px) 26.037 (mAUPRO_px) 0.316 (mF1_px_0.2_0.8_0.1) 0.315 (mAcc_px_0.2_0.8_0.1) 0.315 (mIoU_px_0.2_0.8_0.1) 0.912 (mIoU_max_px) 2024-04-06 15:36:39,382 - ==> Metric Time for pcb3 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.577 (mAUROC_px) 7.911 (mAP_px) 0.924 (mF1_max_px) 25.936 (mAUPRO_px) 0.315 (mF1_px_0.2_0.8_0.1) 0.312 (mAcc_px_0.2_0.8_0.1) 0.322 (mIoU_px_0.2_0.8_0.1) 0.912 (mIoU_max_px) 2024-04-06 15:37:28,551 - ==> Metric Time for pcb4 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.594 (mAUROC_px) 8.030 (mAP_px) 0.916 (mF1_max_px) 27.492 (mAUPRO_px) 0.319 (mF1_px_0.2_0.8_0.1) 0.312 (mAcc_px_0.2_0.8_0.1) 0.311 (mIoU_px_0.2_0.8_0.1) 0.915 (mIoU_max_px) 2024-04-06 15:38:05,133 - ==> Metric Time for pipe_fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.815 (mAUROC_px) 5.839 (mAP_px) 0.675 (mF1_max_px) 20.333 (mAUPRO_px) 0.295 (mF1_px_0.2_0.8_0.1) 0.294 (mAcc_px_0.2_0.8_0.1) 0.298 (mIoU_px_0.2_0.8_0.1) 0.846 (mIoU_max_px) 2024-04-06 15:38:05,149 - | Name | mAUROC_sp_max | mAUROC_sp_max (Max) | mAP_sp_max | mAP_sp_max (Max) | mF1_max_sp_max | mF1_max_sp_max (Max) | mAUROC_px | mAUROC_px (Max) | mAP_px | mAP_px (Max) | mF1_max_px | mF1_max_px (Max) | mAUPRO_px | mAUPRO_px (Max) | mF1_px_0.2_0.8_0.1 | mF1_px_0.2_0.8_0.1 (Max) | mAcc_px_0.2_0.8_0.1 | mAcc_px_0.2_0.8_0.1 (Max) | mIoU_px_0.2_0.8_0.1 | mIoU_px_0.2_0.8_0.1 (Max) | mIoU_max_px | mIoU_max_px (Max) | |:----------:|:---------------:|:---------------------:|:------------:|:------------------:|:----------------:|:----------------------:|:-----------:|:------------------:|:--------:|:------------------:|:------------:|:------------------:|:-----------:|:------------------:|:--------------------:|:--------------------------:|:---------------------:|:---------------------------:|:---------------------:|:---------------------------:|:-------------:|:-------------------:| | candle | 95.890 | 97.100 (50 epoch) | 96.219 | 97.346 (50 epoch) | 89.109 | 91.542 (50 epoch) | 99.045 | 99.098 (250 epoch) | 21.166 | 23.463 (150 epoch) | 31.278 | 32.654 (200 epoch) | 95.184 | 96.087 (300 epoch) | 20.195 | 21.331 (500 epoch) | 39.476 | 41.526 (650 epoch) | 11.504 | 12.185 (500 epoch) | 18.538 | 19.513 (200 epoch) | | capsules | 84.317 | 91.750 (100 epoch) | 89.902 | 94.964 (100 epoch) | 85.446 | 88.780 (100 epoch) | 98.100 | 99.193 (700 epoch) | 48.355 | 61.256 (100 epoch) | 52.142 | 59.868 (250 epoch) | 80.690 | 91.811 (100 epoch) | 23.707 | 33.186 (100 epoch) | 61.676 | 62.576 (50 epoch) | 14.680 | 21.701 (100 epoch) | 35.265 | 42.722 (250 epoch) | | cashew | 93.360 | 94.800 (150 epoch) | 96.119 | 97.492 (150 epoch) | 91.542 | 92.929 (700 epoch) | 95.612 | 96.185 (700 epoch) | 45.748 | 50.952 (250 epoch) | 44.668 | 54.438 (400 epoch) | 85.536 | 89.185 (350 epoch) | 23.943 | 28.352 (350 epoch) | 27.636 | 34.522 (150 epoch) | 14.647 | 17.797 (350 epoch) | 28.757 | 37.398 (400 epoch) | | chewinggum | 97.488 | 97.802 (300 epoch) | 98.758 | 98.939 (300 epoch) | 94.301 | 94.792 (450 epoch) | 97.955 | 98.145 (100 epoch) | 46.408 | 59.823 (50 epoch) | 54.708 | 61.088 (50 epoch) | 79.294 | 80.312 (400 epoch) | 30.776 | 35.748 (50 epoch) | 65.356 | 65.611 (50 epoch) | 19.636 | 23.754 (50 epoch) | 37.653 | 43.976 (50 epoch) | | fryum | 96.360 | 96.560 (650 epoch) | 98.217 | 98.273 (650 epoch) | 93.204 | 93.897 (650 epoch) | 97.308 | 97.308 (750 epoch) | 52.248 | 52.248 (750 epoch) | 53.176 | 53.189 (600 epoch) | 91.740 | 92.340 (550 epoch) | 26.072 | 26.511 (550 epoch) | 27.382 | 28.594 (450 epoch) | 16.351 | 16.529 (550 epoch) | 36.218 | 36.230 (600 epoch) | | macaroni1 | 91.204 | 92.104 (500 epoch) | 86.387 | 90.505 (500 epoch) | 84.314 | 84.314 (750 epoch) | 99.257 | 99.604 (650 epoch) | 7.080 | 20.435 (700 epoch) | 12.454 | 29.513 (650 epoch) | 93.967 | 96.221 (650 epoch) | 7.535 | 16.200 (700 epoch) | 37.573 | 40.765 (50 epoch) | 3.937 | 9.078 (700 epoch) | 6.641 | 17.311 (650 epoch) | | macaroni2 | 81.611 | 82.589 (50 epoch) | 75.053 | 81.196 (50 epoch) | 77.833 | 77.833 (750 epoch) | 98.667 | 99.452 (100 epoch) | 1.927 | 11.272 (50 epoch) | 4.170 | 17.568 (50 epoch) | 93.932 | 96.253 (150 epoch) | 2.403 | 10.086 (50 epoch) | 45.076 | 54.584 (450 epoch) | 1.220 | 5.433 (50 epoch) | 2.129 | 9.630 (50 epoch) | | pcb1 | 96.490 | 96.800 (700 epoch) | 95.218 | 95.568 (700 epoch) | 93.532 | 94.737 (300 epoch) | 99.800 | 99.806 (700 epoch) | 81.595 | 83.141 (450 epoch) | 74.031 | 75.588 (450 epoch) | 94.583 | 94.861 (600 epoch) | 47.279 | 48.256 (700 epoch) | 55.677 | 62.362 (50 epoch) | 33.027 | 34.013 (700 epoch) | 58.770 | 60.756 (450 epoch) | | pcb2 | 95.790 | 95.790 (750 epoch) | 95.654 | 95.692 (550 epoch) | 91.000 | 91.000 (750 epoch) | 99.009 | 99.023 (700 epoch) | 12.703 | 14.130 (50 epoch) | 23.854 | 23.977 (250 epoch) | 90.362 | 91.576 (700 epoch) | 15.350 | 16.150 (200 epoch) | 42.266 | 47.820 (150 epoch) | 8.442 | 8.892 (200 epoch) | 13.542 | 13.621 (250 epoch) | | pcb3 | 95.248 | 95.891 (350 epoch) | 95.547 | 96.056 (350 epoch) | 89.109 | 89.855 (400 epoch) | 99.174 | 99.174 (750 epoch) | 17.975 | 20.249 (150 epoch) | 27.633 | 28.078 (450 epoch) | 90.916 | 92.048 (650 epoch) | 18.955 | 20.042 (150 epoch) | 41.663 | 48.341 (50 epoch) | 10.669 | 11.295 (150 epoch) | 16.032 | 16.332 (450 epoch) | | pcb4 | 99.713 | 99.901 (100 epoch) | 99.714 | 99.901 (100 epoch) | 97.561 | 99.000 (50 epoch) | 98.642 | 98.746 (650 epoch) | 46.302 | 47.966 (250 epoch) | 45.873 | 47.709 (250 epoch) | 89.540 | 90.876 (500 epoch) | 23.350 | 26.925 (200 epoch) | 30.255 | 39.716 (50 epoch) | 14.315 | 16.537 (200 epoch) | 29.763 | 31.328 (250 epoch) | | pipe_fryum | 99.000 | 99.000 (650 epoch) | 99.482 | 99.489 (650 epoch) | 97.030 | 97.561 (200 epoch) | 99.108 | 99.114 (450 epoch) | 55.511 | 56.529 (700 epoch) | 58.245 | 58.733 (450 epoch) | 94.396 | 95.525 (150 epoch) | 30.294 | 34.684 (250 epoch) | 33.568 | 43.014 (100 epoch) | 19.664 | 22.399 (250 epoch) | 41.089 | 41.576 (450 epoch) | | Avg | 93.872 | 94.251 (100 epoch) | 93.856 | 94.481 (100 epoch) | 90.332 | 90.468 (700 epoch) | 98.473 | 98.640 (700 epoch) | 36.418 | 39.701 (250 epoch) | 40.186 | 44.125 (250 epoch) | 90.012 | 91.494 (250 epoch) | 22.488 | 25.164 (100 epoch) | 42.300 | 46.179 (50 epoch) | 14.008 | 15.755 (100 epoch) | 27.033 | 29.990 (250 epoch) | 2024-04-06 15:38:05,536 - ==> Total time: 4 days, 14:37:59 Eta: 1 day, 12:52:39 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 15:39:21,851 - Train: 75.01% [405800/541000] [750.1/1000.0] [batch_t 1.459 (1.474)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-06 15:41:47,364 - Train: 75.03% [405900/541000] [750.3/1000.0] [batch_t 1.460 (1.455)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 15:44:12,966 - Train: 75.05% [406000/541000] [750.5/1000.0] [batch_t 1.460 (1.456)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 15:46:38,686 - Train: 75.06% [406100/541000] [750.6/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 15:49:04,370 - Train: 75.08% [406200/541000] [750.8/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-06 15:51:16,941 - ==> Total time: 4 days, 14:51:11 Eta: 1 day, 12:45:15 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 15:51:35,556 - Train: 75.10% [406300/541000] [751.0/1000.0] [batch_t 1.453 (1.664)] [data_t 0.003] [optim_t 1.450] [lr 0.005000] 2024-04-06 15:54:01,268 - Train: 75.12% [406400/541000] [751.2/1000.0] [batch_t 1.462 (1.457)] [data_t 0.003] [optim_t 1.459] [lr 0.005000] 2024-04-06 15:56:26,945 - Train: 75.14% [406500/541000] [751.4/1000.0] [batch_t 1.451 (1.457)] [data_t 0.003] [optim_t 1.448] [lr 0.005000] 2024-04-06 15:58:52,678 - Train: 75.16% [406600/541000] [751.6/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 16:01:21,132 - Train: 75.18% [406700/541000] [751.8/1000.0] [batch_t 1.455 (1.484)] [data_t 0.003] [optim_t 1.451] [lr 0.005000] 2024-04-06 16:03:46,801 - Train: 75.19% [406800/541000] [751.9/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 16:04:33,399 - ==> Total time: 4 days, 15:04:27 Eta: 1 day, 12:37:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 16:06:15,842 - Train: 75.21% [406900/541000] [752.1/1000.0] [batch_t 1.452 (1.469)] [data_t 0.003] [optim_t 1.449] [lr 0.005000] 2024-04-06 16:08:41,547 - Train: 75.23% [407000/541000] [752.3/1000.0] [batch_t 1.455 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 16:11:07,307 - Train: 75.25% [407100/541000] [752.5/1000.0] [batch_t 1.454 (1.458)] [data_t 0.003] [optim_t 1.450] [lr 0.005000] 2024-04-06 16:13:33,043 - Train: 75.27% [407200/541000] [752.7/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 16:15:58,765 - Train: 75.29% [407300/541000] [752.9/1000.0] [batch_t 1.459 (1.457)] [data_t 0.003] [optim_t 1.456] [lr 0.005000] 2024-04-06 16:17:45,177 - ==> Total time: 4 days, 15:17:39 Eta: 1 day, 12:30:24 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 16:18:26,858 - Train: 75.30% [407400/541000] [753.0/1000.0] [batch_t 1.456 (1.475)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 16:20:52,583 - Train: 75.32% [407500/541000] [753.2/1000.0] [batch_t 1.462 (1.457)] [data_t 0.003] [optim_t 1.459] [lr 0.005000] 2024-04-06 16:23:18,311 - Train: 75.34% [407600/541000] [753.4/1000.0] [batch_t 1.449 (1.457)] [data_t 0.003] [optim_t 1.446] [lr 0.005000] 2024-04-06 16:25:43,997 - Train: 75.36% [407700/541000] [753.6/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 16:28:09,692 - Train: 75.38% [407800/541000] [753.8/1000.0] [batch_t 1.455 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 16:30:35,502 - Train: 75.40% [407900/541000] [754.0/1000.0] [batch_t 1.456 (1.458)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 16:30:55,923 - ==> Total time: 4 days, 15:30:50 Eta: 1 day, 12:22:57 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 16:33:04,767 - Train: 75.42% [408000/541000] [754.2/1000.0] [batch_t 1.450 (1.466)] [data_t 0.003] [optim_t 1.446] [lr 0.005000] 2024-04-06 16:35:30,326 - Train: 75.43% [408100/541000] [754.3/1000.0] [batch_t 1.450 (1.455)] [data_t 0.003] [optim_t 1.447] [lr 0.005000] 2024-04-06 16:37:56,001 - Train: 75.45% [408200/541000] [754.5/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 16:40:21,740 - Train: 75.47% [408300/541000] [754.7/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.005000] 2024-04-06 16:42:47,424 - Train: 75.49% [408400/541000] [754.9/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 16:44:07,584 - ==> Total time: 4 days, 15:44:02 Eta: 1 day, 12:15:28 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 16:45:15,325 - Train: 75.51% [408500/541000] [755.1/1000.0] [batch_t 1.457 (1.464)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-06 16:47:41,091 - Train: 75.53% [408600/541000] [755.3/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 16:50:06,881 - Train: 75.55% [408700/541000] [755.5/1000.0] [batch_t 1.456 (1.458)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 16:52:32,655 - Train: 75.56% [408800/541000] [755.6/1000.0] [batch_t 1.453 (1.458)] [data_t 0.003] [optim_t 1.450] [lr 0.005000] 2024-04-06 16:54:58,367 - Train: 75.58% [408900/541000] [755.8/1000.0] [batch_t 1.450 (1.457)] [data_t 0.003] [optim_t 1.447] [lr 0.005000] 2024-04-06 16:57:18,266 - ==> Total time: 4 days, 15:57:12 Eta: 1 day, 12:07:59 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 16:57:26,409 - Train: 75.60% [409000/541000] [756.0/1000.0] [batch_t 1.455 (1.598)] [data_t 0.003] [optim_t 1.452] [lr 0.005000] 2024-04-06 16:59:52,086 - Train: 75.62% [409100/541000] [756.2/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.005000] 2024-04-06 17:02:17,852 - Train: 75.64% [409200/541000] [756.4/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 17:04:43,641 - Train: 75.66% [409300/541000] [756.6/1000.0] [batch_t 1.457 (1.458)] [data_t 0.003] [optim_t 1.454] [lr 0.005000] 2024-04-06 17:07:09,426 - Train: 75.67% [409400/541000] [756.7/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.005000] 2024-04-06 17:09:35,163 - Train: 75.69% [409500/541000] [756.9/1000.0] [batch_t 1.446 (1.457)] [data_t 0.003] [optim_t 1.443] [lr 0.005000] 2024-04-06 17:10:29,053 - ==> Total time: 4 days, 16:10:23 Eta: 1 day, 12:00:28 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 17:11:56,065 - Train: 75.71% [409600/541000] [757.1/1000.0] [batch_t 0.604 (1.352)] [data_t 0.002] [optim_t 0.602] [lr 0.005000] 2024-04-06 17:12:56,748 - Train: 75.73% [409700/541000] [757.3/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 17:13:57,423 - Train: 75.75% [409800/541000] [757.5/1000.0] [batch_t 0.606 (0.607)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-06 17:14:58,295 - Train: 75.77% [409900/541000] [757.7/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 17:15:59,135 - Train: 75.79% [410000/541000] [757.9/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-06 17:16:46,562 - ==> Total time: 4 days, 16:16:41 Eta: 1 day, 11:50:45 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 17:17:02,346 - Train: 75.80% [410100/541000] [758.0/1000.0] [batch_t 0.608 (0.637)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 17:18:03,157 - Train: 75.82% [410200/541000] [758.2/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 17:19:03,967 - Train: 75.84% [410300/541000] [758.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 17:20:04,760 - Train: 75.86% [410400/541000] [758.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 17:21:05,741 - Train: 75.88% [410500/541000] [758.8/1000.0] [batch_t 0.613 (0.610)] [data_t 0.003] [optim_t 0.611] [lr 0.005000] 2024-04-06 17:22:06,575 - Train: 75.90% [410600/541000] [759.0/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 17:22:18,139 - ==> Total time: 4 days, 16:22:12 Eta: 1 day, 11:40:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 17:23:09,723 - Train: 75.91% [410700/541000] [759.1/1000.0] [batch_t 0.608 (0.613)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 17:24:10,524 - Train: 75.93% [410800/541000] [759.3/1000.0] [batch_t 0.612 (0.608)] [data_t 0.003] [optim_t 0.609] [lr 0.005000] 2024-04-06 17:25:11,370 - Train: 75.95% [410900/541000] [759.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 17:26:12,339 - Train: 75.97% [411000/541000] [759.7/1000.0] [batch_t 0.608 (0.610)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 17:27:13,180 - Train: 75.99% [411100/541000] [759.9/1000.0] [batch_t 0.612 (0.608)] [data_t 0.003] [optim_t 0.610] [lr 0.005000] 2024-04-06 17:27:49,656 - ==> Total time: 4 days, 16:27:44 Eta: 1 day, 11:30:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 17:28:16,452 - Train: 76.01% [411200/541000] [760.1/1000.0] [batch_t 0.607 (0.623)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 17:29:17,279 - Train: 76.03% [411300/541000] [760.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 17:30:18,090 - Train: 76.04% [411400/541000] [760.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 17:31:18,882 - Train: 76.06% [411500/541000] [760.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 17:32:19,818 - Train: 76.08% [411600/541000] [760.8/1000.0] [batch_t 0.607 (0.609)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 17:33:20,658 - Train: 76.10% [411700/541000] [761.0/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 17:33:21,270 - ==> Total time: 4 days, 16:33:15 Eta: 1 day, 11:20:55 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 17:34:24,667 - Train: 76.12% [411800/541000] [761.2/1000.0] [batch_t 0.609 (0.618)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-06 17:35:25,495 - Train: 76.14% [411900/541000] [761.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 17:36:26,362 - Train: 76.16% [412000/541000] [761.6/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 17:37:27,287 - Train: 76.17% [412100/541000] [761.7/1000.0] [batch_t 0.610 (0.609)] [data_t 0.003] [optim_t 0.608] [lr 0.005000] 2024-04-06 17:38:28,112 - Train: 76.19% [412200/541000] [761.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 17:38:53,664 - ==> Total time: 4 days, 16:38:48 Eta: 1 day, 11:11:01 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 17:39:32,427 - Train: 76.21% [412300/541000] [762.1/1000.0] [batch_t 0.607 (0.628)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 17:40:33,206 - Train: 76.23% [412400/541000] [762.3/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 17:41:34,012 - Train: 76.25% [412500/541000] [762.5/1000.0] [batch_t 0.610 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-06 17:42:34,829 - Train: 76.27% [412600/541000] [762.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 17:43:35,814 - Train: 76.28% [412700/541000] [762.8/1000.0] [batch_t 0.608 (0.610)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 17:44:26,269 - ==> Total time: 4 days, 16:44:20 Eta: 1 day, 11:01:06 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 17:44:39,739 - Train: 76.30% [412800/541000] [763.0/1000.0] [batch_t 0.609 (0.650)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 17:45:40,579 - Train: 76.32% [412900/541000] [763.2/1000.0] [batch_t 0.613 (0.608)] [data_t 0.003] [optim_t 0.610] [lr 0.005000] 2024-04-06 17:46:41,417 - Train: 76.34% [413000/541000] [763.4/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 17:47:42,200 - Train: 76.36% [413100/541000] [763.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 17:48:43,021 - Train: 76.38% [413200/541000] [763.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 17:49:43,979 - Train: 76.40% [413300/541000] [764.0/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 17:49:58,575 - ==> Total time: 4 days, 16:49:53 Eta: 1 day, 10:51:13 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 17:50:47,059 - Train: 76.41% [413400/541000] [764.1/1000.0] [batch_t 0.608 (0.612)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 17:51:47,835 - Train: 76.43% [413500/541000] [764.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 17:52:48,616 - Train: 76.45% [413600/541000] [764.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 17:53:49,397 - Train: 76.47% [413700/541000] [764.7/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 17:54:50,212 - Train: 76.49% [413800/541000] [764.9/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.005000] 2024-04-06 17:55:29,793 - ==> Total time: 4 days, 16:55:24 Eta: 1 day, 10:41:19 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 17:55:54,750 - Train: 76.51% [413900/541000] [765.1/1000.0] [batch_t 0.607 (0.635)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 17:56:55,540 - Train: 76.52% [414000/541000] [765.2/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 17:57:56,347 - Train: 76.54% [414100/541000] [765.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 17:58:57,149 - Train: 76.56% [414200/541000] [765.6/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 17:59:57,984 - Train: 76.58% [414300/541000] [765.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 18:00:58,947 - Train: 76.60% [414400/541000] [766.0/1000.0] [batch_t 0.609 (0.610)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-06 18:01:02,596 - ==> Total time: 4 days, 17:00:57 Eta: 1 day, 10:31:27 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 18:02:03,841 - Train: 76.62% [414500/541000] [766.2/1000.0] [batch_t 0.608 (0.612)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 18:03:04,668 - Train: 76.64% [414600/541000] [766.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 18:04:05,482 - Train: 76.65% [414700/541000] [766.5/1000.0] [batch_t 0.610 (0.608)] [data_t 0.003] [optim_t 0.608] [lr 0.005000] 2024-04-06 18:05:06,338 - Train: 76.67% [414800/541000] [766.7/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 18:06:07,291 - Train: 76.69% [414900/541000] [766.9/1000.0] [batch_t 0.609 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 18:06:35,885 - ==> Total time: 4 days, 17:06:30 Eta: 1 day, 10:21:36 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 18:07:10,988 - Train: 76.71% [415000/541000] [767.1/1000.0] [batch_t 0.608 (0.617)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 18:08:11,781 - Train: 76.73% [415100/541000] [767.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 18:09:12,600 - Train: 76.75% [415200/541000] [767.5/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 18:10:13,389 - Train: 76.77% [415300/541000] [767.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 18:11:14,348 - Train: 76.78% [415400/541000] [767.8/1000.0] [batch_t 0.609 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 18:12:07,858 - ==> Total time: 4 days, 17:12:02 Eta: 1 day, 10:11:45 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 18:12:17,383 - Train: 76.80% [415500/541000] [768.0/1000.0] [batch_t 0.608 (0.643)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 18:13:18,186 - Train: 76.82% [415600/541000] [768.2/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 18:14:18,988 - Train: 76.84% [415700/541000] [768.4/1000.0] [batch_t 0.606 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 18:15:19,771 - Train: 76.86% [415800/541000] [768.6/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-06 18:16:20,557 - Train: 76.88% [415900/541000] [768.8/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-06 18:17:21,504 - Train: 76.89% [416000/541000] [768.9/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 18:17:39,147 - ==> Total time: 4 days, 17:17:33 Eta: 1 day, 10:01:55 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 18:18:26,070 - Train: 76.91% [416100/541000] [769.1/1000.0] [batch_t 0.611 (0.621)] [data_t 0.003] [optim_t 0.608] [lr 0.005000] 2024-04-06 18:19:26,878 - Train: 76.93% [416200/541000] [769.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 18:20:27,676 - Train: 76.95% [416300/541000] [769.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 18:21:28,509 - Train: 76.97% [416400/541000] [769.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 18:22:29,440 - Train: 76.99% [416500/541000] [769.9/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 18:23:11,998 - ==> Total time: 4 days, 17:23:06 Eta: 1 day, 9:52:05 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 18:23:32,522 - Train: 77.01% [416600/541000] [770.1/1000.0] [batch_t 0.608 (0.625)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 18:24:33,428 - Train: 77.02% [416700/541000] [770.2/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 18:25:34,263 - Train: 77.04% [416800/541000] [770.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 18:26:35,093 - Train: 77.06% [416900/541000] [770.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 18:27:36,036 - Train: 77.08% [417000/541000] [770.8/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 18:28:36,867 - Train: 77.10% [417100/541000] [771.0/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 18:28:43,559 - ==> Total time: 4 days, 17:28:38 Eta: 1 day, 9:42:16 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 18:29:40,449 - Train: 77.12% [417200/541000] [771.2/1000.0] [batch_t 0.608 (0.614)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 18:30:41,271 - Train: 77.13% [417300/541000] [771.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 18:31:42,091 - Train: 77.15% [417400/541000] [771.5/1000.0] [batch_t 0.614 (0.608)] [data_t 0.003] [optim_t 0.611] [lr 0.005000] 2024-04-06 18:32:43,027 - Train: 77.17% [417500/541000] [771.7/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 18:33:43,845 - Train: 77.19% [417600/541000] [771.9/1000.0] [batch_t 0.610 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-06 18:34:15,470 - ==> Total time: 4 days, 17:34:09 Eta: 1 day, 9:32:28 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 18:34:48,689 - Train: 77.21% [417700/541000] [772.1/1000.0] [batch_t 0.606 (0.632)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 18:35:49,484 - Train: 77.23% [417800/541000] [772.3/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 18:36:50,274 - Train: 77.25% [417900/541000] [772.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 18:37:51,058 - Train: 77.26% [418000/541000] [772.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 18:38:51,995 - Train: 77.28% [418100/541000] [772.8/1000.0] [batch_t 0.609 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 18:39:48,535 - ==> Total time: 4 days, 17:39:42 Eta: 1 day, 9:22:41 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 18:39:55,380 - Train: 77.30% [418200/541000] [773.0/1000.0] [batch_t 0.609 (0.660)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 18:40:56,166 - Train: 77.32% [418300/541000] [773.2/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 18:41:56,977 - Train: 77.34% [418400/541000] [773.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 18:42:57,784 - Train: 77.36% [418500/541000] [773.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 18:43:58,560 - Train: 77.38% [418600/541000] [773.8/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 18:44:59,374 - Train: 77.39% [418700/541000] [773.9/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-06 18:45:20,050 - ==> Total time: 4 days, 17:45:14 Eta: 1 day, 9:12:54 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 18:46:03,473 - Train: 77.41% [418800/541000] [774.1/1000.0] [batch_t 0.608 (0.617)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 18:47:04,300 - Train: 77.43% [418900/541000] [774.3/1000.0] [batch_t 0.614 (0.608)] [data_t 0.003] [optim_t 0.611] [lr 0.005000] 2024-04-06 18:48:05,111 - Train: 77.45% [419000/541000] [774.5/1000.0] [batch_t 0.606 (0.608)] [data_t 0.003] [optim_t 0.603] [lr 0.005000] 2024-04-06 18:49:05,964 - Train: 77.47% [419100/541000] [774.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 18:50:06,786 - Train: 77.49% [419200/541000] [774.9/1000.0] [batch_t 0.610 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-06 18:50:52,382 - ==> Total time: 4 days, 17:50:46 Eta: 1 day, 9:03:07 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 18:51:10,246 - Train: 77.50% [419300/541000] [775.0/1000.0] [batch_t 0.605 (0.628)] [data_t 0.003] [optim_t 0.602] [lr 0.005000] 2024-04-06 18:52:19,166 - Train: 77.52% [419400/541000] [775.2/1000.0] [batch_t 0.607 (0.689)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 18:53:22,580 - Train: 77.54% [419500/541000] [775.4/1000.0] [batch_t 0.608 (0.634)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 18:54:23,380 - Train: 77.56% [419600/541000] [775.6/1000.0] [batch_t 0.606 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 18:55:25,959 - Train: 77.58% [419700/541000] [775.8/1000.0] [batch_t 0.609 (0.626)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 18:56:26,756 - Train: 77.60% [419800/541000] [776.0/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 18:56:36,492 - ==> Total time: 4 days, 17:56:30 Eta: 1 day, 8:53:25 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 18:57:33,003 - Train: 77.62% [419900/541000] [776.2/1000.0] [batch_t 0.608 (0.630)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 18:58:34,268 - Train: 77.63% [420000/541000] [776.3/1000.0] [batch_t 0.608 (0.613)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 18:59:38,830 - Train: 77.65% [420100/541000] [776.5/1000.0] [batch_t 0.875 (0.646)] [data_t 0.269] [optim_t 0.606] [lr 0.005000] 2024-04-06 19:00:51,354 - Train: 77.67% [420200/541000] [776.7/1000.0] [batch_t 0.607 (0.725)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 19:02:12,233 - Train: 77.69% [420300/541000] [776.9/1000.0] [batch_t 0.608 (0.809)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 19:02:58,574 - ==> Total time: 4 days, 18:02:53 Eta: 1 day, 8:43:54 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 19:03:37,105 - Train: 77.71% [420400/541000] [777.1/1000.0] [batch_t 0.609 (0.808)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-06 19:04:41,643 - Train: 77.73% [420500/541000] [777.3/1000.0] [batch_t 0.609 (0.645)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 19:05:44,565 - Train: 77.74% [420600/541000] [777.4/1000.0] [batch_t 0.607 (0.629)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 19:06:50,969 - Train: 77.76% [420700/541000] [777.6/1000.0] [batch_t 0.607 (0.664)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 19:07:57,634 - Train: 77.78% [420800/541000] [777.8/1000.0] [batch_t 0.609 (0.667)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 19:09:06,360 - ==> Total time: 4 days, 18:09:00 Eta: 1 day, 8:34:20 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 19:09:10,419 - Train: 77.80% [420900/541000] [778.0/1000.0] [batch_t 0.608 (0.910)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 19:10:11,245 - Train: 77.82% [421000/541000] [778.2/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 19:11:12,068 - Train: 77.84% [421100/541000] [778.4/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 19:12:12,859 - Train: 77.86% [421200/541000] [778.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 19:13:13,651 - Train: 77.87% [421300/541000] [778.7/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 19:14:14,454 - Train: 77.89% [421400/541000] [778.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 19:14:38,167 - ==> Total time: 4 days, 18:14:32 Eta: 1 day, 8:24:36 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 19:15:18,223 - Train: 77.91% [421500/541000] [779.1/1000.0] [batch_t 0.607 (0.620)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 19:16:19,169 - Train: 77.93% [421600/541000] [779.3/1000.0] [batch_t 0.609 (0.609)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-06 19:17:19,986 - Train: 77.95% [421700/541000] [779.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 19:18:20,810 - Train: 77.97% [421800/541000] [779.7/1000.0] [batch_t 0.610 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-06 19:19:21,642 - Train: 77.99% [421900/541000] [779.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 19:20:10,288 - ==> Total time: 4 days, 18:20:04 Eta: 1 day, 8:14:53 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 19:20:26,354 - Train: 78.00% [422000/541000] [780.0/1000.0] [batch_t 0.608 (0.658)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 19:21:27,298 - Train: 78.02% [422100/541000] [780.2/1000.0] [batch_t 0.607 (0.609)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 19:22:28,111 - Train: 78.04% [422200/541000] [780.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 19:23:28,911 - Train: 78.06% [422300/541000] [780.6/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 19:24:29,732 - Train: 78.08% [422400/541000] [780.8/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 19:25:30,556 - Train: 78.10% [422500/541000] [781.0/1000.0] [batch_t 0.606 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 19:25:43,339 - ==> Total time: 4 days, 18:25:37 Eta: 1 day, 8:05:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 19:26:34,848 - Train: 78.11% [422600/541000] [781.1/1000.0] [batch_t 0.607 (0.619)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-06 19:27:35,803 - Train: 78.13% [422700/541000] [781.3/1000.0] [batch_t 0.607 (0.609)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 19:28:36,621 - Train: 78.15% [422800/541000] [781.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 19:29:37,452 - Train: 78.17% [422900/541000] [781.7/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 19:30:38,261 - Train: 78.19% [423000/541000] [781.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 19:31:15,968 - ==> Total time: 4 days, 18:31:10 Eta: 1 day, 7:55:29 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 19:31:41,822 - Train: 78.21% [423100/541000] [782.1/1000.0] [batch_t 0.611 (0.626)] [data_t 0.003] [optim_t 0.609] [lr 0.005000] 2024-04-06 19:32:42,648 - Train: 78.23% [423200/541000] [782.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 19:33:43,616 - Train: 78.24% [423300/541000] [782.4/1000.0] [batch_t 0.608 (0.610)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 19:34:44,466 - Train: 78.26% [423400/541000] [782.6/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-06 19:35:45,310 - Train: 78.28% [423500/541000] [782.8/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 19:36:46,182 - Train: 78.30% [423600/541000] [783.0/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 19:36:48,011 - ==> Total time: 4 days, 18:36:42 Eta: 1 day, 7:45:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 19:37:50,931 - Train: 78.32% [423700/541000] [783.2/1000.0] [batch_t 0.608 (0.619)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 19:38:51,732 - Train: 78.34% [423800/541000] [783.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 19:39:52,708 - Train: 78.35% [423900/541000] [783.5/1000.0] [batch_t 0.610 (0.610)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-06 19:40:53,514 - Train: 78.37% [424000/541000] [783.7/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 19:41:54,325 - Train: 78.39% [424100/541000] [783.9/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 19:42:21,084 - ==> Total time: 4 days, 18:42:15 Eta: 1 day, 7:36:07 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 19:42:58,725 - Train: 78.41% [424200/541000] [784.1/1000.0] [batch_t 0.612 (0.627)] [data_t 0.003] [optim_t 0.609] [lr 0.005000] 2024-04-06 19:43:59,582 - Train: 78.43% [424300/541000] [784.3/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 19:45:00,392 - Train: 78.45% [424400/541000] [784.5/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 19:46:01,343 - Train: 78.47% [424500/541000] [784.7/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 19:47:02,155 - Train: 78.48% [424600/541000] [784.8/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 19:47:53,845 - ==> Total time: 4 days, 18:47:48 Eta: 1 day, 7:26:28 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 19:48:05,159 - Train: 78.50% [424700/541000] [785.0/1000.0] [batch_t 0.609 (0.634)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 19:49:06,000 - Train: 78.52% [424800/541000] [785.2/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 19:50:06,808 - Train: 78.54% [424900/541000] [785.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 19:51:07,758 - Train: 78.56% [425000/541000] [785.6/1000.0] [batch_t 0.609 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 19:52:08,542 - Train: 78.58% [425100/541000] [785.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 19:53:09,385 - Train: 78.60% [425200/541000] [786.0/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 19:53:25,190 - ==> Total time: 4 days, 18:53:19 Eta: 1 day, 7:16:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 19:54:12,372 - Train: 78.61% [425300/541000] [786.1/1000.0] [batch_t 0.607 (0.614)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 19:55:13,212 - Train: 78.63% [425400/541000] [786.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 19:56:14,064 - Train: 78.65% [425500/541000] [786.5/1000.0] [batch_t 0.610 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.005000] 2024-04-06 19:57:15,023 - Train: 78.67% [425600/541000] [786.7/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 19:58:15,815 - Train: 78.69% [425700/541000] [786.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 19:58:56,595 - ==> Total time: 4 days, 18:58:51 Eta: 1 day, 7:07:09 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 19:59:18,925 - Train: 78.71% [425800/541000] [787.1/1000.0] [batch_t 0.607 (0.621)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 20:00:19,757 - Train: 78.72% [425900/541000] [787.2/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 20:01:20,572 - Train: 78.74% [426000/541000] [787.4/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 20:02:21,387 - Train: 78.76% [426100/541000] [787.6/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 20:03:22,340 - Train: 78.78% [426200/541000] [787.8/1000.0] [batch_t 0.606 (0.609)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 20:04:23,141 - Train: 78.80% [426300/541000] [788.0/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 20:04:28,019 - ==> Total time: 4 days, 19:04:22 Eta: 1 day, 6:57:31 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 20:05:26,244 - Train: 78.82% [426400/541000] [788.2/1000.0] [batch_t 0.608 (0.611)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 20:06:27,020 - Train: 78.84% [426500/541000] [788.4/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 20:07:27,831 - Train: 78.85% [426600/541000] [788.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 20:08:28,746 - Train: 78.87% [426700/541000] [788.7/1000.0] [batch_t 0.609 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 20:09:29,555 - Train: 78.89% [426800/541000] [788.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 20:09:59,328 - ==> Total time: 4 days, 19:09:53 Eta: 1 day, 6:47:53 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 20:10:34,132 - Train: 78.91% [426900/541000] [789.1/1000.0] [batch_t 0.609 (0.646)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 20:11:34,955 - Train: 78.93% [427000/541000] [789.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 20:12:35,691 - Train: 78.95% [427100/541000] [789.5/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 20:13:36,369 - Train: 78.96% [427200/541000] [789.6/1000.0] [batch_t 0.607 (0.607)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 20:14:37,235 - Train: 78.98% [427300/541000] [789.8/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 20:15:31,968 - ==> Total time: 4 days, 19:15:26 Eta: 1 day, 6:38:16 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 20:15:40,092 - Train: 79.00% [427400/541000] [790.0/1000.0] [batch_t 0.609 (0.644)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 20:16:40,910 - Train: 79.02% [427500/541000] [790.2/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 20:17:41,703 - Train: 79.04% [427600/541000] [790.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 20:18:42,505 - Train: 79.06% [427700/541000] [790.6/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 20:19:43,485 - Train: 79.08% [427800/541000] [790.8/1000.0] [batch_t 0.609 (0.610)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 20:20:44,305 - Train: 79.09% [427900/541000] [790.9/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 20:21:03,168 - ==> Total time: 4 days, 19:20:57 Eta: 1 day, 6:28:40 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 20:21:48,554 - Train: 79.11% [428000/541000] [791.1/1000.0] [batch_t 0.608 (0.620)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 20:22:49,379 - Train: 79.13% [428100/541000] [791.3/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 20:23:50,210 - Train: 79.15% [428200/541000] [791.5/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 20:24:51,159 - Train: 79.17% [428300/541000] [791.7/1000.0] [batch_t 0.607 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 20:25:51,983 - Train: 79.19% [428400/541000] [791.9/1000.0] [batch_t 0.611 (0.608)] [data_t 0.003] [optim_t 0.608] [lr 0.005000] 2024-04-06 20:26:35,778 - ==> Total time: 4 days, 19:26:30 Eta: 1 day, 6:19:04 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 20:26:55,197 - Train: 79.21% [428500/541000] [792.1/1000.0] [batch_t 0.608 (0.621)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 20:27:55,984 - Train: 79.22% [428600/541000] [792.2/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 20:28:56,851 - Train: 79.24% [428700/541000] [792.4/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 20:29:57,635 - Train: 79.26% [428800/541000] [792.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 20:30:58,572 - Train: 79.28% [428900/541000] [792.8/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 20:31:59,406 - Train: 79.30% [429000/541000] [793.0/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 20:32:07,322 - ==> Total time: 4 days, 19:32:01 Eta: 1 day, 6:09:29 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 20:33:02,793 - Train: 79.32% [429100/541000] [793.2/1000.0] [batch_t 0.607 (0.612)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 20:34:03,596 - Train: 79.33% [429200/541000] [793.3/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-06 20:35:04,422 - Train: 79.35% [429300/541000] [793.5/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 20:36:05,223 - Train: 79.37% [429400/541000] [793.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 20:37:06,009 - Train: 79.39% [429500/541000] [793.9/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 20:37:38,931 - ==> Total time: 4 days, 19:37:33 Eta: 1 day, 5:59:55 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 20:38:09,382 - Train: 79.41% [429600/541000] [794.1/1000.0] [batch_t 0.608 (0.616)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 20:39:10,191 - Train: 79.43% [429700/541000] [794.3/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 20:40:11,021 - Train: 79.45% [429800/541000] [794.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 20:41:11,828 - Train: 79.46% [429900/541000] [794.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 20:42:12,609 - Train: 79.48% [430000/541000] [794.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 20:43:10,565 - ==> Total time: 4 days, 19:43:05 Eta: 1 day, 5:50:21 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 20:43:15,758 - Train: 79.50% [430100/541000] [795.0/1000.0] [batch_t 0.608 (0.675)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 20:44:16,552 - Train: 79.52% [430200/541000] [795.2/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 20:45:17,379 - Train: 79.54% [430300/541000] [795.4/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 20:46:18,196 - Train: 79.56% [430400/541000] [795.6/1000.0] [batch_t 0.612 (0.608)] [data_t 0.003] [optim_t 0.610] [lr 0.005000] 2024-04-06 20:47:19,028 - Train: 79.57% [430500/541000] [795.7/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 20:48:20,007 - Train: 79.59% [430600/541000] [795.9/1000.0] [batch_t 0.608 (0.610)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 20:48:41,892 - ==> Total time: 4 days, 19:48:36 Eta: 1 day, 5:40:47 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 20:49:22,999 - Train: 79.61% [430700/541000] [796.1/1000.0] [batch_t 0.609 (0.613)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 20:50:23,787 - Train: 79.63% [430800/541000] [796.3/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-06 20:51:24,583 - Train: 79.65% [430900/541000] [796.5/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 20:52:25,344 - Train: 79.67% [431000/541000] [796.7/1000.0] [batch_t 0.607 (0.607)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 20:53:26,327 - Train: 79.69% [431100/541000] [796.9/1000.0] [batch_t 0.606 (0.610)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 20:54:13,186 - ==> Total time: 4 days, 19:54:07 Eta: 1 day, 5:31:15 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 20:54:29,347 - Train: 79.70% [431200/541000] [797.0/1000.0] [batch_t 0.607 (0.624)] [data_t 0.003] [optim_t 0.604] [lr 0.005000] 2024-04-06 20:55:30,138 - Train: 79.72% [431300/541000] [797.2/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 20:56:30,914 - Train: 79.74% [431400/541000] [797.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 20:57:31,754 - Train: 79.76% [431500/541000] [797.6/1000.0] [batch_t 0.605 (0.608)] [data_t 0.002] [optim_t 0.603] [lr 0.005000] 2024-04-06 20:58:32,722 - Train: 79.78% [431600/541000] [797.8/1000.0] [batch_t 0.609 (0.610)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 20:59:33,510 - Train: 79.80% [431700/541000] [798.0/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 20:59:44,458 - ==> Total time: 4 days, 19:59:38 Eta: 1 day, 5:21:42 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 21:00:37,129 - Train: 79.82% [431800/541000] [798.2/1000.0] [batch_t 0.609 (0.617)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 21:01:37,934 - Train: 79.83% [431900/541000] [798.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 21:02:38,792 - Train: 79.85% [432000/541000] [798.5/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-06 21:03:39,620 - Train: 79.87% [432100/541000] [798.7/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-06 21:04:40,526 - Train: 79.89% [432200/541000] [798.9/1000.0] [batch_t 0.609 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.005000] 2024-04-06 21:05:16,394 - ==> Total time: 4 days, 20:05:10 Eta: 1 day, 5:12:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 21:05:44,006 - Train: 79.91% [432300/541000] [799.1/1000.0] [batch_t 0.607 (0.620)] [data_t 0.002] [optim_t 0.604] [lr 0.005000] 2024-04-06 21:06:44,799 - Train: 79.93% [432400/541000] [799.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.005000] 2024-04-06 21:07:45,681 - Train: 79.94% [432500/541000] [799.4/1000.0] [batch_t 0.611 (0.609)] [data_t 0.003] [optim_t 0.609] [lr 0.005000] 2024-04-06 21:08:46,565 - Train: 79.96% [432600/541000] [799.6/1000.0] [batch_t 0.612 (0.609)] [data_t 0.003] [optim_t 0.609] [lr 0.005000] 2024-04-06 21:09:47,584 - Train: 79.98% [432700/541000] [799.8/1000.0] [batch_t 0.608 (0.610)] [data_t 0.003] [optim_t 0.606] [lr 0.005000] 2024-04-06 21:10:48,407 - Train: 80.00% [432800/541000] [800.0/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.005000] 2024-04-06 21:11:00,360 - Test: 36.76% [50/136] [batch_t 0.215 (0.228)] 2024-04-06 21:11:12,105 - Test: 73.53% [100/136] [batch_t 0.208 (0.231)] 2024-04-06 21:11:19,621 - Test: 100.00% [136/136] [batch_t 0.054 (0.225)] 2024-04-06 21:12:07,707 - ==> Metric Time for candle : 0.002 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.590 (mAUROC_px) 8.025 (mAP_px) 0.920 (mF1_max_px) 24.907 (mAUPRO_px) 0.316 (mF1_px_0.2_0.8_0.1) 0.312 (mAcc_px_0.2_0.8_0.1) 0.315 (mIoU_px_0.2_0.8_0.1) 0.911 (mIoU_max_px) 2024-04-06 21:12:44,856 - ==> Metric Time for capsules : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 8.194 (mAUROC_px) 6.285 (mAP_px) 0.729 (mF1_max_px) 20.240 (mAUPRO_px) 0.250 (mF1_px_0.2_0.8_0.1) 0.250 (mAcc_px_0.2_0.8_0.1) 0.246 (mIoU_px_0.2_0.8_0.1) 0.728 (mIoU_max_px) 2024-04-06 21:13:21,313 - ==> Metric Time for cashew : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.822 (mAUROC_px) 6.312 (mAP_px) 0.734 (mF1_max_px) 19.984 (mAUPRO_px) 0.236 (mF1_px_0.2_0.8_0.1) 0.233 (mAcc_px_0.2_0.8_0.1) 0.238 (mIoU_px_0.2_0.8_0.1) 0.687 (mIoU_max_px) 2024-04-06 21:13:57,718 - ==> Metric Time for chewinggum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.789 (mAUROC_px) 6.022 (mAP_px) 0.685 (mF1_max_px) 20.243 (mAUPRO_px) 0.234 (mF1_px_0.2_0.8_0.1) 0.232 (mAcc_px_0.2_0.8_0.1) 0.243 (mIoU_px_0.2_0.8_0.1) 0.747 (mIoU_max_px) 2024-04-06 21:14:34,200 - ==> Metric Time for fryum : 0.002 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.772 (mAUROC_px) 5.680 (mAP_px) 0.692 (mF1_max_px) 20.723 (mAUPRO_px) 0.236 (mF1_px_0.2_0.8_0.1) 0.234 (mAcc_px_0.2_0.8_0.1) 0.237 (mIoU_px_0.2_0.8_0.1) 0.686 (mIoU_max_px) 2024-04-06 21:15:20,830 - ==> Metric Time for macaroni1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.478 (mAUROC_px) 7.870 (mAP_px) 0.942 (mF1_max_px) 24.745 (mAUPRO_px) 0.388 (mF1_px_0.2_0.8_0.1) 0.396 (mAcc_px_0.2_0.8_0.1) 0.400 (mIoU_px_0.2_0.8_0.1) 1.139 (mIoU_max_px) 2024-04-06 21:16:08,354 - ==> Metric Time for macaroni2 : 0.002 (mAUROC_sp_max) 0.001 (mAP_sp_max) 0.001 (mF1_max_sp_max) 11.377 (mAUROC_px) 7.958 (mAP_px) 0.924 (mF1_max_px) 25.059 (mAUPRO_px) 0.317 (mF1_px_0.2_0.8_0.1) 0.317 (mAcc_px_0.2_0.8_0.1) 0.318 (mIoU_px_0.2_0.8_0.1) 0.941 (mIoU_max_px) 2024-04-06 21:16:57,374 - ==> Metric Time for pcb1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.877 (mAUROC_px) 8.150 (mAP_px) 0.944 (mF1_max_px) 26.885 (mAUPRO_px) 0.314 (mF1_px_0.2_0.8_0.1) 0.316 (mAcc_px_0.2_0.8_0.1) 0.316 (mIoU_px_0.2_0.8_0.1) 0.922 (mIoU_max_px) 2024-04-06 21:17:45,274 - ==> Metric Time for pcb2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.708 (mAUROC_px) 8.163 (mAP_px) 0.928 (mF1_max_px) 25.913 (mAUPRO_px) 0.316 (mF1_px_0.2_0.8_0.1) 0.316 (mAcc_px_0.2_0.8_0.1) 0.329 (mIoU_px_0.2_0.8_0.1) 0.947 (mIoU_max_px) 2024-04-06 21:18:32,562 - ==> Metric Time for pcb3 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.526 (mAUROC_px) 7.946 (mAP_px) 0.964 (mF1_max_px) 25.684 (mAUPRO_px) 0.317 (mF1_px_0.2_0.8_0.1) 0.317 (mAcc_px_0.2_0.8_0.1) 0.317 (mIoU_px_0.2_0.8_0.1) 0.933 (mIoU_max_px) 2024-04-06 21:19:22,160 - ==> Metric Time for pcb4 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.672 (mAUROC_px) 8.105 (mAP_px) 0.935 (mF1_max_px) 27.709 (mAUPRO_px) 0.319 (mF1_px_0.2_0.8_0.1) 0.319 (mAcc_px_0.2_0.8_0.1) 0.321 (mIoU_px_0.2_0.8_0.1) 0.936 (mIoU_max_px) 2024-04-06 21:19:58,767 - ==> Metric Time for pipe_fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.867 (mAUROC_px) 5.833 (mAP_px) 0.691 (mF1_max_px) 20.553 (mAUPRO_px) 0.235 (mF1_px_0.2_0.8_0.1) 0.238 (mAcc_px_0.2_0.8_0.1) 0.276 (mIoU_px_0.2_0.8_0.1) 0.697 (mIoU_max_px) 2024-04-06 21:19:58,789 - | Name | mAUROC_sp_max | mAUROC_sp_max (Max) | mAP_sp_max | mAP_sp_max (Max) | mF1_max_sp_max | mF1_max_sp_max (Max) | mAUROC_px | mAUROC_px (Max) | mAP_px | mAP_px (Max) | mF1_max_px | mF1_max_px (Max) | mAUPRO_px | mAUPRO_px (Max) | mF1_px_0.2_0.8_0.1 | mF1_px_0.2_0.8_0.1 (Max) | mAcc_px_0.2_0.8_0.1 | mAcc_px_0.2_0.8_0.1 (Max) | mIoU_px_0.2_0.8_0.1 | mIoU_px_0.2_0.8_0.1 (Max) | mIoU_max_px | mIoU_max_px (Max) | |:----------:|:---------------:|:---------------------:|:------------:|:------------------:|:----------------:|:----------------------:|:-----------:|:------------------:|:--------:|:------------------:|:------------:|:------------------:|:-----------:|:------------------:|:--------------------:|:--------------------------:|:---------------------:|:---------------------------:|:---------------------:|:---------------------------:|:-------------:|:-------------------:| | candle | 95.910 | 97.100 (50 epoch) | 96.168 | 97.346 (50 epoch) | 89.655 | 91.542 (50 epoch) | 99.014 | 99.098 (250 epoch) | 20.576 | 23.463 (150 epoch) | 30.778 | 32.654 (200 epoch) | 95.241 | 96.087 (300 epoch) | 19.989 | 21.331 (500 epoch) | 39.108 | 41.526 (650 epoch) | 11.361 | 12.185 (500 epoch) | 18.188 | 19.513 (200 epoch) | | capsules | 87.367 | 91.750 (100 epoch) | 92.808 | 94.964 (100 epoch) | 85.714 | 88.780 (100 epoch) | 98.825 | 99.193 (700 epoch) | 53.702 | 61.256 (100 epoch) | 57.397 | 59.868 (250 epoch) | 87.454 | 91.811 (100 epoch) | 25.745 | 33.186 (100 epoch) | 49.789 | 62.576 (50 epoch) | 16.436 | 21.701 (100 epoch) | 40.249 | 42.722 (250 epoch) | | cashew | 94.940 | 94.940 (800 epoch) | 97.488 | 97.492 (150 epoch) | 92.708 | 92.929 (700 epoch) | 95.783 | 96.185 (700 epoch) | 35.298 | 50.952 (250 epoch) | 37.468 | 54.438 (400 epoch) | 80.054 | 89.185 (350 epoch) | 21.596 | 28.352 (350 epoch) | 31.167 | 34.522 (150 epoch) | 12.623 | 17.797 (350 epoch) | 23.052 | 37.398 (400 epoch) | | chewinggum | 96.919 | 97.802 (300 epoch) | 98.532 | 98.939 (300 epoch) | 94.359 | 94.792 (450 epoch) | 97.925 | 98.145 (100 epoch) | 39.764 | 59.823 (50 epoch) | 45.192 | 61.088 (50 epoch) | 79.689 | 80.312 (400 epoch) | 26.912 | 35.748 (50 epoch) | 65.723 | 65.723 (800 epoch) | 16.414 | 23.754 (50 epoch) | 29.192 | 43.976 (50 epoch) | | fryum | 96.040 | 96.560 (650 epoch) | 98.064 | 98.273 (650 epoch) | 92.462 | 93.897 (650 epoch) | 97.227 | 97.308 (750 epoch) | 50.688 | 52.248 (750 epoch) | 52.828 | 53.189 (600 epoch) | 92.069 | 92.340 (550 epoch) | 25.901 | 26.511 (550 epoch) | 27.047 | 28.594 (450 epoch) | 16.145 | 16.529 (550 epoch) | 35.896 | 36.230 (600 epoch) | | macaroni1 | 90.606 | 92.104 (500 epoch) | 86.400 | 90.505 (500 epoch) | 81.553 | 84.314 (750 epoch) | 99.180 | 99.604 (650 epoch) | 8.527 | 20.435 (700 epoch) | 15.575 | 29.513 (650 epoch) | 93.874 | 96.221 (650 epoch) | 8.441 | 16.200 (700 epoch) | 36.779 | 40.765 (50 epoch) | 4.464 | 9.078 (700 epoch) | 8.445 | 17.311 (650 epoch) | | macaroni2 | 76.298 | 82.589 (50 epoch) | 66.073 | 81.196 (50 epoch) | 75.455 | 77.833 (750 epoch) | 96.006 | 99.452 (100 epoch) | 0.917 | 11.272 (50 epoch) | 3.669 | 17.568 (50 epoch) | 83.495 | 96.253 (150 epoch) | 1.425 | 10.086 (50 epoch) | 50.494 | 54.584 (450 epoch) | 0.720 | 5.433 (50 epoch) | 1.869 | 9.630 (50 epoch) | | pcb1 | 96.720 | 96.800 (700 epoch) | 95.420 | 95.568 (700 epoch) | 93.839 | 94.737 (300 epoch) | 99.801 | 99.806 (700 epoch) | 81.354 | 83.141 (450 epoch) | 73.798 | 75.588 (450 epoch) | 94.511 | 94.861 (600 epoch) | 46.757 | 48.256 (700 epoch) | 54.934 | 62.362 (50 epoch) | 32.713 | 34.013 (700 epoch) | 58.477 | 60.756 (450 epoch) | | pcb2 | 95.970 | 95.970 (800 epoch) | 95.773 | 95.773 (800 epoch) | 91.542 | 91.542 (800 epoch) | 99.008 | 99.023 (700 epoch) | 12.626 | 14.130 (50 epoch) | 23.803 | 23.977 (250 epoch) | 90.704 | 91.576 (700 epoch) | 15.441 | 16.150 (200 epoch) | 43.071 | 47.820 (150 epoch) | 8.498 | 8.892 (200 epoch) | 13.509 | 13.621 (250 epoch) | | pcb3 | 95.396 | 95.891 (350 epoch) | 95.586 | 96.056 (350 epoch) | 89.552 | 89.855 (400 epoch) | 99.170 | 99.174 (750 epoch) | 17.675 | 20.249 (150 epoch) | 27.577 | 28.078 (450 epoch) | 91.927 | 92.048 (650 epoch) | 19.271 | 20.042 (150 epoch) | 43.123 | 48.341 (50 epoch) | 10.843 | 11.295 (150 epoch) | 15.994 | 16.332 (450 epoch) | | pcb4 | 99.752 | 99.901 (100 epoch) | 99.752 | 99.901 (100 epoch) | 97.561 | 99.000 (50 epoch) | 98.692 | 98.746 (650 epoch) | 46.475 | 47.966 (250 epoch) | 46.912 | 47.709 (250 epoch) | 88.470 | 90.876 (500 epoch) | 22.808 | 26.925 (200 epoch) | 29.135 | 39.716 (50 epoch) | 14.013 | 16.537 (200 epoch) | 30.644 | 31.328 (250 epoch) | | pipe_fryum | 98.900 | 99.000 (650 epoch) | 99.430 | 99.489 (650 epoch) | 97.030 | 97.561 (200 epoch) | 98.929 | 99.114 (450 epoch) | 53.660 | 56.529 (700 epoch) | 53.769 | 58.733 (450 epoch) | 93.159 | 95.525 (150 epoch) | 31.498 | 34.684 (250 epoch) | 36.214 | 43.014 (100 epoch) | 20.228 | 22.399 (250 epoch) | 36.770 | 41.576 (450 epoch) | | Avg | 93.735 | 94.251 (100 epoch) | 93.458 | 94.481 (100 epoch) | 90.119 | 90.468 (700 epoch) | 98.297 | 98.640 (700 epoch) | 35.105 | 39.701 (250 epoch) | 39.064 | 44.125 (250 epoch) | 89.221 | 91.494 (250 epoch) | 22.149 | 25.164 (100 epoch) | 42.215 | 46.179 (50 epoch) | 13.705 | 15.755 (100 epoch) | 26.024 | 29.990 (250 epoch) | 2024-04-06 21:19:59,109 - ==> Total time: 4 days, 20:19:53 Eta: 1 day, 5:04:58 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 21:22:27,289 - Train: 80.02% [432900/541000] [800.2/1000.0] [batch_t 1.457 (1.460)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-06 21:24:52,902 - Train: 80.04% [433000/541000] [800.4/1000.0] [batch_t 1.460 (1.456)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-06 21:27:18,566 - Train: 80.06% [433100/541000] [800.6/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-06 21:29:44,258 - Train: 80.07% [433200/541000] [800.7/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-06 21:32:09,905 - Train: 80.09% [433300/541000] [800.9/1000.0] [batch_t 1.454 (1.456)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-06 21:33:09,656 - ==> Total time: 4 days, 20:33:04 Eta: 1 day, 4:57:21 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 21:34:37,864 - Train: 80.11% [433400/541000] [801.1/1000.0] [batch_t 1.455 (1.464)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-06 21:37:03,483 - Train: 80.13% [433500/541000] [801.3/1000.0] [batch_t 1.456 (1.456)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-06 21:39:29,132 - Train: 80.15% [433600/541000] [801.5/1000.0] [batch_t 1.456 (1.456)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-06 21:41:54,826 - Train: 80.17% [433700/541000] [801.7/1000.0] [batch_t 1.460 (1.457)] [data_t 0.004] [optim_t 1.456] [lr 0.000500] 2024-04-06 21:44:20,410 - Train: 80.18% [433800/541000] [801.8/1000.0] [batch_t 1.460 (1.456)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-06 21:46:19,818 - ==> Total time: 4 days, 20:46:14 Eta: 1 day, 4:49:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 21:46:48,415 - Train: 80.20% [433900/541000] [802.0/1000.0] [batch_t 1.454 (1.479)] [data_t 0.004] [optim_t 1.451] [lr 0.000500] 2024-04-06 21:49:14,061 - Train: 80.22% [434000/541000] [802.2/1000.0] [batch_t 1.453 (1.456)] [data_t 0.003] [optim_t 1.449] [lr 0.000500] 2024-04-06 21:51:39,643 - Train: 80.24% [434100/541000] [802.4/1000.0] [batch_t 1.455 (1.456)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-06 21:54:05,213 - Train: 80.26% [434200/541000] [802.6/1000.0] [batch_t 1.460 (1.456)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-06 21:56:30,933 - Train: 80.28% [434300/541000] [802.8/1000.0] [batch_t 1.453 (1.457)] [data_t 0.003] [optim_t 1.450] [lr 0.000500] 2024-04-06 21:58:56,647 - Train: 80.30% [434400/541000] [803.0/1000.0] [batch_t 1.461 (1.457)] [data_t 0.004] [optim_t 1.457] [lr 0.000500] 2024-04-06 21:59:30,125 - ==> Total time: 4 days, 20:59:24 Eta: 1 day, 4:42:04 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 22:01:24,591 - Train: 80.31% [434500/541000] [803.1/1000.0] [batch_t 1.460 (1.463)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-06 22:03:50,247 - Train: 80.33% [434600/541000] [803.3/1000.0] [batch_t 1.454 (1.456)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-06 22:06:15,918 - Train: 80.35% [434700/541000] [803.5/1000.0] [batch_t 1.457 (1.457)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-06 22:08:41,479 - Train: 80.37% [434800/541000] [803.7/1000.0] [batch_t 1.456 (1.456)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-06 22:11:07,003 - Train: 80.39% [434900/541000] [803.9/1000.0] [batch_t 1.460 (1.455)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-06 22:12:40,179 - ==> Total time: 4 days, 21:12:34 Eta: 1 day, 4:34:24 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 22:13:34,737 - Train: 80.41% [435000/541000] [804.1/1000.0] [batch_t 1.460 (1.465)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-06 22:16:00,407 - Train: 80.43% [435100/541000] [804.3/1000.0] [batch_t 1.457 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-06 22:18:26,048 - Train: 80.44% [435200/541000] [804.4/1000.0] [batch_t 1.455 (1.456)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-06 22:20:51,688 - Train: 80.46% [435300/541000] [804.6/1000.0] [batch_t 1.454 (1.456)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-06 22:23:17,338 - Train: 80.48% [435400/541000] [804.8/1000.0] [batch_t 1.461 (1.456)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-06 22:25:42,935 - Train: 80.50% [435500/541000] [805.0/1000.0] [batch_t 1.456 (1.456)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-06 22:25:50,216 - ==> Total time: 4 days, 21:25:44 Eta: 1 day, 4:26:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 22:28:10,848 - Train: 80.52% [435600/541000] [805.2/1000.0] [batch_t 1.455 (1.461)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-06 22:30:36,551 - Train: 80.54% [435700/541000] [805.4/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-06 22:33:02,210 - Train: 80.55% [435800/541000] [805.5/1000.0] [batch_t 1.458 (1.456)] [data_t 0.003] [optim_t 1.455] [lr 0.000500] 2024-04-06 22:35:27,851 - Train: 80.57% [435900/541000] [805.7/1000.0] [batch_t 1.458 (1.456)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-06 22:37:53,500 - Train: 80.59% [436000/541000] [805.9/1000.0] [batch_t 1.461 (1.456)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-06 22:39:00,462 - ==> Total time: 4 days, 21:38:54 Eta: 1 day, 4:19:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 22:40:22,417 - Train: 80.61% [436100/541000] [806.1/1000.0] [batch_t 1.456 (1.485)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-06 22:42:48,027 - Train: 80.63% [436200/541000] [806.3/1000.0] [batch_t 1.461 (1.456)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-06 22:45:13,677 - Train: 80.65% [436300/541000] [806.5/1000.0] [batch_t 1.455 (1.456)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-06 22:47:39,245 - Train: 80.67% [436400/541000] [806.7/1000.0] [batch_t 1.459 (1.456)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-06 22:50:04,850 - Train: 80.68% [436500/541000] [806.8/1000.0] [batch_t 1.461 (1.456)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-06 22:52:11,489 - ==> Total time: 4 days, 21:52:05 Eta: 1 day, 4:11:20 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 22:52:32,684 - Train: 80.70% [436600/541000] [807.0/1000.0] [batch_t 1.453 (1.488)] [data_t 0.003] [optim_t 1.450] [lr 0.000500] 2024-04-06 22:54:58,341 - Train: 80.72% [436700/541000] [807.2/1000.0] [batch_t 1.454 (1.456)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-06 22:57:24,063 - Train: 80.74% [436800/541000] [807.4/1000.0] [batch_t 1.454 (1.457)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-06 22:59:49,727 - Train: 80.76% [436900/541000] [807.6/1000.0] [batch_t 1.451 (1.457)] [data_t 0.003] [optim_t 1.448] [lr 0.000500] 2024-04-06 23:02:15,393 - Train: 80.78% [437000/541000] [807.8/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-06 23:04:41,084 - Train: 80.79% [437100/541000] [807.9/1000.0] [batch_t 1.458 (1.457)] [data_t 0.003] [optim_t 1.455] [lr 0.000500] 2024-04-06 23:05:21,868 - ==> Total time: 4 days, 22:05:16 Eta: 1 day, 4:03:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 23:07:09,013 - Train: 80.81% [437200/541000] [808.1/1000.0] [batch_t 1.451 (1.463)] [data_t 0.003] [optim_t 1.448] [lr 0.000500] 2024-04-06 23:09:34,605 - Train: 80.83% [437300/541000] [808.3/1000.0] [batch_t 1.451 (1.456)] [data_t 0.003] [optim_t 1.448] [lr 0.000500] 2024-04-06 23:12:00,181 - Train: 80.85% [437400/541000] [808.5/1000.0] [batch_t 1.446 (1.456)] [data_t 0.003] [optim_t 1.442] [lr 0.000500] 2024-04-06 23:14:25,814 - Train: 80.87% [437500/541000] [808.7/1000.0] [batch_t 1.455 (1.456)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-06 23:16:51,470 - Train: 80.89% [437600/541000] [808.9/1000.0] [batch_t 1.456 (1.456)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-06 23:18:31,967 - ==> Total time: 4 days, 22:18:26 Eta: 1 day, 3:55:53 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 23:19:19,302 - Train: 80.91% [437700/541000] [809.1/1000.0] [batch_t 1.458 (1.469)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-06 23:21:44,912 - Train: 80.92% [437800/541000] [809.2/1000.0] [batch_t 1.456 (1.456)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-06 23:24:10,533 - Train: 80.94% [437900/541000] [809.4/1000.0] [batch_t 1.456 (1.456)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-06 23:26:36,195 - Train: 80.96% [438000/541000] [809.6/1000.0] [batch_t 1.453 (1.457)] [data_t 0.003] [optim_t 1.449] [lr 0.000500] 2024-04-06 23:29:01,860 - Train: 80.98% [438100/541000] [809.8/1000.0] [batch_t 1.455 (1.457)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-06 23:31:27,570 - Train: 81.00% [438200/541000] [810.0/1000.0] [batch_t 1.454 (1.457)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-06 23:31:42,129 - ==> Total time: 4 days, 22:31:36 Eta: 1 day, 3:48:09 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 23:33:55,215 - Train: 81.02% [438300/541000] [810.2/1000.0] [batch_t 1.459 (1.459)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-06 23:36:20,839 - Train: 81.04% [438400/541000] [810.4/1000.0] [batch_t 1.458 (1.456)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-06 23:38:46,480 - Train: 81.05% [438500/541000] [810.5/1000.0] [batch_t 1.454 (1.456)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-06 23:41:12,074 - Train: 81.07% [438600/541000] [810.7/1000.0] [batch_t 1.455 (1.456)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-06 23:43:37,684 - Train: 81.09% [438700/541000] [810.9/1000.0] [batch_t 1.457 (1.456)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-06 23:44:51,931 - ==> Total time: 4 days, 22:44:46 Eta: 1 day, 3:40:23 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 23:46:05,581 - Train: 81.11% [438800/541000] [811.1/1000.0] [batch_t 1.450 (1.463)] [data_t 0.003] [optim_t 1.447] [lr 0.000500] 2024-04-06 23:48:31,183 - Train: 81.13% [438900/541000] [811.3/1000.0] [batch_t 1.453 (1.456)] [data_t 0.003] [optim_t 1.450] [lr 0.000500] 2024-04-06 23:50:56,791 - Train: 81.15% [439000/541000] [811.5/1000.0] [batch_t 1.456 (1.456)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-06 23:53:22,422 - Train: 81.16% [439100/541000] [811.6/1000.0] [batch_t 1.457 (1.456)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-06 23:55:48,007 - Train: 81.18% [439200/541000] [811.8/1000.0] [batch_t 1.457 (1.456)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-06 23:58:01,996 - ==> Total time: 4 days, 22:57:56 Eta: 1 day, 3:32:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-06 23:58:18,169 - Train: 81.20% [439300/541000] [812.0/1000.0] [batch_t 1.455 (1.544)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 00:00:43,745 - Train: 81.22% [439400/541000] [812.2/1000.0] [batch_t 1.457 (1.456)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-07 00:03:09,369 - Train: 81.24% [439500/541000] [812.4/1000.0] [batch_t 1.453 (1.456)] [data_t 0.003] [optim_t 1.450] [lr 0.000500] 2024-04-07 00:05:35,006 - Train: 81.26% [439600/541000] [812.6/1000.0] [batch_t 1.451 (1.456)] [data_t 0.003] [optim_t 1.447] [lr 0.000500] 2024-04-07 00:08:00,617 - Train: 81.28% [439700/541000] [812.8/1000.0] [batch_t 1.461 (1.456)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 00:10:26,327 - Train: 81.29% [439800/541000] [812.9/1000.0] [batch_t 1.462 (1.457)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-07 00:11:14,406 - ==> Total time: 4 days, 23:11:08 Eta: 1 day, 3:24:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 00:12:54,343 - Train: 81.31% [439900/541000] [813.1/1000.0] [batch_t 1.460 (1.462)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-07 00:15:20,029 - Train: 81.33% [440000/541000] [813.3/1000.0] [batch_t 1.454 (1.457)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 00:17:45,564 - Train: 81.35% [440100/541000] [813.5/1000.0] [batch_t 1.450 (1.455)] [data_t 0.003] [optim_t 1.447] [lr 0.000500] 2024-04-07 00:20:11,246 - Train: 81.37% [440200/541000] [813.7/1000.0] [batch_t 1.450 (1.457)] [data_t 0.003] [optim_t 1.446] [lr 0.000500] 2024-04-07 00:22:36,891 - Train: 81.39% [440300/541000] [813.9/1000.0] [batch_t 1.454 (1.456)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 00:24:24,591 - ==> Total time: 4 days, 23:24:19 Eta: 1 day, 3:17:03 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 00:25:04,623 - Train: 81.40% [440400/541000] [814.0/1000.0] [batch_t 1.452 (1.468)] [data_t 0.003] [optim_t 1.448] [lr 0.000500] 2024-04-07 00:27:30,164 - Train: 81.42% [440500/541000] [814.2/1000.0] [batch_t 1.451 (1.455)] [data_t 0.003] [optim_t 1.447] [lr 0.000500] 2024-04-07 00:29:55,868 - Train: 81.44% [440600/541000] [814.4/1000.0] [batch_t 1.455 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 00:32:21,560 - Train: 81.46% [440700/541000] [814.6/1000.0] [batch_t 1.454 (1.457)] [data_t 0.003] [optim_t 1.450] [lr 0.000500] 2024-04-07 00:34:47,095 - Train: 81.48% [440800/541000] [814.8/1000.0] [batch_t 1.460 (1.455)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 00:37:12,700 - Train: 81.50% [440900/541000] [815.0/1000.0] [batch_t 1.456 (1.456)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 00:37:34,533 - ==> Total time: 4 days, 23:37:28 Eta: 1 day, 3:09:14 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 00:39:40,470 - Train: 81.52% [441000/541000] [815.2/1000.0] [batch_t 1.455 (1.460)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 00:42:06,077 - Train: 81.53% [441100/541000] [815.3/1000.0] [batch_t 1.459 (1.456)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-07 00:44:31,662 - Train: 81.55% [441200/541000] [815.5/1000.0] [batch_t 1.461 (1.456)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 00:46:57,339 - Train: 81.57% [441300/541000] [815.7/1000.0] [batch_t 1.455 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 00:49:22,954 - Train: 81.59% [441400/541000] [815.9/1000.0] [batch_t 1.456 (1.456)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 00:50:44,479 - ==> Total time: 4 days, 23:50:38 Eta: 1 day, 3:01:25 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 00:51:50,726 - Train: 81.61% [441500/541000] [816.1/1000.0] [batch_t 1.455 (1.465)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 00:54:16,241 - Train: 81.63% [441600/541000] [816.3/1000.0] [batch_t 1.450 (1.455)] [data_t 0.003] [optim_t 1.446] [lr 0.000500] 2024-04-07 00:56:41,879 - Train: 81.65% [441700/541000] [816.5/1000.0] [batch_t 1.450 (1.456)] [data_t 0.003] [optim_t 1.447] [lr 0.000500] 2024-04-07 00:59:07,515 - Train: 81.66% [441800/541000] [816.6/1000.0] [batch_t 1.455 (1.456)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 01:01:33,209 - Train: 81.68% [441900/541000] [816.8/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 01:03:54,505 - ==> Total time: 5 days, 0:03:48 Eta: 1 day, 2:53:35 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 01:04:00,984 - Train: 81.70% [442000/541000] [817.0/1000.0] [batch_t 1.451 (1.583)] [data_t 0.003] [optim_t 1.448] [lr 0.000500] 2024-04-07 01:06:26,654 - Train: 81.72% [442100/541000] [817.2/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 01:08:52,280 - Train: 81.74% [442200/541000] [817.4/1000.0] [batch_t 1.455 (1.456)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 01:11:17,926 - Train: 81.76% [442300/541000] [817.6/1000.0] [batch_t 1.453 (1.456)] [data_t 0.003] [optim_t 1.450] [lr 0.000500] 2024-04-07 01:13:43,595 - Train: 81.77% [442400/541000] [817.7/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 01:16:09,200 - Train: 81.79% [442500/541000] [817.9/1000.0] [batch_t 1.459 (1.456)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-07 01:17:04,531 - ==> Total time: 5 days, 0:16:58 Eta: 1 day, 2:45:44 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 01:18:37,096 - Train: 81.81% [442600/541000] [818.1/1000.0] [batch_t 1.461 (1.462)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 01:21:02,776 - Train: 81.83% [442700/541000] [818.3/1000.0] [batch_t 1.459 (1.457)] [data_t 0.003] [optim_t 1.455] [lr 0.000500] 2024-04-07 01:23:28,424 - Train: 81.85% [442800/541000] [818.5/1000.0] [batch_t 1.452 (1.456)] [data_t 0.003] [optim_t 1.449] [lr 0.000500] 2024-04-07 01:25:54,167 - Train: 81.87% [442900/541000] [818.7/1000.0] [batch_t 1.457 (1.457)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-07 01:28:19,852 - Train: 81.89% [443000/541000] [818.9/1000.0] [batch_t 1.451 (1.457)] [data_t 0.003] [optim_t 1.447] [lr 0.000500] 2024-04-07 01:30:14,868 - ==> Total time: 5 days, 0:30:09 Eta: 1 day, 2:37:52 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 01:30:47,516 - Train: 81.90% [443100/541000] [819.0/1000.0] [batch_t 1.458 (1.473)] [data_t 0.003] [optim_t 1.455] [lr 0.000500] 2024-04-07 01:33:13,145 - Train: 81.92% [443200/541000] [819.2/1000.0] [batch_t 1.457 (1.456)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-07 01:35:38,793 - Train: 81.94% [443300/541000] [819.4/1000.0] [batch_t 1.456 (1.456)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 01:38:04,470 - Train: 81.96% [443400/541000] [819.6/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 01:40:30,148 - Train: 81.98% [443500/541000] [819.8/1000.0] [batch_t 1.455 (1.457)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 01:42:55,728 - Train: 82.00% [443600/541000] [820.0/1000.0] [batch_t 1.454 (1.456)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 01:43:24,866 - ==> Total time: 5 days, 0:43:19 Eta: 1 day, 2:29:59 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 01:45:23,466 - Train: 82.01% [443700/541000] [820.1/1000.0] [batch_t 1.457 (1.460)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 01:47:49,070 - Train: 82.03% [443800/541000] [820.3/1000.0] [batch_t 1.453 (1.456)] [data_t 0.003] [optim_t 1.450] [lr 0.000500] 2024-04-07 01:50:14,727 - Train: 82.05% [443900/541000] [820.5/1000.0] [batch_t 1.455 (1.456)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 01:52:40,266 - Train: 82.07% [444000/541000] [820.7/1000.0] [batch_t 1.456 (1.455)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 01:55:05,895 - Train: 82.09% [444100/541000] [820.9/1000.0] [batch_t 1.455 (1.456)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 01:56:34,704 - ==> Total time: 5 days, 0:56:29 Eta: 1 day, 2:22:06 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 01:57:33,737 - Train: 82.11% [444200/541000] [821.1/1000.0] [batch_t 1.455 (1.464)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 01:59:59,397 - Train: 82.13% [444300/541000] [821.3/1000.0] [batch_t 1.465 (1.456)] [data_t 0.003] [optim_t 1.461] [lr 0.000500] 2024-04-07 02:02:25,018 - Train: 82.14% [444400/541000] [821.4/1000.0] [batch_t 1.454 (1.456)] [data_t 0.003] [optim_t 1.450] [lr 0.000500] 2024-04-07 02:04:50,730 - Train: 82.16% [444500/541000] [821.6/1000.0] [batch_t 1.454 (1.457)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 02:07:16,408 - Train: 82.18% [444600/541000] [821.8/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 02:09:41,982 - Train: 82.20% [444700/541000] [822.0/1000.0] [batch_t 1.449 (1.456)] [data_t 0.003] [optim_t 1.446] [lr 0.000500] 2024-04-07 02:09:44,906 - ==> Total time: 5 days, 1:09:39 Eta: 1 day, 2:14:12 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 02:12:09,847 - Train: 82.22% [444800/541000] [822.2/1000.0] [batch_t 1.449 (1.460)] [data_t 0.003] [optim_t 1.446] [lr 0.000500] 2024-04-07 02:14:35,588 - Train: 82.24% [444900/541000] [822.4/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 02:17:01,309 - Train: 82.26% [445000/541000] [822.6/1000.0] [batch_t 1.459 (1.457)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-07 02:19:27,061 - Train: 82.27% [445100/541000] [822.7/1000.0] [batch_t 1.445 (1.457)] [data_t 0.003] [optim_t 1.442] [lr 0.000500] 2024-04-07 02:21:52,726 - Train: 82.29% [445200/541000] [822.9/1000.0] [batch_t 1.454 (1.457)] [data_t 0.003] [optim_t 1.450] [lr 0.000500] 2024-04-07 02:22:55,352 - ==> Total time: 5 days, 1:22:49 Eta: 1 day, 2:06:17 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 02:24:20,457 - Train: 82.31% [445300/541000] [823.1/1000.0] [batch_t 1.456 (1.462)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 02:26:46,157 - Train: 82.33% [445400/541000] [823.3/1000.0] [batch_t 1.448 (1.457)] [data_t 0.003] [optim_t 1.445] [lr 0.000500] 2024-04-07 02:29:11,803 - Train: 82.35% [445500/541000] [823.5/1000.0] [batch_t 1.462 (1.456)] [data_t 0.004] [optim_t 1.458] [lr 0.000500] 2024-04-07 02:31:37,502 - Train: 82.37% [445600/541000] [823.7/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 02:34:03,086 - Train: 82.38% [445700/541000] [823.8/1000.0] [batch_t 1.451 (1.456)] [data_t 0.003] [optim_t 1.448] [lr 0.000500] 2024-04-07 02:36:05,386 - ==> Total time: 5 days, 1:35:59 Eta: 1 day, 1:58:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 02:36:30,811 - Train: 82.40% [445800/541000] [824.0/1000.0] [batch_t 1.456 (1.478)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 02:38:56,392 - Train: 82.42% [445900/541000] [824.2/1000.0] [batch_t 1.455 (1.456)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 02:41:22,011 - Train: 82.44% [446000/541000] [824.4/1000.0] [batch_t 1.453 (1.456)] [data_t 0.003] [optim_t 1.450] [lr 0.000500] 2024-04-07 02:43:47,596 - Train: 82.46% [446100/541000] [824.6/1000.0] [batch_t 1.461 (1.456)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 02:46:13,273 - Train: 82.48% [446200/541000] [824.8/1000.0] [batch_t 1.458 (1.457)] [data_t 0.003] [optim_t 1.455] [lr 0.000500] 2024-04-07 02:48:38,878 - Train: 82.50% [446300/541000] [825.0/1000.0] [batch_t 1.460 (1.456)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 02:49:15,296 - ==> Total time: 5 days, 1:49:09 Eta: 1 day, 1:50:25 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 02:51:06,880 - Train: 82.51% [446400/541000] [825.1/1000.0] [batch_t 1.454 (1.461)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 02:53:32,540 - Train: 82.53% [446500/541000] [825.3/1000.0] [batch_t 1.456 (1.456)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 02:55:58,297 - Train: 82.55% [446600/541000] [825.5/1000.0] [batch_t 1.450 (1.457)] [data_t 0.003] [optim_t 1.447] [lr 0.000500] 2024-04-07 02:58:24,088 - Train: 82.57% [446700/541000] [825.7/1000.0] [batch_t 1.467 (1.458)] [data_t 0.003] [optim_t 1.464] [lr 0.000500] 2024-04-07 03:00:49,827 - Train: 82.59% [446800/541000] [825.9/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 03:02:25,974 - ==> Total time: 5 days, 2:02:20 Eta: 1 day, 1:42:28 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 03:03:17,691 - Train: 82.61% [446900/541000] [826.1/1000.0] [batch_t 1.457 (1.467)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 03:05:43,374 - Train: 82.62% [447000/541000] [826.2/1000.0] [batch_t 1.455 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 03:08:08,994 - Train: 82.64% [447100/541000] [826.4/1000.0] [batch_t 1.454 (1.456)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 03:10:34,704 - Train: 82.66% [447200/541000] [826.6/1000.0] [batch_t 1.459 (1.457)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-07 03:13:00,285 - Train: 82.68% [447300/541000] [826.8/1000.0] [batch_t 1.455 (1.456)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 03:15:25,888 - Train: 82.70% [447400/541000] [827.0/1000.0] [batch_t 1.463 (1.456)] [data_t 0.004] [optim_t 1.459] [lr 0.000500] 2024-04-07 03:15:36,096 - ==> Total time: 5 days, 2:15:30 Eta: 1 day, 1:34:30 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 03:17:53,672 - Train: 82.72% [447500/541000] [827.2/1000.0] [batch_t 1.454 (1.460)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 03:20:19,276 - Train: 82.74% [447600/541000] [827.4/1000.0] [batch_t 1.455 (1.456)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 03:22:44,966 - Train: 82.75% [447700/541000] [827.5/1000.0] [batch_t 1.462 (1.457)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-07 03:25:10,648 - Train: 82.77% [447800/541000] [827.7/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 03:27:36,291 - Train: 82.79% [447900/541000] [827.9/1000.0] [batch_t 1.457 (1.456)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-07 03:28:46,188 - ==> Total time: 5 days, 2:28:40 Eta: 1 day, 1:26:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 03:30:03,978 - Train: 82.81% [448000/541000] [828.1/1000.0] [batch_t 1.460 (1.462)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-07 03:32:29,646 - Train: 82.83% [448100/541000] [828.3/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 03:34:55,238 - Train: 82.85% [448200/541000] [828.5/1000.0] [batch_t 1.453 (1.456)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 03:37:20,801 - Train: 82.87% [448300/541000] [828.7/1000.0] [batch_t 1.449 (1.456)] [data_t 0.003] [optim_t 1.446] [lr 0.000500] 2024-04-07 03:39:46,355 - Train: 82.88% [448400/541000] [828.8/1000.0] [batch_t 1.454 (1.455)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 03:41:55,922 - ==> Total time: 5 days, 2:41:50 Eta: 1 day, 1:18:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 03:42:14,057 - Train: 82.90% [448500/541000] [829.0/1000.0] [batch_t 1.459 (1.487)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-07 03:44:39,658 - Train: 82.92% [448600/541000] [829.2/1000.0] [batch_t 1.452 (1.456)] [data_t 0.003] [optim_t 1.449] [lr 0.000500] 2024-04-07 03:47:05,347 - Train: 82.94% [448700/541000] [829.4/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 03:49:30,948 - Train: 82.96% [448800/541000] [829.6/1000.0] [batch_t 1.456 (1.456)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 03:51:56,524 - Train: 82.98% [448900/541000] [829.8/1000.0] [batch_t 1.451 (1.456)] [data_t 0.003] [optim_t 1.447] [lr 0.000500] 2024-04-07 03:54:22,106 - Train: 82.99% [449000/541000] [829.9/1000.0] [batch_t 1.455 (1.456)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 03:55:05,783 - ==> Total time: 5 days, 2:55:00 Eta: 1 day, 1:10:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 03:56:49,860 - Train: 83.01% [449100/541000] [830.1/1000.0] [batch_t 1.460 (1.462)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 03:59:15,482 - Train: 83.03% [449200/541000] [830.3/1000.0] [batch_t 1.452 (1.456)] [data_t 0.003] [optim_t 1.448] [lr 0.000500] 2024-04-07 04:01:41,087 - Train: 83.05% [449300/541000] [830.5/1000.0] [batch_t 1.455 (1.456)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 04:04:06,821 - Train: 83.07% [449400/541000] [830.7/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 04:06:32,440 - Train: 83.09% [449500/541000] [830.9/1000.0] [batch_t 1.455 (1.456)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 04:08:15,829 - ==> Total time: 5 days, 3:08:10 Eta: 1 day, 1:02:31 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 04:09:00,213 - Train: 83.11% [449600/541000] [831.1/1000.0] [batch_t 1.452 (1.467)] [data_t 0.003] [optim_t 1.449] [lr 0.000500] 2024-04-07 04:11:25,928 - Train: 83.12% [449700/541000] [831.2/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 04:13:51,634 - Train: 83.14% [449800/541000] [831.4/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 04:16:17,354 - Train: 83.16% [449900/541000] [831.6/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 04:18:42,900 - Train: 83.18% [450000/541000] [831.8/1000.0] [batch_t 1.460 (1.455)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 04:21:08,504 - Train: 83.20% [450100/541000] [832.0/1000.0] [batch_t 1.457 (1.456)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-07 04:21:25,980 - ==> Total time: 5 days, 3:21:20 Eta: 1 day, 0:54:30 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 04:23:36,327 - Train: 83.22% [450200/541000] [832.2/1000.0] [batch_t 1.459 (1.461)] [data_t 0.003] [optim_t 1.455] [lr 0.000500] 2024-04-07 04:26:02,018 - Train: 83.23% [450300/541000] [832.3/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 04:28:27,736 - Train: 83.25% [450400/541000] [832.5/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 04:30:53,368 - Train: 83.27% [450500/541000] [832.7/1000.0] [batch_t 1.460 (1.456)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 04:33:18,995 - Train: 83.29% [450600/541000] [832.9/1000.0] [batch_t 1.461 (1.456)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 04:34:36,143 - ==> Total time: 5 days, 3:34:30 Eta: 1 day, 0:46:27 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 04:35:46,707 - Train: 83.31% [450700/541000] [833.1/1000.0] [batch_t 1.454 (1.464)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 04:38:12,252 - Train: 83.33% [450800/541000] [833.3/1000.0] [batch_t 1.460 (1.455)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-07 04:40:37,910 - Train: 83.35% [450900/541000] [833.5/1000.0] [batch_t 1.457 (1.456)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-07 04:43:03,558 - Train: 83.36% [451000/541000] [833.6/1000.0] [batch_t 1.460 (1.456)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 04:45:29,135 - Train: 83.38% [451100/541000] [833.8/1000.0] [batch_t 1.455 (1.456)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 04:47:46,000 - ==> Total time: 5 days, 3:47:40 Eta: 1 day, 0:38:24 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 04:47:56,865 - Train: 83.40% [451200/541000] [834.0/1000.0] [batch_t 1.455 (1.517)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 04:50:22,475 - Train: 83.42% [451300/541000] [834.2/1000.0] [batch_t 1.451 (1.456)] [data_t 0.003] [optim_t 1.448] [lr 0.000500] 2024-04-07 04:52:47,996 - Train: 83.44% [451400/541000] [834.4/1000.0] [batch_t 1.459 (1.455)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-07 04:55:13,598 - Train: 83.46% [451500/541000] [834.6/1000.0] [batch_t 1.451 (1.456)] [data_t 0.003] [optim_t 1.448] [lr 0.000500] 2024-04-07 04:57:39,323 - Train: 83.48% [451600/541000] [834.8/1000.0] [batch_t 1.454 (1.457)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 05:00:05,037 - Train: 83.49% [451700/541000] [834.9/1000.0] [batch_t 1.462 (1.457)] [data_t 0.004] [optim_t 1.458] [lr 0.000500] 2024-04-07 05:00:56,038 - ==> Total time: 5 days, 4:00:50 Eta: 1 day, 0:30:20 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 05:02:32,861 - Train: 83.51% [451800/541000] [835.1/1000.0] [batch_t 1.455 (1.461)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 05:04:58,481 - Train: 83.53% [451900/541000] [835.3/1000.0] [batch_t 1.455 (1.456)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 05:07:24,135 - Train: 83.55% [452000/541000] [835.5/1000.0] [batch_t 1.461 (1.456)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 05:09:49,844 - Train: 83.57% [452100/541000] [835.7/1000.0] [batch_t 1.451 (1.457)] [data_t 0.003] [optim_t 1.447] [lr 0.000500] 2024-04-07 05:12:12,457 - Train: 83.59% [452200/541000] [835.9/1000.0] [batch_t 0.607 (1.426)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 05:12:58,574 - ==> Total time: 5 days, 4:12:53 Eta: 1 day, 0:22:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 05:13:15,205 - Train: 83.60% [452300/541000] [836.0/1000.0] [batch_t 0.607 (0.621)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 05:14:15,860 - Train: 83.62% [452400/541000] [836.2/1000.0] [batch_t 0.605 (0.606)] [data_t 0.002] [optim_t 0.603] [lr 0.000500] 2024-04-07 05:15:16,591 - Train: 83.64% [452500/541000] [836.4/1000.0] [batch_t 0.608 (0.607)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 05:16:17,563 - Train: 83.66% [452600/541000] [836.6/1000.0] [batch_t 0.610 (0.610)] [data_t 0.002] [optim_t 0.608] [lr 0.000500] 2024-04-07 05:17:18,382 - Train: 83.68% [452700/541000] [836.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 05:18:19,169 - Train: 83.70% [452800/541000] [837.0/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 05:18:29,513 - ==> Total time: 5 days, 4:18:23 Eta: 1 day, 0:12:28 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 05:19:22,058 - Train: 83.72% [452900/541000] [837.2/1000.0] [batch_t 0.605 (0.612)] [data_t 0.002] [optim_t 0.602] [lr 0.000500] 2024-04-07 05:20:22,829 - Train: 83.73% [453000/541000] [837.3/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 05:21:23,647 - Train: 83.75% [453100/541000] [837.5/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 05:22:24,561 - Train: 83.77% [453200/541000] [837.7/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 05:23:25,409 - Train: 83.79% [453300/541000] [837.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 05:24:00,722 - ==> Total time: 5 days, 4:23:55 Eta: 1 day, 0:02:54 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 05:24:29,286 - Train: 83.81% [453400/541000] [838.1/1000.0] [batch_t 0.606 (0.639)] [data_t 0.002] [optim_t 0.604] [lr 0.000500] 2024-04-07 05:25:30,069 - Train: 83.83% [453500/541000] [838.3/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 05:26:30,891 - Train: 83.84% [453600/541000] [838.4/1000.0] [batch_t 0.611 (0.608)] [data_t 0.003] [optim_t 0.608] [lr 0.000500] 2024-04-07 05:27:31,881 - Train: 83.86% [453700/541000] [838.6/1000.0] [batch_t 0.610 (0.610)] [data_t 0.003] [optim_t 0.607] [lr 0.000500] 2024-04-07 05:28:32,716 - Train: 83.88% [453800/541000] [838.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 05:29:32,900 - ==> Total time: 5 days, 4:29:27 Eta: 23:53:21 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 05:29:35,689 - Train: 83.90% [453900/541000] [839.0/1000.0] [batch_t 0.974 (0.974)] [data_t 0.364] [optim_t 0.611] [lr 0.000500] 2024-04-07 05:30:36,473 - Train: 83.92% [454000/541000] [839.2/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 05:31:37,265 - Train: 83.94% [454100/541000] [839.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 05:32:38,053 - Train: 83.96% [454200/541000] [839.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 05:33:38,971 - Train: 83.97% [454300/541000] [839.7/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 05:34:39,800 - Train: 83.99% [454400/541000] [839.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 05:35:04,120 - ==> Total time: 5 days, 4:34:58 Eta: 23:43:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 05:35:42,898 - Train: 84.01% [454500/541000] [840.1/1000.0] [batch_t 0.608 (0.615)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 05:36:43,702 - Train: 84.03% [454600/541000] [840.3/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.000500] 2024-04-07 05:37:44,542 - Train: 84.05% [454700/541000] [840.5/1000.0] [batch_t 0.611 (0.608)] [data_t 0.003] [optim_t 0.608] [lr 0.000500] 2024-04-07 05:38:45,486 - Train: 84.07% [454800/541000] [840.7/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 05:39:46,265 - Train: 84.09% [454900/541000] [840.9/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.000500] 2024-04-07 05:40:35,521 - ==> Total time: 5 days, 4:40:29 Eta: 23:34:16 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 05:40:49,306 - Train: 84.10% [455000/541000] [841.0/1000.0] [batch_t 0.608 (0.629)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 05:41:50,126 - Train: 84.12% [455100/541000] [841.2/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 05:42:50,897 - Train: 84.14% [455200/541000] [841.4/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 05:43:51,709 - Train: 84.16% [455300/541000] [841.6/1000.0] [batch_t 0.610 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.000500] 2024-04-07 05:44:52,681 - Train: 84.18% [455400/541000] [841.8/1000.0] [batch_t 0.607 (0.610)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 05:45:53,482 - Train: 84.20% [455500/541000] [842.0/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 05:46:06,858 - ==> Total time: 5 days, 4:46:01 Eta: 23:24:44 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 05:46:56,288 - Train: 84.21% [455600/541000] [842.1/1000.0] [batch_t 0.607 (0.611)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 05:47:57,071 - Train: 84.23% [455700/541000] [842.3/1000.0] [batch_t 0.610 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.000500] 2024-04-07 05:48:57,841 - Train: 84.25% [455800/541000] [842.5/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 05:49:58,637 - Train: 84.27% [455900/541000] [842.7/1000.0] [batch_t 0.606 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 05:50:59,549 - Train: 84.29% [456000/541000] [842.9/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 05:51:37,827 - ==> Total time: 5 days, 4:51:32 Eta: 23:15:13 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 05:52:02,361 - Train: 84.31% [456100/541000] [843.1/1000.0] [batch_t 0.608 (0.617)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 05:53:03,158 - Train: 84.33% [456200/541000] [843.3/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 05:54:03,976 - Train: 84.34% [456300/541000] [843.4/1000.0] [batch_t 0.615 (0.608)] [data_t 0.003] [optim_t 0.612] [lr 0.000500] 2024-04-07 05:55:04,805 - Train: 84.36% [456400/541000] [843.6/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 05:56:05,611 - Train: 84.38% [456500/541000] [843.8/1000.0] [batch_t 0.605 (0.608)] [data_t 0.003] [optim_t 0.603] [lr 0.000500] 2024-04-07 05:57:06,455 - Train: 84.40% [456600/541000] [844.0/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 05:57:08,888 - ==> Total time: 5 days, 4:57:03 Eta: 23:05:42 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 05:58:09,679 - Train: 84.42% [456700/541000] [844.2/1000.0] [batch_t 0.606 (0.612)] [data_t 0.003] [optim_t 0.603] [lr 0.000500] 2024-04-07 05:59:10,520 - Train: 84.44% [456800/541000] [844.4/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 06:00:11,321 - Train: 84.45% [456900/541000] [844.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 06:01:12,208 - Train: 84.47% [457000/541000] [844.7/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 06:02:13,039 - Train: 84.49% [457100/541000] [844.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 06:02:40,389 - ==> Total time: 5 days, 5:02:34 Eta: 22:56:12 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 06:03:15,951 - Train: 84.51% [457200/541000] [845.1/1000.0] [batch_t 0.607 (0.615)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 06:04:16,904 - Train: 84.53% [457300/541000] [845.3/1000.0] [batch_t 0.609 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 06:05:17,717 - Train: 84.55% [457400/541000] [845.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 06:06:18,521 - Train: 84.57% [457500/541000] [845.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 06:07:19,354 - Train: 84.58% [457600/541000] [845.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 06:08:11,651 - ==> Total time: 5 days, 5:08:06 Eta: 22:46:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 06:08:22,369 - Train: 84.60% [457700/541000] [846.0/1000.0] [batch_t 0.613 (0.634)] [data_t 0.003] [optim_t 0.610] [lr 0.000500] 2024-04-07 06:09:23,178 - Train: 84.62% [457800/541000] [846.2/1000.0] [batch_t 0.610 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.000500] 2024-04-07 06:10:24,156 - Train: 84.64% [457900/541000] [846.4/1000.0] [batch_t 0.608 (0.610)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 06:11:24,987 - Train: 84.66% [458000/541000] [846.6/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 06:12:25,795 - Train: 84.68% [458100/541000] [846.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 06:13:26,612 - Train: 84.70% [458200/541000] [847.0/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 06:13:43,036 - ==> Total time: 5 days, 5:13:37 Eta: 22:37:14 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 06:14:29,547 - Train: 84.71% [458300/541000] [847.1/1000.0] [batch_t 0.606 (0.613)] [data_t 0.002] [optim_t 0.604] [lr 0.000500] 2024-04-07 06:15:30,367 - Train: 84.73% [458400/541000] [847.3/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 06:16:31,302 - Train: 84.75% [458500/541000] [847.5/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 06:17:32,116 - Train: 84.77% [458600/541000] [847.7/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 06:18:32,924 - Train: 84.79% [458700/541000] [847.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 06:19:14,264 - ==> Total time: 5 days, 5:19:08 Eta: 22:27:46 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 06:19:35,804 - Train: 84.81% [458800/541000] [848.1/1000.0] [batch_t 0.608 (0.621)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 06:20:36,631 - Train: 84.82% [458900/541000] [848.2/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 06:21:37,453 - Train: 84.84% [459000/541000] [848.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 06:22:38,407 - Train: 84.86% [459100/541000] [848.6/1000.0] [batch_t 0.606 (0.609)] [data_t 0.002] [optim_t 0.604] [lr 0.000500] 2024-04-07 06:23:39,216 - Train: 84.88% [459200/541000] [848.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 06:24:40,029 - Train: 84.90% [459300/541000] [849.0/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 06:24:45,516 - ==> Total time: 5 days, 5:24:39 Eta: 22:18:18 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 06:25:42,968 - Train: 84.92% [459400/541000] [849.2/1000.0] [batch_t 0.609 (0.612)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 06:26:43,811 - Train: 84.94% [459500/541000] [849.4/1000.0] [batch_t 0.605 (0.608)] [data_t 0.003] [optim_t 0.602] [lr 0.000500] 2024-04-07 06:27:44,776 - Train: 84.95% [459600/541000] [849.5/1000.0] [batch_t 0.610 (0.610)] [data_t 0.003] [optim_t 0.607] [lr 0.000500] 2024-04-07 06:28:45,599 - Train: 84.97% [459700/541000] [849.7/1000.0] [batch_t 0.606 (0.608)] [data_t 0.003] [optim_t 0.603] [lr 0.000500] 2024-04-07 06:29:46,385 - Train: 84.99% [459800/541000] [849.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 06:30:28,960 - Test: 36.76% [50/136] [batch_t 0.217 (0.228)] 2024-04-07 06:30:39,655 - Test: 73.53% [100/136] [batch_t 0.220 (0.221)] 2024-04-07 06:30:47,186 - Test: 100.00% [136/136] [batch_t 0.053 (0.218)] 2024-04-07 06:31:38,535 - ==> Metric Time for candle : 0.003 (mAUROC_sp_max) 0.001 (mAP_sp_max) 0.000 (mF1_max_sp_max) 11.225 (mAUROC_px) 8.187 (mAP_px) 0.941 (mF1_max_px) 26.851 (mAUPRO_px) 0.381 (mF1_px_0.2_0.8_0.1) 0.381 (mAcc_px_0.2_0.8_0.1) 0.382 (mIoU_px_0.2_0.8_0.1) 1.115 (mIoU_max_px) 2024-04-07 06:32:16,513 - ==> Metric Time for capsules : 0.002 (mAUROC_sp_max) 0.001 (mAP_sp_max) 0.001 (mF1_max_sp_max) 8.681 (mAUROC_px) 6.323 (mAP_px) 0.757 (mF1_max_px) 20.449 (mAUPRO_px) 0.258 (mF1_px_0.2_0.8_0.1) 0.256 (mAcc_px_0.2_0.8_0.1) 0.261 (mIoU_px_0.2_0.8_0.1) 0.750 (mIoU_max_px) 2024-04-07 06:32:52,775 - ==> Metric Time for cashew : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.931 (mAUROC_px) 6.215 (mAP_px) 0.706 (mF1_max_px) 19.771 (mAUPRO_px) 0.242 (mF1_px_0.2_0.8_0.1) 0.243 (mAcc_px_0.2_0.8_0.1) 0.243 (mIoU_px_0.2_0.8_0.1) 0.700 (mIoU_max_px) 2024-04-07 06:33:29,408 - ==> Metric Time for chewinggum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.795 (mAUROC_px) 6.213 (mAP_px) 0.720 (mF1_max_px) 20.254 (mAUPRO_px) 0.238 (mF1_px_0.2_0.8_0.1) 0.242 (mAcc_px_0.2_0.8_0.1) 0.243 (mIoU_px_0.2_0.8_0.1) 0.708 (mIoU_max_px) 2024-04-07 06:34:06,880 - ==> Metric Time for fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 8.140 (mAUROC_px) 5.833 (mAP_px) 0.744 (mF1_max_px) 21.110 (mAUPRO_px) 0.242 (mF1_px_0.2_0.8_0.1) 0.243 (mAcc_px_0.2_0.8_0.1) 0.243 (mIoU_px_0.2_0.8_0.1) 0.705 (mIoU_max_px) 2024-04-07 06:34:53,977 - ==> Metric Time for macaroni1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.771 (mAUROC_px) 8.008 (mAP_px) 0.946 (mF1_max_px) 25.178 (mAUPRO_px) 0.322 (mF1_px_0.2_0.8_0.1) 0.324 (mAcc_px_0.2_0.8_0.1) 0.325 (mIoU_px_0.2_0.8_0.1) 0.945 (mIoU_max_px) 2024-04-07 06:35:40,660 - ==> Metric Time for macaroni2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.702 (mAUROC_px) 7.853 (mAP_px) 0.938 (mF1_max_px) 24.980 (mAUPRO_px) 0.320 (mF1_px_0.2_0.8_0.1) 0.321 (mAcc_px_0.2_0.8_0.1) 0.323 (mIoU_px_0.2_0.8_0.1) 0.964 (mIoU_max_px) 2024-04-07 06:36:28,537 - ==> Metric Time for pcb1 : 0.002 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.741 (mAUROC_px) 8.023 (mAP_px) 0.941 (mF1_max_px) 25.966 (mAUPRO_px) 0.323 (mF1_px_0.2_0.8_0.1) 0.327 (mAcc_px_0.2_0.8_0.1) 0.327 (mIoU_px_0.2_0.8_0.1) 0.950 (mIoU_max_px) 2024-04-07 06:37:16,373 - ==> Metric Time for pcb2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.845 (mAUROC_px) 8.099 (mAP_px) 0.963 (mF1_max_px) 25.738 (mAUPRO_px) 0.321 (mF1_px_0.2_0.8_0.1) 0.323 (mAcc_px_0.2_0.8_0.1) 0.324 (mIoU_px_0.2_0.8_0.1) 0.936 (mIoU_max_px) 2024-04-07 06:38:09,189 - ==> Metric Time for pcb3 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.641 (mAUROC_px) 7.946 (mAP_px) 1.191 (mF1_max_px) 30.850 (mAUPRO_px) 0.321 (mF1_px_0.2_0.8_0.1) 0.324 (mAcc_px_0.2_0.8_0.1) 0.321 (mIoU_px_0.2_0.8_0.1) 0.942 (mIoU_max_px) 2024-04-07 06:38:58,119 - ==> Metric Time for pcb4 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.724 (mAUROC_px) 8.083 (mAP_px) 0.947 (mF1_max_px) 26.979 (mAUPRO_px) 0.323 (mF1_px_0.2_0.8_0.1) 0.327 (mAcc_px_0.2_0.8_0.1) 0.325 (mIoU_px_0.2_0.8_0.1) 0.943 (mIoU_max_px) 2024-04-07 06:39:34,212 - ==> Metric Time for pipe_fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.903 (mAUROC_px) 5.842 (mAP_px) 0.696 (mF1_max_px) 20.025 (mAUPRO_px) 0.238 (mF1_px_0.2_0.8_0.1) 0.240 (mAcc_px_0.2_0.8_0.1) 0.238 (mIoU_px_0.2_0.8_0.1) 0.698 (mIoU_max_px) 2024-04-07 06:39:34,233 - | Name | mAUROC_sp_max | mAUROC_sp_max (Max) | mAP_sp_max | mAP_sp_max (Max) | mF1_max_sp_max | mF1_max_sp_max (Max) | mAUROC_px | mAUROC_px (Max) | mAP_px | mAP_px (Max) | mF1_max_px | mF1_max_px (Max) | mAUPRO_px | mAUPRO_px (Max) | mF1_px_0.2_0.8_0.1 | mF1_px_0.2_0.8_0.1 (Max) | mAcc_px_0.2_0.8_0.1 | mAcc_px_0.2_0.8_0.1 (Max) | mIoU_px_0.2_0.8_0.1 | mIoU_px_0.2_0.8_0.1 (Max) | mIoU_max_px | mIoU_max_px (Max) | |:----------:|:---------------:|:---------------------:|:------------:|:------------------:|:----------------:|:----------------------:|:-----------:|:------------------:|:--------:|:------------------:|:------------:|:------------------:|:-----------:|:------------------:|:--------------------:|:--------------------------:|:---------------------:|:---------------------------:|:---------------------:|:---------------------------:|:-------------:|:-------------------:| | candle | 96.170 | 97.100 (50 epoch) | 96.379 | 97.346 (50 epoch) | 89.855 | 91.542 (50 epoch) | 99.073 | 99.098 (250 epoch) | 20.798 | 23.463 (150 epoch) | 30.796 | 32.654 (200 epoch) | 95.657 | 96.087 (300 epoch) | 20.090 | 21.331 (500 epoch) | 38.228 | 41.526 (650 epoch) | 11.436 | 12.185 (500 epoch) | 18.201 | 19.513 (200 epoch) | | capsules | 72.683 | 91.750 (100 epoch) | 74.643 | 94.964 (100 epoch) | 82.143 | 88.780 (100 epoch) | 96.753 | 99.193 (700 epoch) | 24.651 | 61.256 (100 epoch) | 38.840 | 59.868 (250 epoch) | 71.804 | 91.811 (100 epoch) | 14.244 | 33.186 (100 epoch) | 40.962 | 62.576 (50 epoch) | 8.261 | 21.701 (100 epoch) | 24.100 | 42.722 (250 epoch) | | cashew | 94.400 | 94.940 (800 epoch) | 97.105 | 97.492 (150 epoch) | 92.157 | 92.929 (700 epoch) | 95.055 | 96.185 (700 epoch) | 26.038 | 50.952 (250 epoch) | 28.520 | 54.438 (400 epoch) | 80.592 | 89.185 (350 epoch) | 17.507 | 28.352 (350 epoch) | 28.271 | 34.522 (150 epoch) | 9.935 | 17.797 (350 epoch) | 16.632 | 37.398 (400 epoch) | | chewinggum | 96.684 | 97.802 (300 epoch) | 98.471 | 98.939 (300 epoch) | 94.845 | 94.845 (850 epoch) | 97.924 | 98.145 (100 epoch) | 40.716 | 59.823 (50 epoch) | 46.289 | 61.088 (50 epoch) | 79.585 | 80.312 (400 epoch) | 27.208 | 35.748 (50 epoch) | 64.569 | 65.723 (800 epoch) | 16.661 | 23.754 (50 epoch) | 30.115 | 43.976 (50 epoch) | | fryum | 96.160 | 96.560 (650 epoch) | 98.118 | 98.273 (650 epoch) | 92.537 | 93.897 (650 epoch) | 97.275 | 97.308 (750 epoch) | 51.288 | 52.248 (750 epoch) | 53.514 | 53.514 (850 epoch) | 92.310 | 92.340 (550 epoch) | 25.609 | 26.511 (550 epoch) | 25.361 | 28.594 (450 epoch) | 16.038 | 16.529 (550 epoch) | 36.532 | 36.532 (850 epoch) | | macaroni1 | 91.173 | 92.104 (500 epoch) | 86.821 | 90.505 (500 epoch) | 82.297 | 84.314 (750 epoch) | 99.318 | 99.604 (650 epoch) | 6.108 | 20.435 (700 epoch) | 10.610 | 29.513 (650 epoch) | 95.014 | 96.221 (650 epoch) | 7.037 | 16.200 (700 epoch) | 36.648 | 40.765 (50 epoch) | 3.657 | 9.078 (700 epoch) | 5.602 | 17.311 (650 epoch) | | macaroni2 | 75.562 | 82.589 (50 epoch) | 63.527 | 81.196 (50 epoch) | 75.576 | 77.833 (750 epoch) | 95.195 | 99.452 (100 epoch) | 0.509 | 11.272 (50 epoch) | 2.739 | 17.568 (50 epoch) | 81.174 | 96.253 (150 epoch) | 1.041 | 10.086 (50 epoch) | 49.210 | 54.584 (450 epoch) | 0.525 | 5.433 (50 epoch) | 1.388 | 9.630 (50 epoch) | | pcb1 | 96.880 | 96.880 (850 epoch) | 95.627 | 95.627 (850 epoch) | 94.286 | 94.737 (300 epoch) | 99.800 | 99.806 (700 epoch) | 81.019 | 83.141 (450 epoch) | 73.941 | 75.588 (450 epoch) | 93.431 | 94.861 (600 epoch) | 47.018 | 48.256 (700 epoch) | 53.680 | 62.362 (50 epoch) | 32.778 | 34.013 (700 epoch) | 58.656 | 60.756 (450 epoch) | | pcb2 | 96.050 | 96.050 (850 epoch) | 95.888 | 95.888 (850 epoch) | 91.000 | 91.542 (800 epoch) | 99.020 | 99.023 (700 epoch) | 12.598 | 14.130 (50 epoch) | 23.813 | 23.977 (250 epoch) | 89.161 | 91.576 (700 epoch) | 15.463 | 16.150 (200 epoch) | 42.230 | 47.820 (150 epoch) | 8.511 | 8.892 (200 epoch) | 13.516 | 13.621 (250 epoch) | | pcb3 | 95.842 | 95.891 (350 epoch) | 96.066 | 96.066 (850 epoch) | 90.291 | 90.291 (850 epoch) | 99.134 | 99.174 (750 epoch) | 16.943 | 20.249 (150 epoch) | 26.963 | 28.078 (450 epoch) | 92.258 | 92.258 (850 epoch) | 18.979 | 20.042 (150 epoch) | 43.028 | 48.341 (50 epoch) | 10.651 | 11.295 (150 epoch) | 15.582 | 16.332 (450 epoch) | | pcb4 | 99.713 | 99.901 (100 epoch) | 99.716 | 99.901 (100 epoch) | 97.537 | 99.000 (50 epoch) | 98.722 | 98.746 (650 epoch) | 47.366 | 47.966 (250 epoch) | 47.341 | 47.709 (250 epoch) | 91.033 | 91.033 (850 epoch) | 23.490 | 26.925 (200 epoch) | 29.112 | 39.716 (50 epoch) | 14.461 | 16.537 (200 epoch) | 31.011 | 31.328 (250 epoch) | | pipe_fryum | 99.020 | 99.020 (850 epoch) | 99.491 | 99.491 (850 epoch) | 97.537 | 97.561 (200 epoch) | 98.819 | 99.114 (450 epoch) | 51.984 | 56.529 (700 epoch) | 53.504 | 58.733 (450 epoch) | 92.467 | 95.525 (150 epoch) | 30.668 | 34.684 (250 epoch) | 34.854 | 43.014 (100 epoch) | 19.575 | 22.399 (250 epoch) | 36.523 | 41.576 (450 epoch) | | Avg | 92.528 | 94.251 (100 epoch) | 91.821 | 94.481 (100 epoch) | 90.005 | 90.468 (700 epoch) | 98.007 | 98.640 (700 epoch) | 31.668 | 39.701 (250 epoch) | 36.406 | 44.125 (250 epoch) | 87.874 | 91.494 (250 epoch) | 20.696 | 25.164 (100 epoch) | 40.513 | 46.179 (50 epoch) | 12.708 | 15.755 (100 epoch) | 23.988 | 29.990 (250 epoch) | 2024-04-07 06:39:34,622 - ==> Total time: 5 days, 5:39:29 Eta: 22:10:29 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 06:40:50,133 - Train: 85.01% [459900/541000] [850.1/1000.0] [batch_t 1.453 (1.461)] [data_t 0.003] [optim_t 1.449] [lr 0.000500] 2024-04-07 06:43:15,929 - Train: 85.03% [460000/541000] [850.3/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-07 06:45:41,710 - Train: 85.05% [460100/541000] [850.5/1000.0] [batch_t 1.455 (1.458)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 06:48:07,434 - Train: 85.06% [460200/541000] [850.6/1000.0] [batch_t 1.464 (1.457)] [data_t 0.004] [optim_t 1.460] [lr 0.000500] 2024-04-07 06:50:33,140 - Train: 85.08% [460300/541000] [850.8/1000.0] [batch_t 1.447 (1.457)] [data_t 0.003] [optim_t 1.443] [lr 0.000500] 2024-04-07 06:52:45,758 - ==> Total time: 5 days, 5:52:40 Eta: 22:02:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 06:53:00,948 - Train: 85.10% [460400/541000] [851.0/1000.0] [batch_t 1.456 (1.495)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 06:55:26,650 - Train: 85.12% [460500/541000] [851.2/1000.0] [batch_t 1.455 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 06:57:52,416 - Train: 85.14% [460600/541000] [851.4/1000.0] [batch_t 1.454 (1.458)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 07:00:18,108 - Train: 85.16% [460700/541000] [851.6/1000.0] [batch_t 1.458 (1.457)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-07 07:02:43,834 - Train: 85.18% [460800/541000] [851.8/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 07:05:09,542 - Train: 85.19% [460900/541000] [851.9/1000.0] [batch_t 1.454 (1.457)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 07:05:56,156 - ==> Total time: 5 days, 6:05:50 Eta: 21:54:15 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 07:07:37,505 - Train: 85.21% [461000/541000] [852.1/1000.0] [batch_t 1.460 (1.463)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-07 07:10:03,248 - Train: 85.23% [461100/541000] [852.3/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 07:12:28,994 - Train: 85.25% [461200/541000] [852.5/1000.0] [batch_t 1.467 (1.457)] [data_t 0.003] [optim_t 1.465] [lr 0.000500] 2024-04-07 07:14:54,707 - Train: 85.27% [461300/541000] [852.7/1000.0] [batch_t 1.459 (1.457)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-07 07:17:20,435 - Train: 85.29% [461400/541000] [852.9/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 07:19:06,841 - ==> Total time: 5 days, 6:19:01 Eta: 21:46:06 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 07:19:48,516 - Train: 85.30% [461500/541000] [853.0/1000.0] [batch_t 1.466 (1.469)] [data_t 0.003] [optim_t 1.462] [lr 0.000500] 2024-04-07 07:22:14,170 - Train: 85.32% [461600/541000] [853.2/1000.0] [batch_t 1.457 (1.456)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-07 07:24:39,881 - Train: 85.34% [461700/541000] [853.4/1000.0] [batch_t 1.459 (1.457)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-07 07:27:05,589 - Train: 85.36% [461800/541000] [853.6/1000.0] [batch_t 1.454 (1.457)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 07:29:31,350 - Train: 85.38% [461900/541000] [853.8/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 07:31:57,114 - Train: 85.40% [462000/541000] [854.0/1000.0] [batch_t 1.457 (1.458)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-07 07:32:17,526 - ==> Total time: 5 days, 6:32:11 Eta: 21:37:57 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 07:34:25,308 - Train: 85.42% [462100/541000] [854.2/1000.0] [batch_t 1.462 (1.463)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-07 07:36:51,122 - Train: 85.43% [462200/541000] [854.3/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 07:39:16,875 - Train: 85.45% [462300/541000] [854.5/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 07:41:42,594 - Train: 85.47% [462400/541000] [854.7/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 07:44:08,284 - Train: 85.49% [462500/541000] [854.9/1000.0] [batch_t 1.454 (1.457)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 07:45:28,415 - ==> Total time: 5 days, 6:45:22 Eta: 21:29:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 07:46:36,164 - Train: 85.51% [462600/541000] [855.1/1000.0] [batch_t 1.462 (1.467)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-07 07:49:01,931 - Train: 85.53% [462700/541000] [855.3/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 07:51:27,683 - Train: 85.55% [462800/541000] [855.5/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 07:53:53,317 - Train: 85.56% [462900/541000] [855.6/1000.0] [batch_t 1.456 (1.456)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 07:56:19,025 - Train: 85.58% [463000/541000] [855.8/1000.0] [batch_t 1.462 (1.457)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-07 07:58:38,906 - ==> Total time: 5 days, 6:58:33 Eta: 21:21:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 07:58:46,942 - Train: 85.60% [463100/541000] [856.0/1000.0] [batch_t 1.455 (1.544)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 08:01:12,665 - Train: 85.62% [463200/541000] [856.2/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 08:03:38,420 - Train: 85.64% [463300/541000] [856.4/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 08:06:04,175 - Train: 85.66% [463400/541000] [856.6/1000.0] [batch_t 1.457 (1.457)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-07 08:08:29,925 - Train: 85.67% [463500/541000] [856.7/1000.0] [batch_t 1.462 (1.457)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-07 08:10:55,699 - Train: 85.69% [463600/541000] [856.9/1000.0] [batch_t 1.458 (1.458)] [data_t 0.003] [optim_t 1.455] [lr 0.000500] 2024-04-07 08:11:49,577 - ==> Total time: 5 days, 7:11:44 Eta: 21:13:26 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 08:12:43,172 - Train: 85.71% [463700/541000] [857.1/1000.0] [batch_t 0.608 (0.823)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 08:13:51,083 - Train: 85.73% [463800/541000] [857.3/1000.0] [batch_t 0.608 (0.679)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 08:14:52,032 - Train: 85.75% [463900/541000] [857.5/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 08:15:52,817 - Train: 85.77% [464000/541000] [857.7/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 08:16:53,629 - Train: 85.79% [464100/541000] [857.9/1000.0] [batch_t 0.610 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.000500] 2024-04-07 08:17:41,059 - ==> Total time: 5 days, 7:17:35 Eta: 21:04:01 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 08:17:56,630 - Train: 85.80% [464200/541000] [858.0/1000.0] [batch_t 0.610 (0.624)] [data_t 0.003] [optim_t 0.607] [lr 0.000500] 2024-04-07 08:18:57,504 - Train: 85.82% [464300/541000] [858.2/1000.0] [batch_t 0.609 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 08:19:58,294 - Train: 85.84% [464400/541000] [858.4/1000.0] [batch_t 0.610 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.000500] 2024-04-07 08:20:59,226 - Train: 85.86% [464500/541000] [858.6/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 08:22:00,040 - Train: 85.88% [464600/541000] [858.8/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 08:23:00,899 - Train: 85.90% [464700/541000] [859.0/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 08:23:12,460 - ==> Total time: 5 days, 7:23:06 Eta: 20:54:34 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 08:24:03,997 - Train: 85.91% [464800/541000] [859.1/1000.0] [batch_t 0.606 (0.613)] [data_t 0.003] [optim_t 0.603] [lr 0.000500] 2024-04-07 08:25:04,802 - Train: 85.93% [464900/541000] [859.3/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 08:26:05,695 - Train: 85.95% [465000/541000] [859.5/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 08:27:06,467 - Train: 85.97% [465100/541000] [859.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 08:28:07,287 - Train: 85.99% [465200/541000] [859.9/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 08:28:43,749 - ==> Total time: 5 days, 7:28:38 Eta: 20:45:07 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 08:29:10,219 - Train: 86.01% [465300/541000] [860.1/1000.0] [batch_t 0.609 (0.619)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 08:30:11,111 - Train: 86.03% [465400/541000] [860.3/1000.0] [batch_t 0.609 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 08:31:11,928 - Train: 86.04% [465500/541000] [860.4/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 08:32:12,900 - Train: 86.06% [465600/541000] [860.6/1000.0] [batch_t 0.607 (0.610)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 08:33:13,676 - Train: 86.08% [465700/541000] [860.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 08:34:14,458 - Train: 86.10% [465800/541000] [861.0/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 08:34:15,070 - ==> Total time: 5 days, 7:34:09 Eta: 20:35:41 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 08:35:17,365 - Train: 86.12% [465900/541000] [861.2/1000.0] [batch_t 0.608 (0.612)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 08:36:18,154 - Train: 86.14% [466000/541000] [861.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 08:37:18,943 - Train: 86.16% [466100/541000] [861.6/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.000500] 2024-04-07 08:38:19,867 - Train: 86.17% [466200/541000] [861.7/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 08:39:20,654 - Train: 86.19% [466300/541000] [861.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 08:39:46,193 - ==> Total time: 5 days, 7:39:40 Eta: 20:26:15 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 08:40:23,579 - Train: 86.21% [466400/541000] [862.1/1000.0] [batch_t 0.608 (0.615)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 08:41:24,380 - Train: 86.23% [466500/541000] [862.3/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 08:42:25,189 - Train: 86.25% [466600/541000] [862.5/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 08:43:26,039 - Train: 86.27% [466700/541000] [862.7/1000.0] [batch_t 0.614 (0.608)] [data_t 0.003] [optim_t 0.611] [lr 0.000500] 2024-04-07 08:44:26,996 - Train: 86.28% [466800/541000] [862.8/1000.0] [batch_t 0.606 (0.609)] [data_t 0.003] [optim_t 0.603] [lr 0.000500] 2024-04-07 08:45:17,489 - ==> Total time: 5 days, 7:45:11 Eta: 20:16:50 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 08:45:29,966 - Train: 86.30% [466900/541000] [863.0/1000.0] [batch_t 0.608 (0.627)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 08:46:30,770 - Train: 86.32% [467000/541000] [863.2/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 08:47:31,630 - Train: 86.34% [467100/541000] [863.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 08:48:32,453 - Train: 86.36% [467200/541000] [863.6/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 08:49:33,302 - Train: 86.38% [467300/541000] [863.8/1000.0] [batch_t 0.613 (0.608)] [data_t 0.003] [optim_t 0.610] [lr 0.000500] 2024-04-07 08:50:34,292 - Train: 86.40% [467400/541000] [864.0/1000.0] [batch_t 0.609 (0.610)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 08:50:48,894 - ==> Total time: 5 days, 7:50:43 Eta: 20:07:25 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 08:51:37,183 - Train: 86.41% [467500/541000] [864.1/1000.0] [batch_t 0.609 (0.612)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 08:52:38,002 - Train: 86.43% [467600/541000] [864.3/1000.0] [batch_t 0.606 (0.608)] [data_t 0.003] [optim_t 0.603] [lr 0.000500] 2024-04-07 08:53:38,880 - Train: 86.45% [467700/541000] [864.5/1000.0] [batch_t 0.610 (0.609)] [data_t 0.003] [optim_t 0.607] [lr 0.000500] 2024-04-07 08:54:39,726 - Train: 86.47% [467800/541000] [864.7/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 08:55:40,713 - Train: 86.49% [467900/541000] [864.9/1000.0] [batch_t 0.609 (0.610)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 08:56:20,259 - ==> Total time: 5 days, 7:56:14 Eta: 19:58:01 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 08:56:43,725 - Train: 86.51% [468000/541000] [865.1/1000.0] [batch_t 0.608 (0.618)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 08:57:44,528 - Train: 86.52% [468100/541000] [865.2/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 08:58:45,346 - Train: 86.54% [468200/541000] [865.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 08:59:46,128 - Train: 86.56% [468300/541000] [865.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 09:00:47,106 - Train: 86.58% [468400/541000] [865.8/1000.0] [batch_t 0.608 (0.610)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 09:01:47,907 - Train: 86.60% [468500/541000] [866.0/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.000500] 2024-04-07 09:01:51,556 - ==> Total time: 5 days, 8:01:46 Eta: 19:48:38 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 09:02:50,857 - Train: 86.62% [468600/541000] [866.2/1000.0] [batch_t 0.608 (0.612)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 09:03:51,688 - Train: 86.64% [468700/541000] [866.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 09:04:52,484 - Train: 86.65% [468800/541000] [866.5/1000.0] [batch_t 0.611 (0.608)] [data_t 0.003] [optim_t 0.608] [lr 0.000500] 2024-04-07 09:05:53,273 - Train: 86.67% [468900/541000] [866.7/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 09:06:54,099 - Train: 86.69% [469000/541000] [866.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 09:07:22,793 - ==> Total time: 5 days, 8:07:17 Eta: 19:39:14 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 09:07:57,236 - Train: 86.71% [469100/541000] [867.1/1000.0] [batch_t 0.611 (0.616)] [data_t 0.003] [optim_t 0.609] [lr 0.000500] 2024-04-07 09:08:58,035 - Train: 86.73% [469200/541000] [867.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 09:09:58,850 - Train: 86.75% [469300/541000] [867.5/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 09:10:59,645 - Train: 86.77% [469400/541000] [867.7/1000.0] [batch_t 0.610 (0.608)] [data_t 0.002] [optim_t 0.608] [lr 0.000500] 2024-04-07 09:12:00,506 - Train: 86.78% [469500/541000] [867.8/1000.0] [batch_t 0.610 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.000500] 2024-04-07 09:12:54,222 - ==> Total time: 5 days, 8:12:48 Eta: 19:29:52 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 09:13:03,704 - Train: 86.80% [469600/541000] [868.0/1000.0] [batch_t 0.607 (0.636)] [data_t 0.002] [optim_t 0.604] [lr 0.000500] 2024-04-07 09:14:04,497 - Train: 86.82% [469700/541000] [868.2/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.000500] 2024-04-07 09:15:05,285 - Train: 86.84% [469800/541000] [868.4/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 09:16:06,068 - Train: 86.86% [469900/541000] [868.6/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 09:17:06,859 - Train: 86.88% [470000/541000] [868.8/1000.0] [batch_t 0.610 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.000500] 2024-04-07 09:18:07,809 - Train: 86.89% [470100/541000] [868.9/1000.0] [batch_t 0.607 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 09:18:25,449 - ==> Total time: 5 days, 8:18:19 Eta: 19:20:30 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 09:19:10,907 - Train: 86.91% [470200/541000] [869.1/1000.0] [batch_t 0.609 (0.613)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 09:20:11,692 - Train: 86.93% [470300/541000] [869.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 09:21:12,488 - Train: 86.95% [470400/541000] [869.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 09:22:13,284 - Train: 86.97% [470500/541000] [869.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 09:23:14,234 - Train: 86.99% [470600/541000] [869.9/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 09:23:56,789 - ==> Total time: 5 days, 8:23:51 Eta: 19:11:09 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 09:24:17,198 - Train: 87.01% [470700/541000] [870.1/1000.0] [batch_t 0.609 (0.621)] [data_t 0.003] [optim_t 0.607] [lr 0.000500] 2024-04-07 09:25:18,019 - Train: 87.02% [470800/541000] [870.2/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 09:26:18,860 - Train: 87.04% [470900/541000] [870.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 09:27:19,693 - Train: 87.06% [471000/541000] [870.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 09:28:20,658 - Train: 87.08% [471100/541000] [870.8/1000.0] [batch_t 0.607 (0.610)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 09:29:21,482 - Train: 87.10% [471200/541000] [871.0/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 09:29:28,182 - ==> Total time: 5 days, 8:29:22 Eta: 19:01:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 09:30:24,565 - Train: 87.12% [471300/541000] [871.2/1000.0] [batch_t 0.610 (0.612)] [data_t 0.003] [optim_t 0.607] [lr 0.000500] 2024-04-07 09:31:25,397 - Train: 87.13% [471400/541000] [871.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 09:32:26,217 - Train: 87.15% [471500/541000] [871.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 09:33:27,167 - Train: 87.17% [471600/541000] [871.7/1000.0] [batch_t 0.613 (0.609)] [data_t 0.002] [optim_t 0.611] [lr 0.000500] 2024-04-07 09:34:27,956 - Train: 87.19% [471700/541000] [871.9/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.000500] 2024-04-07 09:34:59,555 - ==> Total time: 5 days, 8:34:54 Eta: 18:52:27 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 09:35:30,914 - Train: 87.21% [471800/541000] [872.1/1000.0] [batch_t 0.607 (0.616)] [data_t 0.002] [optim_t 0.604] [lr 0.000500] 2024-04-07 09:36:31,758 - Train: 87.23% [471900/541000] [872.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 09:37:32,558 - Train: 87.25% [472000/541000] [872.5/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 09:38:33,362 - Train: 87.26% [472100/541000] [872.6/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 09:39:34,320 - Train: 87.28% [472200/541000] [872.8/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 09:40:30,859 - ==> Total time: 5 days, 8:40:25 Eta: 18:43:07 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 09:40:37,387 - Train: 87.30% [472300/541000] [873.0/1000.0] [batch_t 0.608 (0.659)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 09:41:38,163 - Train: 87.32% [472400/541000] [873.2/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 09:42:38,965 - Train: 87.34% [472500/541000] [873.4/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 09:43:39,786 - Train: 87.36% [472600/541000] [873.6/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.000500] 2024-04-07 09:44:40,750 - Train: 87.38% [472700/541000] [873.8/1000.0] [batch_t 0.608 (0.610)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 09:45:41,529 - Train: 87.39% [472800/541000] [873.9/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.000500] 2024-04-07 09:46:02,198 - ==> Total time: 5 days, 8:45:56 Eta: 18:33:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 09:46:45,806 - Train: 87.41% [472900/541000] [874.1/1000.0] [batch_t 0.607 (0.617)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 09:47:46,581 - Train: 87.43% [473000/541000] [874.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 09:48:47,356 - Train: 87.45% [473100/541000] [874.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 09:49:48,135 - Train: 87.47% [473200/541000] [874.7/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 09:50:48,962 - Train: 87.49% [473300/541000] [874.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 09:51:34,636 - ==> Total time: 5 days, 8:51:29 Eta: 18:24:29 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 09:51:52,013 - Train: 87.50% [473400/541000] [875.0/1000.0] [batch_t 0.608 (0.621)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 09:52:52,779 - Train: 87.52% [473500/541000] [875.2/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 09:53:53,604 - Train: 87.54% [473600/541000] [875.4/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 09:54:54,432 - Train: 87.56% [473700/541000] [875.6/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 09:55:55,243 - Train: 87.58% [473800/541000] [875.8/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 09:56:56,045 - Train: 87.60% [473900/541000] [876.0/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 09:57:05,780 - ==> Total time: 5 days, 8:57:00 Eta: 18:15:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 09:57:59,036 - Train: 87.62% [474000/541000] [876.2/1000.0] [batch_t 0.607 (0.613)] [data_t 0.002] [optim_t 0.604] [lr 0.000500] 2024-04-07 09:58:59,872 - Train: 87.63% [474100/541000] [876.3/1000.0] [batch_t 0.611 (0.608)] [data_t 0.003] [optim_t 0.608] [lr 0.000500] 2024-04-07 10:00:00,674 - Train: 87.65% [474200/541000] [876.5/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 10:01:01,525 - Train: 87.67% [474300/541000] [876.7/1000.0] [batch_t 0.606 (0.608)] [data_t 0.003] [optim_t 0.603] [lr 0.000500] 2024-04-07 10:02:02,309 - Train: 87.69% [474400/541000] [876.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 10:02:37,092 - ==> Total time: 5 days, 9:02:31 Eta: 18:05:53 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 10:03:05,319 - Train: 87.71% [474500/541000] [877.1/1000.0] [batch_t 0.609 (0.615)] [data_t 0.003] [optim_t 0.607] [lr 0.000500] 2024-04-07 10:04:06,134 - Train: 87.73% [474600/541000] [877.3/1000.0] [batch_t 0.613 (0.608)] [data_t 0.003] [optim_t 0.611] [lr 0.000500] 2024-04-07 10:05:06,926 - Train: 87.74% [474700/541000] [877.4/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 10:06:07,734 - Train: 87.76% [474800/541000] [877.6/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.000500] 2024-04-07 10:07:08,539 - Train: 87.78% [474900/541000] [877.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 10:08:08,100 - ==> Total time: 5 days, 9:08:02 Eta: 17:56:36 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 10:08:11,455 - Train: 87.80% [475000/541000] [878.0/1000.0] [batch_t 0.609 (0.768)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 10:09:12,375 - Train: 87.82% [475100/541000] [878.2/1000.0] [batch_t 0.609 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 10:10:13,190 - Train: 87.84% [475200/541000] [878.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 10:11:13,981 - Train: 87.86% [475300/541000] [878.6/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 10:12:14,787 - Train: 87.87% [475400/541000] [878.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 10:13:15,556 - Train: 87.89% [475500/541000] [878.9/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 10:13:39,272 - ==> Total time: 5 days, 9:13:33 Eta: 17:47:19 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 10:14:18,716 - Train: 87.91% [475600/541000] [879.1/1000.0] [batch_t 0.607 (0.613)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 10:15:19,511 - Train: 87.93% [475700/541000] [879.3/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 10:16:20,274 - Train: 87.95% [475800/541000] [879.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 10:17:21,078 - Train: 87.97% [475900/541000] [879.7/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 10:18:21,926 - Train: 87.99% [476000/541000] [879.9/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 10:19:10,558 - ==> Total time: 5 days, 9:19:05 Eta: 17:38:03 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 10:19:24,897 - Train: 88.00% [476100/541000] [880.0/1000.0] [batch_t 0.609 (0.629)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 10:20:25,882 - Train: 88.02% [476200/541000] [880.2/1000.0] [batch_t 0.609 (0.610)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 10:21:26,644 - Train: 88.04% [476300/541000] [880.4/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 10:22:27,439 - Train: 88.06% [476400/541000] [880.6/1000.0] [batch_t 0.611 (0.608)] [data_t 0.003] [optim_t 0.608] [lr 0.000500] 2024-04-07 10:23:28,254 - Train: 88.08% [476500/541000] [880.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 10:24:29,041 - Train: 88.10% [476600/541000] [881.0/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 10:24:48,536 - ==> Total time: 5 days, 9:24:42 Eta: 17:28:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 10:26:02,049 - Train: 88.11% [476700/541000] [881.1/1000.0] [batch_t 0.608 (0.880)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 10:27:55,545 - Train: 88.13% [476800/541000] [881.3/1000.0] [batch_t 0.608 (1.135)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 10:30:31,136 - Train: 88.15% [476900/541000] [881.5/1000.0] [batch_t 1.699 (1.556)] [data_t 1.094] [optim_t 0.605] [lr 0.000500] 2024-04-07 10:33:17,418 - Train: 88.17% [477000/541000] [881.7/1000.0] [batch_t 0.606 (1.663)] [data_t 0.003] [optim_t 0.603] [lr 0.000500] 2024-04-07 10:35:57,735 - Train: 88.19% [477100/541000] [881.9/1000.0] [batch_t 0.607 (1.603)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 10:37:41,245 - ==> Total time: 5 days, 9:37:35 Eta: 17:20:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 10:38:46,786 - Train: 88.21% [477200/541000] [882.1/1000.0] [batch_t 0.609 (1.559)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 10:40:55,297 - Train: 88.23% [477300/541000] [882.3/1000.0] [batch_t 0.609 (1.285)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 10:43:17,701 - Train: 88.24% [477400/541000] [882.4/1000.0] [batch_t 0.611 (1.424)] [data_t 0.003] [optim_t 0.608] [lr 0.000500] 2024-04-07 10:45:07,317 - Train: 88.26% [477500/541000] [882.6/1000.0] [batch_t 0.612 (1.096)] [data_t 0.002] [optim_t 0.609] [lr 0.000500] 2024-04-07 10:47:13,301 - Train: 88.28% [477600/541000] [882.8/1000.0] [batch_t 0.608 (1.260)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 10:48:56,030 - Train: 88.30% [477700/541000] [883.0/1000.0] [batch_t 0.606 (1.027)] [data_t 0.002] [optim_t 0.604] [lr 0.000500] 2024-04-07 10:48:59,286 - ==> Total time: 5 days, 9:48:53 Eta: 17:12:03 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 10:50:55,113 - Train: 88.32% [477800/541000] [883.2/1000.0] [batch_t 0.607 (1.142)] [data_t 0.002] [optim_t 0.604] [lr 0.000500] 2024-04-07 10:53:49,228 - Train: 88.34% [477900/541000] [883.4/1000.0] [batch_t 0.608 (1.741)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 10:55:00,808 - Train: 88.35% [478000/541000] [883.5/1000.0] [batch_t 0.607 (0.716)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 10:56:01,569 - Train: 88.37% [478100/541000] [883.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 10:57:02,340 - Train: 88.39% [478200/541000] [883.9/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.000500] 2024-04-07 10:57:29,099 - ==> Total time: 5 days, 9:57:23 Eta: 17:03:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 10:58:06,404 - Train: 88.41% [478300/541000] [884.1/1000.0] [batch_t 0.608 (0.628)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 10:59:07,167 - Train: 88.43% [478400/541000] [884.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 11:00:07,934 - Train: 88.45% [478500/541000] [884.5/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 11:01:08,862 - Train: 88.47% [478600/541000] [884.7/1000.0] [batch_t 0.754 (0.609)] [data_t 0.002] [optim_t 0.752] [lr 0.000500] 2024-04-07 11:02:09,648 - Train: 88.48% [478700/541000] [884.8/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 11:03:01,301 - ==> Total time: 5 days, 10:02:55 Eta: 16:53:56 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 11:03:15,578 - Train: 88.50% [478800/541000] [885.0/1000.0] [batch_t 0.612 (0.654)] [data_t 0.003] [optim_t 0.609] [lr 0.000500] 2024-04-07 11:04:16,381 - Train: 88.52% [478900/541000] [885.2/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 11:05:17,206 - Train: 88.54% [479000/541000] [885.4/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 11:06:18,007 - Train: 88.56% [479100/541000] [885.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 11:07:18,805 - Train: 88.58% [479200/541000] [885.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 11:08:19,747 - Train: 88.60% [479300/541000] [886.0/1000.0] [batch_t 0.609 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 11:08:35,555 - ==> Total time: 5 days, 10:08:30 Eta: 16:44:42 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 11:09:23,907 - Train: 88.61% [479400/541000] [886.1/1000.0] [batch_t 0.607 (0.623)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 11:10:24,663 - Train: 88.63% [479500/541000] [886.3/1000.0] [batch_t 0.609 (0.607)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 11:11:25,435 - Train: 88.65% [479600/541000] [886.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 11:12:26,220 - Train: 88.67% [479700/541000] [886.7/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 11:13:27,015 - Train: 88.69% [479800/541000] [886.9/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 11:14:07,956 - ==> Total time: 5 days, 10:14:02 Eta: 16:35:28 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 11:14:31,843 - Train: 88.71% [479900/541000] [887.1/1000.0] [batch_t 0.613 (0.653)] [data_t 0.003] [optim_t 0.610] [lr 0.000500] 2024-04-07 11:15:32,718 - Train: 88.72% [480000/541000] [887.2/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 11:16:33,530 - Train: 88.74% [480100/541000] [887.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 11:17:34,330 - Train: 88.76% [480200/541000] [887.6/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.000500] 2024-04-07 11:18:35,135 - Train: 88.78% [480300/541000] [887.8/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 11:19:36,067 - Train: 88.80% [480400/541000] [888.0/1000.0] [batch_t 0.609 (0.609)] [data_t 0.003] [optim_t 0.607] [lr 0.000500] 2024-04-07 11:19:40,931 - ==> Total time: 5 days, 10:19:35 Eta: 16:26:15 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 11:20:40,885 - Train: 88.82% [480500/541000] [888.2/1000.0] [batch_t 0.608 (0.622)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 11:21:41,677 - Train: 88.84% [480600/541000] [888.4/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 11:22:42,447 - Train: 88.85% [480700/541000] [888.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 11:23:43,227 - Train: 88.87% [480800/541000] [888.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 11:24:43,966 - Train: 88.89% [480900/541000] [888.9/1000.0] [batch_t 0.606 (0.607)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 11:25:13,742 - ==> Total time: 5 days, 10:25:08 Eta: 16:17:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 11:25:47,226 - Train: 88.91% [481000/541000] [889.1/1000.0] [batch_t 0.608 (0.617)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 11:26:47,980 - Train: 88.93% [481100/541000] [889.3/1000.0] [batch_t 0.608 (0.607)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 11:27:48,737 - Train: 88.95% [481200/541000] [889.5/1000.0] [batch_t 0.608 (0.607)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 11:28:49,512 - Train: 88.96% [481300/541000] [889.6/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.000500] 2024-04-07 11:29:50,276 - Train: 88.98% [481400/541000] [889.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 11:30:44,953 - ==> Total time: 5 days, 10:30:39 Eta: 16:07:50 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 11:30:53,213 - Train: 89.00% [481500/541000] [890.0/1000.0] [batch_t 0.606 (0.645)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 11:31:54,116 - Train: 89.02% [481600/541000] [890.2/1000.0] [batch_t 0.609 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 11:32:54,884 - Train: 89.04% [481700/541000] [890.4/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.603] [lr 0.000500] 2024-04-07 11:33:55,682 - Train: 89.06% [481800/541000] [890.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 11:34:56,462 - Train: 89.08% [481900/541000] [890.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 11:35:57,231 - Train: 89.09% [482000/541000] [890.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 11:36:16,065 - ==> Total time: 5 days, 10:36:10 Eta: 15:58:38 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 11:37:00,548 - Train: 89.11% [482100/541000] [891.1/1000.0] [batch_t 0.608 (0.616)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 11:38:01,320 - Train: 89.13% [482200/541000] [891.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 11:39:03,051 - Train: 89.15% [482300/541000] [891.5/1000.0] [batch_t 0.608 (0.617)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 11:40:03,816 - Train: 89.17% [482400/541000] [891.7/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 11:41:04,573 - Train: 89.19% [482500/541000] [891.9/1000.0] [batch_t 0.608 (0.607)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 11:41:50,668 - ==> Total time: 5 days, 10:41:45 Eta: 15:49:26 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 11:42:13,391 - Train: 89.21% [482600/541000] [892.1/1000.0] [batch_t 0.607 (0.714)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 11:43:37,060 - Train: 89.22% [482700/541000] [892.2/1000.0] [batch_t 0.606 (0.837)] [data_t 0.003] [optim_t 0.603] [lr 0.000500] 2024-04-07 11:46:32,362 - Train: 89.24% [482800/541000] [892.4/1000.0] [batch_t 0.611 (1.753)] [data_t 0.003] [optim_t 0.608] [lr 0.000500] 2024-04-07 11:48:56,496 - Train: 89.26% [482900/541000] [892.6/1000.0] [batch_t 0.607 (1.441)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 11:50:25,334 - Train: 89.28% [483000/541000] [892.8/1000.0] [batch_t 0.606 (0.888)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 11:51:28,439 - Train: 89.30% [483100/541000] [893.0/1000.0] [batch_t 0.612 (0.631)] [data_t 0.003] [optim_t 0.610] [lr 0.000500] 2024-04-07 11:51:36,338 - ==> Total time: 5 days, 10:51:30 Eta: 15:40:46 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 11:52:35,743 - Train: 89.32% [483200/541000] [893.2/1000.0] [batch_t 0.609 (0.649)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 11:53:58,104 - Train: 89.33% [483300/541000] [893.3/1000.0] [batch_t 0.607 (0.824)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 11:55:29,841 - Train: 89.35% [483400/541000] [893.5/1000.0] [batch_t 0.608 (0.917)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 11:56:54,223 - Train: 89.37% [483500/541000] [893.7/1000.0] [batch_t 0.607 (0.844)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 11:58:26,784 - Train: 89.39% [483600/541000] [893.9/1000.0] [batch_t 0.608 (0.926)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 11:59:18,782 - ==> Total time: 5 days, 10:59:13 Eta: 15:31:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 12:00:06,739 - Train: 89.41% [483700/541000] [894.1/1000.0] [batch_t 1.051 (0.937)] [data_t 0.447] [optim_t 0.604] [lr 0.000500] 2024-04-07 12:01:36,209 - Train: 89.43% [483800/541000] [894.3/1000.0] [batch_t 0.846 (0.895)] [data_t 0.236] [optim_t 0.610] [lr 0.000500] 2024-04-07 12:03:19,005 - Train: 89.45% [483900/541000] [894.5/1000.0] [batch_t 2.509 (1.028)] [data_t 1.900] [optim_t 0.608] [lr 0.000500] 2024-04-07 12:05:00,680 - Train: 89.46% [484000/541000] [894.6/1000.0] [batch_t 1.463 (1.017)] [data_t 0.857] [optim_t 0.606] [lr 0.000500] 2024-04-07 12:06:29,369 - Train: 89.48% [484100/541000] [894.8/1000.0] [batch_t 2.545 (0.887)] [data_t 1.940] [optim_t 0.605] [lr 0.000500] 2024-04-07 12:07:54,320 - ==> Total time: 5 days, 11:07:48 Eta: 15:23:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 12:08:06,558 - Train: 89.50% [484200/541000] [895.0/1000.0] [batch_t 0.609 (1.625)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 12:10:36,298 - Train: 89.52% [484300/541000] [895.2/1000.0] [batch_t 3.115 (1.497)] [data_t 2.509] [optim_t 0.606] [lr 0.000500] 2024-04-07 12:13:16,585 - Train: 89.54% [484400/541000] [895.4/1000.0] [batch_t 1.520 (1.603)] [data_t 0.915] [optim_t 0.605] [lr 0.000500] 2024-04-07 12:14:47,335 - Train: 89.56% [484500/541000] [895.6/1000.0] [batch_t 0.608 (0.907)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 12:16:12,595 - Train: 89.57% [484600/541000] [895.7/1000.0] [batch_t 0.787 (0.853)] [data_t 0.182] [optim_t 0.606] [lr 0.000500] 2024-04-07 12:17:33,152 - Train: 89.59% [484700/541000] [895.9/1000.0] [batch_t 0.608 (0.805)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 12:17:59,424 - ==> Total time: 5 days, 11:17:53 Eta: 15:14:23 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 12:18:47,023 - Train: 89.61% [484800/541000] [896.1/1000.0] [batch_t 0.607 (0.690)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 12:19:47,951 - Train: 89.63% [484900/541000] [896.3/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 12:20:48,765 - Train: 89.65% [485000/541000] [896.5/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 12:21:49,546 - Train: 89.67% [485100/541000] [896.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 12:22:50,366 - Train: 89.69% [485200/541000] [896.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 12:23:37,178 - ==> Total time: 5 days, 11:23:31 Eta: 15:05:14 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 12:23:54,890 - Train: 89.70% [485300/541000] [897.0/1000.0] [batch_t 0.609 (0.657)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 12:24:55,844 - Train: 89.72% [485400/541000] [897.2/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 12:25:56,633 - Train: 89.74% [485500/541000] [897.4/1000.0] [batch_t 0.606 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 12:26:57,439 - Train: 89.76% [485600/541000] [897.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 12:27:58,259 - Train: 89.78% [485700/541000] [897.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 12:28:59,104 - Train: 89.80% [485800/541000] [898.0/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 12:29:10,051 - ==> Total time: 5 days, 11:29:04 Eta: 14:56:05 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 12:30:02,678 - Train: 89.82% [485900/541000] [898.2/1000.0] [batch_t 0.608 (0.617)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 12:31:03,476 - Train: 89.83% [486000/541000] [898.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 12:32:04,313 - Train: 89.85% [486100/541000] [898.5/1000.0] [batch_t 0.610 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.000500] 2024-04-07 12:33:05,121 - Train: 89.87% [486200/541000] [898.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 12:34:05,903 - Train: 89.89% [486300/541000] [898.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 12:34:41,784 - ==> Total time: 5 days, 11:34:36 Eta: 14:46:56 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 12:35:10,551 - Train: 89.91% [486400/541000] [899.1/1000.0] [batch_t 0.606 (0.631)] [data_t 0.002] [optim_t 0.604] [lr 0.000500] 2024-04-07 12:36:11,358 - Train: 89.93% [486500/541000] [899.3/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.000500] 2024-04-07 12:37:12,162 - Train: 89.94% [486600/541000] [899.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 12:38:12,949 - Train: 89.96% [486700/541000] [899.6/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.000500] 2024-04-07 12:39:13,742 - Train: 89.98% [486800/541000] [899.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 12:40:14,509 - Train: 90.00% [486900/541000] [900.0/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 12:40:26,671 - Test: 36.76% [50/136] [batch_t 0.213 (0.229)] 2024-04-07 12:40:37,309 - Test: 73.53% [100/136] [batch_t 0.211 (0.221)] 2024-04-07 12:40:45,014 - Test: 100.00% [136/136] [batch_t 0.065 (0.219)] 2024-04-07 12:41:38,171 - ==> Metric Time for candle : 0.002 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.431 (mAUROC_px) 8.468 (mAP_px) 1.112 (mF1_max_px) 28.979 (mAUPRO_px) 0.403 (mF1_px_0.2_0.8_0.1) 0.404 (mAcc_px_0.2_0.8_0.1) 0.408 (mIoU_px_0.2_0.8_0.1) 1.172 (mIoU_max_px) 2024-04-07 12:42:17,496 - ==> Metric Time for capsules : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 9.049 (mAUROC_px) 6.287 (mAP_px) 0.749 (mF1_max_px) 21.487 (mAUPRO_px) 0.258 (mF1_px_0.2_0.8_0.1) 0.259 (mAcc_px_0.2_0.8_0.1) 0.259 (mIoU_px_0.2_0.8_0.1) 0.748 (mIoU_max_px) 2024-04-07 12:42:53,771 - ==> Metric Time for cashew : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.755 (mAUROC_px) 5.930 (mAP_px) 0.709 (mF1_max_px) 20.257 (mAUPRO_px) 0.242 (mF1_px_0.2_0.8_0.1) 0.241 (mAcc_px_0.2_0.8_0.1) 0.249 (mIoU_px_0.2_0.8_0.1) 0.703 (mIoU_max_px) 2024-04-07 12:43:30,703 - ==> Metric Time for chewinggum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.763 (mAUROC_px) 5.949 (mAP_px) 0.705 (mF1_max_px) 20.881 (mAUPRO_px) 0.244 (mF1_px_0.2_0.8_0.1) 0.244 (mAcc_px_0.2_0.8_0.1) 0.244 (mIoU_px_0.2_0.8_0.1) 0.710 (mIoU_max_px) 2024-04-07 12:44:07,947 - ==> Metric Time for fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.657 (mAUROC_px) 5.721 (mAP_px) 0.714 (mF1_max_px) 21.525 (mAUPRO_px) 0.243 (mF1_px_0.2_0.8_0.1) 0.239 (mAcc_px_0.2_0.8_0.1) 0.248 (mIoU_px_0.2_0.8_0.1) 0.705 (mIoU_max_px) 2024-04-07 12:44:54,885 - ==> Metric Time for macaroni1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.506 (mAUROC_px) 7.820 (mAP_px) 0.988 (mF1_max_px) 25.343 (mAUPRO_px) 0.344 (mF1_px_0.2_0.8_0.1) 0.344 (mAcc_px_0.2_0.8_0.1) 0.338 (mIoU_px_0.2_0.8_0.1) 0.981 (mIoU_max_px) 2024-04-07 12:45:42,253 - ==> Metric Time for macaroni2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.480 (mAUROC_px) 7.833 (mAP_px) 1.004 (mF1_max_px) 25.749 (mAUPRO_px) 0.342 (mF1_px_0.2_0.8_0.1) 0.336 (mAcc_px_0.2_0.8_0.1) 0.348 (mIoU_px_0.2_0.8_0.1) 0.993 (mIoU_max_px) 2024-04-07 12:46:31,029 - ==> Metric Time for pcb1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.645 (mAUROC_px) 7.951 (mAP_px) 1.056 (mF1_max_px) 26.788 (mAUPRO_px) 0.347 (mF1_px_0.2_0.8_0.1) 0.347 (mAcc_px_0.2_0.8_0.1) 0.347 (mIoU_px_0.2_0.8_0.1) 1.007 (mIoU_max_px) 2024-04-07 12:47:23,723 - ==> Metric Time for pcb2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.830 (mAUROC_px) 8.234 (mAP_px) 0.992 (mF1_max_px) 30.290 (mAUPRO_px) 0.350 (mF1_px_0.2_0.8_0.1) 0.349 (mAcc_px_0.2_0.8_0.1) 0.346 (mIoU_px_0.2_0.8_0.1) 1.010 (mIoU_max_px) 2024-04-07 12:48:12,250 - ==> Metric Time for pcb3 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.639 (mAUROC_px) 7.942 (mAP_px) 1.006 (mF1_max_px) 26.612 (mAUPRO_px) 0.343 (mF1_px_0.2_0.8_0.1) 0.342 (mAcc_px_0.2_0.8_0.1) 0.339 (mIoU_px_0.2_0.8_0.1) 1.008 (mIoU_max_px) 2024-04-07 12:49:03,092 - ==> Metric Time for pcb4 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.706 (mAUROC_px) 8.157 (mAP_px) 1.023 (mF1_max_px) 28.594 (mAUPRO_px) 0.346 (mF1_px_0.2_0.8_0.1) 0.344 (mAcc_px_0.2_0.8_0.1) 0.368 (mIoU_px_0.2_0.8_0.1) 1.010 (mIoU_max_px) 2024-04-07 12:49:40,232 - ==> Metric Time for pipe_fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.827 (mAUROC_px) 5.870 (mAP_px) 0.709 (mF1_max_px) 21.100 (mAUPRO_px) 0.242 (mF1_px_0.2_0.8_0.1) 0.244 (mAcc_px_0.2_0.8_0.1) 0.245 (mIoU_px_0.2_0.8_0.1) 0.699 (mIoU_max_px) 2024-04-07 12:49:40,249 - | Name | mAUROC_sp_max | mAUROC_sp_max (Max) | mAP_sp_max | mAP_sp_max (Max) | mF1_max_sp_max | mF1_max_sp_max (Max) | mAUROC_px | mAUROC_px (Max) | mAP_px | mAP_px (Max) | mF1_max_px | mF1_max_px (Max) | mAUPRO_px | mAUPRO_px (Max) | mF1_px_0.2_0.8_0.1 | mF1_px_0.2_0.8_0.1 (Max) | mAcc_px_0.2_0.8_0.1 | mAcc_px_0.2_0.8_0.1 (Max) | mIoU_px_0.2_0.8_0.1 | mIoU_px_0.2_0.8_0.1 (Max) | mIoU_max_px | mIoU_max_px (Max) | |:----------:|:---------------:|:---------------------:|:------------:|:------------------:|:----------------:|:----------------------:|:-----------:|:------------------:|:--------:|:------------------:|:------------:|:------------------:|:-----------:|:------------------:|:--------------------:|:--------------------------:|:---------------------:|:---------------------------:|:---------------------:|:---------------------------:|:-------------:|:-------------------:| | candle | 96.210 | 97.100 (50 epoch) | 96.447 | 97.346 (50 epoch) | 90.196 | 91.542 (50 epoch) | 99.062 | 99.098 (250 epoch) | 20.815 | 23.463 (150 epoch) | 30.810 | 32.654 (200 epoch) | 95.722 | 96.087 (300 epoch) | 20.225 | 21.331 (500 epoch) | 38.705 | 41.526 (650 epoch) | 11.507 | 12.185 (500 epoch) | 18.210 | 19.513 (200 epoch) | | capsules | 72.100 | 91.750 (100 epoch) | 74.323 | 94.964 (100 epoch) | 83.262 | 88.780 (100 epoch) | 95.907 | 99.193 (700 epoch) | 18.273 | 61.256 (100 epoch) | 32.962 | 59.868 (250 epoch) | 67.153 | 91.811 (100 epoch) | 11.381 | 33.186 (100 epoch) | 37.420 | 62.576 (50 epoch) | 6.424 | 21.701 (100 epoch) | 19.733 | 42.722 (250 epoch) | | cashew | 94.440 | 94.940 (800 epoch) | 97.080 | 97.492 (150 epoch) | 92.611 | 92.929 (700 epoch) | 95.222 | 96.185 (700 epoch) | 26.090 | 50.952 (250 epoch) | 27.440 | 54.438 (400 epoch) | 78.644 | 89.185 (350 epoch) | 17.796 | 28.352 (350 epoch) | 29.107 | 34.522 (150 epoch) | 10.076 | 17.797 (350 epoch) | 15.902 | 37.398 (400 epoch) | | chewinggum | 96.841 | 97.802 (300 epoch) | 98.549 | 98.939 (300 epoch) | 94.845 | 94.845 (850 epoch) | 97.982 | 98.145 (100 epoch) | 43.992 | 59.823 (50 epoch) | 52.123 | 61.088 (50 epoch) | 79.862 | 80.312 (400 epoch) | 28.145 | 35.748 (50 epoch) | 61.687 | 65.723 (800 epoch) | 17.652 | 23.754 (50 epoch) | 35.248 | 43.976 (50 epoch) | | fryum | 96.200 | 96.560 (650 epoch) | 98.143 | 98.273 (650 epoch) | 92.537 | 93.897 (650 epoch) | 97.314 | 97.314 (900 epoch) | 51.465 | 52.248 (750 epoch) | 53.679 | 53.679 (900 epoch) | 92.191 | 92.340 (550 epoch) | 25.550 | 26.511 (550 epoch) | 25.141 | 28.594 (450 epoch) | 16.020 | 16.529 (550 epoch) | 36.686 | 36.686 (900 epoch) | | macaroni1 | 91.123 | 92.104 (500 epoch) | 86.794 | 90.505 (500 epoch) | 82.178 | 84.314 (750 epoch) | 99.302 | 99.604 (650 epoch) | 5.586 | 20.435 (700 epoch) | 9.112 | 29.513 (650 epoch) | 94.840 | 96.221 (650 epoch) | 6.374 | 16.200 (700 epoch) | 36.078 | 40.765 (50 epoch) | 3.298 | 9.078 (700 epoch) | 4.773 | 17.311 (650 epoch) | | macaroni2 | 74.786 | 82.589 (50 epoch) | 61.522 | 81.196 (50 epoch) | 75.336 | 77.833 (750 epoch) | 93.306 | 99.452 (100 epoch) | 0.297 | 11.272 (50 epoch) | 1.948 | 17.568 (50 epoch) | 74.562 | 96.253 (150 epoch) | 0.667 | 10.086 (50 epoch) | 30.924 | 54.584 (450 epoch) | 0.335 | 5.433 (50 epoch) | 0.984 | 9.630 (50 epoch) | | pcb1 | 96.790 | 96.880 (850 epoch) | 95.439 | 95.627 (850 epoch) | 94.231 | 94.737 (300 epoch) | 99.800 | 99.806 (700 epoch) | 81.250 | 83.141 (450 epoch) | 74.013 | 75.588 (450 epoch) | 94.908 | 94.908 (900 epoch) | 47.015 | 48.256 (700 epoch) | 53.633 | 62.362 (50 epoch) | 32.847 | 34.013 (700 epoch) | 58.747 | 60.756 (450 epoch) | | pcb2 | 96.120 | 96.120 (900 epoch) | 95.880 | 95.888 (850 epoch) | 91.542 | 91.542 (800 epoch) | 99.021 | 99.023 (700 epoch) | 12.594 | 14.130 (50 epoch) | 23.723 | 23.977 (250 epoch) | 89.137 | 91.576 (700 epoch) | 15.538 | 16.150 (200 epoch) | 42.479 | 47.820 (150 epoch) | 8.553 | 8.892 (200 epoch) | 13.458 | 13.621 (250 epoch) | | pcb3 | 95.891 | 95.891 (350 epoch) | 96.061 | 96.066 (850 epoch) | 89.524 | 90.291 (850 epoch) | 99.135 | 99.174 (750 epoch) | 16.881 | 20.249 (150 epoch) | 26.897 | 28.078 (450 epoch) | 92.078 | 92.258 (850 epoch) | 18.789 | 20.042 (150 epoch) | 42.592 | 48.341 (50 epoch) | 10.537 | 11.295 (150 epoch) | 15.538 | 16.332 (450 epoch) | | pcb4 | 99.703 | 99.901 (100 epoch) | 99.706 | 99.901 (100 epoch) | 97.537 | 99.000 (50 epoch) | 98.689 | 98.746 (650 epoch) | 46.158 | 47.966 (250 epoch) | 46.087 | 47.709 (250 epoch) | 90.366 | 91.033 (850 epoch) | 23.179 | 26.925 (200 epoch) | 28.886 | 39.716 (50 epoch) | 14.205 | 16.537 (200 epoch) | 29.944 | 31.328 (250 epoch) | | pipe_fryum | 99.040 | 99.040 (900 epoch) | 99.503 | 99.503 (900 epoch) | 97.087 | 97.561 (200 epoch) | 98.816 | 99.114 (450 epoch) | 50.780 | 56.529 (700 epoch) | 52.532 | 58.733 (450 epoch) | 91.478 | 95.525 (150 epoch) | 29.343 | 34.684 (250 epoch) | 32.940 | 43.014 (100 epoch) | 18.659 | 22.399 (250 epoch) | 35.623 | 41.576 (450 epoch) | | Avg | 92.437 | 94.251 (100 epoch) | 91.621 | 94.481 (100 epoch) | 90.074 | 90.468 (700 epoch) | 97.796 | 98.640 (700 epoch) | 31.182 | 39.701 (250 epoch) | 35.944 | 44.125 (250 epoch) | 86.745 | 91.494 (250 epoch) | 20.333 | 25.164 (100 epoch) | 38.299 | 46.179 (50 epoch) | 12.509 | 15.755 (100 epoch) | 23.737 | 29.990 (250 epoch) | 2024-04-07 12:49:40,627 - ==> Total time: 5 days, 11:49:35 Eta: 14:38:50 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 12:52:10,457 - Train: 90.02% [487000/541000] [900.2/1000.0] [batch_t 1.455 (1.466)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 12:54:36,373 - Train: 90.04% [487100/541000] [900.4/1000.0] [batch_t 1.456 (1.459)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 12:57:02,253 - Train: 90.06% [487200/541000] [900.6/1000.0] [batch_t 1.456 (1.459)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 12:59:28,224 - Train: 90.07% [487300/541000] [900.7/1000.0] [batch_t 1.453 (1.460)] [data_t 0.003] [optim_t 1.449] [lr 0.000500] 2024-04-07 13:01:54,166 - Train: 90.09% [487400/541000] [900.9/1000.0] [batch_t 1.456 (1.459)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 13:02:53,979 - ==> Total time: 5 days, 12:02:48 Eta: 14:30:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 13:04:22,319 - Train: 90.11% [487500/541000] [901.1/1000.0] [batch_t 1.461 (1.465)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 13:06:48,203 - Train: 90.13% [487600/541000] [901.3/1000.0] [batch_t 1.456 (1.459)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 13:09:14,034 - Train: 90.15% [487700/541000] [901.5/1000.0] [batch_t 1.462 (1.458)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-07 13:11:39,896 - Train: 90.17% [487800/541000] [901.7/1000.0] [batch_t 1.461 (1.459)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 13:14:05,859 - Train: 90.18% [487900/541000] [901.8/1000.0] [batch_t 1.461 (1.460)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 13:16:05,468 - ==> Total time: 5 days, 12:15:59 Eta: 14:22:13 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 13:16:34,127 - Train: 90.20% [488000/541000] [902.0/1000.0] [batch_t 1.463 (1.487)] [data_t 0.003] [optim_t 1.460] [lr 0.000500] 2024-04-07 13:18:59,983 - Train: 90.22% [488100/541000] [902.2/1000.0] [batch_t 1.462 (1.458)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-07 13:21:25,792 - Train: 90.24% [488200/541000] [902.4/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 13:23:51,630 - Train: 90.26% [488300/541000] [902.6/1000.0] [batch_t 1.453 (1.458)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 13:26:17,473 - Train: 90.28% [488400/541000] [902.8/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 13:28:43,311 - Train: 90.30% [488500/541000] [903.0/1000.0] [batch_t 1.451 (1.458)] [data_t 0.003] [optim_t 1.447] [lr 0.000500] 2024-04-07 13:29:16,864 - ==> Total time: 5 days, 12:29:11 Eta: 14:13:53 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 13:31:11,375 - Train: 90.31% [488600/541000] [903.1/1000.0] [batch_t 1.456 (1.465)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 13:33:37,272 - Train: 90.33% [488700/541000] [903.3/1000.0] [batch_t 1.455 (1.459)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 13:36:03,111 - Train: 90.35% [488800/541000] [903.5/1000.0] [batch_t 1.456 (1.458)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 13:38:28,977 - Train: 90.37% [488900/541000] [903.7/1000.0] [batch_t 1.457 (1.459)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-07 13:40:54,757 - Train: 90.39% [489000/541000] [903.9/1000.0] [batch_t 1.462 (1.458)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-07 13:42:28,085 - ==> Total time: 5 days, 12:42:22 Eta: 14:05:33 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 13:43:22,792 - Train: 90.41% [489100/541000] [904.1/1000.0] [batch_t 1.457 (1.469)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-07 13:45:48,681 - Train: 90.43% [489200/541000] [904.3/1000.0] [batch_t 1.458 (1.459)] [data_t 0.003] [optim_t 1.455] [lr 0.000500] 2024-04-07 13:48:14,552 - Train: 90.44% [489300/541000] [904.4/1000.0] [batch_t 1.460 (1.459)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 13:50:40,375 - Train: 90.46% [489400/541000] [904.6/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 13:53:06,146 - Train: 90.48% [489500/541000] [904.8/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 13:55:31,982 - Train: 90.50% [489600/541000] [905.0/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 13:55:39,277 - ==> Total time: 5 days, 12:55:33 Eta: 13:57:12 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 13:57:59,891 - Train: 90.52% [489700/541000] [905.2/1000.0] [batch_t 1.461 (1.462)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 14:00:25,618 - Train: 90.54% [489800/541000] [905.4/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 14:02:51,436 - Train: 90.55% [489900/541000] [905.5/1000.0] [batch_t 1.456 (1.458)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 14:05:17,285 - Train: 90.57% [490000/541000] [905.7/1000.0] [batch_t 1.457 (1.458)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 14:07:43,085 - Train: 90.59% [490100/541000] [905.9/1000.0] [batch_t 1.463 (1.458)] [data_t 0.003] [optim_t 1.460] [lr 0.000500] 2024-04-07 14:08:50,159 - ==> Total time: 5 days, 13:08:44 Eta: 13:48:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 14:10:10,975 - Train: 90.61% [490200/541000] [906.1/1000.0] [batch_t 1.457 (1.465)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-07 14:12:28,894 - Train: 90.63% [490300/541000] [906.3/1000.0] [batch_t 0.608 (1.379)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 14:13:52,114 - Train: 90.65% [490400/541000] [906.5/1000.0] [batch_t 0.609 (0.832)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 14:15:25,982 - Train: 90.67% [490500/541000] [906.7/1000.0] [batch_t 0.612 (0.939)] [data_t 0.003] [optim_t 0.609] [lr 0.000500] 2024-04-07 14:17:35,113 - Train: 90.68% [490600/541000] [906.8/1000.0] [batch_t 0.608 (1.291)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 14:19:15,597 - ==> Total time: 5 days, 13:19:10 Eta: 13:40:12 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 14:19:36,671 - Train: 90.70% [490700/541000] [907.0/1000.0] [batch_t 2.158 (1.253)] [data_t 1.553] [optim_t 0.605] [lr 0.000500] 2024-04-07 14:22:17,025 - Train: 90.72% [490800/541000] [907.2/1000.0] [batch_t 2.381 (1.603)] [data_t 1.777] [optim_t 0.604] [lr 0.000500] 2024-04-07 14:24:27,764 - Train: 90.74% [490900/541000] [907.4/1000.0] [batch_t 2.949 (1.307)] [data_t 2.347] [optim_t 0.602] [lr 0.000500] 2024-04-07 14:27:04,982 - Train: 90.76% [491000/541000] [907.6/1000.0] [batch_t 0.608 (1.572)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 14:28:05,914 - Train: 90.78% [491100/541000] [907.8/1000.0] [batch_t 0.607 (0.609)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 14:29:06,734 - Train: 90.79% [491200/541000] [907.9/1000.0] [batch_t 0.606 (0.608)] [data_t 0.003] [optim_t 0.603] [lr 0.000500] 2024-04-07 14:29:23,750 - ==> Total time: 5 days, 13:29:18 Eta: 13:31:30 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 14:30:10,029 - Train: 90.81% [491300/541000] [908.1/1000.0] [batch_t 0.608 (0.616)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 14:31:10,814 - Train: 90.83% [491400/541000] [908.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 14:32:11,641 - Train: 90.85% [491500/541000] [908.5/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 14:33:12,708 - Train: 90.87% [491600/541000] [908.7/1000.0] [batch_t 0.609 (0.611)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 14:34:13,571 - Train: 90.89% [491700/541000] [908.9/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 14:34:55,560 - ==> Total time: 5 days, 13:34:50 Eta: 13:22:21 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 14:35:17,147 - Train: 90.91% [491800/541000] [909.1/1000.0] [batch_t 0.607 (0.629)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 14:36:17,952 - Train: 90.92% [491900/541000] [909.2/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 14:37:18,807 - Train: 90.94% [492000/541000] [909.4/1000.0] [batch_t 0.604 (0.608)] [data_t 0.003] [optim_t 0.602] [lr 0.000500] 2024-04-07 14:38:19,616 - Train: 90.96% [492100/541000] [909.6/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 14:39:20,429 - Train: 90.98% [492200/541000] [909.8/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 14:40:21,410 - Train: 91.00% [492300/541000] [910.0/1000.0] [batch_t 0.608 (0.610)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 14:40:27,504 - ==> Total time: 5 days, 13:40:21 Eta: 13:13:13 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 14:41:25,430 - Train: 91.02% [492400/541000] [910.2/1000.0] [batch_t 0.608 (0.618)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 14:42:26,245 - Train: 91.04% [492500/541000] [910.4/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 14:43:27,078 - Train: 91.05% [492600/541000] [910.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 14:44:27,885 - Train: 91.07% [492700/541000] [910.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 14:45:28,704 - Train: 91.09% [492800/541000] [910.9/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 14:45:59,703 - ==> Total time: 5 days, 13:45:54 Eta: 13:04:05 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 14:46:34,728 - Train: 91.11% [492900/541000] [911.1/1000.0] [batch_t 0.608 (0.628)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 14:47:35,526 - Train: 91.13% [493000/541000] [911.3/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 14:48:36,325 - Train: 91.15% [493100/541000] [911.5/1000.0] [batch_t 0.606 (0.608)] [data_t 0.003] [optim_t 0.603] [lr 0.000500] 2024-04-07 14:49:37,119 - Train: 91.16% [493200/541000] [911.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 14:50:37,929 - Train: 91.18% [493300/541000] [911.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 14:51:33,840 - ==> Total time: 5 days, 13:51:28 Eta: 12:54:57 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 14:51:41,300 - Train: 91.20% [493400/541000] [912.0/1000.0] [batch_t 0.608 (0.684)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 14:52:42,206 - Train: 91.22% [493500/541000] [912.2/1000.0] [batch_t 0.610 (0.609)] [data_t 0.003] [optim_t 0.607] [lr 0.000500] 2024-04-07 14:53:43,021 - Train: 91.24% [493600/541000] [912.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 14:54:43,834 - Train: 91.26% [493700/541000] [912.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 14:55:44,652 - Train: 91.28% [493800/541000] [912.8/1000.0] [batch_t 0.606 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 14:56:45,456 - Train: 91.29% [493900/541000] [912.9/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 14:57:05,516 - ==> Total time: 5 days, 13:56:59 Eta: 12:45:50 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 14:57:48,569 - Train: 91.31% [494000/541000] [913.1/1000.0] [batch_t 0.607 (0.615)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 14:58:49,370 - Train: 91.33% [494100/541000] [913.3/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 14:59:50,151 - Train: 91.35% [494200/541000] [913.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 15:00:50,970 - Train: 91.37% [494300/541000] [913.7/1000.0] [batch_t 0.611 (0.608)] [data_t 0.002] [optim_t 0.609] [lr 0.000500] 2024-04-07 15:01:51,784 - Train: 91.39% [494400/541000] [913.9/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.607] [lr 0.000500] 2024-04-07 15:02:36,749 - ==> Total time: 5 days, 14:02:31 Eta: 12:36:44 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 15:02:54,908 - Train: 91.40% [494500/541000] [914.0/1000.0] [batch_t 0.608 (0.630)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 15:03:55,676 - Train: 91.42% [494600/541000] [914.2/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 15:04:56,576 - Train: 91.44% [494700/541000] [914.4/1000.0] [batch_t 0.607 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 15:05:57,378 - Train: 91.46% [494800/541000] [914.6/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 15:06:58,142 - Train: 91.48% [494900/541000] [914.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 15:07:58,925 - Train: 91.50% [495000/541000] [915.0/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 15:08:08,054 - ==> Total time: 5 days, 14:08:02 Eta: 12:27:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 15:09:01,837 - Train: 91.52% [495100/541000] [915.2/1000.0] [batch_t 0.608 (0.612)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 15:10:02,738 - Train: 91.53% [495200/541000] [915.3/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 15:11:03,567 - Train: 91.55% [495300/541000] [915.5/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 15:12:04,376 - Train: 91.57% [495400/541000] [915.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 15:13:05,205 - Train: 91.59% [495500/541000] [915.9/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.000500] 2024-04-07 15:13:39,252 - ==> Total time: 5 days, 14:13:33 Eta: 12:18:32 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 15:14:08,137 - Train: 91.61% [495600/541000] [916.1/1000.0] [batch_t 0.608 (0.617)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 15:15:09,043 - Train: 91.63% [495700/541000] [916.3/1000.0] [batch_t 0.607 (0.609)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 15:16:09,856 - Train: 91.65% [495800/541000] [916.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 15:17:10,653 - Train: 91.66% [495900/541000] [916.6/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 15:18:11,448 - Train: 91.68% [496000/541000] [916.8/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 15:19:10,429 - ==> Total time: 5 days, 14:19:04 Eta: 12:09:26 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 15:19:14,511 - Train: 91.70% [496100/541000] [917.0/1000.0] [batch_t 0.611 (0.767)] [data_t 0.003] [optim_t 0.608] [lr 0.000500] 2024-04-07 15:20:15,452 - Train: 91.72% [496200/541000] [917.2/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 15:21:16,257 - Train: 91.74% [496300/541000] [917.4/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.000500] 2024-04-07 15:22:17,052 - Train: 91.76% [496400/541000] [917.6/1000.0] [batch_t 0.615 (0.608)] [data_t 0.002] [optim_t 0.612] [lr 0.000500] 2024-04-07 15:23:17,856 - Train: 91.77% [496500/541000] [917.7/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 15:24:18,657 - Train: 91.79% [496600/541000] [917.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 15:24:41,760 - ==> Total time: 5 days, 14:24:36 Eta: 12:00:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 15:25:21,610 - Train: 91.81% [496700/541000] [918.1/1000.0] [batch_t 0.608 (0.613)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 15:26:22,600 - Train: 91.83% [496800/541000] [918.3/1000.0] [batch_t 0.608 (0.610)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 15:27:23,413 - Train: 91.85% [496900/541000] [918.5/1000.0] [batch_t 0.613 (0.608)] [data_t 0.003] [optim_t 0.610] [lr 0.000500] 2024-04-07 15:28:24,251 - Train: 91.87% [497000/541000] [918.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 15:29:25,069 - Train: 91.89% [497100/541000] [918.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 15:30:13,144 - ==> Total time: 5 days, 14:30:07 Eta: 11:51:17 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 15:30:28,020 - Train: 91.90% [497200/541000] [919.0/1000.0] [batch_t 0.609 (0.625)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 15:31:28,859 - Train: 91.92% [497300/541000] [919.2/1000.0] [batch_t 0.606 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 15:32:29,825 - Train: 91.94% [497400/541000] [919.4/1000.0] [batch_t 0.610 (0.610)] [data_t 0.003] [optim_t 0.607] [lr 0.000500] 2024-04-07 15:33:30,637 - Train: 91.96% [497500/541000] [919.6/1000.0] [batch_t 0.611 (0.608)] [data_t 0.003] [optim_t 0.608] [lr 0.000500] 2024-04-07 15:34:31,484 - Train: 91.98% [497600/541000] [919.8/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 15:35:32,287 - Train: 92.00% [497700/541000] [920.0/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 15:35:44,442 - ==> Total time: 5 days, 14:35:38 Eta: 11:42:13 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 15:36:35,056 - Train: 92.01% [497800/541000] [920.1/1000.0] [batch_t 0.605 (0.612)] [data_t 0.002] [optim_t 0.603] [lr 0.000500] 2024-04-07 15:37:35,988 - Train: 92.03% [497900/541000] [920.3/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 15:38:36,771 - Train: 92.05% [498000/541000] [920.5/1000.0] [batch_t 0.610 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.000500] 2024-04-07 15:39:43,588 - Train: 92.07% [498100/541000] [920.7/1000.0] [batch_t 0.607 (0.668)] [data_t 0.002] [optim_t 0.604] [lr 0.000500] 2024-04-07 15:40:44,417 - Train: 92.09% [498200/541000] [920.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 15:41:21,938 - ==> Total time: 5 days, 14:41:16 Eta: 11:33:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 15:41:50,733 - Train: 92.11% [498300/541000] [921.1/1000.0] [batch_t 0.606 (0.654)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 15:42:58,507 - Train: 92.13% [498400/541000] [921.3/1000.0] [batch_t 0.610 (0.678)] [data_t 0.003] [optim_t 0.608] [lr 0.000500] 2024-04-07 15:44:05,422 - Train: 92.14% [498500/541000] [921.4/1000.0] [batch_t 0.606 (0.669)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 15:45:16,925 - Train: 92.16% [498600/541000] [921.6/1000.0] [batch_t 0.608 (0.715)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 15:46:25,525 - Train: 92.18% [498700/541000] [921.8/1000.0] [batch_t 0.608 (0.686)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 15:47:26,354 - Train: 92.20% [498800/541000] [922.0/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.000500] 2024-04-07 15:47:27,574 - ==> Total time: 5 days, 14:47:22 Eta: 11:24:10 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 15:48:42,695 - Train: 92.22% [498900/541000] [922.2/1000.0] [batch_t 0.609 (0.728)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 15:49:43,726 - Train: 92.24% [499000/541000] [922.4/1000.0] [batch_t 0.607 (0.610)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 15:50:44,849 - Train: 92.26% [499100/541000] [922.6/1000.0] [batch_t 0.610 (0.611)] [data_t 0.003] [optim_t 0.607] [lr 0.000500] 2024-04-07 15:51:45,690 - Train: 92.27% [499200/541000] [922.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 15:52:46,480 - Train: 92.29% [499300/541000] [922.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 15:53:12,641 - ==> Total time: 5 days, 14:53:07 Eta: 11:15:09 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 15:53:51,666 - Train: 92.31% [499400/541000] [923.1/1000.0] [batch_t 0.608 (0.639)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 15:54:52,487 - Train: 92.33% [499500/541000] [923.3/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 15:55:53,464 - Train: 92.35% [499600/541000] [923.5/1000.0] [batch_t 0.607 (0.610)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 15:56:54,311 - Train: 92.37% [499700/541000] [923.7/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.000500] 2024-04-07 15:57:55,187 - Train: 92.38% [499800/541000] [923.8/1000.0] [batch_t 0.611 (0.609)] [data_t 0.003] [optim_t 0.608] [lr 0.000500] 2024-04-07 15:58:46,345 - ==> Total time: 5 days, 14:58:40 Eta: 11:06:07 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 15:58:59,799 - Train: 92.40% [499900/541000] [924.0/1000.0] [batch_t 0.608 (0.661)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 16:00:00,617 - Train: 92.42% [500000/541000] [924.2/1000.0] [batch_t 0.606 (0.608)] [data_t 0.003] [optim_t 0.603] [lr 0.000500] 2024-04-07 16:01:01,549 - Train: 92.44% [500100/541000] [924.4/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 16:02:02,401 - Train: 92.46% [500200/541000] [924.6/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 16:03:03,235 - Train: 92.48% [500300/541000] [924.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 16:04:04,067 - Train: 92.50% [500400/541000] [925.0/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 16:04:19,283 - ==> Total time: 5 days, 15:04:13 Eta: 10:57:05 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 16:05:07,579 - Train: 92.51% [500500/541000] [925.1/1000.0] [batch_t 0.608 (0.619)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 16:06:08,407 - Train: 92.53% [500600/541000] [925.3/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 16:07:09,338 - Train: 92.55% [500700/541000] [925.5/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 16:08:10,171 - Train: 92.57% [500800/541000] [925.7/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 16:09:11,015 - Train: 92.59% [500900/541000] [925.9/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.000500] 2024-04-07 16:09:51,179 - ==> Total time: 5 days, 15:09:45 Eta: 10:48:04 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 16:10:14,111 - Train: 92.61% [501000/541000] [926.1/1000.0] [batch_t 0.609 (0.622)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 16:11:14,977 - Train: 92.62% [501100/541000] [926.2/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 16:12:15,872 - Train: 92.64% [501200/541000] [926.4/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 16:13:16,882 - Train: 92.66% [501300/541000] [926.6/1000.0] [batch_t 0.608 (0.610)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 16:14:17,799 - Train: 92.68% [501400/541000] [926.8/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 16:15:18,700 - Train: 92.70% [501500/541000] [927.0/1000.0] [batch_t 0.610 (0.609)] [data_t 0.003] [optim_t 0.607] [lr 0.000500] 2024-04-07 16:15:22,965 - ==> Total time: 5 days, 15:15:17 Eta: 10:39:04 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 16:16:21,888 - Train: 92.72% [501600/541000] [927.2/1000.0] [batch_t 0.607 (0.613)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 16:17:22,727 - Train: 92.74% [501700/541000] [927.4/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 16:18:23,695 - Train: 92.75% [501800/541000] [927.5/1000.0] [batch_t 0.612 (0.610)] [data_t 0.002] [optim_t 0.610] [lr 0.000500] 2024-04-07 16:19:24,534 - Train: 92.77% [501900/541000] [927.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 16:20:25,387 - Train: 92.79% [502000/541000] [927.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 16:20:54,629 - ==> Total time: 5 days, 15:20:49 Eta: 10:30:03 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 16:21:28,960 - Train: 92.81% [502100/541000] [928.1/1000.0] [batch_t 0.608 (0.620)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 16:22:29,864 - Train: 92.83% [502200/541000] [928.3/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 16:23:30,759 - Train: 92.85% [502300/541000] [928.5/1000.0] [batch_t 0.606 (0.609)] [data_t 0.002] [optim_t 0.604] [lr 0.000500] 2024-04-07 16:24:31,580 - Train: 92.87% [502400/541000] [928.7/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 16:25:32,561 - Train: 92.88% [502500/541000] [928.8/1000.0] [batch_t 0.609 (0.610)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 16:26:26,715 - ==> Total time: 5 days, 15:26:21 Eta: 10:21:04 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 16:26:35,838 - Train: 92.90% [502600/541000] [929.0/1000.0] [batch_t 0.607 (0.651)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 16:27:36,702 - Train: 92.92% [502700/541000] [929.2/1000.0] [batch_t 0.607 (0.609)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 16:28:37,548 - Train: 92.94% [502800/541000] [929.4/1000.0] [batch_t 0.606 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 16:29:38,429 - Train: 92.96% [502900/541000] [929.6/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 16:30:39,436 - Train: 92.98% [503000/541000] [929.8/1000.0] [batch_t 0.610 (0.610)] [data_t 0.003] [optim_t 0.607] [lr 0.000500] 2024-04-07 16:31:40,335 - Train: 92.99% [503100/541000] [929.9/1000.0] [batch_t 0.607 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 16:31:58,640 - ==> Total time: 5 days, 15:31:53 Eta: 10:12:04 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 16:32:43,357 - Train: 93.01% [503200/541000] [930.1/1000.0] [batch_t 0.608 (0.614)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 16:33:44,209 - Train: 93.03% [503300/541000] [930.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 16:34:45,023 - Train: 93.05% [503400/541000] [930.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 16:35:45,883 - Train: 93.07% [503500/541000] [930.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 16:36:46,925 - Train: 93.09% [503600/541000] [930.9/1000.0] [batch_t 0.607 (0.610)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 16:37:30,168 - ==> Total time: 5 days, 15:37:24 Eta: 10:03:05 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 16:37:50,186 - Train: 93.11% [503700/541000] [931.1/1000.0] [batch_t 0.607 (0.621)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 16:38:51,044 - Train: 93.12% [503800/541000] [931.2/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 16:39:51,919 - Train: 93.14% [503900/541000] [931.4/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 16:40:52,796 - Train: 93.16% [504000/541000] [931.6/1000.0] [batch_t 0.610 (0.609)] [data_t 0.003] [optim_t 0.607] [lr 0.000500] 2024-04-07 16:41:53,641 - Train: 93.18% [504100/541000] [931.8/1000.0] [batch_t 0.610 (0.608)] [data_t 0.003] [optim_t 0.608] [lr 0.000500] 2024-04-07 16:42:54,611 - Train: 93.20% [504200/541000] [932.0/1000.0] [batch_t 0.610 (0.610)] [data_t 0.003] [optim_t 0.607] [lr 0.000500] 2024-04-07 16:43:01,909 - ==> Total time: 5 days, 15:42:56 Eta: 9:54:07 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 16:43:59,697 - Train: 93.22% [504300/541000] [932.2/1000.0] [batch_t 0.609 (0.616)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 16:45:00,523 - Train: 93.23% [504400/541000] [932.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 16:46:01,405 - Train: 93.25% [504500/541000] [932.5/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 16:47:02,257 - Train: 93.27% [504600/541000] [932.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 16:48:03,131 - Train: 93.29% [504700/541000] [932.9/1000.0] [batch_t 0.609 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 16:48:35,395 - ==> Total time: 5 days, 15:48:29 Eta: 9:45:09 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 16:49:06,636 - Train: 93.31% [504800/541000] [933.1/1000.0] [batch_t 0.606 (0.618)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 16:50:07,443 - Train: 93.33% [504900/541000] [933.3/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 16:51:08,277 - Train: 93.35% [505000/541000] [933.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 16:52:09,050 - Train: 93.36% [505100/541000] [933.6/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 16:53:09,843 - Train: 93.38% [505200/541000] [933.8/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 16:54:07,010 - ==> Total time: 5 days, 15:54:01 Eta: 9:36:11 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 16:54:13,007 - Train: 93.40% [505300/541000] [934.0/1000.0] [batch_t 0.607 (0.701)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 16:55:13,808 - Train: 93.42% [505400/541000] [934.2/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 16:56:14,704 - Train: 93.44% [505500/541000] [934.4/1000.0] [batch_t 0.607 (0.609)] [data_t 0.002] [optim_t 0.604] [lr 0.000500] 2024-04-07 16:57:15,485 - Train: 93.46% [505600/541000] [934.6/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.604] [lr 0.000500] 2024-04-07 16:58:16,262 - Train: 93.48% [505700/541000] [934.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 16:59:17,029 - Train: 93.49% [505800/541000] [934.9/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 16:59:38,296 - ==> Total time: 5 days, 15:59:32 Eta: 9:27:14 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 17:00:20,122 - Train: 93.51% [505900/541000] [935.1/1000.0] [batch_t 0.608 (0.615)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 17:01:21,029 - Train: 93.53% [506000/541000] [935.3/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 17:02:21,804 - Train: 93.55% [506100/541000] [935.5/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 17:03:22,585 - Train: 93.57% [506200/541000] [935.7/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 17:04:23,376 - Train: 93.59% [506300/541000] [935.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 17:05:09,568 - ==> Total time: 5 days, 16:05:04 Eta: 9:18:17 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 17:05:26,246 - Train: 93.60% [506400/541000] [936.0/1000.0] [batch_t 0.609 (0.628)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 17:06:27,001 - Train: 93.62% [506500/541000] [936.2/1000.0] [batch_t 0.608 (0.607)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 17:07:27,773 - Train: 93.64% [506600/541000] [936.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 17:08:28,724 - Train: 93.66% [506700/541000] [936.6/1000.0] [batch_t 0.607 (0.609)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 17:09:29,503 - Train: 93.68% [506800/541000] [936.8/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.603] [lr 0.000500] 2024-04-07 17:10:30,290 - Train: 93.70% [506900/541000] [937.0/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 17:10:40,625 - ==> Total time: 5 days, 16:10:35 Eta: 9:09:21 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 17:11:33,007 - Train: 93.72% [507000/541000] [937.2/1000.0] [batch_t 0.605 (0.612)] [data_t 0.002] [optim_t 0.603] [lr 0.000500] 2024-04-07 17:12:33,776 - Train: 93.73% [507100/541000] [937.3/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 17:13:34,692 - Train: 93.75% [507200/541000] [937.5/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 17:14:35,521 - Train: 93.77% [507300/541000] [937.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 17:15:36,333 - Train: 93.79% [507400/541000] [937.9/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 17:16:11,594 - ==> Total time: 5 days, 16:16:06 Eta: 9:00:25 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 17:16:39,375 - Train: 93.81% [507500/541000] [938.1/1000.0] [batch_t 0.608 (0.619)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 17:17:40,167 - Train: 93.83% [507600/541000] [938.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 17:18:40,972 - Train: 93.84% [507700/541000] [938.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 17:19:41,916 - Train: 93.86% [507800/541000] [938.6/1000.0] [batch_t 0.610 (0.609)] [data_t 0.003] [optim_t 0.608] [lr 0.000500] 2024-04-07 17:20:42,721 - Train: 93.88% [507900/541000] [938.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 17:21:42,967 - ==> Total time: 5 days, 16:21:37 Eta: 8:51:30 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 17:21:45,717 - Train: 93.90% [508000/541000] [939.0/1000.0] [batch_t 0.934 (0.934)] [data_t 0.315] [optim_t 0.619] [lr 0.000500] 2024-04-07 17:22:46,538 - Train: 93.92% [508100/541000] [939.2/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 17:23:47,338 - Train: 93.94% [508200/541000] [939.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 17:24:48,257 - Train: 93.96% [508300/541000] [939.6/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 17:25:49,038 - Train: 93.97% [508400/541000] [939.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 17:26:49,822 - Train: 93.99% [508500/541000] [939.9/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 17:27:14,142 - ==> Total time: 5 days, 16:27:08 Eta: 8:42:35 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 17:27:52,960 - Train: 94.01% [508600/541000] [940.1/1000.0] [batch_t 0.608 (0.617)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 17:28:53,734 - Train: 94.03% [508700/541000] [940.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 17:29:54,510 - Train: 94.05% [508800/541000] [940.5/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 17:30:55,337 - Train: 94.07% [508900/541000] [940.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 17:31:56,297 - Train: 94.09% [509000/541000] [940.9/1000.0] [batch_t 0.606 (0.609)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 17:32:45,542 - ==> Total time: 5 days, 16:32:39 Eta: 8:33:40 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 17:32:59,111 - Train: 94.10% [509100/541000] [941.0/1000.0] [batch_t 0.608 (0.630)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 17:33:59,957 - Train: 94.12% [509200/541000] [941.2/1000.0] [batch_t 0.607 (0.608)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 17:35:00,777 - Train: 94.14% [509300/541000] [941.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 17:36:01,762 - Train: 94.16% [509400/541000] [941.6/1000.0] [batch_t 0.609 (0.610)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 17:37:02,610 - Train: 94.18% [509500/541000] [941.8/1000.0] [batch_t 0.613 (0.608)] [data_t 0.003] [optim_t 0.610] [lr 0.000500] 2024-04-07 17:38:03,465 - Train: 94.20% [509600/541000] [942.0/1000.0] [batch_t 0.609 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 17:38:16,859 - ==> Total time: 5 days, 16:38:11 Eta: 8:24:46 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 17:39:06,344 - Train: 94.21% [509700/541000] [942.1/1000.0] [batch_t 0.608 (0.613)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 17:40:07,200 - Train: 94.23% [509800/541000] [942.3/1000.0] [batch_t 0.611 (0.608)] [data_t 0.003] [optim_t 0.608] [lr 0.000500] 2024-04-07 17:41:08,040 - Train: 94.25% [509900/541000] [942.5/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 17:42:08,912 - Train: 94.27% [510000/541000] [942.7/1000.0] [batch_t 0.612 (0.609)] [data_t 0.003] [optim_t 0.610] [lr 0.000500] 2024-04-07 17:43:09,894 - Train: 94.29% [510100/541000] [942.9/1000.0] [batch_t 0.607 (0.610)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 17:43:48,256 - ==> Total time: 5 days, 16:43:42 Eta: 8:15:52 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 17:44:12,786 - Train: 94.31% [510200/541000] [943.1/1000.0] [batch_t 0.608 (0.618)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 17:45:13,609 - Train: 94.33% [510300/541000] [943.3/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 17:46:14,450 - Train: 94.34% [510400/541000] [943.4/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 17:47:15,276 - Train: 94.36% [510500/541000] [943.6/1000.0] [batch_t 0.607 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 17:48:16,100 - Train: 94.38% [510600/541000] [943.8/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 17:49:16,971 - Train: 94.40% [510700/541000] [944.0/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 17:49:19,407 - ==> Total time: 5 days, 16:49:13 Eta: 8:06:59 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 17:50:20,382 - Train: 94.42% [510800/541000] [944.2/1000.0] [batch_t 0.608 (0.612)] [data_t 0.002] [optim_t 0.605] [lr 0.000500] 2024-04-07 17:51:21,244 - Train: 94.44% [510900/541000] [944.4/1000.0] [batch_t 0.606 (0.609)] [data_t 0.003] [optim_t 0.603] [lr 0.000500] 2024-04-07 17:52:22,036 - Train: 94.45% [511000/541000] [944.5/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.603] [lr 0.000500] 2024-04-07 17:53:22,874 - Train: 94.47% [511100/541000] [944.7/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 17:54:23,707 - Train: 94.49% [511200/541000] [944.9/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 17:54:51,094 - ==> Total time: 5 days, 16:54:45 Eta: 7:58:06 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 17:55:26,697 - Train: 94.51% [511300/541000] [945.1/1000.0] [batch_t 0.608 (0.615)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 17:56:27,689 - Train: 94.53% [511400/541000] [945.3/1000.0] [batch_t 0.608 (0.610)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 17:57:28,541 - Train: 94.55% [511500/541000] [945.5/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 17:58:29,395 - Train: 94.57% [511600/541000] [945.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 17:59:30,238 - Train: 94.58% [511700/541000] [945.8/1000.0] [batch_t 0.610 (0.608)] [data_t 0.003] [optim_t 0.608] [lr 0.000500] 2024-04-07 18:00:22,585 - ==> Total time: 5 days, 17:00:17 Eta: 7:49:14 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 18:00:33,252 - Train: 94.60% [511800/541000] [946.0/1000.0] [batch_t 0.609 (0.638)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 18:01:34,069 - Train: 94.62% [511900/541000] [946.2/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 18:02:34,992 - Train: 94.64% [512000/541000] [946.4/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 18:03:35,773 - Train: 94.66% [512100/541000] [946.6/1000.0] [batch_t 0.606 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 18:04:36,604 - Train: 94.68% [512200/541000] [946.8/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 18:05:37,418 - Train: 94.70% [512300/541000] [947.0/1000.0] [batch_t 0.608 (0.608)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 18:05:53,861 - ==> Total time: 5 days, 17:05:48 Eta: 7:40:22 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 18:06:40,341 - Train: 94.71% [512400/541000] [947.1/1000.0] [batch_t 0.609 (0.614)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 18:07:41,315 - Train: 94.73% [512500/541000] [947.3/1000.0] [batch_t 0.607 (0.610)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 18:08:42,132 - Train: 94.75% [512600/541000] [947.5/1000.0] [batch_t 0.606 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 18:09:42,966 - Train: 94.77% [512700/541000] [947.7/1000.0] [batch_t 0.606 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 18:10:43,826 - Train: 94.79% [512800/541000] [947.9/1000.0] [batch_t 0.608 (0.609)] [data_t 0.002] [optim_t 0.606] [lr 0.000500] 2024-04-07 18:11:25,192 - ==> Total time: 5 days, 17:11:19 Eta: 7:31:30 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 18:11:47,039 - Train: 94.81% [512900/541000] [948.1/1000.0] [batch_t 0.608 (0.623)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 18:12:47,865 - Train: 94.82% [513000/541000] [948.2/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 18:13:48,796 - Train: 94.84% [513100/541000] [948.4/1000.0] [batch_t 0.726 (0.609)] [data_t 0.003] [optim_t 0.723] [lr 0.000500] 2024-04-07 18:14:49,654 - Train: 94.86% [513200/541000] [948.6/1000.0] [batch_t 0.606 (0.608)] [data_t 0.003] [optim_t 0.604] [lr 0.000500] 2024-04-07 18:15:50,491 - Train: 94.88% [513300/541000] [948.8/1000.0] [batch_t 0.610 (0.608)] [data_t 0.003] [optim_t 0.607] [lr 0.000500] 2024-04-07 18:16:51,357 - Train: 94.90% [513400/541000] [949.0/1000.0] [batch_t 0.608 (0.609)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 18:16:56,837 - ==> Total time: 5 days, 17:16:51 Eta: 7:22:39 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 18:17:54,215 - Train: 94.92% [513500/541000] [949.2/1000.0] [batch_t 0.609 (0.612)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 18:18:55,208 - Train: 94.94% [513600/541000] [949.4/1000.0] [batch_t 0.608 (0.610)] [data_t 0.003] [optim_t 0.605] [lr 0.000500] 2024-04-07 18:19:55,994 - Train: 94.95% [513700/541000] [949.5/1000.0] [batch_t 0.609 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 18:20:56,790 - Train: 94.97% [513800/541000] [949.7/1000.0] [batch_t 0.608 (0.608)] [data_t 0.003] [optim_t 0.606] [lr 0.000500] 2024-04-07 18:21:57,611 - Train: 94.99% [513900/541000] [949.9/1000.0] [batch_t 0.606 (0.608)] [data_t 0.002] [optim_t 0.603] [lr 0.000500] 2024-04-07 18:22:39,976 - Test: 36.76% [50/136] [batch_t 0.213 (0.225)] 2024-04-07 18:22:50,678 - Test: 73.53% [100/136] [batch_t 0.213 (0.220)] 2024-04-07 18:22:58,224 - Test: 100.00% [136/136] [batch_t 0.055 (0.217)] 2024-04-07 18:23:52,752 - ==> Metric Time for candle : 0.002 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 11.202 (mAUROC_px) 8.523 (mAP_px) 1.076 (mF1_max_px) 29.752 (mAUPRO_px) 0.365 (mF1_px_0.2_0.8_0.1) 0.362 (mAcc_px_0.2_0.8_0.1) 0.367 (mIoU_px_0.2_0.8_0.1) 1.065 (mIoU_max_px) 2024-04-07 18:24:30,698 - ==> Metric Time for capsules : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 8.294 (mAUROC_px) 6.386 (mAP_px) 0.788 (mF1_max_px) 20.643 (mAUPRO_px) 0.271 (mF1_px_0.2_0.8_0.1) 0.264 (mAcc_px_0.2_0.8_0.1) 0.271 (mIoU_px_0.2_0.8_0.1) 0.791 (mIoU_max_px) 2024-04-07 18:25:06,749 - ==> Metric Time for cashew : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.783 (mAUROC_px) 5.996 (mAP_px) 0.744 (mF1_max_px) 19.845 (mAUPRO_px) 0.250 (mF1_px_0.2_0.8_0.1) 0.247 (mAcc_px_0.2_0.8_0.1) 0.243 (mIoU_px_0.2_0.8_0.1) 0.722 (mIoU_max_px) 2024-04-07 18:25:45,485 - ==> Metric Time for chewinggum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.679 (mAUROC_px) 5.890 (mAP_px) 0.760 (mF1_max_px) 22.666 (mAUPRO_px) 0.250 (mF1_px_0.2_0.8_0.1) 0.255 (mAcc_px_0.2_0.8_0.1) 0.255 (mIoU_px_0.2_0.8_0.1) 0.763 (mIoU_max_px) 2024-04-07 18:26:22,890 - ==> Metric Time for fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.765 (mAUROC_px) 5.743 (mAP_px) 0.720 (mF1_max_px) 21.509 (mAUPRO_px) 0.244 (mF1_px_0.2_0.8_0.1) 0.244 (mAcc_px_0.2_0.8_0.1) 0.246 (mIoU_px_0.2_0.8_0.1) 0.707 (mIoU_max_px) 2024-04-07 18:27:10,394 - ==> Metric Time for macaroni1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.673 (mAUROC_px) 7.976 (mAP_px) 0.971 (mF1_max_px) 25.687 (mAUPRO_px) 0.322 (mF1_px_0.2_0.8_0.1) 0.323 (mAcc_px_0.2_0.8_0.1) 0.327 (mIoU_px_0.2_0.8_0.1) 0.945 (mIoU_max_px) 2024-04-07 18:27:57,578 - ==> Metric Time for macaroni2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.551 (mAUROC_px) 7.915 (mAP_px) 0.951 (mF1_max_px) 25.555 (mAUPRO_px) 0.326 (mF1_px_0.2_0.8_0.1) 0.324 (mAcc_px_0.2_0.8_0.1) 0.329 (mIoU_px_0.2_0.8_0.1) 0.946 (mIoU_max_px) 2024-04-07 18:28:46,819 - ==> Metric Time for pcb1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.866 (mAUROC_px) 8.173 (mAP_px) 0.958 (mF1_max_px) 27.035 (mAUPRO_px) 0.325 (mF1_px_0.2_0.8_0.1) 0.327 (mAcc_px_0.2_0.8_0.1) 0.327 (mIoU_px_0.2_0.8_0.1) 0.945 (mIoU_max_px) 2024-04-07 18:29:35,321 - ==> Metric Time for pcb2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.773 (mAUROC_px) 8.181 (mAP_px) 0.958 (mF1_max_px) 26.388 (mAUPRO_px) 0.325 (mF1_px_0.2_0.8_0.1) 0.328 (mAcc_px_0.2_0.8_0.1) 0.326 (mIoU_px_0.2_0.8_0.1) 0.940 (mIoU_max_px) 2024-04-07 18:30:23,193 - ==> Metric Time for pcb3 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.548 (mAUROC_px) 7.974 (mAP_px) 0.956 (mF1_max_px) 26.185 (mAUPRO_px) 0.326 (mF1_px_0.2_0.8_0.1) 0.326 (mAcc_px_0.2_0.8_0.1) 0.328 (mIoU_px_0.2_0.8_0.1) 0.948 (mIoU_max_px) 2024-04-07 18:31:13,544 - ==> Metric Time for pcb4 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.836 (mAUROC_px) 8.025 (mAP_px) 0.946 (mF1_max_px) 28.326 (mAUPRO_px) 0.327 (mF1_px_0.2_0.8_0.1) 0.326 (mAcc_px_0.2_0.8_0.1) 0.330 (mIoU_px_0.2_0.8_0.1) 0.952 (mIoU_max_px) 2024-04-07 18:31:50,032 - ==> Metric Time for pipe_fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.863 (mAUROC_px) 5.829 (mAP_px) 0.699 (mF1_max_px) 20.466 (mAUPRO_px) 0.238 (mF1_px_0.2_0.8_0.1) 0.238 (mAcc_px_0.2_0.8_0.1) 0.239 (mIoU_px_0.2_0.8_0.1) 0.693 (mIoU_max_px) 2024-04-07 18:31:50,053 - | Name | mAUROC_sp_max | mAUROC_sp_max (Max) | mAP_sp_max | mAP_sp_max (Max) | mF1_max_sp_max | mF1_max_sp_max (Max) | mAUROC_px | mAUROC_px (Max) | mAP_px | mAP_px (Max) | mF1_max_px | mF1_max_px (Max) | mAUPRO_px | mAUPRO_px (Max) | mF1_px_0.2_0.8_0.1 | mF1_px_0.2_0.8_0.1 (Max) | mAcc_px_0.2_0.8_0.1 | mAcc_px_0.2_0.8_0.1 (Max) | mIoU_px_0.2_0.8_0.1 | mIoU_px_0.2_0.8_0.1 (Max) | mIoU_max_px | mIoU_max_px (Max) | |:----------:|:---------------:|:---------------------:|:------------:|:------------------:|:----------------:|:----------------------:|:-----------:|:------------------:|:--------:|:------------------:|:------------:|:------------------:|:-----------:|:------------------:|:--------------------:|:--------------------------:|:---------------------:|:---------------------------:|:---------------------:|:---------------------------:|:-------------:|:-------------------:| | candle | 96.200 | 97.100 (50 epoch) | 96.429 | 97.346 (50 epoch) | 89.756 | 91.542 (50 epoch) | 99.071 | 99.098 (250 epoch) | 20.760 | 23.463 (150 epoch) | 30.731 | 32.654 (200 epoch) | 95.605 | 96.087 (300 epoch) | 20.135 | 21.331 (500 epoch) | 38.270 | 41.526 (650 epoch) | 11.456 | 12.185 (500 epoch) | 18.155 | 19.513 (200 epoch) | | capsules | 74.167 | 91.750 (100 epoch) | 76.087 | 94.964 (100 epoch) | 82.096 | 88.780 (100 epoch) | 95.929 | 99.193 (700 epoch) | 18.313 | 61.256 (100 epoch) | 32.467 | 59.868 (250 epoch) | 69.349 | 91.811 (100 epoch) | 11.562 | 33.186 (100 epoch) | 40.497 | 62.576 (50 epoch) | 6.505 | 21.701 (100 epoch) | 19.380 | 42.722 (250 epoch) | | cashew | 94.220 | 94.940 (800 epoch) | 96.987 | 97.492 (150 epoch) | 92.233 | 92.929 (700 epoch) | 95.447 | 96.185 (700 epoch) | 28.181 | 50.952 (250 epoch) | 29.743 | 54.438 (400 epoch) | 76.614 | 89.185 (350 epoch) | 19.818 | 28.352 (350 epoch) | 30.233 | 34.522 (150 epoch) | 11.305 | 17.797 (350 epoch) | 17.469 | 37.398 (400 epoch) | | chewinggum | 96.801 | 97.802 (300 epoch) | 98.531 | 98.939 (300 epoch) | 95.385 | 95.385 (950 epoch) | 97.926 | 98.145 (100 epoch) | 42.084 | 59.823 (50 epoch) | 48.570 | 61.088 (50 epoch) | 79.320 | 80.312 (400 epoch) | 27.557 | 35.748 (50 epoch) | 64.381 | 65.723 (800 epoch) | 16.988 | 23.754 (50 epoch) | 32.074 | 43.976 (50 epoch) | | fryum | 96.360 | 96.560 (650 epoch) | 98.205 | 98.273 (650 epoch) | 93.000 | 93.897 (650 epoch) | 97.316 | 97.316 (950 epoch) | 51.416 | 52.248 (750 epoch) | 53.667 | 53.679 (900 epoch) | 92.166 | 92.340 (550 epoch) | 25.404 | 26.511 (550 epoch) | 24.920 | 28.594 (450 epoch) | 15.935 | 16.529 (550 epoch) | 36.675 | 36.686 (900 epoch) | | macaroni1 | 91.153 | 92.104 (500 epoch) | 86.445 | 90.505 (500 epoch) | 83.019 | 84.314 (750 epoch) | 99.279 | 99.604 (650 epoch) | 4.511 | 20.435 (700 epoch) | 7.022 | 29.513 (650 epoch) | 94.744 | 96.221 (650 epoch) | 5.722 | 16.200 (700 epoch) | 36.168 | 40.765 (50 epoch) | 2.947 | 9.078 (700 epoch) | 3.639 | 17.311 (650 epoch) | | macaroni2 | 75.239 | 82.589 (50 epoch) | 62.103 | 81.196 (50 epoch) | 75.349 | 77.833 (750 epoch) | 93.305 | 99.452 (100 epoch) | 0.327 | 11.272 (50 epoch) | 2.875 | 17.568 (50 epoch) | 73.839 | 96.253 (150 epoch) | 0.710 | 10.086 (50 epoch) | 30.911 | 54.584 (450 epoch) | 0.357 | 5.433 (50 epoch) | 1.458 | 9.630 (50 epoch) | | pcb1 | 96.680 | 96.880 (850 epoch) | 95.228 | 95.627 (850 epoch) | 94.686 | 94.737 (300 epoch) | 99.800 | 99.806 (700 epoch) | 81.337 | 83.141 (450 epoch) | 74.087 | 75.588 (450 epoch) | 94.723 | 94.908 (900 epoch) | 46.901 | 48.256 (700 epoch) | 53.362 | 62.362 (50 epoch) | 32.771 | 34.013 (700 epoch) | 58.840 | 60.756 (450 epoch) | | pcb2 | 96.200 | 96.200 (950 epoch) | 95.984 | 95.984 (950 epoch) | 91.000 | 91.542 (800 epoch) | 99.019 | 99.023 (700 epoch) | 12.631 | 14.130 (50 epoch) | 23.793 | 23.977 (250 epoch) | 89.231 | 91.576 (700 epoch) | 15.748 | 16.150 (200 epoch) | 43.215 | 47.820 (150 epoch) | 8.672 | 8.892 (200 epoch) | 13.503 | 13.621 (250 epoch) | | pcb3 | 95.921 | 95.921 (950 epoch) | 96.142 | 96.142 (950 epoch) | 90.291 | 90.291 (850 epoch) | 99.124 | 99.174 (750 epoch) | 16.785 | 20.249 (150 epoch) | 26.695 | 28.078 (450 epoch) | 91.920 | 92.258 (850 epoch) | 18.747 | 20.042 (150 epoch) | 42.834 | 48.341 (50 epoch) | 10.503 | 11.295 (150 epoch) | 15.404 | 16.332 (450 epoch) | | pcb4 | 99.733 | 99.901 (100 epoch) | 99.735 | 99.901 (100 epoch) | 97.537 | 99.000 (50 epoch) | 98.644 | 98.746 (650 epoch) | 44.840 | 47.966 (250 epoch) | 45.267 | 47.709 (250 epoch) | 90.167 | 91.033 (850 epoch) | 23.406 | 26.925 (200 epoch) | 29.490 | 39.716 (50 epoch) | 14.288 | 16.537 (200 epoch) | 29.255 | 31.328 (250 epoch) | | pipe_fryum | 99.020 | 99.040 (900 epoch) | 99.494 | 99.503 (900 epoch) | 97.087 | 97.561 (200 epoch) | 98.838 | 99.114 (450 epoch) | 49.588 | 56.529 (700 epoch) | 51.428 | 58.733 (450 epoch) | 91.816 | 95.525 (150 epoch) | 28.065 | 34.684 (250 epoch) | 31.469 | 43.014 (100 epoch) | 17.777 | 22.399 (250 epoch) | 34.615 | 41.576 (450 epoch) | | Avg | 92.641 | 94.251 (100 epoch) | 91.781 | 94.481 (100 epoch) | 90.120 | 90.468 (700 epoch) | 97.808 | 98.640 (700 epoch) | 30.898 | 39.701 (250 epoch) | 35.529 | 44.125 (250 epoch) | 86.625 | 91.494 (250 epoch) | 20.315 | 25.164 (100 epoch) | 38.813 | 46.179 (50 epoch) | 12.459 | 15.755 (100 epoch) | 23.372 | 29.990 (250 epoch) | 2024-04-07 18:31:50,406 - ==> Total time: 5 days, 17:31:44 Eta: 7:14:18 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 18:33:05,882 - Train: 95.01% [514000/541000] [950.1/1000.0] [batch_t 1.456 (1.464)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 18:35:31,660 - Train: 95.03% [514100/541000] [950.3/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 18:37:57,371 - Train: 95.05% [514200/541000] [950.5/1000.0] [batch_t 1.455 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 18:40:22,981 - Train: 95.06% [514300/541000] [950.6/1000.0] [batch_t 1.456 (1.456)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 18:42:48,589 - Train: 95.08% [514400/541000] [950.8/1000.0] [batch_t 1.456 (1.456)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 18:45:01,102 - ==> Total time: 5 days, 17:44:55 Eta: 7:05:50 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 18:45:18,280 - Train: 95.10% [514500/541000] [951.0/1000.0] [batch_t 1.461 (1.548)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 18:47:43,834 - Train: 95.12% [514600/541000] [951.2/1000.0] [batch_t 1.456 (1.455)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 18:50:09,467 - Train: 95.14% [514700/541000] [951.4/1000.0] [batch_t 1.460 (1.456)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 18:52:35,027 - Train: 95.16% [514800/541000] [951.6/1000.0] [batch_t 1.457 (1.456)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-07 18:55:00,633 - Train: 95.18% [514900/541000] [951.8/1000.0] [batch_t 1.456 (1.456)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 18:57:26,236 - Train: 95.19% [515000/541000] [951.9/1000.0] [batch_t 1.460 (1.456)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 18:58:12,874 - ==> Total time: 5 days, 17:58:07 Eta: 6:57:23 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 18:59:54,005 - Train: 95.21% [515100/541000] [952.1/1000.0] [batch_t 1.455 (1.461)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 19:02:19,658 - Train: 95.23% [515200/541000] [952.3/1000.0] [batch_t 1.452 (1.456)] [data_t 0.003] [optim_t 1.448] [lr 0.000500] 2024-04-07 19:04:45,332 - Train: 95.25% [515300/541000] [952.5/1000.0] [batch_t 1.451 (1.457)] [data_t 0.003] [optim_t 1.448] [lr 0.000500] 2024-04-07 19:07:10,954 - Train: 95.27% [515400/541000] [952.7/1000.0] [batch_t 1.459 (1.456)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-07 19:09:36,555 - Train: 95.29% [515500/541000] [952.9/1000.0] [batch_t 1.461 (1.456)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 19:11:22,908 - ==> Total time: 5 days, 18:11:17 Eta: 6:48:54 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 19:12:04,414 - Train: 95.30% [515600/541000] [953.0/1000.0] [batch_t 1.460 (1.472)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 19:14:29,956 - Train: 95.32% [515700/541000] [953.2/1000.0] [batch_t 1.459 (1.455)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-07 19:16:55,645 - Train: 95.34% [515800/541000] [953.4/1000.0] [batch_t 1.466 (1.457)] [data_t 0.003] [optim_t 1.463] [lr 0.000500] 2024-04-07 19:19:21,369 - Train: 95.36% [515900/541000] [953.6/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 19:21:47,117 - Train: 95.38% [516000/541000] [953.8/1000.0] [batch_t 1.459 (1.457)] [data_t 0.004] [optim_t 1.455] [lr 0.000500] 2024-04-07 19:24:12,851 - Train: 95.40% [516100/541000] [954.0/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 19:24:33,239 - ==> Total time: 5 days, 18:24:27 Eta: 6:40:25 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 19:26:40,501 - Train: 95.42% [516200/541000] [954.2/1000.0] [batch_t 1.453 (1.461)] [data_t 0.003] [optim_t 1.450] [lr 0.000500] 2024-04-07 19:29:06,257 - Train: 95.43% [516300/541000] [954.3/1000.0] [batch_t 1.454 (1.457)] [data_t 0.003] [optim_t 1.450] [lr 0.000500] 2024-04-07 19:31:31,925 - Train: 95.45% [516400/541000] [954.5/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 19:33:57,661 - Train: 95.47% [516500/541000] [954.7/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 19:36:23,309 - Train: 95.49% [516600/541000] [954.9/1000.0] [batch_t 1.456 (1.456)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 19:37:43,510 - ==> Total time: 5 days, 18:37:37 Eta: 6:31:55 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 19:38:51,292 - Train: 95.51% [516700/541000] [955.1/1000.0] [batch_t 1.456 (1.463)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 19:41:17,179 - Train: 95.53% [516800/541000] [955.3/1000.0] [batch_t 1.458 (1.459)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-07 19:43:43,013 - Train: 95.55% [516900/541000] [955.5/1000.0] [batch_t 1.466 (1.458)] [data_t 0.003] [optim_t 1.463] [lr 0.000500] 2024-04-07 19:46:08,840 - Train: 95.56% [517000/541000] [955.6/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 19:48:34,634 - Train: 95.58% [517100/541000] [955.8/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 19:50:54,673 - ==> Total time: 5 days, 18:50:49 Eta: 6:23:25 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 19:51:02,758 - Train: 95.60% [517200/541000] [956.0/1000.0] [batch_t 1.452 (1.574)] [data_t 0.003] [optim_t 1.449] [lr 0.000500] 2024-04-07 19:53:28,544 - Train: 95.62% [517300/541000] [956.2/1000.0] [batch_t 1.456 (1.458)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 19:55:54,261 - Train: 95.64% [517400/541000] [956.4/1000.0] [batch_t 1.452 (1.457)] [data_t 0.003] [optim_t 1.448] [lr 0.000500] 2024-04-07 19:58:20,118 - Train: 95.66% [517500/541000] [956.6/1000.0] [batch_t 1.465 (1.458)] [data_t 0.003] [optim_t 1.461] [lr 0.000500] 2024-04-07 20:00:45,950 - Train: 95.67% [517600/541000] [956.7/1000.0] [batch_t 1.463 (1.458)] [data_t 0.003] [optim_t 1.460] [lr 0.000500] 2024-04-07 20:03:11,720 - Train: 95.69% [517700/541000] [956.9/1000.0] [batch_t 1.456 (1.458)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 20:04:05,688 - ==> Total time: 5 days, 19:04:00 Eta: 6:14:54 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 20:05:40,153 - Train: 95.71% [517800/541000] [957.1/1000.0] [batch_t 1.461 (1.465)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 20:08:05,947 - Train: 95.73% [517900/541000] [957.3/1000.0] [batch_t 1.455 (1.458)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 20:10:31,824 - Train: 95.75% [518000/541000] [957.5/1000.0] [batch_t 1.461 (1.459)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 20:12:57,601 - Train: 95.77% [518100/541000] [957.7/1000.0] [batch_t 1.455 (1.458)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 20:15:23,471 - Train: 95.79% [518200/541000] [957.9/1000.0] [batch_t 1.457 (1.459)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 20:17:17,132 - ==> Total time: 5 days, 19:17:11 Eta: 6:06:23 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 20:17:51,535 - Train: 95.80% [518300/541000] [958.0/1000.0] [batch_t 1.460 (1.471)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 20:20:17,218 - Train: 95.82% [518400/541000] [958.2/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 20:22:43,005 - Train: 95.84% [518500/541000] [958.4/1000.0] [batch_t 1.459 (1.458)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-07 20:25:08,829 - Train: 95.86% [518600/541000] [958.6/1000.0] [batch_t 1.451 (1.458)] [data_t 0.003] [optim_t 1.448] [lr 0.000500] 2024-04-07 20:27:34,553 - Train: 95.88% [518700/541000] [958.8/1000.0] [batch_t 1.459 (1.457)] [data_t 0.003] [optim_t 1.455] [lr 0.000500] 2024-04-07 20:30:00,334 - Train: 95.90% [518800/541000] [959.0/1000.0] [batch_t 1.462 (1.458)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-07 20:30:28,051 - ==> Total time: 5 days, 19:30:22 Eta: 5:57:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 20:32:28,579 - Train: 95.91% [518900/541000] [959.1/1000.0] [batch_t 1.454 (1.462)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 20:34:54,342 - Train: 95.93% [519000/541000] [959.3/1000.0] [batch_t 1.462 (1.458)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-07 20:37:20,142 - Train: 95.95% [519100/541000] [959.5/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 20:39:46,036 - Train: 95.97% [519200/541000] [959.7/1000.0] [batch_t 1.456 (1.459)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 20:42:11,792 - Train: 95.99% [519300/541000] [959.9/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 20:43:39,318 - ==> Total time: 5 days, 19:43:33 Eta: 5:49:18 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 20:44:40,136 - Train: 96.01% [519400/541000] [960.1/1000.0] [batch_t 1.463 (1.465)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-07 20:47:05,887 - Train: 96.03% [519500/541000] [960.3/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 20:49:31,728 - Train: 96.04% [519600/541000] [960.4/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 20:51:57,473 - Train: 96.06% [519700/541000] [960.6/1000.0] [batch_t 1.459 (1.457)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-07 20:54:23,204 - Train: 96.08% [519800/541000] [960.8/1000.0] [batch_t 1.459 (1.457)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-07 20:56:48,982 - Train: 96.10% [519900/541000] [961.0/1000.0] [batch_t 1.448 (1.458)] [data_t 0.002] [optim_t 1.446] [lr 0.000500] 2024-04-07 20:56:50,440 - ==> Total time: 5 days, 19:56:44 Eta: 5:40:45 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 20:59:16,907 - Train: 96.12% [520000/541000] [961.2/1000.0] [batch_t 1.455 (1.460)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 21:01:42,634 - Train: 96.14% [520100/541000] [961.4/1000.0] [batch_t 1.458 (1.457)] [data_t 0.003] [optim_t 1.455] [lr 0.000500] 2024-04-07 21:04:08,386 - Train: 96.16% [520200/541000] [961.6/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 21:06:34,130 - Train: 96.17% [520300/541000] [961.7/1000.0] [batch_t 1.462 (1.457)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-07 21:08:59,912 - Train: 96.19% [520400/541000] [961.9/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 21:10:01,125 - ==> Total time: 5 days, 20:09:55 Eta: 5:32:12 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 21:11:29,423 - Train: 96.21% [520500/541000] [962.1/1000.0] [batch_t 1.459 (1.493)] [data_t 0.003] [optim_t 1.455] [lr 0.000500] 2024-04-07 21:13:55,141 - Train: 96.23% [520600/541000] [962.3/1000.0] [batch_t 1.455 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 21:16:20,831 - Train: 96.25% [520700/541000] [962.5/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 21:18:46,637 - Train: 96.27% [520800/541000] [962.7/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 21:21:12,424 - Train: 96.28% [520900/541000] [962.8/1000.0] [batch_t 1.454 (1.458)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 21:23:13,521 - ==> Total time: 5 days, 20:23:07 Eta: 5:23:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 21:23:40,504 - Train: 96.30% [521000/541000] [963.0/1000.0] [batch_t 1.461 (1.480)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 21:26:06,370 - Train: 96.32% [521100/541000] [963.2/1000.0] [batch_t 1.461 (1.459)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 21:28:32,165 - Train: 96.34% [521200/541000] [963.4/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 21:30:57,938 - Train: 96.36% [521300/541000] [963.6/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-07 21:33:23,636 - Train: 96.38% [521400/541000] [963.8/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 21:35:49,430 - Train: 96.40% [521500/541000] [964.0/1000.0] [batch_t 1.454 (1.458)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 21:36:24,381 - ==> Total time: 5 days, 20:36:18 Eta: 5:15:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 21:38:17,354 - Train: 96.41% [521600/541000] [964.1/1000.0] [batch_t 1.463 (1.461)] [data_t 0.003] [optim_t 1.460] [lr 0.000500] 2024-04-07 21:40:43,117 - Train: 96.43% [521700/541000] [964.3/1000.0] [batch_t 1.457 (1.458)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-07 21:43:08,930 - Train: 96.45% [521800/541000] [964.5/1000.0] [batch_t 1.456 (1.458)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 21:45:34,701 - Train: 96.47% [521900/541000] [964.7/1000.0] [batch_t 1.459 (1.458)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-07 21:48:00,482 - Train: 96.49% [522000/541000] [964.9/1000.0] [batch_t 1.459 (1.458)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-07 21:49:35,190 - ==> Total time: 5 days, 20:49:29 Eta: 5:06:27 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 21:50:28,566 - Train: 96.51% [522100/541000] [965.1/1000.0] [batch_t 1.452 (1.466)] [data_t 0.003] [optim_t 1.449] [lr 0.000500] 2024-04-07 21:52:54,349 - Train: 96.52% [522200/541000] [965.2/1000.0] [batch_t 1.457 (1.458)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 21:55:20,218 - Train: 96.54% [522300/541000] [965.4/1000.0] [batch_t 1.462 (1.459)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-07 21:57:46,048 - Train: 96.56% [522400/541000] [965.6/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 22:00:11,825 - Train: 96.58% [522500/541000] [965.8/1000.0] [batch_t 1.458 (1.458)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-07 22:02:37,681 - Train: 96.60% [522600/541000] [966.0/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 22:02:46,443 - ==> Total time: 5 days, 21:02:40 Eta: 4:57:51 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 22:05:05,696 - Train: 96.62% [522700/541000] [966.2/1000.0] [batch_t 1.460 (1.462)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 22:07:31,478 - Train: 96.64% [522800/541000] [966.4/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 22:09:57,294 - Train: 96.65% [522900/541000] [966.5/1000.0] [batch_t 1.460 (1.458)] [data_t 0.004] [optim_t 1.456] [lr 0.000500] 2024-04-07 22:12:23,102 - Train: 96.67% [523000/541000] [966.7/1000.0] [batch_t 1.462 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 22:14:49,000 - Train: 96.69% [523100/541000] [966.9/1000.0] [batch_t 1.458 (1.459)] [data_t 0.003] [optim_t 1.455] [lr 0.000500] 2024-04-07 22:15:57,591 - ==> Total time: 5 days, 21:15:52 Eta: 4:49:14 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 22:17:17,326 - Train: 96.71% [523200/541000] [967.1/1000.0] [batch_t 1.456 (1.466)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 22:19:43,240 - Train: 96.73% [523300/541000] [967.3/1000.0] [batch_t 1.462 (1.459)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 22:22:09,057 - Train: 96.75% [523400/541000] [967.5/1000.0] [batch_t 1.455 (1.458)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 22:24:34,833 - Train: 96.77% [523500/541000] [967.7/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 22:27:00,645 - Train: 96.78% [523600/541000] [967.8/1000.0] [batch_t 1.459 (1.458)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-07 22:29:08,985 - ==> Total time: 5 days, 21:29:03 Eta: 4:40:37 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 22:29:28,622 - Train: 96.80% [523700/541000] [968.0/1000.0] [batch_t 1.458 (1.488)] [data_t 0.003] [optim_t 1.455] [lr 0.000500] 2024-04-07 22:31:54,497 - Train: 96.82% [523800/541000] [968.2/1000.0] [batch_t 1.461 (1.459)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 22:34:20,343 - Train: 96.84% [523900/541000] [968.4/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 22:36:46,189 - Train: 96.86% [524000/541000] [968.6/1000.0] [batch_t 1.457 (1.458)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-07 22:39:12,100 - Train: 96.88% [524100/541000] [968.8/1000.0] [batch_t 1.466 (1.459)] [data_t 0.003] [optim_t 1.462] [lr 0.000500] 2024-04-07 22:41:37,925 - Train: 96.89% [524200/541000] [968.9/1000.0] [batch_t 1.459 (1.458)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-07 22:42:20,205 - ==> Total time: 5 days, 21:42:14 Eta: 4:32:00 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 22:44:05,909 - Train: 96.91% [524300/541000] [969.1/1000.0] [batch_t 1.462 (1.462)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-07 22:46:31,758 - Train: 96.93% [524400/541000] [969.3/1000.0] [batch_t 1.457 (1.458)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-07 22:48:57,609 - Train: 96.95% [524500/541000] [969.5/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 22:51:23,443 - Train: 96.97% [524600/541000] [969.7/1000.0] [batch_t 1.463 (1.458)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-07 22:53:49,243 - Train: 96.99% [524700/541000] [969.9/1000.0] [batch_t 1.455 (1.458)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-07 22:55:31,298 - ==> Total time: 5 days, 21:55:25 Eta: 4:23:21 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 22:56:17,489 - Train: 97.01% [524800/541000] [970.1/1000.0] [batch_t 1.454 (1.471)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 22:58:43,293 - Train: 97.02% [524900/541000] [970.2/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 23:01:09,122 - Train: 97.04% [525000/541000] [970.4/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 23:03:34,950 - Train: 97.06% [525100/541000] [970.6/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 23:06:00,728 - Train: 97.08% [525200/541000] [970.8/1000.0] [batch_t 1.459 (1.458)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-07 23:08:26,605 - Train: 97.10% [525300/541000] [971.0/1000.0] [batch_t 1.462 (1.459)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-07 23:08:42,653 - ==> Total time: 5 days, 22:08:37 Eta: 4:14:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 23:10:54,630 - Train: 97.12% [525400/541000] [971.2/1000.0] [batch_t 1.462 (1.462)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-07 23:13:20,551 - Train: 97.13% [525500/541000] [971.3/1000.0] [batch_t 1.461 (1.459)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 23:15:46,361 - Train: 97.15% [525600/541000] [971.5/1000.0] [batch_t 1.455 (1.458)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 23:18:12,123 - Train: 97.17% [525700/541000] [971.7/1000.0] [batch_t 1.454 (1.458)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 23:20:37,870 - Train: 97.19% [525800/541000] [971.9/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 23:21:53,695 - ==> Total time: 5 days, 22:21:48 Eta: 4:06:03 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 23:23:05,991 - Train: 97.21% [525900/541000] [972.1/1000.0] [batch_t 1.454 (1.468)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 23:25:31,722 - Train: 97.23% [526000/541000] [972.3/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-07 23:27:57,554 - Train: 97.25% [526100/541000] [972.5/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-07 23:30:23,500 - Train: 97.26% [526200/541000] [972.6/1000.0] [batch_t 1.461 (1.459)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 23:32:49,414 - Train: 97.28% [526300/541000] [972.8/1000.0] [batch_t 1.459 (1.459)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-07 23:35:05,106 - ==> Total time: 5 days, 22:34:59 Eta: 3:57:23 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 23:35:17,646 - Train: 97.30% [526400/541000] [973.0/1000.0] [batch_t 1.462 (1.510)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-07 23:37:43,528 - Train: 97.32% [526500/541000] [973.2/1000.0] [batch_t 1.461 (1.459)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 23:40:09,363 - Train: 97.34% [526600/541000] [973.4/1000.0] [batch_t 1.440 (1.458)] [data_t 0.003] [optim_t 1.437] [lr 0.000500] 2024-04-07 23:42:35,162 - Train: 97.36% [526700/541000] [973.6/1000.0] [batch_t 1.465 (1.458)] [data_t 0.003] [optim_t 1.462] [lr 0.000500] 2024-04-07 23:45:00,975 - Train: 97.38% [526800/541000] [973.8/1000.0] [batch_t 1.462 (1.458)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-07 23:47:26,797 - Train: 97.39% [526900/541000] [973.9/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-07 23:48:16,428 - ==> Total time: 5 days, 22:48:10 Eta: 3:48:43 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-07 23:49:54,759 - Train: 97.41% [527000/541000] [974.1/1000.0] [batch_t 1.454 (1.463)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-07 23:52:20,478 - Train: 97.43% [527100/541000] [974.3/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-07 23:54:46,299 - Train: 97.45% [527200/541000] [974.5/1000.0] [batch_t 1.457 (1.458)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-07 23:57:12,159 - Train: 97.47% [527300/541000] [974.7/1000.0] [batch_t 1.450 (1.459)] [data_t 0.003] [optim_t 1.447] [lr 0.000500] 2024-04-07 23:59:37,951 - Train: 97.49% [527400/541000] [974.9/1000.0] [batch_t 1.454 (1.458)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-08 00:01:27,318 - ==> Total time: 5 days, 23:01:21 Eta: 3:40:02 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-08 00:02:06,061 - Train: 97.50% [527500/541000] [975.0/1000.0] [batch_t 1.456 (1.474)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-08 00:04:31,902 - Train: 97.52% [527600/541000] [975.2/1000.0] [batch_t 1.462 (1.458)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-08 00:06:57,725 - Train: 97.54% [527700/541000] [975.4/1000.0] [batch_t 1.466 (1.458)] [data_t 0.003] [optim_t 1.463] [lr 0.000500] 2024-04-08 00:09:23,526 - Train: 97.56% [527800/541000] [975.6/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-08 00:11:49,281 - Train: 97.58% [527900/541000] [975.8/1000.0] [batch_t 1.454 (1.457)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-08 00:14:15,049 - Train: 97.60% [528000/541000] [976.0/1000.0] [batch_t 1.463 (1.458)] [data_t 0.003] [optim_t 1.460] [lr 0.000500] 2024-04-08 00:14:38,376 - ==> Total time: 5 days, 23:14:32 Eta: 3:31:20 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-08 00:16:43,359 - Train: 97.62% [528100/541000] [976.2/1000.0] [batch_t 1.457 (1.464)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-08 00:19:09,084 - Train: 97.63% [528200/541000] [976.3/1000.0] [batch_t 1.458 (1.457)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-08 00:21:34,839 - Train: 97.65% [528300/541000] [976.5/1000.0] [batch_t 1.465 (1.457)] [data_t 0.003] [optim_t 1.462] [lr 0.000500] 2024-04-08 00:24:00,555 - Train: 97.67% [528400/541000] [976.7/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-08 00:26:26,385 - Train: 97.69% [528500/541000] [976.9/1000.0] [batch_t 1.459 (1.458)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-08 00:27:49,522 - ==> Total time: 5 days, 23:27:43 Eta: 3:22:38 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-08 00:28:54,313 - Train: 97.71% [528600/541000] [977.1/1000.0] [batch_t 1.463 (1.466)] [data_t 0.003] [optim_t 1.460] [lr 0.000500] 2024-04-08 00:31:20,030 - Train: 97.73% [528700/541000] [977.3/1000.0] [batch_t 1.457 (1.457)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-08 00:33:45,714 - Train: 97.74% [528800/541000] [977.4/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-08 00:36:11,461 - Train: 97.76% [528900/541000] [977.6/1000.0] [batch_t 1.457 (1.457)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-08 00:38:37,184 - Train: 97.78% [529000/541000] [977.8/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-08 00:40:59,998 - ==> Total time: 5 days, 23:40:54 Eta: 3:13:55 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-08 00:41:06,892 - Train: 97.80% [529100/541000] [978.0/1000.0] [batch_t 1.452 (2.486)] [data_t 0.003] [optim_t 1.449] [lr 0.000500] 2024-04-08 00:43:32,675 - Train: 97.82% [529200/541000] [978.2/1000.0] [batch_t 1.462 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-08 00:45:58,385 - Train: 97.84% [529300/541000] [978.4/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-08 00:48:24,214 - Train: 97.86% [529400/541000] [978.6/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-08 00:50:49,963 - Train: 97.87% [529500/541000] [978.7/1000.0] [batch_t 1.457 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-08 00:53:15,731 - Train: 97.89% [529600/541000] [978.9/1000.0] [batch_t 1.455 (1.458)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-08 00:54:12,597 - ==> Total time: 5 days, 23:54:07 Eta: 3:05:12 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-08 00:55:43,911 - Train: 97.91% [529700/541000] [979.1/1000.0] [batch_t 1.466 (1.462)] [data_t 0.003] [optim_t 1.463] [lr 0.000500] 2024-04-08 00:58:09,656 - Train: 97.93% [529800/541000] [979.3/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-08 01:00:35,467 - Train: 97.95% [529900/541000] [979.5/1000.0] [batch_t 1.454 (1.458)] [data_t 0.003] [optim_t 1.450] [lr 0.000500] 2024-04-08 01:03:01,253 - Train: 97.97% [530000/541000] [979.7/1000.0] [batch_t 1.462 (1.458)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-08 01:05:27,053 - Train: 97.99% [530100/541000] [979.9/1000.0] [batch_t 1.457 (1.458)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-08 01:07:23,663 - ==> Total time: 6 days, 0:07:18 Eta: 2:56:28 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-08 01:07:55,106 - Train: 98.00% [530200/541000] [980.0/1000.0] [batch_t 1.455 (1.480)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-08 01:10:20,809 - Train: 98.02% [530300/541000] [980.2/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-08 01:12:46,494 - Train: 98.04% [530400/541000] [980.4/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-08 01:15:12,249 - Train: 98.06% [530500/541000] [980.6/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-08 01:17:38,034 - Train: 98.08% [530600/541000] [980.8/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-08 01:20:03,784 - Train: 98.10% [530700/541000] [981.0/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-08 01:20:34,388 - ==> Total time: 6 days, 0:20:28 Eta: 2:47:44 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-08 01:22:31,914 - Train: 98.11% [530800/541000] [981.1/1000.0] [batch_t 1.457 (1.465)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-08 01:24:57,676 - Train: 98.13% [530900/541000] [981.3/1000.0] [batch_t 1.454 (1.458)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-08 01:27:23,447 - Train: 98.15% [531000/541000] [981.5/1000.0] [batch_t 1.462 (1.458)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-08 01:29:49,178 - Train: 98.17% [531100/541000] [981.7/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-08 01:32:14,909 - Train: 98.19% [531200/541000] [981.9/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-08 01:33:45,286 - ==> Total time: 6 days, 0:33:39 Eta: 2:38:59 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-08 01:34:42,757 - Train: 98.21% [531300/541000] [982.1/1000.0] [batch_t 1.455 (1.467)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-08 01:37:08,463 - Train: 98.23% [531400/541000] [982.3/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-08 01:39:34,218 - Train: 98.24% [531500/541000] [982.4/1000.0] [batch_t 1.463 (1.457)] [data_t 0.003] [optim_t 1.460] [lr 0.000500] 2024-04-08 01:41:59,954 - Train: 98.26% [531600/541000] [982.6/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-08 01:44:25,766 - Train: 98.28% [531700/541000] [982.8/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-08 01:46:51,579 - Train: 98.30% [531800/541000] [983.0/1000.0] [batch_t 1.462 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-08 01:46:55,965 - ==> Total time: 6 days, 0:46:50 Eta: 2:30:13 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-08 01:49:19,588 - Train: 98.32% [531900/541000] [983.2/1000.0] [batch_t 1.462 (1.461)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-08 01:51:45,301 - Train: 98.34% [532000/541000] [983.4/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-08 01:54:11,022 - Train: 98.35% [532100/541000] [983.5/1000.0] [batch_t 1.459 (1.457)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-08 01:56:36,757 - Train: 98.37% [532200/541000] [983.7/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-08 01:59:02,581 - Train: 98.39% [532300/541000] [983.9/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-08 02:00:06,701 - ==> Total time: 6 days, 1:00:01 Eta: 2:21:27 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-08 02:01:30,535 - Train: 98.41% [532400/541000] [984.1/1000.0] [batch_t 1.455 (1.464)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-08 02:03:56,259 - Train: 98.43% [532500/541000] [984.3/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-08 02:06:21,961 - Train: 98.45% [532600/541000] [984.5/1000.0] [batch_t 1.455 (1.457)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-08 02:08:47,676 - Train: 98.47% [532700/541000] [984.7/1000.0] [batch_t 1.462 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-08 02:11:13,445 - Train: 98.48% [532800/541000] [984.8/1000.0] [batch_t 1.459 (1.458)] [data_t 0.003] [optim_t 1.455] [lr 0.000500] 2024-04-08 02:13:17,325 - ==> Total time: 6 days, 1:13:11 Eta: 2:12:41 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-08 02:13:41,268 - Train: 98.50% [532900/541000] [985.0/1000.0] [batch_t 1.455 (1.483)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-08 02:16:07,056 - Train: 98.52% [533000/541000] [985.2/1000.0] [batch_t 1.455 (1.458)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-08 02:18:32,811 - Train: 98.54% [533100/541000] [985.4/1000.0] [batch_t 1.462 (1.457)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-08 02:20:58,549 - Train: 98.56% [533200/541000] [985.6/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-08 02:23:24,345 - Train: 98.58% [533300/541000] [985.8/1000.0] [batch_t 1.452 (1.458)] [data_t 0.003] [optim_t 1.449] [lr 0.000500] 2024-04-08 02:25:50,050 - Train: 98.60% [533400/541000] [986.0/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-08 02:26:27,930 - ==> Total time: 6 days, 1:26:22 Eta: 2:03:54 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-08 02:28:17,988 - Train: 98.61% [533500/541000] [986.1/1000.0] [batch_t 1.459 (1.464)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-08 02:30:43,777 - Train: 98.63% [533600/541000] [986.3/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-08 02:33:09,559 - Train: 98.65% [533700/541000] [986.5/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-08 02:35:35,343 - Train: 98.67% [533800/541000] [986.7/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-08 02:38:01,158 - Train: 98.69% [533900/541000] [986.9/1000.0] [batch_t 1.458 (1.458)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-08 02:39:38,770 - ==> Total time: 6 days, 1:39:33 Eta: 1:55:06 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-08 02:40:29,216 - Train: 98.71% [534000/541000] [987.1/1000.0] [batch_t 1.455 (1.468)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-08 02:42:55,000 - Train: 98.72% [534100/541000] [987.2/1000.0] [batch_t 1.451 (1.458)] [data_t 0.003] [optim_t 1.448] [lr 0.000500] 2024-04-08 02:45:20,778 - Train: 98.74% [534200/541000] [987.4/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-08 02:47:46,529 - Train: 98.76% [534300/541000] [987.6/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-08 02:50:12,284 - Train: 98.78% [534400/541000] [987.8/1000.0] [batch_t 1.462 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-08 02:52:38,096 - Train: 98.80% [534500/541000] [988.0/1000.0] [batch_t 1.463 (1.458)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-08 02:52:49,764 - ==> Total time: 6 days, 1:52:44 Eta: 1:46:18 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-08 02:55:06,220 - Train: 98.82% [534600/541000] [988.2/1000.0] [batch_t 1.457 (1.462)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-08 02:57:31,972 - Train: 98.84% [534700/541000] [988.4/1000.0] [batch_t 1.454 (1.457)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-08 02:59:57,703 - Train: 98.85% [534800/541000] [988.5/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-08 03:02:23,469 - Train: 98.87% [534900/541000] [988.7/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-08 03:04:49,283 - Train: 98.89% [535000/541000] [988.9/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-08 03:06:00,677 - ==> Total time: 6 days, 2:05:55 Eta: 1:37:29 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-08 03:07:18,495 - Train: 98.91% [535100/541000] [989.1/1000.0] [batch_t 1.451 (1.489)] [data_t 0.003] [optim_t 1.448] [lr 0.000500] 2024-04-08 03:09:44,309 - Train: 98.93% [535200/541000] [989.3/1000.0] [batch_t 1.462 (1.458)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-08 03:12:10,074 - Train: 98.95% [535300/541000] [989.5/1000.0] [batch_t 1.445 (1.458)] [data_t 0.003] [optim_t 1.442] [lr 0.000500] 2024-04-08 03:14:36,042 - Train: 98.96% [535400/541000] [989.6/1000.0] [batch_t 1.461 (1.460)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-08 03:17:01,807 - Train: 98.98% [535500/541000] [989.8/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-08 03:19:12,990 - ==> Total time: 6 days, 2:19:07 Eta: 1:28:40 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-08 03:19:29,850 - Train: 99.00% [535600/541000] [990.0/1000.0] [batch_t 1.463 (1.489)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-08 03:21:55,613 - Train: 99.02% [535700/541000] [990.2/1000.0] [batch_t 1.456 (1.458)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-08 03:24:21,388 - Train: 99.04% [535800/541000] [990.4/1000.0] [batch_t 1.459 (1.458)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-08 03:26:47,151 - Train: 99.06% [535900/541000] [990.6/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-08 03:29:12,870 - Train: 99.08% [536000/541000] [990.8/1000.0] [batch_t 1.463 (1.457)] [data_t 0.003] [optim_t 1.460] [lr 0.000500] 2024-04-08 03:31:38,578 - Train: 99.09% [536100/541000] [990.9/1000.0] [batch_t 1.460 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-08 03:32:23,762 - ==> Total time: 6 days, 2:32:18 Eta: 1:19:50 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-08 03:34:06,511 - Train: 99.11% [536200/541000] [991.1/1000.0] [batch_t 1.460 (1.464)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-08 03:36:32,270 - Train: 99.13% [536300/541000] [991.3/1000.0] [batch_t 1.459 (1.457)] [data_t 0.004] [optim_t 1.455] [lr 0.000500] 2024-04-08 03:38:58,029 - Train: 99.15% [536400/541000] [991.5/1000.0] [batch_t 1.457 (1.457)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-08 03:41:23,736 - Train: 99.17% [536500/541000] [991.7/1000.0] [batch_t 1.458 (1.457)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-08 03:43:49,695 - Train: 99.19% [536600/541000] [991.9/1000.0] [batch_t 1.456 (1.459)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-08 03:45:34,700 - ==> Total time: 6 days, 2:45:29 Eta: 1:11:00 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-08 03:46:17,734 - Train: 99.21% [536700/541000] [992.1/1000.0] [batch_t 1.461 (1.471)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-08 03:48:43,498 - Train: 99.22% [536800/541000] [992.2/1000.0] [batch_t 1.462 (1.458)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-08 03:51:09,249 - Train: 99.24% [536900/541000] [992.4/1000.0] [batch_t 1.462 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-08 03:53:35,035 - Train: 99.26% [537000/541000] [992.6/1000.0] [batch_t 1.454 (1.458)] [data_t 0.004] [optim_t 1.450] [lr 0.000500] 2024-04-08 03:56:00,830 - Train: 99.28% [537100/541000] [992.8/1000.0] [batch_t 1.458 (1.458)] [data_t 0.003] [optim_t 1.455] [lr 0.000500] 2024-04-08 03:58:26,586 - Train: 99.30% [537200/541000] [993.0/1000.0] [batch_t 1.459 (1.457)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-08 03:58:45,549 - ==> Total time: 6 days, 2:58:39 Eta: 1:02:09 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-08 04:00:54,637 - Train: 99.32% [537300/541000] [993.2/1000.0] [batch_t 1.463 (1.461)] [data_t 0.003] [optim_t 1.460] [lr 0.000500] 2024-04-08 04:03:20,473 - Train: 99.33% [537400/541000] [993.3/1000.0] [batch_t 1.451 (1.458)] [data_t 0.003] [optim_t 1.448] [lr 0.000500] 2024-04-08 04:05:46,225 - Train: 99.35% [537500/541000] [993.5/1000.0] [batch_t 1.463 (1.457)] [data_t 0.003] [optim_t 1.460] [lr 0.000500] 2024-04-08 04:08:11,948 - Train: 99.37% [537600/541000] [993.7/1000.0] [batch_t 1.451 (1.457)] [data_t 0.003] [optim_t 1.448] [lr 0.000500] 2024-04-08 04:10:37,761 - Train: 99.39% [537700/541000] [993.9/1000.0] [batch_t 1.455 (1.458)] [data_t 0.003] [optim_t 1.452] [lr 0.000500] 2024-04-08 04:11:56,463 - ==> Total time: 6 days, 3:11:50 Eta: 0:53:18 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-08 04:13:05,530 - Train: 99.41% [537800/541000] [994.1/1000.0] [batch_t 1.456 (1.464)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-08 04:15:31,303 - Train: 99.43% [537900/541000] [994.3/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-08 04:17:57,008 - Train: 99.45% [538000/541000] [994.5/1000.0] [batch_t 1.457 (1.457)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-08 04:20:22,850 - Train: 99.46% [538100/541000] [994.6/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-08 04:22:48,598 - Train: 99.48% [538200/541000] [994.8/1000.0] [batch_t 1.457 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-08 04:25:07,103 - ==> Total time: 6 days, 3:25:01 Eta: 0:44:26 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-08 04:25:16,677 - Train: 99.50% [538300/541000] [995.0/1000.0] [batch_t 1.459 (1.529)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-08 04:27:42,457 - Train: 99.52% [538400/541000] [995.2/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-08 04:30:08,122 - Train: 99.54% [538500/541000] [995.4/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-08 04:32:33,903 - Train: 99.56% [538600/541000] [995.6/1000.0] [batch_t 1.456 (1.458)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-08 04:34:59,703 - Train: 99.57% [538700/541000] [995.7/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-08 04:37:25,502 - Train: 99.59% [538800/541000] [995.9/1000.0] [batch_t 1.462 (1.458)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-08 04:38:17,938 - ==> Total time: 6 days, 3:38:12 Eta: 0:35:34 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-08 04:39:53,588 - Train: 99.61% [538900/541000] [996.1/1000.0] [batch_t 1.460 (1.463)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-08 04:42:19,371 - Train: 99.63% [539000/541000] [996.3/1000.0] [batch_t 1.459 (1.458)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-08 04:44:45,190 - Train: 99.65% [539100/541000] [996.5/1000.0] [batch_t 1.462 (1.458)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-08 04:47:10,974 - Train: 99.67% [539200/541000] [996.7/1000.0] [batch_t 1.457 (1.458)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-08 04:49:36,705 - Train: 99.69% [539300/541000] [996.9/1000.0] [batch_t 1.457 (1.457)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-08 04:51:28,987 - ==> Total time: 6 days, 3:51:23 Eta: 0:26:41 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-08 04:52:04,922 - Train: 99.70% [539400/541000] [997.0/1000.0] [batch_t 1.454 (1.473)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-08 04:54:30,745 - Train: 99.72% [539500/541000] [997.2/1000.0] [batch_t 1.454 (1.458)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-08 04:56:56,552 - Train: 99.74% [539600/541000] [997.4/1000.0] [batch_t 1.461 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-08 04:59:22,272 - Train: 99.76% [539700/541000] [997.6/1000.0] [batch_t 1.459 (1.457)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-08 05:01:47,963 - Train: 99.78% [539800/541000] [997.8/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-08 05:04:13,750 - Train: 99.80% [539900/541000] [998.0/1000.0] [batch_t 1.454 (1.458)] [data_t 0.003] [optim_t 1.451] [lr 0.000500] 2024-04-08 05:04:40,008 - ==> Total time: 6 days, 4:04:34 Eta: 0:17:48 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-08 05:06:41,976 - Train: 99.82% [540000/541000] [998.2/1000.0] [batch_t 1.460 (1.466)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-08 05:09:07,700 - Train: 99.83% [540100/541000] [998.3/1000.0] [batch_t 1.462 (1.457)] [data_t 0.003] [optim_t 1.459] [lr 0.000500] 2024-04-08 05:11:33,420 - Train: 99.85% [540200/541000] [998.5/1000.0] [batch_t 1.459 (1.457)] [data_t 0.003] [optim_t 1.455] [lr 0.000500] 2024-04-08 05:13:59,162 - Train: 99.87% [540300/541000] [998.7/1000.0] [batch_t 1.461 (1.457)] [data_t 0.003] [optim_t 1.458] [lr 0.000500] 2024-04-08 05:16:24,864 - Train: 99.89% [540400/541000] [998.9/1000.0] [batch_t 1.456 (1.457)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-08 05:17:50,906 - ==> Total time: 6 days, 4:17:45 Eta: 0:08:54 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-08 05:18:53,070 - Train: 99.91% [540500/541000] [999.1/1000.0] [batch_t 1.456 (1.467)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-08 05:21:18,810 - Train: 99.93% [540600/541000] [999.3/1000.0] [batch_t 1.457 (1.457)] [data_t 0.003] [optim_t 1.454] [lr 0.000500] 2024-04-08 05:23:44,581 - Train: 99.94% [540700/541000] [999.4/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.457] [lr 0.000500] 2024-04-08 05:26:10,376 - Train: 99.96% [540800/541000] [999.6/1000.0] [batch_t 1.456 (1.458)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-08 05:28:36,152 - Train: 99.98% [540900/541000] [999.8/1000.0] [batch_t 1.460 (1.458)] [data_t 0.003] [optim_t 1.456] [lr 0.000500] 2024-04-08 05:31:01,963 - Train: 100.00% [541000/541000] [1000.0/1000.0] [batch_t 1.456 (1.458)] [data_t 0.003] [optim_t 1.453] [lr 0.000500] 2024-04-08 05:31:26,021 - Test: 36.76% [50/136] [batch_t 0.427 (0.445)] 2024-04-08 05:31:47,284 - Test: 73.53% [100/136] [batch_t 0.427 (0.435)] 2024-04-08 05:32:02,234 - Test: 100.00% [136/136] [batch_t 0.075 (0.430)] 2024-04-08 05:32:52,126 - ==> Metric Time for candle : 0.002 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 11.247 (mAUROC_px) 8.216 (mAP_px) 0.947 (mF1_max_px) 25.642 (mAUPRO_px) 0.325 (mF1_px_0.2_0.8_0.1) 0.321 (mAcc_px_0.2_0.8_0.1) 0.321 (mIoU_px_0.2_0.8_0.1) 0.943 (mIoU_max_px) 2024-04-08 05:33:29,824 - ==> Metric Time for capsules : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 8.297 (mAUROC_px) 6.389 (mAP_px) 0.745 (mF1_max_px) 20.526 (mAUPRO_px) 0.257 (mF1_px_0.2_0.8_0.1) 0.250 (mAcc_px_0.2_0.8_0.1) 0.258 (mIoU_px_0.2_0.8_0.1) 0.745 (mIoU_max_px) 2024-04-08 05:34:07,968 - ==> Metric Time for cashew : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.860 (mAUROC_px) 6.000 (mAP_px) 0.695 (mF1_max_px) 21.752 (mAUPRO_px) 0.271 (mF1_px_0.2_0.8_0.1) 0.272 (mAcc_px_0.2_0.8_0.1) 0.279 (mIoU_px_0.2_0.8_0.1) 0.802 (mIoU_max_px) 2024-04-08 05:34:44,786 - ==> Metric Time for chewinggum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.914 (mAUROC_px) 5.947 (mAP_px) 0.704 (mF1_max_px) 20.594 (mAUPRO_px) 0.240 (mF1_px_0.2_0.8_0.1) 0.240 (mAcc_px_0.2_0.8_0.1) 0.245 (mIoU_px_0.2_0.8_0.1) 0.700 (mIoU_max_px) 2024-04-08 05:35:22,921 - ==> Metric Time for fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 8.034 (mAUROC_px) 6.250 (mAP_px) 0.778 (mF1_max_px) 21.428 (mAUPRO_px) 0.238 (mF1_px_0.2_0.8_0.1) 0.240 (mAcc_px_0.2_0.8_0.1) 0.244 (mIoU_px_0.2_0.8_0.1) 0.709 (mIoU_max_px) 2024-04-08 05:36:10,880 - ==> Metric Time for macaroni1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.732 (mAUROC_px) 8.011 (mAP_px) 1.026 (mF1_max_px) 25.996 (mAUPRO_px) 0.325 (mF1_px_0.2_0.8_0.1) 0.323 (mAcc_px_0.2_0.8_0.1) 0.325 (mIoU_px_0.2_0.8_0.1) 0.940 (mIoU_max_px) 2024-04-08 05:36:59,133 - ==> Metric Time for macaroni2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 11.434 (mAUROC_px) 8.276 (mAP_px) 0.946 (mF1_max_px) 25.406 (mAUPRO_px) 0.321 (mF1_px_0.2_0.8_0.1) 0.322 (mAcc_px_0.2_0.8_0.1) 0.325 (mIoU_px_0.2_0.8_0.1) 0.940 (mIoU_max_px) 2024-04-08 05:37:53,087 - ==> Metric Time for pcb1 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.791 (mAUROC_px) 8.997 (mAP_px) 1.160 (mF1_max_px) 30.808 (mAUPRO_px) 0.323 (mF1_px_0.2_0.8_0.1) 0.325 (mAcc_px_0.2_0.8_0.1) 0.324 (mIoU_px_0.2_0.8_0.1) 0.941 (mIoU_max_px) 2024-04-08 05:38:41,556 - ==> Metric Time for pcb2 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.895 (mAUROC_px) 8.329 (mAP_px) 0.922 (mF1_max_px) 26.181 (mAUPRO_px) 0.317 (mF1_px_0.2_0.8_0.1) 0.314 (mAcc_px_0.2_0.8_0.1) 0.315 (mIoU_px_0.2_0.8_0.1) 0.912 (mIoU_max_px) 2024-04-08 05:39:29,608 - ==> Metric Time for pcb3 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.685 (mAUROC_px) 8.024 (mAP_px) 0.936 (mF1_max_px) 26.243 (mAUPRO_px) 0.318 (mF1_px_0.2_0.8_0.1) 0.317 (mAcc_px_0.2_0.8_0.1) 0.317 (mIoU_px_0.2_0.8_0.1) 0.927 (mIoU_max_px) 2024-04-08 05:40:19,474 - ==> Metric Time for pcb4 : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 10.842 (mAUROC_px) 8.171 (mAP_px) 0.932 (mF1_max_px) 27.755 (mAUPRO_px) 0.318 (mF1_px_0.2_0.8_0.1) 0.317 (mAcc_px_0.2_0.8_0.1) 0.319 (mIoU_px_0.2_0.8_0.1) 0.928 (mIoU_max_px) 2024-04-08 05:40:58,041 - ==> Metric Time for pipe_fryum : 0.001 (mAUROC_sp_max) 0.000 (mAP_sp_max) 0.000 (mF1_max_sp_max) 7.904 (mAUROC_px) 5.887 (mAP_px) 0.683 (mF1_max_px) 22.096 (mAUPRO_px) 0.308 (mF1_px_0.2_0.8_0.1) 0.310 (mAcc_px_0.2_0.8_0.1) 0.304 (mIoU_px_0.2_0.8_0.1) 0.858 (mIoU_max_px) 2024-04-08 05:40:58,068 - | Name | mAUROC_sp_max | mAUROC_sp_max (Max) | mAP_sp_max | mAP_sp_max (Max) | mF1_max_sp_max | mF1_max_sp_max (Max) | mAUROC_px | mAUROC_px (Max) | mAP_px | mAP_px (Max) | mF1_max_px | mF1_max_px (Max) | mAUPRO_px | mAUPRO_px (Max) | mF1_px_0.2_0.8_0.1 | mF1_px_0.2_0.8_0.1 (Max) | mAcc_px_0.2_0.8_0.1 | mAcc_px_0.2_0.8_0.1 (Max) | mIoU_px_0.2_0.8_0.1 | mIoU_px_0.2_0.8_0.1 (Max) | mIoU_max_px | mIoU_max_px (Max) | |:----------:|:---------------:|:---------------------:|:------------:|:------------------:|:----------------:|:----------------------:|:-----------:|:------------------:|:--------:|:------------------:|:------------:|:-------------------:|:-----------:|:-------------------:|:--------------------:|:--------------------------:|:---------------------:|:---------------------------:|:---------------------:|:---------------------------:|:-------------:|:-------------------:| | candle | 96.170 | 97.100 (50 epoch) | 96.441 | 97.346 (50 epoch) | 89.952 | 91.542 (50 epoch) | 99.057 | 99.098 (250 epoch) | 20.643 | 23.463 (150 epoch) | 30.670 | 32.654 (200 epoch) | 95.741 | 96.087 (300 epoch) | 20.240 | 21.331 (500 epoch) | 38.848 | 41.526 (650 epoch) | 11.508 | 12.185 (500 epoch) | 18.113 | 19.513 (200 epoch) | | capsules | 73.967 | 91.750 (100 epoch) | 76.751 | 94.964 (100 epoch) | 82.609 | 88.780 (100 epoch) | 95.440 | 99.193 (700 epoch) | 19.302 | 61.256 (100 epoch) | 32.191 | 59.868 (250 epoch) | 67.448 | 91.811 (100 epoch) | 12.112 | 33.186 (100 epoch) | 46.290 | 62.576 (50 epoch) | 6.782 | 21.701 (100 epoch) | 19.183 | 42.722 (250 epoch) | | cashew | 94.400 | 94.940 (800 epoch) | 97.076 | 97.492 (150 epoch) | 91.707 | 92.929 (700 epoch) | 95.271 | 96.185 (700 epoch) | 26.329 | 50.952 (250 epoch) | 28.123 | 54.438 (400 epoch) | 80.699 | 89.185 (350 epoch) | 19.111 | 28.352 (350 epoch) | 29.125 | 34.522 (150 epoch) | 10.866 | 17.797 (350 epoch) | 16.362 | 37.398 (400 epoch) | | chewinggum | 96.625 | 97.802 (300 epoch) | 98.455 | 98.939 (300 epoch) | 94.359 | 95.385 (950 epoch) | 97.857 | 98.145 (100 epoch) | 38.976 | 59.823 (50 epoch) | 46.168 | 61.088 (50 epoch) | 78.711 | 80.312 (400 epoch) | 25.970 | 35.748 (50 epoch) | 62.978 | 65.723 (800 epoch) | 15.812 | 23.754 (50 epoch) | 30.012 | 43.976 (50 epoch) | | fryum | 96.460 | 96.560 (650 epoch) | 98.259 | 98.273 (650 epoch) | 92.386 | 93.897 (650 epoch) | 97.307 | 97.316 (950 epoch) | 51.534 | 52.248 (750 epoch) | 53.743 | 53.743 (1000 epoch) | 91.961 | 92.340 (550 epoch) | 25.314 | 26.511 (550 epoch) | 24.710 | 28.594 (450 epoch) | 15.883 | 16.529 (550 epoch) | 36.746 | 36.746 (1000 epoch) | | macaroni1 | 91.173 | 92.104 (500 epoch) | 86.480 | 90.505 (500 epoch) | 82.629 | 84.314 (750 epoch) | 99.256 | 99.604 (650 epoch) | 3.482 | 20.435 (700 epoch) | 5.815 | 29.513 (650 epoch) | 94.230 | 96.221 (650 epoch) | 4.873 | 16.200 (700 epoch) | 35.507 | 40.765 (50 epoch) | 2.498 | 9.078 (700 epoch) | 2.995 | 17.311 (650 epoch) | | macaroni2 | 75.179 | 82.589 (50 epoch) | 61.180 | 81.196 (50 epoch) | 75.336 | 77.833 (750 epoch) | 92.249 | 99.452 (100 epoch) | 0.230 | 11.272 (50 epoch) | 1.757 | 17.568 (50 epoch) | 71.927 | 96.253 (150 epoch) | 0.472 | 10.086 (50 epoch) | 29.014 | 54.584 (450 epoch) | 0.237 | 5.433 (50 epoch) | 0.886 | 9.630 (50 epoch) | | pcb1 | 96.730 | 96.880 (850 epoch) | 95.375 | 95.627 (850 epoch) | 94.231 | 94.737 (300 epoch) | 99.801 | 99.806 (700 epoch) | 81.249 | 83.141 (450 epoch) | 74.116 | 75.588 (450 epoch) | 94.742 | 94.908 (900 epoch) | 46.858 | 48.256 (700 epoch) | 53.357 | 62.362 (50 epoch) | 32.726 | 34.013 (700 epoch) | 58.876 | 60.756 (450 epoch) | | pcb2 | 96.060 | 96.200 (950 epoch) | 95.860 | 95.984 (950 epoch) | 91.626 | 91.626 (1000 epoch) | 99.019 | 99.023 (700 epoch) | 12.541 | 14.130 (50 epoch) | 23.779 | 23.977 (250 epoch) | 91.639 | 91.639 (1000 epoch) | 15.628 | 16.150 (200 epoch) | 42.797 | 47.820 (150 epoch) | 8.603 | 8.892 (200 epoch) | 13.494 | 13.621 (250 epoch) | | pcb3 | 95.891 | 95.921 (950 epoch) | 96.054 | 96.142 (950 epoch) | 90.291 | 90.291 (850 epoch) | 99.117 | 99.174 (750 epoch) | 16.488 | 20.249 (150 epoch) | 26.655 | 28.078 (450 epoch) | 91.934 | 92.258 (850 epoch) | 18.811 | 20.042 (150 epoch) | 43.299 | 48.341 (50 epoch) | 10.539 | 11.295 (150 epoch) | 15.377 | 16.332 (450 epoch) | | pcb4 | 99.733 | 99.901 (100 epoch) | 99.735 | 99.901 (100 epoch) | 97.537 | 99.000 (50 epoch) | 98.669 | 98.746 (650 epoch) | 45.633 | 47.966 (250 epoch) | 45.668 | 47.709 (250 epoch) | 90.005 | 91.033 (850 epoch) | 23.203 | 26.925 (200 epoch) | 28.990 | 39.716 (50 epoch) | 14.196 | 16.537 (200 epoch) | 29.591 | 31.328 (250 epoch) | | pipe_fryum | 98.980 | 99.040 (900 epoch) | 99.472 | 99.503 (900 epoch) | 97.087 | 97.561 (200 epoch) | 98.886 | 99.114 (450 epoch) | 51.927 | 56.529 (700 epoch) | 54.047 | 58.733 (450 epoch) | 93.120 | 95.525 (150 epoch) | 30.240 | 34.684 (250 epoch) | 33.973 | 43.014 (100 epoch) | 19.278 | 22.399 (250 epoch) | 37.031 | 41.576 (450 epoch) | | Avg | 92.614 | 94.251 (100 epoch) | 91.762 | 94.481 (100 epoch) | 89.979 | 90.468 (700 epoch) | 97.661 | 98.640 (700 epoch) | 30.695 | 39.701 (250 epoch) | 35.228 | 44.125 (250 epoch) | 86.846 | 91.494 (250 epoch) | 20.236 | 25.164 (100 epoch) | 39.074 | 46.179 (50 epoch) | 12.411 | 15.755 (100 epoch) | 23.222 | 29.990 (250 epoch) | 2024-04-08 05:40:58,513 - ==> Total time: 6 days, 4:40:52 Eta: 0:00:00 Logged in 'runs/MAMBAADTrainer_configs_mambaad_mambaad_visa_nhcs57c1_20240402-010005' 2024-04-08 05:41:00,784 - finish training