nipunsadvilkar's picture
Saving weights and logs of step 500
41593c6
raw
history blame
74.6 kB
2021-07-09 13:29:42,013 INFO MainThread:41762 [internal.py:wandb_internal():88] W&B internal server running at pid: 41762, started at: 2021-07-09 13:29:42.013449
2021-07-09 13:29:42,015 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: check_version
2021-07-09 13:29:42,015 INFO WriterThread:41762 [datastore.py:open_for_write():80] open: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/run-xrzp5klf.wandb
2021-07-09 13:29:42,016 DEBUG SenderThread:41762 [sender.py:send():179] send: header
2021-07-09 13:29:42,017 DEBUG SenderThread:41762 [sender.py:send_request():193] send_request: check_version
2021-07-09 13:29:42,081 DEBUG SenderThread:41762 [sender.py:send():179] send: run
2021-07-09 13:29:42,149 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: run_start
2021-07-09 13:29:42,150 INFO SenderThread:41762 [dir_watcher.py:__init__():168] watching files in: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files
2021-07-09 13:29:42,150 INFO SenderThread:41762 [sender.py:_start_run_threads():716] run started: xrzp5klf with start time 1625837381
2021-07-09 13:29:42,150 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:29:42,151 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:29:43,152 INFO Thread-8 :41762 [dir_watcher.py:_on_file_created():216] file/dir created: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/wandb-summary.json
2021-07-09 13:29:43,181 DEBUG HandlerThread:41762 [meta.py:__init__():39] meta init
2021-07-09 13:29:43,181 DEBUG HandlerThread:41762 [meta.py:__init__():53] meta init done
2021-07-09 13:29:43,181 DEBUG HandlerThread:41762 [meta.py:probe():210] probe
2021-07-09 13:29:43,182 DEBUG HandlerThread:41762 [meta.py:_setup_git():200] setup git
2021-07-09 13:29:43,208 DEBUG HandlerThread:41762 [meta.py:_setup_git():207] setup git done
2021-07-09 13:29:43,208 DEBUG HandlerThread:41762 [meta.py:_save_pip():57] save pip
2021-07-09 13:29:43,208 DEBUG HandlerThread:41762 [meta.py:_save_pip():71] save pip done
2021-07-09 13:29:43,208 DEBUG HandlerThread:41762 [meta.py:probe():252] probe done
2021-07-09 13:29:43,211 DEBUG SenderThread:41762 [sender.py:send():179] send: files
2021-07-09 13:29:43,212 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-metadata.json with policy now
2021-07-09 13:29:43,219 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 13:29:43,220 DEBUG SenderThread:41762 [sender.py:send_request():193] send_request: stop_status
2021-07-09 13:29:43,277 INFO HandlerThread:41762 [handler.py:handle_tbrecord():547] handling tbrecord: tbrecord {
log_dir: "."
save: true
}
2021-07-09 13:29:43,278 DEBUG HandlerThread:41762 [config_util.py:dict_from_config_file():101] no default config file found in config-defaults.yaml
2021-07-09 13:29:43,286 DEBUG SenderThread:41762 [sender.py:send():179] send: tbrecord
2021-07-09 13:29:43,287 DEBUG SenderThread:41762 [sender.py:send():179] send: files
2021-07-09 13:29:43,287 INFO SenderThread:41762 [sender.py:_save_file():841] saving file events.out.tfevents.1625837383.t1v-n-112df4a9-w-0.40707.3.v2 with policy live
2021-07-09 13:29:43,411 INFO Thread-11 :41762 [upload_job.py:push():137] Uploaded file /tmp/tmp2vrml8zwwandb/2sklq52i-wandb-metadata.json
2021-07-09 13:29:43,472 INFO Thread-14 :41762 [upload_job.py:push():137] Uploaded file /tmp/tmp2vrml8zwwandb/zd2prnhv-events.out.tfevents.1625837383.t1v-n-112df4a9-w-0.40707.3.v2
2021-07-09 13:29:44,151 INFO Thread-8 :41762 [dir_watcher.py:_on_file_created():216] file/dir created: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:29:44,152 INFO Thread-8 :41762 [dir_watcher.py:_on_file_created():216] file/dir created: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/events.out.tfevents.1625837383.t1v-n-112df4a9-w-0.40707.3.v2
2021-07-09 13:29:44,152 INFO Thread-8 :41762 [dir_watcher.py:_on_file_created():216] file/dir created: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/wandb-metadata.json
2021-07-09 13:29:44,152 INFO Thread-8 :41762 [dir_watcher.py:_on_file_created():216] file/dir created: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/requirements.txt
2021-07-09 13:29:46,152 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:29:50,154 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:29:52,155 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:29:58,263 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 13:29:58,264 DEBUG SenderThread:41762 [sender.py:send_request():193] send_request: stop_status
2021-07-09 13:30:11,262 DEBUG SenderThread:41762 [sender.py:send():179] send: stats
2021-07-09 13:30:13,309 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 13:30:13,310 DEBUG SenderThread:41762 [sender.py:send_request():193] send_request: stop_status
2021-07-09 13:30:28,345 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 13:30:28,345 DEBUG SenderThread:41762 [sender.py:send_request():193] send_request: stop_status
2021-07-09 13:30:41,343 DEBUG SenderThread:41762 [sender.py:send():179] send: stats
2021-07-09 13:30:43,379 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 13:30:43,379 DEBUG SenderThread:41762 [sender.py:send_request():193] send_request: stop_status
2021-07-09 13:30:58,411 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 13:30:58,412 DEBUG SenderThread:41762 [sender.py:send_request():193] send_request: stop_status
2021-07-09 13:31:04,189 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:31:06,190 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:31:08,191 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:31:10,192 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:31:11,423 DEBUG SenderThread:41762 [sender.py:send():179] send: stats
2021-07-09 13:31:12,193 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:31:13,451 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 13:31:13,452 DEBUG SenderThread:41762 [sender.py:send_request():193] send_request: stop_status
2021-07-09 13:31:14,194 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:31:16,196 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:31:18,197 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:31:19,197 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/events.out.tfevents.1625837383.t1v-n-112df4a9-w-0.40707.3.v2
2021-07-09 13:31:19,409 INFO Thread-15 :41762 [upload_job.py:push():137] Uploaded file /tmp/tmp2vrml8zwwandb/y4htk22s-events.out.tfevents.1625837383.t1v-n-112df4a9-w-0.40707.3.v2
2021-07-09 13:31:19,477 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,480 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,491 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,502 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,502 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,505 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,510 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,515 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,523 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,523 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,523 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,537 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,537 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,537 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,537 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,538 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,538 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,538 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,538 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,538 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,539 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,539 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,539 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,539 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,540 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,540 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,540 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,540 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,540 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,541 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,541 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,541 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,541 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,541 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,541 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,542 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,542 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,542 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,542 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,542 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,543 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,543 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,543 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,543 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,544 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,544 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,544 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,544 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,544 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,544 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,545 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,545 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,545 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,545 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,545 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,545 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,546 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,546 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,546 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,546 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,546 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,546 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,547 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,547 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,547 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,548 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,548 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,548 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,548 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,548 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,548 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,549 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,549 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,549 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,549 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,549 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,550 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,550 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,550 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,550 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,550 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,551 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,551 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,551 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,551 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,551 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,552 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,552 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,552 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,552 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,552 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,552 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,553 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,553 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,553 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,553 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,553 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,553 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,554 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,554 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,554 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,554 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,554 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,554 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,555 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,555 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,555 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,555 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,555 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,555 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,556 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,556 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,556 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,556 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,556 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,556 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,557 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,557 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,557 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,557 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,557 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,557 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,558 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,558 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,558 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,558 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,558 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,559 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,559 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,559 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,559 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,559 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,560 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,560 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,560 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,560 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,560 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,560 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,561 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,561 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,561 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,561 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,561 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,561 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,562 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,562 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,562 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,562 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,562 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,562 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,563 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,563 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,563 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,563 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,563 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,563 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,564 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,564 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,564 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,564 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,564 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,565 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,565 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,565 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,565 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,565 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,565 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,565 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,566 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,566 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,566 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,566 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,566 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,567 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,567 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,567 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,567 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,567 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,567 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,568 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,568 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,568 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,568 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,568 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,568 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,569 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,569 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,569 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,569 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,569 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,569 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,570 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,570 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,570 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,570 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,571 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,571 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,571 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,571 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,571 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,571 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,572 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,572 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,572 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,572 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,572 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,573 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,573 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,573 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,573 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,573 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,573 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,573 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,574 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,574 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,574 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,574 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,574 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,574 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,575 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,575 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,575 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,575 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,575 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,575 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,576 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,576 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,576 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,576 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,576 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,576 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,576 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,577 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,577 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,577 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,577 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,577 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,577 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,578 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,578 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,578 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,578 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,578 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,578 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,578 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,579 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,579 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,579 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,579 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,579 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,579 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,580 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,580 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,580 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,580 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,580 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,581 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,581 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,581 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,581 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,581 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,582 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,582 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,582 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,582 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,582 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,582 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,583 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,583 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,583 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,583 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,583 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,583 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,584 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,584 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,584 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,584 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,584 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,584 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,584 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,585 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,585 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,585 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,585 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,585 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,585 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,586 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,586 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,586 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,586 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,586 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,586 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,587 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,587 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,587 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,587 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,587 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,587 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,588 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,588 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,588 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,588 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,588 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,588 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,589 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,589 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,589 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,589 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,589 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,590 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,590 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,590 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,590 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,590 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,590 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,591 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,591 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,591 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,591 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,591 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,592 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,592 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,592 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,592 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,592 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,592 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,593 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,593 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,593 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,593 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,593 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,593 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,593 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,594 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,594 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,594 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,594 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,594 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,595 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,595 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,595 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,595 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,595 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,595 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,596 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,596 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,596 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,596 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,596 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,596 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,597 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,597 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,597 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,597 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,597 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,597 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,598 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,598 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,598 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,598 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,598 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,598 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,599 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,599 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,599 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,599 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,599 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,599 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,600 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,600 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,600 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,600 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,600 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,600 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,601 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,601 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,601 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,601 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,601 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,602 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,602 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,602 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,602 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,602 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,603 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,603 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,603 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,603 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,603 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,603 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,604 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,604 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,604 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,604 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,604 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,604 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,605 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,605 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,605 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,605 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,605 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,606 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,606 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,606 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,606 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,606 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,606 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,607 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,607 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,607 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,607 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,607 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,607 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,608 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,608 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,608 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,608 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,608 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,609 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,609 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,609 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,609 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,609 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,609 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,610 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,610 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,610 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,610 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,610 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,611 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,611 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,611 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,611 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,611 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,612 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:19,612 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:19,612 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:19,612 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:20,198 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/wandb-summary.json
2021-07-09 13:31:20,198 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:31:22,199 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:31:24,200 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/events.out.tfevents.1625837383.t1v-n-112df4a9-w-0.40707.3.v2
2021-07-09 13:31:24,201 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:31:26,201 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:31:28,506 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 13:31:28,507 DEBUG SenderThread:41762 [sender.py:send_request():193] send_request: stop_status
2021-07-09 13:31:36,206 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:31:38,207 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:31:40,208 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:31:41,499 DEBUG SenderThread:41762 [sender.py:send():179] send: stats
2021-07-09 13:31:42,209 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:31:43,556 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 13:31:43,556 DEBUG SenderThread:41762 [sender.py:send_request():193] send_request: stop_status
2021-07-09 13:31:44,210 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:31:46,211 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:31:48,212 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:31:50,213 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:31:52,214 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:31:53,536 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:53,536 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:53,541 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:53,541 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:53,542 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:53,542 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:53,542 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:53,543 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:53,543 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:53,543 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:53,543 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:53,544 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:53,544 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:53,544 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:53,544 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:53,544 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:53,545 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:53,545 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:53,545 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:53,545 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:53,545 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:53,546 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:53,546 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:53,546 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:53,546 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:53,546 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:53,546 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:53,547 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:53,547 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:53,547 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:53,547 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:31:53,547 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:31:53,548 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:31:54,215 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/events.out.tfevents.1625837383.t1v-n-112df4a9-w-0.40707.3.v2
2021-07-09 13:31:54,216 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/wandb-summary.json
2021-07-09 13:31:54,216 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:31:56,216 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:31:58,588 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: stop_status
2021-07-09 13:31:58,589 DEBUG SenderThread:41762 [sender.py:send_request():193] send_request: stop_status
2021-07-09 13:32:06,221 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:32:08,222 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:32:10,223 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:32:11,569 DEBUG SenderThread:41762 [sender.py:send():179] send: stats
2021-07-09 13:32:11,972 DEBUG SenderThread:41762 [sender.py:send():179] send: telemetry
2021-07-09 13:32:11,972 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-09 13:32:11,972 DEBUG SenderThread:41762 [sender.py:send():179] send: exit
2021-07-09 13:32:11,973 INFO SenderThread:41762 [sender.py:send_exit():287] handling exit code: 0
2021-07-09 13:32:11,973 INFO SenderThread:41762 [sender.py:send_exit():295] send defer
2021-07-09 13:32:11,973 DEBUG SenderThread:41762 [sender.py:send_request():193] send_request: poll_exit
2021-07-09 13:32:11,974 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: defer
2021-07-09 13:32:11,974 INFO HandlerThread:41762 [handler.py:handle_request_defer():141] handle defer: 0
2021-07-09 13:32:11,974 DEBUG SenderThread:41762 [sender.py:send_request():193] send_request: defer
2021-07-09 13:32:11,974 INFO SenderThread:41762 [sender.py:send_request_defer():304] handle sender defer: 0
2021-07-09 13:32:11,974 INFO SenderThread:41762 [sender.py:transition_state():308] send defer: 1
2021-07-09 13:32:11,974 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: defer
2021-07-09 13:32:11,974 INFO HandlerThread:41762 [handler.py:handle_request_defer():141] handle defer: 1
2021-07-09 13:32:12,071 DEBUG SenderThread:41762 [sender.py:send_request():193] send_request: defer
2021-07-09 13:32:12,071 INFO SenderThread:41762 [sender.py:send_request_defer():304] handle sender defer: 1
2021-07-09 13:32:12,071 INFO SenderThread:41762 [sender.py:transition_state():308] send defer: 2
2021-07-09 13:32:12,071 DEBUG SenderThread:41762 [sender.py:send():179] send: stats
2021-07-09 13:32:12,072 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: defer
2021-07-09 13:32:12,072 INFO HandlerThread:41762 [handler.py:handle_request_defer():141] handle defer: 2
2021-07-09 13:32:12,224 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:32:13,225 INFO Thread-8 :41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/config.yaml
2021-07-09 13:32:18,572 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-09 13:32:18,572 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-09 13:32:18,573 DEBUG SenderThread:41762 [sender.py:send_request():193] send_request: defer
2021-07-09 13:32:18,573 INFO SenderThread:41762 [sender.py:send_request_defer():304] handle sender defer: 2
2021-07-09 13:32:18,573 INFO SenderThread:41762 [sender.py:transition_state():308] send defer: 3
2021-07-09 13:32:18,573 DEBUG SenderThread:41762 [sender.py:send_request():193] send_request: poll_exit
2021-07-09 13:32:18,573 DEBUG SenderThread:41762 [sender.py:send_request():193] send_request: poll_exit
2021-07-09 13:32:18,574 DEBUG SenderThread:41762 [sender.py:send():179] send: history
2021-07-09 13:32:18,574 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:32:18,576 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: defer
2021-07-09 13:32:18,576 INFO HandlerThread:41762 [handler.py:handle_request_defer():141] handle defer: 3
2021-07-09 13:32:18,577 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:32:18,577 DEBUG SenderThread:41762 [sender.py:send():179] send: summary
2021-07-09 13:32:18,578 INFO SenderThread:41762 [sender.py:_save_file():841] saving file wandb-summary.json with policy end
2021-07-09 13:32:18,578 DEBUG SenderThread:41762 [sender.py:send_request():193] send_request: defer
2021-07-09 13:32:18,578 INFO SenderThread:41762 [sender.py:send_request_defer():304] handle sender defer: 3
2021-07-09 13:32:18,578 INFO SenderThread:41762 [sender.py:transition_state():308] send defer: 4
2021-07-09 13:32:18,578 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: defer
2021-07-09 13:32:18,578 INFO HandlerThread:41762 [handler.py:handle_request_defer():141] handle defer: 4
2021-07-09 13:32:18,579 DEBUG SenderThread:41762 [sender.py:send_request():193] send_request: defer
2021-07-09 13:32:18,579 INFO SenderThread:41762 [sender.py:send_request_defer():304] handle sender defer: 4
2021-07-09 13:32:18,579 INFO SenderThread:41762 [sender.py:transition_state():308] send defer: 5
2021-07-09 13:32:18,579 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: defer
2021-07-09 13:32:18,579 INFO HandlerThread:41762 [handler.py:handle_request_defer():141] handle defer: 5
2021-07-09 13:32:18,579 DEBUG SenderThread:41762 [sender.py:send_request():193] send_request: defer
2021-07-09 13:32:18,579 INFO SenderThread:41762 [sender.py:send_request_defer():304] handle sender defer: 5
2021-07-09 13:32:18,579 INFO SenderThread:41762 [dir_watcher.py:finish():282] shutting down directory watcher
2021-07-09 13:32:18,678 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-09 13:32:19,227 INFO SenderThread:41762 [dir_watcher.py:_on_file_modified():229] file/dir modified: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/wandb-summary.json
2021-07-09 13:32:19,227 INFO SenderThread:41762 [dir_watcher.py:finish():312] scan: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files
2021-07-09 13:32:19,227 INFO SenderThread:41762 [dir_watcher.py:finish():318] scan save: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/requirements.txt requirements.txt
2021-07-09 13:32:19,228 INFO SenderThread:41762 [dir_watcher.py:finish():318] scan save: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/events.out.tfevents.1625837383.t1v-n-112df4a9-w-0.40707.3.v2 events.out.tfevents.1625837383.t1v-n-112df4a9-w-0.40707.3.v2
2021-07-09 13:32:19,228 INFO SenderThread:41762 [dir_watcher.py:finish():318] scan save: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log output.log
2021-07-09 13:32:19,228 INFO SenderThread:41762 [dir_watcher.py:finish():318] scan save: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/wandb-metadata.json wandb-metadata.json
2021-07-09 13:32:19,228 INFO SenderThread:41762 [dir_watcher.py:finish():318] scan save: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/config.yaml config.yaml
2021-07-09 13:32:19,229 INFO SenderThread:41762 [dir_watcher.py:finish():318] scan save: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/wandb-summary.json wandb-summary.json
2021-07-09 13:32:19,229 INFO SenderThread:41762 [sender.py:transition_state():308] send defer: 6
2021-07-09 13:32:19,229 DEBUG SenderThread:41762 [sender.py:send_request():193] send_request: poll_exit
2021-07-09 13:32:19,233 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: defer
2021-07-09 13:32:19,233 INFO HandlerThread:41762 [handler.py:handle_request_defer():141] handle defer: 6
2021-07-09 13:32:19,233 DEBUG SenderThread:41762 [sender.py:send_request():193] send_request: defer
2021-07-09 13:32:19,233 INFO SenderThread:41762 [sender.py:send_request_defer():304] handle sender defer: 6
2021-07-09 13:32:19,233 INFO SenderThread:41762 [file_pusher.py:finish():177] shutting down file pusher
2021-07-09 13:32:19,331 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-09 13:32:19,331 DEBUG SenderThread:41762 [sender.py:send_request():193] send_request: poll_exit
2021-07-09 13:32:19,433 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-09 13:32:19,433 DEBUG SenderThread:41762 [sender.py:send_request():193] send_request: poll_exit
2021-07-09 13:32:19,444 INFO Thread-19 :41762 [upload_job.py:push():137] Uploaded file /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/config.yaml
2021-07-09 13:32:19,448 INFO Thread-18 :41762 [upload_job.py:push():137] Uploaded file /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/output.log
2021-07-09 13:32:19,449 INFO Thread-16 :41762 [upload_job.py:push():137] Uploaded file /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/requirements.txt
2021-07-09 13:32:19,460 INFO Thread-17 :41762 [upload_job.py:push():137] Uploaded file /tmp/tmp2vrml8zwwandb/1110a0ll-events.out.tfevents.1625837383.t1v-n-112df4a9-w-0.40707.3.v2
2021-07-09 13:32:19,462 INFO Thread-20 :41762 [upload_job.py:push():137] Uploaded file /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/files/wandb-summary.json
2021-07-09 13:32:19,535 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-09 13:32:19,535 DEBUG SenderThread:41762 [sender.py:send_request():193] send_request: poll_exit
2021-07-09 13:32:19,636 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-09 13:32:19,637 DEBUG SenderThread:41762 [sender.py:send_request():193] send_request: poll_exit
2021-07-09 13:32:19,663 INFO Thread-7 :41762 [sender.py:transition_state():308] send defer: 7
2021-07-09 13:32:19,663 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: defer
2021-07-09 13:32:19,663 INFO HandlerThread:41762 [handler.py:handle_request_defer():141] handle defer: 7
2021-07-09 13:32:19,664 DEBUG SenderThread:41762 [sender.py:send_request():193] send_request: defer
2021-07-09 13:32:19,664 INFO SenderThread:41762 [sender.py:send_request_defer():304] handle sender defer: 7
2021-07-09 13:32:19,711 INFO SenderThread:41762 [sender.py:transition_state():308] send defer: 8
2021-07-09 13:32:19,712 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: defer
2021-07-09 13:32:19,712 INFO HandlerThread:41762 [handler.py:handle_request_defer():141] handle defer: 8
2021-07-09 13:32:19,712 DEBUG SenderThread:41762 [sender.py:send_request():193] send_request: defer
2021-07-09 13:32:19,712 INFO SenderThread:41762 [sender.py:send_request_defer():304] handle sender defer: 8
2021-07-09 13:32:19,712 INFO SenderThread:41762 [sender.py:transition_state():308] send defer: 9
2021-07-09 13:32:19,713 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: defer
2021-07-09 13:32:19,713 INFO HandlerThread:41762 [handler.py:handle_request_defer():141] handle defer: 9
2021-07-09 13:32:19,713 DEBUG SenderThread:41762 [sender.py:send():179] send: final
2021-07-09 13:32:19,713 DEBUG SenderThread:41762 [sender.py:send():179] send: footer
2021-07-09 13:32:19,713 DEBUG SenderThread:41762 [sender.py:send_request():193] send_request: defer
2021-07-09 13:32:19,713 INFO SenderThread:41762 [sender.py:send_request_defer():304] handle sender defer: 9
2021-07-09 13:32:19,738 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: poll_exit
2021-07-09 13:32:19,739 DEBUG SenderThread:41762 [sender.py:send_request():193] send_request: poll_exit
2021-07-09 13:32:19,739 INFO SenderThread:41762 [file_pusher.py:join():182] waiting for file pusher
2021-07-09 13:32:19,740 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: get_summary
2021-07-09 13:32:19,741 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: sampled_history
2021-07-09 13:32:19,743 DEBUG HandlerThread:41762 [handler.py:handle_request():124] handle_request: shutdown
2021-07-09 13:32:19,743 INFO HandlerThread:41762 [handler.py:finish():638] shutting down handler
2021-07-09 13:32:20,713 INFO WriterThread:41762 [datastore.py:close():288] close: /home/nipunsadvilkar/sample/roberta-base-mr/wandb/run-20210709_132941-xrzp5klf/run-xrzp5klf.wandb
2021-07-09 13:32:20,739 INFO SenderThread:41762 [sender.py:finish():945] shutting down sender
2021-07-09 13:32:20,740 INFO SenderThread:41762 [file_pusher.py:finish():177] shutting down file pusher
2021-07-09 13:32:20,740 INFO SenderThread:41762 [file_pusher.py:join():182] waiting for file pusher
2021-07-09 13:32:20,742 INFO MainThread:41762 [internal.py:handle_exit():78] Internal process exited