2023-09-08 04:25:27,759 INFO StreamThr :896010 [internal.py:wandb_internal():86] W&B internal server running at pid: 896010, started at: 2023-09-08 04:25:27.756517 2023-09-08 04:25:27,761 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status 2023-09-08 04:25:27,773 INFO WriterThread:896010 [datastore.py:open_for_write():85] open: ./wandb/run-20230908_042527-jggbeix7/run-jggbeix7.wandb 2023-09-08 04:25:27,775 DEBUG SenderThread:896010 [sender.py:send():369] send: header 2023-09-08 04:25:27,781 DEBUG SenderThread:896010 [sender.py:send():369] send: run 2023-09-08 04:25:27,787 INFO SenderThread:896010 [sender.py:_maybe_setup_resume():737] checking resume status for ckadirt/brain2music/jggbeix7 2023-09-08 04:25:28,326 INFO SenderThread:896010 [dir_watcher.py:__init__():211] watching files in: ./wandb/run-20230908_042527-jggbeix7/files 2023-09-08 04:25:28,326 INFO SenderThread:896010 [sender.py:_start_run_threads():1100] run started: jggbeix7 with start time 1694147127.764063 2023-09-08 04:25:28,327 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:25:28,330 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:25:28,334 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: check_version 2023-09-08 04:25:28,334 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: check_version 2023-09-08 04:25:28,397 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: run_start 2023-09-08 04:25:28,407 DEBUG HandlerThread:896010 [system_info.py:__init__():31] System info init 2023-09-08 04:25:28,407 DEBUG HandlerThread:896010 [system_info.py:__init__():46] System info init done 2023-09-08 04:25:28,407 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 04:25:28,408 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 04:25:28,408 INFO HandlerThread:896010 [system_monitor.py:probe():201] Collecting system info 2023-09-08 04:25:28,408 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 04:25:28,409 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 04:25:28,409 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 04:25:28,410 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 04:25:28,410 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 04:25:28,462 DEBUG HandlerThread:896010 [system_info.py:probe():195] Probing system 2023-09-08 04:25:28,465 DEBUG HandlerThread:896010 [system_info.py:_probe_git():180] Probing git 2023-09-08 04:25:28,471 ERROR HandlerThread:896010 [gitlib.py:root():89] git root error: Cmd('git') failed due to: exit code(128) cmdline: git rev-parse --show-toplevel stderr: 'fatal: detected dubious ownership in repository at '/fsx' To add an exception for this directory, call: git config --global --add safe.directory /fsx' 2023-09-08 04:25:28,471 DEBUG HandlerThread:896010 [system_info.py:_probe_git():188] Probing git done 2023-09-08 04:25:28,472 DEBUG HandlerThread:896010 [system_info.py:probe():240] Probing system done 2023-09-08 04:25:28,472 DEBUG HandlerThread:896010 [system_monitor.py:probe():210] {'os': 'Linux-5.15.0-1037-aws-x86_64-with-glibc2.31', 'python': '3.10.8', 'heartbeatAt': '2023-09-08T04:25:28.462900', 'startedAt': '2023-09-08T04:25:27.718376', 'docker': None, 'cuda': None, 'args': None, 'state': 'running', 'program': 'b2m/src/MLPencoder.ipynb', 'root': '/fsx/proj-fmri/ckadirt', 'git': {'remote': None, 'commit': None}, 'email': None, 'host': 'ip-26-0-150-76', 'username': 'ckadirt', 'executable': '/admin/home-ckadirt/miniconda3/envs/mindeye/bin/python', 'cpu_count': 48, 'cpu_count_logical': 96, 'cpu_freq': {'current': 2983.9045833333334, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2815.061, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}, {'current': 2999.998, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 232.5829315185547, 'used': 89.16302490234375}, 'gpu': 'NVIDIA A100-SXM4-40GB', 'gpu_count': 8, 'gpu_devices': [{'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}], 'memory': {'total': 1121.815086364746}} 2023-09-08 04:25:28,472 INFO HandlerThread:896010 [system_monitor.py:probe():211] Finished collecting system info 2023-09-08 04:25:28,472 INFO HandlerThread:896010 [system_monitor.py:probe():214] Publishing system info 2023-09-08 04:25:28,472 DEBUG HandlerThread:896010 [system_info.py:_save_pip():51] Saving list of pip packages installed into the current environment 2023-09-08 04:25:28,475 DEBUG HandlerThread:896010 [system_info.py:_save_pip():67] Saving pip packages done 2023-09-08 04:25:28,475 DEBUG HandlerThread:896010 [system_info.py:_save_conda():74] Saving list of conda packages installed into the current environment 2023-09-08 04:25:29,330 INFO Thread-12 :896010 [dir_watcher.py:_on_file_created():272] file/dir created: ./wandb/run-20230908_042527-jggbeix7/files/requirements.txt 2023-09-08 04:25:29,330 INFO Thread-12 :896010 [dir_watcher.py:_on_file_created():272] file/dir created: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:25:29,330 INFO Thread-12 :896010 [dir_watcher.py:_on_file_created():272] file/dir created: ./wandb/run-20230908_042527-jggbeix7/files/conda-environment.yaml 2023-09-08 04:25:35,230 DEBUG HandlerThread:896010 [system_info.py:_save_conda():86] Saving conda packages done 2023-09-08 04:25:35,230 DEBUG HandlerThread:896010 [system_info.py:_save_code():89] Saving code 2023-09-08 04:25:35,230 WARNING HandlerThread:896010 [system_info.py:_save_code():91] unable to save code -- program entry not found 2023-09-08 04:25:35,230 DEBUG HandlerThread:896010 [system_info.py:_save_patches():127] Saving git patches 2023-09-08 04:25:35,235 ERROR HandlerThread:896010 [gitlib.py:root():89] git root error: Cmd('git') failed due to: exit code(128) cmdline: git rev-parse --show-toplevel stderr: 'fatal: detected dubious ownership in repository at '/fsx' To add an exception for this directory, call: git config --global --add safe.directory /fsx' 2023-09-08 04:25:35,244 DEBUG HandlerThread:896010 [system_info.py:_save_patches():169] Saving git patches done 2023-09-08 04:25:35,248 INFO HandlerThread:896010 [system_monitor.py:probe():216] Finished publishing system info 2023-09-08 04:25:35,252 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:25:35,252 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: keepalive 2023-09-08 04:25:35,253 DEBUG SenderThread:896010 [sender.py:send():369] send: files 2023-09-08 04:25:35,253 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-metadata.json with policy now 2023-09-08 04:25:35,267 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:25:35,267 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:25:35,338 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/conda-environment.yaml 2023-09-08 04:25:35,338 INFO Thread-12 :896010 [dir_watcher.py:_on_file_created():272] file/dir created: ./wandb/run-20230908_042527-jggbeix7/files/wandb-metadata.json 2023-09-08 04:25:35,622 DEBUG SenderThread:896010 [sender.py:send():369] send: telemetry 2023-09-08 04:25:35,622 DEBUG SenderThread:896010 [sender.py:send():369] send: metric 2023-09-08 04:25:35,622 DEBUG SenderThread:896010 [sender.py:send():369] send: telemetry 2023-09-08 04:25:35,623 DEBUG SenderThread:896010 [sender.py:send():369] send: metric 2023-09-08 04:25:35,623 WARNING SenderThread:896010 [sender.py:send_metric():1305] Seen metric with glob (shouldn't happen) 2023-09-08 04:25:35,878 INFO wandb-upload_0:896010 [upload_job.py:push():133] Uploaded file /tmp/tmprs4v0lymwandb/7p6ghcn7-wandb-metadata.json 2023-09-08 04:25:36,341 INFO Thread-12 :896010 [dir_watcher.py:_on_file_created():272] file/dir created: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:25:38,344 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:25:38,675 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:25:39,345 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:25:41,347 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:25:41,871 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:25:41,873 DEBUG SenderThread:896010 [sender.py:send():369] send: metric 2023-09-08 04:25:41,873 DEBUG SenderThread:896010 [sender.py:send():369] send: metric 2023-09-08 04:25:41,874 DEBUG SenderThread:896010 [sender.py:send():369] send: metric 2023-09-08 04:25:41,874 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:25:41,874 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:25:41,876 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:25:42,350 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:25:42,350 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:25:43,351 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:25:43,682 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:25:43,932 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:25:43,933 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:25:43,933 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:25:43,936 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:25:44,355 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:25:45,357 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:25:45,977 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:25:45,978 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:25:45,978 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:25:45,980 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:25:46,359 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:25:47,361 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:25:48,001 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:25:48,002 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:25:48,002 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:25:48,005 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:25:48,364 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:25:48,789 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:25:49,366 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:25:50,048 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:25:50,049 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:25:50,049 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:25:50,051 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:25:50,268 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:25:50,268 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:25:50,369 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:25:51,371 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:25:52,087 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:25:52,088 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:25:52,089 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:25:52,091 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:25:52,374 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:25:53,384 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:25:53,857 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:25:54,094 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:25:54,095 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:25:54,095 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:25:54,097 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:25:54,387 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:25:55,389 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:25:56,075 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:25:56,077 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:25:56,077 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:25:56,080 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:25:56,393 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:25:57,394 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:25:58,118 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:25:58,119 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:25:58,119 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:25:58,122 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:25:58,397 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:25:58,893 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:25:59,400 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:25:59,400 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/config.yaml 2023-09-08 04:26:00,138 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:26:00,139 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:26:00,139 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:26:00,142 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:26:00,403 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:26:01,404 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:26:02,271 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:26:02,272 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:26:02,273 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:26:02,275 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:26:02,407 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:26:03,409 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:26:04,049 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:26:04,299 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:26:04,300 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:26:04,300 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:26:04,302 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:26:04,412 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:26:05,267 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:26:05,268 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:26:05,364 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:26:05,413 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:26:05,440 DEBUG SenderThread:896010 [sender.py:send():369] send: metric 2023-09-08 04:26:05,440 DEBUG SenderThread:896010 [sender.py:send():369] send: metric 2023-09-08 04:26:05,440 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:26:05,441 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:26:05,443 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:26:06,416 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:26:07,417 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:26:09,444 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:26:14,445 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:26:19,446 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:26:22,400 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:26:22,400 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:26:24,571 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:26:28,411 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 04:26:28,415 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:26:30,419 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:26:31,451 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/config.yaml 2023-09-08 04:26:33,456 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:26:34,228 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:26:34,229 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:26:34,230 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:26:34,233 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:26:34,459 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:26:35,461 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:26:35,617 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:26:35,784 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:26:35,785 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:26:36,271 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:26:36,272 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:26:36,272 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:26:36,274 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:26:36,470 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:26:37,466 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:26:38,303 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:26:38,305 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:26:38,305 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:26:38,308 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:26:38,469 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:26:39,472 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:26:40,312 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:26:40,314 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:26:40,314 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:26:40,324 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:26:40,475 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:26:40,678 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:26:41,477 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:26:42,344 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:26:42,345 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:26:42,345 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:26:42,349 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:26:42,482 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:26:43,484 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:26:44,380 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:26:44,381 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:26:44,382 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:26:44,385 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:26:44,488 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:26:45,489 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:26:45,738 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:26:46,392 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:26:46,393 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:26:46,394 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:26:46,396 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:26:46,492 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:26:47,494 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:26:48,423 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:26:48,424 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:26:48,424 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:26:48,427 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:26:48,497 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:26:49,499 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:26:50,457 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:26:50,458 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:26:50,458 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:26:50,462 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:26:50,503 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:26:50,784 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:26:50,785 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:26:50,955 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:26:51,504 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:26:52,507 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:26:52,508 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:26:52,508 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:26:52,511 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:26:53,509 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:26:53,510 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:26:54,542 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:26:54,553 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:26:54,554 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:26:54,556 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:26:55,514 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:26:55,515 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:26:55,960 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:26:56,606 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:26:56,606 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:26:56,607 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:26:56,609 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:26:57,519 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:26:57,520 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:26:57,696 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:26:57,697 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:26:57,697 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:26:57,702 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:26:58,417 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:26:58,523 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:26:59,524 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:27:01,418 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:27:06,419 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:27:08,664 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:27:08,664 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:27:11,846 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:27:16,847 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:27:20,912 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:27:20,912 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:27:22,082 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:27:25,563 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:27:26,449 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:27:26,450 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:27:26,450 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:27:26,455 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:27:26,566 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:27:27,219 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:27:27,568 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:27:28,420 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:27:28,478 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:27:28,479 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:27:28,480 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:27:28,482 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:27:28,571 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:27:29,573 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:27:30,516 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:27:30,517 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:27:30,517 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:27:30,520 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:27:30,576 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:27:31,578 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:27:32,297 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:27:32,545 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:27:32,546 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:27:32,546 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:27:32,549 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:27:32,581 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:27:33,584 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:27:34,581 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:27:34,582 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:27:34,582 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:27:34,586 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:27:34,587 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:27:35,588 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:27:35,938 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:27:35,938 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:27:36,580 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:27:36,581 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:27:36,582 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:27:36,584 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:27:36,592 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:27:37,345 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:27:37,593 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:27:38,579 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:27:38,581 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:27:38,582 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:27:38,585 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:27:38,598 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:27:39,600 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:27:40,601 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:27:40,603 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:27:40,603 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:27:40,605 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:27:41,605 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:27:41,605 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:27:42,386 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:27:42,636 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:27:42,637 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:27:42,638 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:27:42,641 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:27:43,610 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:27:43,610 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:27:44,660 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:27:44,661 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:27:44,661 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:27:44,664 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:27:45,615 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:27:45,615 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:27:46,699 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:27:46,700 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:27:46,700 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:27:46,703 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:27:47,454 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:27:47,620 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:27:47,621 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:27:48,716 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:27:48,716 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:27:48,717 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:27:48,719 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:27:49,626 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:27:49,626 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:27:49,771 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:27:49,772 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:27:49,772 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:27:49,777 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:27:50,629 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:27:51,630 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:27:52,783 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:27:53,356 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:27:53,357 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:27:58,423 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:27:58,424 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:28:03,424 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:28:06,931 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:28:06,932 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:28:08,883 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:28:13,884 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:28:18,657 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:28:18,658 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:28:18,659 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:28:18,664 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:28:18,673 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:28:19,023 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:28:19,675 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:28:20,699 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:28:20,700 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:28:20,701 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:28:20,703 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:28:21,680 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:28:21,680 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:28:21,833 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:28:21,834 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:28:22,736 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:28:22,737 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:28:22,737 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:28:22,740 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:28:23,685 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:28:23,685 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:28:24,121 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:28:24,777 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:28:24,778 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:28:24,778 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:28:24,781 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:28:25,690 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:28:25,691 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:28:26,806 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:28:26,807 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:28:26,807 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:28:26,810 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:28:27,695 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:28:27,696 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:28:28,426 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:28:28,847 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:28:28,848 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:28:28,849 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:28:28,853 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:28:29,212 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:28:29,701 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:28:29,702 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:28:30,895 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:28:30,897 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:28:30,897 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:28:30,901 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:28:31,707 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:28:31,707 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:28:32,935 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:28:32,936 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:28:32,936 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:28:32,939 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:28:33,711 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:28:33,712 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:28:34,320 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:28:34,988 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:28:34,989 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:28:34,989 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:28:34,992 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:28:35,716 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:28:35,717 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:28:36,834 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:28:36,834 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:28:37,012 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:28:37,013 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:28:37,013 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:28:37,015 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:28:37,722 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:28:37,722 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:28:39,061 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:28:39,062 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:28:39,062 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:28:39,065 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:28:39,428 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:28:39,727 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:28:39,728 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:28:41,154 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:28:41,155 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:28:41,155 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:28:41,159 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:28:41,733 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:28:41,733 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:28:42,268 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:28:42,269 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:28:42,269 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:28:42,272 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:28:42,736 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:28:43,738 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:28:45,288 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:28:50,289 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:28:53,544 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:28:53,544 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:28:55,718 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:28:58,427 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:29:01,429 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:29:06,429 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:29:08,124 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:29:08,125 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:29:11,678 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:29:13,711 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:29:13,712 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:29:13,712 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:29:13,716 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:29:13,785 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:29:13,785 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:29:15,764 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:29:15,765 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:29:15,766 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:29:15,769 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:29:15,790 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:29:15,791 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:29:16,780 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:29:17,794 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:29:17,853 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:29:17,854 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:29:17,854 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:29:17,857 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:29:18,798 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:29:19,800 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:29:19,899 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:29:19,900 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:29:19,900 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:29:19,903 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:29:20,803 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:29:21,805 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:29:21,887 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:29:21,933 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:29:21,934 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:29:21,934 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:29:21,937 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:29:22,809 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:29:23,125 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:29:23,125 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:29:23,811 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:29:23,969 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:29:23,970 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:29:23,970 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:29:23,974 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:29:24,815 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:29:25,817 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:29:26,011 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:29:26,012 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:29:26,012 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:29:26,015 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:29:26,820 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:29:26,985 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:29:27,823 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:29:28,063 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:29:28,064 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:29:28,065 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:29:28,069 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:29:28,430 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:29:28,827 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:29:29,828 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:29:30,145 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:29:30,146 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:29:30,146 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:29:30,149 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:29:30,832 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:29:31,834 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:29:32,131 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:29:32,178 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:29:32,179 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:29:32,180 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:29:32,183 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:29:32,837 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:29:33,839 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:29:34,225 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:29:34,226 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:29:34,226 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:29:34,229 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:29:34,843 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:29:35,845 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:29:36,265 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:29:36,265 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:29:36,265 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:29:36,268 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:29:36,853 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:29:37,140 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:29:37,359 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:29:37,360 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:29:37,360 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:29:37,362 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:29:37,856 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:29:37,856 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:29:39,860 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:29:40,938 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:29:40,939 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:29:43,104 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:29:48,105 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:29:53,106 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:29:55,584 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:29:55,585 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:29:58,515 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:29:58,517 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:30:03,517 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:30:05,901 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:30:06,072 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:30:06,073 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:30:06,073 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:30:06,086 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:30:06,904 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:30:07,906 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:30:08,071 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:30:08,072 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:30:08,073 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:30:08,075 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:30:08,645 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:30:08,909 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:30:09,911 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:30:10,128 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:30:10,129 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:30:10,130 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:30:10,133 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:30:10,585 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:30:10,585 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:30:10,914 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:30:11,916 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:30:12,189 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:30:12,190 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:30:12,191 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:30:12,193 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:30:12,920 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:30:13,785 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:30:13,922 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:30:14,236 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:30:14,237 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:30:14,238 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:30:14,240 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:30:14,925 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:30:15,928 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:30:16,243 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:30:16,244 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:30:16,244 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:30:16,248 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:30:16,931 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:30:17,933 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:30:18,275 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:30:18,276 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:30:18,277 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:30:18,279 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:30:18,847 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:30:18,936 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:30:19,939 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:30:20,379 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:30:20,380 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:30:20,380 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:30:20,384 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:30:20,942 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:30:21,944 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:30:22,428 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:30:22,429 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:30:22,429 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:30:22,432 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:30:22,947 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:30:23,949 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:30:24,017 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:30:24,472 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:30:24,473 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:30:24,473 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:30:24,477 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:30:24,953 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:30:25,585 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:30:25,585 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:30:25,955 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:30:26,519 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:30:26,521 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:30:26,521 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:30:26,524 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:30:26,958 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:30:27,960 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:30:28,434 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:30:28,556 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:30:28,557 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:30:28,557 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:30:28,560 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:30:28,963 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:30:29,069 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:30:29,631 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:30:29,632 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:30:29,632 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:30:29,635 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:30:29,966 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:30:29,966 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:30:31,970 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:30:34,644 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:30:39,645 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:30:40,982 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:30:40,982 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:30:45,104 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:30:50,105 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:30:55,415 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:30:55,609 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:30:55,610 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:30:58,437 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:30:59,062 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:30:59,064 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:30:59,064 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:30:59,067 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:31:00,016 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:31:00,016 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:31:00,440 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:31:01,091 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:31:01,092 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:31:01,092 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:31:01,095 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:31:02,021 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:31:02,021 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:31:03,127 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:31:03,128 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:31:03,129 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:31:03,132 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:31:04,027 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:31:04,027 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:31:05,170 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:31:05,171 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:31:05,171 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:31:05,174 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:31:05,549 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:31:06,033 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:31:06,033 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:31:07,227 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:31:07,228 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:31:07,228 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:31:07,232 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:31:08,038 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:31:08,038 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:31:09,254 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:31:09,255 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:31:09,255 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:31:09,258 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:31:10,043 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:31:10,044 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:31:10,610 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:31:10,610 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:31:10,781 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:31:11,294 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:31:11,295 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:31:11,295 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:31:11,298 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:31:12,048 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:31:12,049 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:31:13,343 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:31:13,344 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:31:13,344 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:31:13,347 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:31:14,058 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:31:14,059 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:31:15,403 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:31:15,404 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:31:15,405 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:31:15,409 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:31:15,975 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:31:16,063 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:31:16,064 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:31:17,430 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:31:17,431 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:31:17,431 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:31:17,434 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:31:18,069 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:31:18,069 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:31:19,447 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:31:19,448 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:31:19,448 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:31:19,451 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:31:20,074 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:31:20,074 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:31:21,002 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:31:21,441 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:31:21,442 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:31:21,442 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:31:21,444 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:31:22,079 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:31:22,080 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:31:22,544 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:31:22,545 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:31:22,545 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:31:22,550 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:31:23,083 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:31:24,085 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:31:26,005 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:31:26,005 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:31:26,102 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:31:28,440 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:31:31,441 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:31:36,442 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:31:40,827 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:31:40,828 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:31:41,998 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:31:46,998 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:31:52,187 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:31:53,463 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:31:53,464 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:31:53,464 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:31:53,467 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:31:54,130 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:31:54,130 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:31:55,534 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:31:55,535 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:31:55,535 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:31:55,538 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:31:55,827 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:31:55,828 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:31:56,134 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:31:56,135 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:31:57,350 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:31:57,596 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:31:57,597 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:31:57,597 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:31:57,600 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:31:58,139 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:31:58,140 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:31:58,443 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:31:59,758 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:31:59,759 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:31:59,759 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:31:59,767 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:32:00,144 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:32:00,144 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:32:01,785 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:32:01,786 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:32:01,787 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:32:01,789 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:32:02,149 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:32:02,149 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:32:02,354 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:32:03,803 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:32:03,805 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:32:03,805 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:32:03,808 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:32:04,153 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:32:04,154 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:32:05,851 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:32:05,852 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:32:05,852 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:32:05,854 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:32:06,158 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:32:06,158 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:32:07,437 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:32:07,884 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:32:07,885 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:32:07,885 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:32:07,888 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:32:08,163 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:32:08,164 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:32:09,932 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:32:09,933 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:32:09,934 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:32:09,936 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:32:10,168 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:32:10,168 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:32:10,827 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:32:10,828 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:32:11,947 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:32:11,948 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:32:11,948 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:32:11,951 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:32:12,173 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:32:12,174 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:32:12,504 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:32:13,967 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:32:13,968 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:32:13,968 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:32:13,972 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:32:14,178 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:32:14,178 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:32:15,999 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:32:15,999 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:32:16,000 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:32:16,002 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:32:16,183 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:32:16,183 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:32:17,122 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:32:17,123 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:32:17,123 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:32:17,128 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:32:17,185 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:32:18,143 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:32:18,186 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:32:20,189 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:32:23,144 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:32:28,220 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:32:28,221 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:32:28,390 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:32:28,444 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:32:33,445 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:32:38,446 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:32:41,284 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:32:41,284 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:32:43,819 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:32:45,866 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:32:45,867 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:32:45,867 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:32:45,871 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:32:46,228 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:32:46,228 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:32:47,894 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:32:47,895 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:32:47,896 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:32:47,898 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:32:48,232 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:32:48,233 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:32:48,872 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:32:49,934 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:32:49,935 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:32:49,935 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:32:49,938 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:32:50,237 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:32:50,238 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:32:51,967 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:32:51,969 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:32:51,969 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:32:51,971 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:32:52,242 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:32:52,242 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:32:53,963 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:32:54,010 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:32:54,011 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:32:54,012 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:32:54,015 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:32:54,247 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:32:54,247 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:32:56,042 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:32:56,043 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:32:56,044 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:32:56,046 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:32:56,252 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:32:56,252 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:32:56,284 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:32:56,284 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:32:58,055 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:32:58,056 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:32:58,056 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:32:58,060 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:32:58,256 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:32:58,257 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:32:58,447 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:32:59,015 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:33:00,073 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:33:00,074 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:33:00,074 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:33:00,077 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:33:00,261 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:33:00,261 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:33:02,153 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:33:02,154 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:33:02,155 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:33:02,158 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:33:02,265 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:33:02,266 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:33:04,208 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:33:04,255 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:33:04,256 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:33:04,257 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:33:04,259 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:33:04,270 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:33:04,271 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:33:06,273 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:33:06,320 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:33:06,321 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:33:06,321 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:33:06,324 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:33:07,280 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:33:08,282 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:33:08,350 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:33:08,351 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:33:08,351 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:33:08,354 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:33:09,258 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:33:09,285 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:33:09,411 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:33:09,412 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:33:09,412 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:33:09,415 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:33:10,288 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:33:10,288 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:33:12,290 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:33:12,779 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:33:12,780 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:33:14,950 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:33:19,951 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:33:24,952 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:33:27,510 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:33:27,510 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:33:28,448 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:33:30,449 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:33:35,450 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:33:38,123 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:33:38,124 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:33:38,124 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:33:38,128 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:33:38,327 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:33:38,328 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:33:40,167 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:33:40,168 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:33:40,168 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:33:40,170 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:33:40,332 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:33:40,332 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:33:40,537 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:33:42,212 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:33:42,213 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:33:42,213 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:33:42,215 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:33:42,336 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:33:42,336 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:33:42,509 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:33:42,510 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:33:44,254 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:33:44,255 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:33:44,255 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:33:44,257 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:33:44,340 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:33:44,341 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:33:45,615 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:33:46,275 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:33:46,276 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:33:46,277 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:33:46,280 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:33:46,349 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:33:46,349 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:33:48,308 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:33:48,309 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:33:48,310 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:33:48,312 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:33:48,353 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:33:48,354 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:33:50,356 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:33:50,359 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:33:50,360 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:33:50,360 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:33:50,362 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:33:50,733 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:33:51,359 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:33:52,361 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:33:52,419 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:33:52,420 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:33:52,421 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:33:52,423 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:33:53,364 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:33:54,366 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:33:54,455 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:33:54,456 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:33:54,456 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:33:54,459 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:33:55,369 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:33:55,842 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:33:56,370 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:33:56,521 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:33:56,522 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:33:56,523 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:33:56,525 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:33:57,373 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:33:57,509 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:33:57,510 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:33:58,375 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:33:58,451 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:33:58,563 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:33:58,564 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:33:58,564 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:33:58,577 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:33:59,378 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:34:00,380 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:34:00,608 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:34:00,608 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:34:00,608 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:34:00,610 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:34:00,894 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:34:01,383 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:34:01,667 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:34:01,669 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:34:01,669 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:34:01,672 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:34:02,386 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:34:02,386 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:34:04,388 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:34:06,680 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:34:11,681 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:34:12,988 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:34:12,988 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:34:17,160 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:34:22,160 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:34:27,161 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:34:27,537 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:34:27,538 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:34:28,453 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:34:30,424 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:34:30,601 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:34:30,602 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:34:30,603 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:34:30,606 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:34:31,427 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:34:32,177 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:34:32,429 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:34:32,632 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:34:32,633 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:34:32,633 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:34:32,636 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:34:33,432 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:34:34,433 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:34:34,666 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:34:34,667 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:34:34,667 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:34:34,669 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:34:35,436 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:34:36,437 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:34:36,687 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:34:36,688 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:34:36,688 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:34:36,690 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:34:37,250 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:34:37,441 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:34:38,443 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:34:38,732 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:34:38,733 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:34:38,734 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:34:38,737 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:34:39,446 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:34:40,448 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:34:40,804 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:34:40,805 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:34:40,806 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:34:40,808 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:34:41,451 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:34:42,441 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:34:42,453 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:34:42,538 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:34:42,538 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:34:42,897 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:34:42,898 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:34:42,898 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:34:42,901 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:34:43,456 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:34:44,458 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:34:44,914 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:34:44,915 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:34:44,916 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:34:44,918 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:34:45,461 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:34:46,463 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:34:46,958 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:34:46,959 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:34:46,959 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:34:46,962 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:34:47,466 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:34:47,525 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:34:48,468 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:34:49,003 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:34:49,004 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:34:49,005 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:34:49,007 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:34:49,471 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:34:50,473 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:34:51,064 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:34:51,065 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:34:51,065 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:34:51,068 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:34:51,476 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:34:52,478 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:34:52,641 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:34:53,093 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:34:53,093 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:34:53,094 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:34:53,096 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:34:53,481 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:34:54,152 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:34:54,153 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:34:54,153 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:34:54,155 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:34:54,483 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:34:54,483 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:34:56,486 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:34:57,711 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:34:57,711 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:34:57,881 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:34:58,454 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:35:03,455 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:35:08,456 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:35:12,575 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:35:12,575 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:35:13,745 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:35:18,746 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:35:23,966 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:35:24,525 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:35:25,824 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:35:25,825 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:35:25,825 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:35:25,828 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:35:26,529 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:35:26,530 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:35:27,593 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:35:27,593 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:35:27,847 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:35:27,848 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:35:27,848 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:35:27,850 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:35:28,457 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:35:28,534 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:35:28,535 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:35:29,029 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:35:29,887 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:35:29,888 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:35:29,888 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:35:29,891 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:35:30,539 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:35:30,539 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:35:31,976 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:35:31,977 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:35:31,978 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:35:31,980 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:35:32,543 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:35:32,543 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:35:34,012 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:35:34,013 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:35:34,013 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:35:34,016 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:35:34,177 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:35:34,549 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:35:34,549 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:35:36,103 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:35:36,104 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:35:36,104 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:35:36,107 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:35:36,553 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:35:36,554 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:35:38,146 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:35:38,147 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:35:38,147 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:35:38,149 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:35:38,559 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:35:38,559 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:35:39,326 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:35:40,183 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:35:40,184 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:35:40,184 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:35:40,188 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:35:40,564 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:35:40,564 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:35:42,248 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:35:42,250 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:35:42,250 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:35:42,254 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:35:42,569 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:35:42,570 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:35:42,611 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:35:42,611 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:35:44,297 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:35:44,298 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:35:44,298 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:35:44,301 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:35:44,463 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:35:44,574 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:35:44,575 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:35:46,413 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:35:46,415 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:35:46,415 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:35:46,418 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:35:46,579 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:35:46,579 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:35:48,538 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:35:48,539 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:35:48,539 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:35:48,541 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:35:48,588 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:35:48,589 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:35:49,514 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:35:49,620 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:35:49,621 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:35:49,622 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:35:49,627 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:35:50,593 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:35:50,593 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:35:52,595 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:35:54,638 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:35:58,459 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:36:00,460 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:36:00,831 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:36:00,832 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:36:06,003 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:36:11,004 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:36:13,867 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:36:13,868 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:36:16,037 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:36:18,495 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:36:18,497 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:36:18,498 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:36:18,502 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:36:18,632 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:36:18,633 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:36:20,572 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:36:20,573 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:36:20,574 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:36:20,577 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:36:20,637 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:36:20,637 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:36:21,129 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:36:22,603 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:36:22,604 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:36:22,604 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:36:22,607 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:36:22,641 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:36:22,642 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:36:24,645 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:36:24,672 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:36:24,673 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:36:24,673 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:36:24,675 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:36:25,648 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:36:26,289 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:36:26,650 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:36:26,745 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:36:26,746 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:36:26,747 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:36:26,749 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:36:27,653 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:36:28,462 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:36:28,655 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:36:28,799 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:36:28,800 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:36:28,800 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:36:28,803 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:36:28,868 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:36:28,868 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:36:29,659 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:36:30,660 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:36:30,852 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:36:30,853 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:36:30,853 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:36:30,856 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:36:31,421 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:36:31,664 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:36:32,665 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:36:32,877 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:36:32,878 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:36:32,878 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:36:32,882 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:36:33,668 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:36:34,670 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:36:34,916 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:36:34,917 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:36:34,918 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:36:34,920 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:36:35,673 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:36:36,525 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:36:36,674 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:36:36,981 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:36:36,982 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:36:36,983 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:36:36,985 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:36:37,677 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:36:38,679 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:36:39,008 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:36:39,009 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:36:39,010 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:36:39,012 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:36:39,683 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:36:40,685 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:36:41,051 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:36:41,052 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:36:41,052 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:36:41,055 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:36:41,556 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:36:41,688 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:36:42,127 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:36:42,128 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:36:42,128 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:36:42,131 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:36:42,690 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:36:42,690 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:36:44,693 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:36:45,747 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:36:45,748 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:36:46,918 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:36:51,920 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:36:56,920 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:36:58,472 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:37:00,465 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:37:00,466 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:37:02,635 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:37:07,636 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:37:11,706 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:37:11,707 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:37:11,707 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:37:11,711 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:37:11,730 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:37:12,679 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:37:12,732 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:37:13,720 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:37:13,721 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:37:13,721 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:37:13,723 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:37:13,735 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:37:14,736 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:37:15,466 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:37:15,466 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:37:15,760 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:37:15,761 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:37:15,761 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:37:15,764 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:37:16,740 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:37:16,740 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:37:17,758 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:37:17,805 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:37:17,806 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:37:17,806 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:37:17,808 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:37:18,744 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:37:18,745 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:37:19,859 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:37:19,860 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:37:19,860 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:37:19,862 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:37:20,749 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:37:20,750 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:37:21,900 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:37:21,901 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:37:21,901 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:37:21,903 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:37:22,754 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:37:22,755 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:37:22,889 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:37:24,003 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:37:24,004 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:37:24,004 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:37:24,009 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:37:24,760 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:37:24,760 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:37:26,070 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:37:26,071 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:37:26,071 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:37:26,074 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:37:26,765 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:37:26,765 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:37:28,073 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:37:28,120 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:37:28,121 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:37:28,121 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:37:28,124 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:37:28,467 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:37:28,769 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:37:28,770 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:37:30,102 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:37:30,103 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:37:30,103 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:37:30,107 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:37:30,465 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:37:30,466 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:37:30,774 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:37:30,775 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:37:32,136 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:37:32,137 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:37:32,138 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:37:32,145 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:37:32,779 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:37:32,780 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:37:33,099 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:37:34,170 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:37:34,171 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:37:34,171 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:37:34,174 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:37:34,784 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:37:34,785 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:37:35,331 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:37:35,333 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:37:35,333 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:37:35,335 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:37:35,787 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:37:36,788 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:37:38,351 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:37:43,352 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:37:46,792 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:37:46,793 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:37:48,962 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:37:53,963 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:37:58,468 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:37:59,469 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:38:01,302 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:38:01,303 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:38:04,671 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:38:04,718 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:38:04,719 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:38:04,719 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:38:04,723 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:38:04,829 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:38:04,829 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:38:06,832 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:38:06,834 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:38:06,835 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:38:06,835 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:38:06,838 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:38:07,835 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:38:08,836 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:38:08,891 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:38:08,892 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:38:08,892 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:38:08,894 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:38:09,842 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:38:09,842 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:38:09,867 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:38:10,930 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:38:10,931 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:38:10,931 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:38:10,934 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:38:11,848 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:38:11,849 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:38:13,009 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:38:13,010 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:38:13,011 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:38:13,013 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:38:13,853 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:38:13,854 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:38:14,989 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:38:15,036 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:38:15,037 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:38:15,037 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:38:15,040 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:38:15,859 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:38:15,859 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:38:16,303 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:38:16,303 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:38:17,032 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:38:17,033 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:38:17,033 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:38:17,037 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:38:17,869 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:38:17,870 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:38:19,040 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:38:19,041 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:38:19,041 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:38:19,044 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:38:19,874 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:38:19,874 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:38:20,009 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:38:21,067 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:38:21,068 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:38:21,069 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:38:21,071 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:38:21,878 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:38:21,879 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:38:23,096 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:38:23,097 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:38:23,097 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:38:23,100 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:38:23,883 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:38:23,883 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:38:25,111 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:38:25,149 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:38:25,150 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:38:25,151 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:38:25,154 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:38:25,888 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:38:25,888 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:38:27,240 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:38:27,241 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:38:27,241 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:38:27,243 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:38:27,899 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:38:27,899 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:38:28,402 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:38:28,403 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:38:28,404 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:38:28,407 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:38:28,470 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:38:28,901 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:38:29,903 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:38:30,471 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:38:31,700 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:38:31,701 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:38:36,031 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:38:41,032 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:38:46,561 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:38:46,561 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:38:46,732 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:38:51,733 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:38:55,939 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:38:56,872 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:38:57,536 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:38:57,537 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:38:57,538 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:38:57,541 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:38:57,943 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:38:57,944 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:38:58,483 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:38:59,565 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:38:59,566 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:38:59,567 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:38:59,569 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:38:59,948 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:38:59,948 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:39:01,561 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:39:01,561 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:39:01,654 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:39:01,739 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:39:01,739 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:39:01,742 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:39:01,954 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:39:01,954 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:39:02,035 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:39:03,741 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:39:03,742 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:39:03,742 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:39:03,745 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:39:03,959 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:39:03,959 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:39:05,792 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:39:05,793 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:39:05,793 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:39:05,796 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:39:05,964 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:39:05,964 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:39:07,188 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:39:07,841 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:39:07,842 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:39:07,843 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:39:07,846 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:39:07,968 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:39:07,969 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:39:09,906 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:39:09,907 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:39:09,907 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:39:09,910 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:39:09,973 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:39:09,973 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:39:11,954 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:39:11,955 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:39:11,955 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:39:11,959 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:39:11,977 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:39:11,978 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:39:12,322 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:39:13,980 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:39:13,996 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:39:13,997 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:39:13,997 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:39:14,000 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:39:14,984 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:39:15,985 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:39:16,021 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:39:16,022 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:39:16,022 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:39:16,027 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:39:16,561 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:39:16,561 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:39:16,989 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:39:17,413 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:39:17,990 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:39:18,072 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:39:18,073 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:39:18,073 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:39:18,076 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:39:18,994 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:39:19,996 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:39:20,133 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:39:20,133 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:39:20,134 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:39:20,136 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:39:20,999 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:39:21,251 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:39:21,252 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:39:21,252 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:39:21,256 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:39:22,002 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:39:22,002 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:39:23,290 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:39:28,290 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:39:28,474 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:39:32,396 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:39:32,397 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:39:33,568 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:39:38,569 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:39:43,569 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:39:47,139 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:39:47,140 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:39:48,657 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:39:50,042 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:39:50,115 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:39:50,116 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:39:50,117 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:39:50,120 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:39:51,050 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:39:52,047 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:39:52,190 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:39:52,191 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:39:52,191 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:39:52,194 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:39:53,050 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:39:53,761 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:39:54,052 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:39:54,216 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:39:54,217 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:39:54,217 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:39:54,220 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:39:55,055 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:39:56,057 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:39:56,277 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:39:56,278 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:39:56,278 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:39:56,281 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:39:57,060 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:39:58,061 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:39:58,315 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:39:58,316 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:39:58,317 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:39:58,319 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:39:58,477 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:39:58,897 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:39:59,064 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:40:00,067 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:40:00,382 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:40:00,383 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:40:00,383 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:40:00,386 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:40:01,070 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:40:02,072 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:40:02,140 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:40:02,140 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:40:02,427 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:40:02,428 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:40:02,428 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:40:02,431 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:40:03,075 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:40:04,077 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:40:04,094 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:40:04,549 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:40:04,559 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:40:04,560 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:40:04,562 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:40:05,080 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:40:06,082 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:40:06,671 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:40:06,672 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:40:06,672 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:40:06,674 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:40:07,085 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:40:08,086 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:40:08,718 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:40:08,719 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:40:08,719 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:40:08,722 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:40:09,089 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:40:09,287 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:40:10,091 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:40:10,755 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:40:10,756 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:40:10,756 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:40:10,759 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:40:11,094 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:40:12,096 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:40:12,812 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:40:12,813 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:40:12,813 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:40:12,817 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:40:13,099 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:40:13,881 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:40:13,883 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:40:13,883 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:40:13,891 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:40:14,102 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:40:14,102 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:40:14,892 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:40:16,105 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:40:17,510 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:40:17,510 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:40:20,683 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:40:25,684 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:40:28,479 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:40:31,481 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:40:32,167 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:40:32,167 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:40:37,338 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:40:42,338 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:40:44,143 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:40:45,214 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:40:45,215 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:40:45,215 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:40:45,219 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:40:46,148 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:40:46,148 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:40:47,167 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:40:47,168 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:40:47,263 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:40:47,339 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:40:47,339 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:40:47,339 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:40:47,342 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:40:48,156 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:40:48,156 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:40:49,312 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:40:49,313 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:40:49,313 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:40:49,316 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:40:50,161 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:40:50,161 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:40:51,347 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:40:51,348 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:40:51,348 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:40:51,351 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:40:52,165 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:40:52,166 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:40:52,500 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:40:53,359 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:40:53,360 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:40:53,360 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:40:53,362 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:40:54,170 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:40:54,171 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:40:55,390 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:40:55,391 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:40:55,391 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:40:55,394 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:40:56,176 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:40:56,176 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:40:57,423 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:40:57,424 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:40:57,424 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:40:57,427 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:40:57,588 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:40:58,181 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:40:58,181 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:40:58,483 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:40:59,511 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:40:59,512 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:40:59,512 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:40:59,515 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:41:00,186 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:41:00,186 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:41:01,572 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:41:01,573 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:41:01,573 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:41:01,575 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:41:02,190 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:41:02,191 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:41:02,199 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:41:02,199 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:41:02,607 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:41:03,637 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:41:03,638 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:41:03,639 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:41:03,642 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:41:04,195 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:41:04,195 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:41:05,676 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:41:05,677 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:41:05,677 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:41:05,681 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:41:06,199 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:41:06,200 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:41:07,648 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:41:07,695 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:41:07,696 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:41:07,696 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:41:07,698 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:41:08,204 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:41:08,205 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:41:08,760 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:41:08,761 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:41:08,761 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:41:08,765 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:41:09,207 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:41:10,209 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:41:12,773 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:41:17,773 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:41:20,112 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:41:20,113 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:41:23,285 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:41:28,286 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:41:28,493 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:41:32,981 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:41:32,982 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:41:34,154 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:41:36,246 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:41:37,822 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:41:37,823 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:41:37,823 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:41:37,827 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:41:38,251 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:41:38,252 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:41:39,212 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:41:39,868 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:41:39,869 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:41:39,869 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:41:39,871 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:41:40,256 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:41:40,256 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:41:41,915 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:41:41,916 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:41:41,916 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:41:41,919 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:41:42,267 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:41:42,267 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:41:43,988 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:41:43,990 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:41:43,990 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:41:43,997 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:41:44,265 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:41:44,266 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:41:44,426 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:41:46,138 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:41:46,139 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:41:46,139 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:41:46,142 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:41:46,270 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:41:46,271 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:41:47,982 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:41:47,982 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:41:48,225 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:41:48,226 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:41:48,226 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:41:48,230 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:41:48,275 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:41:48,275 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:41:49,617 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:41:50,275 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:41:50,276 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:41:50,276 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:41:50,279 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:41:50,280 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:41:50,280 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:41:52,284 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:41:52,292 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:41:52,293 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:41:52,293 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:41:52,296 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:41:53,287 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:41:54,288 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:41:54,340 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:41:54,341 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:41:54,341 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:41:54,344 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:41:54,706 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:41:55,292 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:41:56,294 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:41:56,348 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:41:56,349 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:41:56,349 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:41:56,352 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:41:57,297 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:41:58,299 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:41:58,360 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:41:58,361 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:41:58,361 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:41:58,364 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:41:58,488 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:41:59,302 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:41:59,747 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:42:00,305 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:42:00,411 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:42:00,411 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:42:00,411 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:42:00,413 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:42:01,309 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:42:01,515 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:42:01,516 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:42:01,516 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:42:01,518 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:42:02,311 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:42:02,311 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:42:04,794 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:42:04,981 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:42:04,981 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:42:10,152 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:42:15,153 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:42:19,802 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:42:19,802 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:42:20,972 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:42:25,973 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:42:28,491 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:42:30,352 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:42:30,511 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:42:30,512 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:42:30,513 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:42:30,516 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:42:31,081 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:42:31,356 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:42:32,357 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:42:32,552 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:42:32,553 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:42:32,553 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:42:32,556 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:42:33,360 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:42:34,362 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:42:34,584 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:42:34,585 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:42:34,585 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:42:34,587 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:42:34,802 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:42:34,802 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:42:35,364 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:42:36,158 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:42:36,366 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:42:36,590 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:42:36,591 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:42:36,591 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:42:36,594 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:42:37,369 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:42:38,371 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:42:38,661 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:42:38,662 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:42:38,662 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:42:38,664 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:42:39,374 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:42:40,376 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:42:40,702 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:42:40,704 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:42:40,704 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:42:40,707 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:42:41,272 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:42:41,379 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:42:42,381 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:42:42,748 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:42:42,749 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:42:42,750 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:42:42,752 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:42:43,383 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:42:44,386 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:42:44,778 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:42:44,779 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:42:44,779 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:42:44,781 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:42:45,389 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:42:46,360 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:42:46,391 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:42:46,814 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:42:46,815 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:42:46,815 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:42:46,818 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:42:47,394 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:42:48,396 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:42:48,865 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:42:48,866 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:42:48,866 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:42:48,870 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:42:49,399 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:42:49,802 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:42:49,803 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:42:50,400 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:42:50,941 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:42:50,942 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:42:50,942 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:42:50,944 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:42:51,403 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:42:51,521 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:42:52,405 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:42:52,993 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:42:52,994 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:42:52,994 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:42:52,996 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:42:53,408 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:42:54,060 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:42:54,061 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:42:54,062 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:42:54,065 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:42:54,411 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:42:54,411 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:42:56,414 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:42:57,076 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:42:58,493 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:43:02,494 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:43:05,586 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:43:05,587 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:43:07,758 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:43:12,759 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:43:17,759 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:43:20,181 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:43:20,181 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:43:22,449 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:43:22,879 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:43:22,926 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:43:22,927 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:43:22,927 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:43:22,931 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:43:23,452 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:43:24,454 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:43:25,056 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:43:25,057 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:43:25,057 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:43:25,059 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:43:25,457 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:43:26,459 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:43:27,118 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:43:27,119 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:43:27,119 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:43:27,122 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:43:27,462 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:43:27,880 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:43:28,463 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:43:28,496 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:43:29,135 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:43:29,136 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:43:29,136 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:43:29,138 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:43:29,467 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:43:30,469 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:43:31,166 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:43:31,167 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:43:31,167 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:43:31,170 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:43:31,472 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:43:32,474 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:43:32,925 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:43:33,175 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:43:33,176 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:43:33,177 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:43:33,181 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:43:33,477 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:43:34,479 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:43:35,209 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:43:35,210 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:43:35,210 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:43:35,211 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:43:35,213 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:43:35,214 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:43:35,482 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:43:36,484 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:43:37,258 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:43:37,259 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:43:37,259 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:43:37,262 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:43:37,487 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:43:38,054 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:43:38,489 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:43:39,324 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:43:39,325 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:43:39,326 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:43:39,328 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:43:39,492 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:43:40,494 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:43:41,373 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:43:41,374 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:43:41,375 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:43:41,378 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:43:41,498 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:43:42,499 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:43:43,157 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:43:43,417 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:43:43,418 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:43:43,419 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:43:43,421 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:43:43,503 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:43:44,505 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:43:45,511 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:43:45,512 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:43:45,512 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:43:45,514 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:43:46,510 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:43:46,510 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:43:46,595 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:43:46,597 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:43:46,597 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:43:46,599 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:43:47,512 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:43:48,514 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:43:48,607 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:43:50,270 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:43:50,271 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:43:54,399 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:43:58,505 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:43:59,506 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:44:04,507 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:44:05,219 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:44:05,219 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:44:10,391 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:44:14,549 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:44:15,403 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:44:15,861 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:44:15,862 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:44:15,863 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:44:15,866 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:44:16,554 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:44:16,555 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:44:17,932 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:44:17,933 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:44:17,933 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:44:17,935 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:44:18,559 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:44:18,559 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:44:19,977 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:44:19,978 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:44:19,979 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:44:19,981 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:44:20,219 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:44:20,219 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:44:20,563 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:44:20,563 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:44:20,565 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:44:22,023 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:44:22,024 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:44:22,025 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:44:22,027 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:44:22,567 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:44:22,568 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:44:24,055 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:44:24,056 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:44:24,056 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:44:24,060 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:44:24,573 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:44:24,573 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:44:25,633 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:44:26,079 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:44:26,080 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:44:26,080 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:44:26,083 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:44:26,577 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:44:26,578 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:44:28,213 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:44:28,214 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:44:28,214 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:44:28,217 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:44:28,499 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:44:28,582 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:44:28,582 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:44:30,280 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:44:30,281 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:44:30,281 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:44:30,285 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:44:30,587 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:44:30,587 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:44:30,655 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:44:32,272 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:44:32,274 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:44:32,274 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:44:32,276 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:44:32,591 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:44:32,592 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:44:34,333 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:44:34,334 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:44:34,334 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:44:34,337 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:44:34,596 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:44:34,596 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:44:35,219 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:44:35,220 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:44:35,714 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:44:36,368 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:44:36,369 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:44:36,369 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:44:36,372 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:44:36,600 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:44:36,601 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:44:38,408 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:44:38,408 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:44:38,409 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:44:38,410 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:44:38,605 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:44:38,605 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:44:39,511 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:44:39,512 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:44:39,512 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:44:39,515 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:44:39,610 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:44:40,611 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:44:41,526 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:44:46,527 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:44:50,696 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:44:50,696 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:44:51,873 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:44:56,874 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:44:58,500 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:45:02,501 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:45:05,380 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:45:05,380 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:45:07,620 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:45:08,473 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:45:08,474 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:45:08,475 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:45:08,480 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:45:08,658 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:45:08,659 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:45:10,541 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:45:10,542 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:45:10,542 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:45:10,544 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:45:10,657 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:45:10,657 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:45:12,569 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:45:12,570 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:45:12,570 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:45:12,573 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:45:12,662 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:45:12,662 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:45:12,738 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:45:14,600 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:45:14,601 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:45:14,601 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:45:14,604 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:45:14,666 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:45:14,666 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:45:16,648 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:45:16,649 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:45:16,649 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:45:16,652 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:45:16,670 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:45:16,671 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:45:17,834 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:45:18,671 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:45:18,673 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:45:18,673 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:45:18,675 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:45:18,676 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:45:18,676 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:45:20,380 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:45:20,381 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:45:20,680 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:45:20,726 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:45:20,726 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:45:20,727 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:45:20,730 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:45:21,683 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:45:22,685 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:45:22,730 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:45:22,731 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:45:22,731 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:45:22,734 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:45:22,898 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:45:23,688 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:45:24,689 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:45:24,774 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:45:24,775 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:45:24,775 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:45:24,778 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:45:25,693 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:45:26,694 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:45:26,903 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:45:26,904 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:45:26,904 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:45:26,907 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:45:27,697 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:45:28,079 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:45:28,514 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:45:28,699 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:45:28,940 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:45:28,941 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:45:28,941 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:45:28,943 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:45:29,702 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:45:30,705 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:45:30,972 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:45:30,973 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:45:30,973 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:45:30,976 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:45:31,707 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:45:32,048 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:45:32,049 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:45:32,049 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:45:32,052 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:45:32,711 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:45:32,711 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:45:34,070 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:45:35,511 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:45:35,511 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:45:39,713 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:45:44,714 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:45:49,714 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:45:50,387 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:45:50,387 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:45:55,579 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:45:58,512 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:46:00,682 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:46:00,749 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:46:01,738 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:46:01,739 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:46:01,739 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:46:01,742 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:46:01,752 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:46:02,754 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:46:03,775 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:46:03,776 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:46:03,777 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:46:03,779 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:46:04,758 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:46:04,759 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:46:05,387 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:46:05,387 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:46:05,789 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:46:05,835 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:46:05,836 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:46:05,837 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:46:05,839 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:46:06,763 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:46:06,764 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:46:07,891 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:46:07,893 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:46:07,893 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:46:07,896 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:46:08,768 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:46:08,769 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:46:09,916 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:46:09,917 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:46:09,917 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:46:09,919 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:46:10,773 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:46:10,774 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:46:10,895 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:46:11,954 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:46:11,955 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:46:11,955 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:46:11,958 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:46:12,778 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:46:12,778 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:46:14,020 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:46:14,021 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:46:14,022 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:46:14,024 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:46:14,783 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:46:14,783 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:46:16,003 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:46:16,050 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:46:16,051 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:46:16,051 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:46:16,054 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:46:16,788 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:46:16,788 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:46:18,086 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:46:18,087 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:46:18,087 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:46:18,090 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:46:18,793 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:46:18,794 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:46:20,117 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:46:20,118 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:46:20,118 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:46:20,121 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:46:20,387 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:46:20,388 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:46:20,798 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:46:20,798 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:46:21,094 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:46:22,161 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:46:22,162 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:46:22,163 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:46:22,165 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:46:22,803 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:46:22,803 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:46:24,212 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:46:24,213 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:46:24,213 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:46:24,216 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:46:24,807 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:46:24,808 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:46:25,315 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:46:25,316 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:46:25,316 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:46:25,319 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:46:25,810 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:46:26,328 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:46:26,812 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:46:28,511 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:46:31,512 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:46:36,480 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:46:36,481 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:46:36,651 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:46:41,748 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:46:46,748 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:46:51,144 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:46:51,144 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:46:52,271 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:46:54,351 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:46:54,352 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:46:54,352 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:46:54,357 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:46:54,851 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:46:54,851 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:46:56,372 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:46:56,373 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:46:56,374 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:46:56,376 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:46:56,855 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:46:56,856 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:46:57,349 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:46:58,423 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:46:58,424 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:46:58,424 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:46:58,427 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:46:58,514 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:46:58,860 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:46:58,860 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:47:00,462 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:47:00,463 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:47:00,464 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:47:00,466 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:47:00,864 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:47:00,864 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:47:02,480 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:47:02,527 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:47:02,528 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:47:02,528 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:47:02,531 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:47:02,869 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:47:02,869 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:47:04,547 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:47:04,548 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:47:04,549 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:47:04,551 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:47:04,873 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:47:04,873 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:47:06,175 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:47:06,176 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:47:06,580 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:47:06,581 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:47:06,582 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:47:06,584 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:47:06,877 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:47:06,878 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:47:07,533 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:47:08,585 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:47:08,586 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:47:08,586 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:47:08,590 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:47:08,882 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:47:08,883 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:47:10,681 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:47:10,682 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:47:10,682 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:47:10,685 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:47:10,889 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:47:10,890 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:47:12,732 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:47:12,779 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:47:12,780 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:47:12,781 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:47:12,783 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:47:12,894 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:47:12,894 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:47:14,822 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:47:14,823 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:47:14,823 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:47:14,826 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:47:14,898 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:47:14,898 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:47:16,851 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:47:16,852 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:47:16,852 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:47:16,855 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:47:16,903 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:47:16,903 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:47:17,773 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:47:17,926 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:47:17,927 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:47:17,928 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:47:17,930 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:47:18,907 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:47:18,907 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:47:21,475 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:47:21,475 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:47:23,648 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:47:28,515 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:47:29,516 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:47:34,517 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:47:36,169 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:47:36,170 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:47:40,339 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:47:45,350 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:47:46,817 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:47:46,818 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:47:46,818 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:47:46,822 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:47:46,946 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:47:46,947 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:47:48,849 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:47:48,850 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:47:48,851 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:47:48,853 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:47:48,950 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:47:48,951 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:47:50,447 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:47:50,889 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:47:50,890 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:47:50,890 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:47:50,894 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:47:50,955 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:47:50,955 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:47:51,169 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:47:51,170 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:47:52,904 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:47:52,905 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:47:52,905 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:47:52,908 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:47:52,960 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:47:52,960 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:47:54,956 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:47:54,957 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:47:54,957 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:47:54,959 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:47:54,964 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:47:54,964 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:47:55,523 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:47:56,967 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:47:56,996 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:47:56,997 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:47:56,997 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:47:57,000 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:47:57,970 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:47:58,527 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:47:58,972 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:47:59,050 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:47:59,051 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:47:59,051 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:47:59,054 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:47:59,975 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:48:00,663 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:48:00,977 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:48:01,111 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:48:01,112 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:48:01,112 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:48:01,115 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:48:01,980 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:48:02,981 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:48:03,167 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:48:03,168 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:48:03,168 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:48:03,170 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:48:03,984 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:48:04,986 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:48:05,203 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:48:05,204 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:48:05,204 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:48:05,207 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:48:05,768 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:48:05,988 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:48:06,169 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:48:06,170 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:48:06,990 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:48:07,218 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:48:07,219 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:48:07,219 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:48:07,221 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:48:07,993 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:48:08,994 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:48:09,247 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:48:09,247 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:48:09,248 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:48:09,251 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:48:09,998 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:48:10,323 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:48:10,324 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:48:10,324 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:48:10,326 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:48:11,000 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:48:11,000 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:48:11,339 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:48:16,339 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:48:21,340 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:48:21,466 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:48:21,467 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:48:26,638 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:48:28,528 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:48:32,530 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:48:36,247 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:48:36,247 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:48:37,722 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:48:39,038 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:48:39,389 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:48:39,390 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:48:39,390 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:48:39,394 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:48:40,042 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:48:41,044 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:48:41,444 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:48:41,445 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:48:41,446 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:48:41,448 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:48:42,046 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:48:42,821 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:48:43,048 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:48:43,485 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:48:43,486 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:48:43,487 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:48:43,489 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:48:44,050 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:48:45,052 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:48:45,507 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:48:45,508 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:48:45,508 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:48:45,510 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:48:46,055 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:48:47,057 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:48:47,541 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:48:47,542 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:48:47,542 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:48:47,544 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:48:47,909 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:48:48,061 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:48:49,062 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:48:49,635 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:48:49,636 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:48:49,637 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:48:49,640 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:48:50,065 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:48:51,067 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:48:51,247 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:48:51,248 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:48:51,732 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:48:51,733 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:48:51,733 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:48:51,736 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:48:52,070 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:48:52,909 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:48:53,072 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:48:53,782 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:48:53,783 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:48:53,784 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:48:53,786 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:48:54,074 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:48:55,076 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:48:55,804 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:48:55,805 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:48:55,805 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:48:55,807 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:48:56,079 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:48:57,080 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:48:57,823 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:48:57,824 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:48:57,824 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:48:57,829 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:48:57,986 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:48:58,083 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:48:58,523 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:48:59,085 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:48:59,880 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:48:59,881 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:48:59,881 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:48:59,884 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:49:00,088 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:49:01,090 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:49:01,889 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:49:01,890 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:49:01,890 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:49:01,892 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:49:02,093 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:49:02,965 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:49:02,966 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:49:02,967 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:49:02,969 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:49:03,099 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:49:03,099 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:49:03,982 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:49:05,102 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:49:06,573 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:49:06,573 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:49:09,750 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:49:14,751 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:49:19,752 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:49:21,455 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:49:21,455 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:49:25,626 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:49:28,524 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:49:31,525 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:49:34,935 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:49:34,936 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:49:34,936 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:49:34,940 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:49:35,149 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:49:35,155 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:49:36,455 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:49:36,455 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:49:36,627 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:49:36,967 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:49:36,968 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:49:36,969 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:49:36,971 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:49:37,153 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:49:37,154 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:49:39,005 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:49:39,006 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:49:39,006 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:49:39,008 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:49:39,158 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:49:39,159 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:49:41,045 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:49:41,046 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:49:41,047 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:49:41,049 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:49:41,163 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:49:41,163 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:49:41,817 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:49:43,086 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:49:43,087 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:49:43,087 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:49:43,090 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:49:43,167 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:49:43,168 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:49:45,128 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:49:45,129 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:49:45,129 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:49:45,133 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:49:45,173 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:49:45,173 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:49:46,913 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:49:47,163 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:49:47,164 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:49:47,164 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:49:47,166 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:49:47,177 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:49:47,178 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:49:49,181 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:49:49,192 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:49:49,193 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:49:49,193 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:49:49,195 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:49:50,183 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:49:51,185 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:49:51,248 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:49:51,249 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:49:51,249 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:49:51,252 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:49:51,455 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:49:51,455 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:49:52,021 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:49:52,188 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:49:53,190 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:49:53,296 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:49:53,297 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:49:53,297 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:49:53,299 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:49:54,193 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:49:55,194 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:49:55,371 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:49:55,372 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:49:55,373 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:49:55,375 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:49:56,197 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:49:57,025 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:49:57,199 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:49:57,477 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:49:57,478 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:49:57,478 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:49:57,481 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:49:58,202 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:49:58,527 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:49:58,554 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:49:58,555 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:49:58,556 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:49:58,558 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:49:59,205 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:49:59,205 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:50:02,566 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:50:07,566 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:50:09,510 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:50:09,510 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:50:12,681 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:50:17,682 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:50:21,837 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:50:21,838 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:50:23,007 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:50:28,106 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:50:28,527 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:50:29,246 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:50:29,692 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:50:29,694 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:50:29,694 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:50:29,697 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:50:30,250 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:50:31,252 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:50:31,730 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:50:31,731 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:50:31,732 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:50:31,734 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:50:32,255 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:50:33,116 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:50:33,257 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:50:33,769 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:50:33,770 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:50:33,770 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:50:33,773 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:50:34,260 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:50:35,262 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:50:35,790 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:50:35,791 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:50:35,792 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:50:35,794 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:50:36,265 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:50:36,838 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:50:36,838 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:50:37,266 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:50:37,837 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:50:37,838 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:50:37,838 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:50:37,841 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:50:38,211 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:50:38,270 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:50:39,271 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:50:39,874 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:50:39,875 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:50:39,875 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:50:39,878 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:50:40,275 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:50:41,276 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:50:41,917 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:50:41,918 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:50:41,918 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:50:41,921 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:50:42,279 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:50:43,280 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:50:43,282 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:50:43,935 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:50:43,938 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:50:43,939 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:50:43,942 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:50:44,285 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:50:45,286 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:50:45,964 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:50:45,965 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:50:45,966 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:50:45,968 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:50:46,290 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:50:47,291 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:50:48,012 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:50:48,013 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:50:48,013 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:50:48,016 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:50:48,295 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:50:48,377 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:50:49,296 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:50:50,061 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:50:50,062 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:50:50,062 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:50:50,065 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:50:50,299 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:50:51,301 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:50:51,839 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:50:51,839 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:50:52,143 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:50:52,144 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:50:52,144 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:50:52,147 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:50:52,304 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:50:53,244 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:50:53,245 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:50:53,245 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:50:53,248 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:50:53,306 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:50:53,307 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:50:54,260 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:50:55,309 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:50:58,538 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:50:59,539 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:51:04,540 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:51:09,541 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:51:10,542 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:51:10,542 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:51:14,716 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:51:19,717 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:51:23,346 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:51:23,868 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:51:23,870 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:51:23,870 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:51:23,874 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:51:24,349 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:51:24,824 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:51:25,328 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:51:25,329 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:51:25,351 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:51:25,864 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:51:25,865 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:51:25,865 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:51:25,867 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:51:26,354 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:51:27,356 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:51:27,902 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:51:27,903 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:51:27,903 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:51:27,906 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:51:28,359 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:51:28,532 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:51:29,361 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:51:29,901 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:51:29,948 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:51:29,948 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:51:29,949 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:51:29,951 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:51:30,364 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:51:31,365 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:51:31,989 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:51:31,990 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:51:31,990 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:51:31,993 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:51:32,368 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:51:33,370 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:51:34,086 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:51:34,087 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:51:34,087 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:51:34,090 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:51:34,374 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:51:35,007 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:51:35,379 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:51:36,283 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:51:36,284 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:51:36,284 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:51:36,287 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:51:36,382 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:51:37,384 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:51:38,314 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:51:38,315 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:51:38,316 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:51:38,318 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:51:38,387 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:51:39,389 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:51:40,093 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:51:40,342 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:51:40,343 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:51:40,343 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:51:40,344 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:51:40,346 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:51:40,347 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:51:40,391 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:51:41,393 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:51:42,415 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:51:42,416 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:51:42,417 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:51:42,419 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:51:43,398 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:51:43,398 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:51:44,444 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:51:44,445 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:51:44,445 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:51:44,447 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:51:45,214 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:51:45,403 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:51:45,403 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:51:46,481 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:51:46,482 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:51:46,482 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:51:46,485 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:51:47,408 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:51:47,409 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:51:47,544 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:51:47,546 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:51:47,546 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:51:47,549 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:51:48,411 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:51:49,412 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:51:50,560 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:51:56,230 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:51:58,533 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:51:58,590 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:51:58,590 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:52:01,760 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:52:06,760 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:52:10,805 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:52:10,806 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:52:11,965 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:52:15,447 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:52:16,540 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:52:16,542 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:52:16,542 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:52:16,546 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:52:17,100 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:52:17,451 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:52:17,452 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:52:18,552 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:52:18,553 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:52:18,553 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:52:18,555 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:52:19,456 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:52:19,456 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:52:20,586 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:52:20,587 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:52:20,588 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:52:20,590 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:52:21,461 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:52:21,461 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:52:22,165 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:52:22,606 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:52:22,607 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:52:22,607 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:52:22,610 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:52:23,466 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:52:23,466 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:52:24,657 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:52:24,658 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:52:24,659 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:52:24,661 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:52:25,470 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:52:25,471 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:52:25,805 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:52:25,806 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:52:26,706 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:52:26,707 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:52:26,707 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:52:26,710 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:52:27,275 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:52:27,475 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:52:27,475 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:52:28,546 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:52:28,752 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:52:28,753 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:52:28,753 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:52:28,755 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:52:29,481 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:52:29,481 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:52:30,803 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:52:30,804 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:52:30,804 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:52:30,807 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:52:31,486 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:52:31,486 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:52:32,394 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:52:32,848 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:52:32,849 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:52:32,849 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:52:32,853 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:52:33,491 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:52:33,492 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:52:34,872 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:52:34,873 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:52:34,873 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:52:34,875 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:52:35,496 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:52:35,496 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:52:36,909 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:52:36,910 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:52:36,911 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:52:36,913 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:52:37,474 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:52:37,501 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:52:37,501 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:52:38,954 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:52:38,954 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:52:38,954 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:52:38,957 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:52:39,506 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:52:39,507 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:52:40,103 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:52:40,104 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:52:40,105 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:52:40,110 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:52:40,509 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:52:41,510 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:52:43,126 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:52:43,573 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:52:43,574 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:52:48,745 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:52:53,746 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:52:57,360 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:52:57,361 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:52:58,538 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:52:59,539 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:53:04,539 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:53:09,368 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:53:09,374 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:53:09,374 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:53:09,378 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:53:09,553 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:53:09,553 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:53:09,562 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:53:11,394 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:53:11,395 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:53:11,395 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:53:11,397 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:53:11,557 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:53:11,558 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:53:12,208 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:53:12,208 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:53:13,443 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:53:13,444 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:53:13,444 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:53:13,447 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:53:13,562 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:53:13,562 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:53:14,641 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:53:15,499 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:53:15,500 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:53:15,500 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:53:15,504 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:53:15,567 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:53:15,567 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:53:17,535 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:53:17,536 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:53:17,537 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:53:17,539 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:53:17,571 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:53:17,572 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:53:19,575 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:53:19,575 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:53:19,576 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:53:19,576 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:53:19,580 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:53:19,739 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:53:20,578 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:53:21,579 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:53:21,605 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:53:21,606 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:53:21,606 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:53:21,608 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:53:22,583 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:53:23,584 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:53:23,648 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:53:23,649 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:53:23,649 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:53:23,651 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:53:24,587 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:53:24,836 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:53:25,589 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:53:25,697 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:53:25,698 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:53:25,698 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:53:25,701 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:53:26,592 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:53:27,208 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:53:27,208 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:53:27,594 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:53:27,722 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:53:27,723 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:53:27,723 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:53:27,726 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:53:28,541 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:53:28,597 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:53:29,599 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:53:29,756 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:53:29,757 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:53:29,758 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:53:29,761 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:53:29,921 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:53:30,602 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:53:31,604 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:53:31,769 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:53:31,770 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:53:31,770 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:53:31,771 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:53:32,615 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:53:32,826 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:53:32,827 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:53:32,827 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:53:32,830 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:53:33,618 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:53:33,618 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:53:35,842 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:53:41,645 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:53:43,925 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:53:43,926 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:53:47,097 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:53:52,097 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:53:57,568 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:53:57,686 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:53:57,686 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:53:58,542 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:54:01,589 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:54:01,590 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:54:01,590 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:54:01,593 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:54:01,659 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:54:01,659 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:54:02,764 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:54:03,615 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:54:03,616 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:54:03,616 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:54:03,618 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:54:03,664 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:54:03,664 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:54:05,667 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:54:05,676 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:54:05,677 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:54:05,677 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:54:05,679 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:54:06,670 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:54:07,671 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:54:07,726 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:54:07,727 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:54:07,727 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:54:07,730 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:54:07,889 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:54:08,674 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:54:09,676 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:54:09,750 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:54:09,751 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:54:09,751 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:54:09,753 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:54:10,679 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:54:11,681 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:54:11,788 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:54:11,789 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:54:11,789 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:54:11,791 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:54:12,686 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:54:12,686 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:54:12,687 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:54:12,965 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:54:13,686 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:54:13,835 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:54:13,836 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:54:13,836 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:54:13,839 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:54:14,688 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:54:15,690 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:54:15,868 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:54:15,869 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:54:15,869 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:54:15,871 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:54:16,693 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:54:17,695 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:54:17,900 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:54:17,901 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:54:17,902 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:54:17,904 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:54:18,065 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:54:18,698 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:54:19,699 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:54:19,967 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:54:19,968 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:54:19,969 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:54:19,971 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:54:20,702 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:54:21,704 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:54:21,983 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:54:21,984 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:54:21,984 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:54:21,987 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:54:22,707 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:54:23,145 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:54:23,709 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:54:23,993 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:54:23,994 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:54:23,994 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:54:23,996 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:54:24,712 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:54:25,139 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:54:25,140 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:54:25,140 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:54:25,142 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:54:25,715 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:54:25,715 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:54:27,718 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:54:28,157 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:54:28,544 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:54:28,764 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:54:28,764 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:54:33,936 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:54:38,937 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:54:43,293 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:54:43,294 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:54:44,465 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:54:49,466 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:54:53,752 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:54:54,201 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:54:54,202 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:54:54,203 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:54:54,206 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:54:54,560 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:54:54,755 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:54:55,756 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:54:56,221 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:54:56,222 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:54:56,223 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:54:56,225 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:54:56,759 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:54:57,761 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:54:58,267 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:54:58,268 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:54:58,268 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:54:58,270 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:54:58,293 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:54:58,294 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:54:58,547 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:54:58,764 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:54:59,641 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:54:59,766 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:55:00,286 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:55:00,287 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:55:00,287 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:55:00,290 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:55:00,768 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:55:01,771 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:55:02,331 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:55:02,332 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:55:02,332 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:55:02,336 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:55:02,774 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:55:03,776 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:55:04,372 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:55:04,373 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:55:04,373 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:55:04,375 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:55:04,741 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:55:04,779 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:55:05,780 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:55:06,418 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:55:06,419 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:55:06,419 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:55:06,422 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:55:06,783 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:55:07,785 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:55:08,453 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:55:08,455 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:55:08,455 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:55:08,457 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:55:08,788 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:55:09,790 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:55:09,850 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:55:10,509 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:55:10,510 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:55:10,510 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:55:10,514 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:55:10,792 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:55:11,794 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:55:12,540 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:55:12,541 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:55:12,542 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:55:12,544 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:55:12,797 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:55:13,311 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:55:13,311 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:55:13,799 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:55:14,594 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:55:14,595 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:55:14,596 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:55:14,598 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:55:14,802 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:55:14,958 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:55:15,804 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:55:16,643 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:55:16,644 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:55:16,644 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:55:16,647 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:55:16,807 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:55:17,737 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:55:17,738 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:55:17,738 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:55:17,742 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:55:17,810 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:55:17,810 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:55:19,813 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:55:20,753 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:55:26,644 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:55:28,558 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:55:28,987 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:55:28,988 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:55:32,158 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:55:37,159 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:55:42,598 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:55:43,650 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:55:43,650 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:55:47,356 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:55:47,357 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:55:47,358 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:55:47,361 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:55:47,744 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:55:47,853 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:55:47,853 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:55:49,403 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:55:49,405 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:55:49,405 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:55:49,407 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:55:49,858 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:55:49,858 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:55:51,433 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:55:51,434 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:55:51,435 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:55:51,437 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:55:51,862 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:55:51,862 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:55:52,809 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:55:53,474 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:55:53,475 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:55:53,475 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:55:53,478 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:55:53,866 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:55:53,867 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:55:55,519 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:55:55,520 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:55:55,520 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:55:55,523 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:55:55,871 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:55:55,872 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:55:57,555 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:55:57,564 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:55:57,564 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:55:57,567 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:55:57,885 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:55:57,886 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:55:57,935 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:55:58,551 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:55:58,650 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:55:58,651 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:55:59,626 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:55:59,627 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:55:59,627 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:55:59,631 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:55:59,890 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:55:59,891 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:56:01,648 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:56:01,649 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:56:01,649 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:56:01,652 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:56:01,898 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:56:01,898 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:56:03,028 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:56:03,684 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:56:03,685 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:56:03,685 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:56:03,688 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:56:03,902 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:56:03,903 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:56:05,723 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:56:05,724 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:56:05,725 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:56:05,727 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:56:05,907 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:56:05,908 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:56:07,743 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:56:07,744 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:56:07,744 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:56:07,747 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:56:07,912 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:56:07,912 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:56:08,109 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:56:09,769 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:56:09,770 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:56:09,770 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:56:09,772 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:56:09,916 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:56:09,917 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:56:10,869 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:56:10,870 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:56:10,871 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:56:10,875 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:56:10,919 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:56:11,921 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:56:13,891 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:56:14,330 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:56:14,330 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:56:19,519 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:56:24,520 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:56:28,552 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:56:28,986 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:56:28,987 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:56:30,160 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:56:35,159 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:56:39,719 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:56:39,720 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:56:39,720 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:56:39,724 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:56:39,962 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:56:39,962 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:56:40,210 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:56:41,763 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:56:41,764 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:56:41,764 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:56:41,767 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:56:41,966 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:56:41,967 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:56:43,818 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:56:43,819 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:56:43,819 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:56:43,822 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:56:43,971 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:56:43,971 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:56:43,986 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:56:43,987 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:56:45,396 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:56:45,842 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:56:45,843 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:56:45,843 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:56:45,846 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:56:45,975 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:56:45,976 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:56:47,874 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:56:47,875 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:56:47,876 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:56:47,878 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:56:47,980 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:56:47,980 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:56:49,914 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:56:49,915 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:56:49,916 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:56:49,918 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:56:49,985 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:56:49,985 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:56:50,490 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:56:51,983 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:56:51,984 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:56:51,984 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:56:51,987 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:56:51,990 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:56:51,990 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:56:53,992 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:56:54,025 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:56:54,026 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:56:54,026 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:56:54,029 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:56:54,996 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:56:55,652 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:56:55,998 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:56:56,105 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:56:56,106 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:56:56,107 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:56:56,111 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:56:57,001 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:56:58,003 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:56:58,154 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:56:58,155 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:56:58,155 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:56:58,157 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:56:58,555 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:56:58,987 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:56:58,987 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:56:59,006 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:57:00,008 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:57:00,193 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:57:00,195 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:57:00,195 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:57:00,198 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:57:00,761 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:57:01,011 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:57:02,012 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:57:02,248 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:57:02,249 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:57:02,249 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:57:02,251 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:57:03,018 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:57:03,330 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:57:03,331 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:57:03,331 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:57:03,334 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:57:04,020 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:57:04,021 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:57:06,023 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:57:06,341 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:57:12,310 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:57:14,509 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:57:14,510 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:57:17,681 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:57:22,682 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:57:28,235 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:57:28,557 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:57:29,133 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:57:29,133 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:57:33,312 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:57:34,060 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:57:34,161 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:57:34,162 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:57:34,162 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:57:34,166 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:57:35,063 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:57:36,064 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:57:36,275 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:57:36,276 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:57:36,276 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:57:36,279 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:57:37,067 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:57:38,069 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:57:38,307 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:57:38,308 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:57:38,308 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:57:38,310 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:57:38,472 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:57:39,072 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:57:40,073 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:57:40,348 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:57:40,349 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:57:40,349 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:57:40,351 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:57:41,076 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:57:42,078 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:57:42,390 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:57:42,391 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:57:42,391 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:57:42,393 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:57:43,081 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:57:43,573 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:57:44,083 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:57:44,133 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:57:44,134 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:57:44,423 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:57:44,424 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:57:44,424 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:57:44,426 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:57:45,086 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:57:46,088 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:57:46,464 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:57:46,465 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:57:46,466 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:57:46,469 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:57:47,091 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:57:48,093 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:57:48,497 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:57:48,498 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:57:48,498 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:57:48,501 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:57:48,647 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:57:49,096 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:57:50,097 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:57:50,549 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:57:50,550 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:57:50,550 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:57:50,553 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:57:51,100 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:57:52,102 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:57:52,589 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:57:52,590 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:57:52,591 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:57:52,593 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:57:53,105 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:57:53,771 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:57:54,106 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:57:54,629 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:57:54,630 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:57:54,630 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:57:54,633 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:57:55,110 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:57:56,112 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:57:56,662 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:57:56,663 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:57:56,663 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:57:56,666 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:57:57,115 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:57:57,736 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:57:57,737 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:57:57,737 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:57:57,741 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:57:58,118 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:57:58,118 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:57:58,558 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:57:59,559 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:58:00,121 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:58:01,013 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:58:01,013 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:58:05,184 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:58:10,184 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:58:15,148 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:58:15,148 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:58:15,319 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:58:20,319 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:58:25,320 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:58:28,160 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:58:28,561 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:58:29,043 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:58:29,044 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:58:29,044 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:58:29,048 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:58:29,163 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:58:29,830 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:58:29,831 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:58:30,165 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:58:30,436 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:58:31,089 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:58:31,090 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:58:31,090 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:58:31,093 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:58:31,179 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:58:32,170 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:58:33,129 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:58:33,131 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:58:33,131 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:58:33,133 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:58:33,178 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:58:34,180 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:58:35,176 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:58:35,177 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:58:35,178 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:58:35,181 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:58:35,183 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:58:35,541 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:58:36,186 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:58:37,195 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:58:37,196 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:58:37,197 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:58:37,200 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:58:38,190 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:58:38,190 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:58:39,239 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:58:39,240 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:58:39,241 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:58:39,243 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:58:40,194 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:58:40,195 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:58:40,665 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:58:41,318 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:58:41,319 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:58:41,320 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:58:41,322 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:58:42,199 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:58:42,199 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:58:43,324 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:58:43,325 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:58:43,325 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:58:43,327 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:58:44,203 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:58:44,204 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:58:44,831 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:58:44,831 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:58:45,400 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:58:45,401 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:58:45,402 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:58:45,404 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:58:45,767 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:58:46,208 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:58:46,208 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:58:47,435 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:58:47,436 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:58:47,436 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:58:47,438 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:58:48,212 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:58:48,213 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:58:49,484 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:58:49,485 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:58:49,485 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:58:49,488 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:58:50,217 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:58:50,218 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:58:50,870 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:58:51,530 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:58:51,530 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:58:51,531 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:58:51,533 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:58:52,222 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:58:52,222 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:58:52,637 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:58:52,638 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:58:52,638 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:58:52,641 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:58:53,224 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:58:54,225 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:58:56,650 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:58:58,562 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:58:59,908 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:58:59,909 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:59:02,081 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:59:07,098 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:59:12,098 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:59:14,839 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:59:14,839 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:59:18,020 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:59:22,204 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:59:22,205 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:59:22,205 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:59:22,208 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:59:22,270 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:59:22,270 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:59:23,178 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:59:24,227 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:59:24,228 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:59:24,229 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:59:24,231 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:59:24,274 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:59:24,274 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:59:26,246 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:59:26,247 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:59:26,247 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:59:26,250 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:59:26,278 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:59:26,279 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:59:28,244 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:59:28,282 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:59:28,291 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:59:28,291 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:59:28,292 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:59:28,294 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:59:28,566 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:59:29,285 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:59:29,839 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:59:29,839 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:59:30,287 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:59:30,355 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:59:30,356 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:59:30,357 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:59:30,360 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:59:31,290 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:59:32,292 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:59:32,371 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:59:32,372 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:59:32,372 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:59:32,375 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:59:33,295 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:59:33,362 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:59:34,297 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:59:34,413 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:59:34,414 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:59:34,414 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:59:34,417 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:59:35,300 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:59:36,301 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:59:36,470 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:59:36,472 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:59:36,472 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:59:36,474 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:59:37,304 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:59:38,306 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:59:38,488 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:59:38,535 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:59:38,536 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:59:38,537 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:59:38,539 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:59:39,309 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:59:40,311 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:59:40,566 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:59:40,567 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:59:40,567 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:59:40,571 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:59:41,314 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:59:42,316 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:59:42,605 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:59:42,606 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:59:42,607 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:59:42,609 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:59:43,319 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:59:43,580 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:59:44,320 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:59:44,599 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:59:44,600 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:59:44,600 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:59:44,602 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:59:44,839 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 04:59:44,839 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 04:59:45,324 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:59:45,661 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 04:59:45,662 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 04:59:45,663 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 04:59:45,667 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 04:59:46,326 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:59:46,327 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 04:59:48,330 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 04:59:48,690 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:59:53,691 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 04:59:58,577 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 04:59:59,578 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:00:02,979 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:00:02,979 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:00:05,151 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:00:10,152 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:00:14,365 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:00:14,671 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 05:00:14,672 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 05:00:14,673 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 05:00:14,676 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 05:00:15,134 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:00:15,135 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:00:15,307 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:00:15,368 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 05:00:16,369 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:00:16,724 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 05:00:16,725 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 05:00:16,725 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 05:00:16,727 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 05:00:17,373 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 05:00:18,374 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:00:18,783 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 05:00:18,785 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 05:00:18,785 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 05:00:18,788 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 05:00:19,378 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 05:00:20,379 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:00:20,384 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:00:20,836 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 05:00:20,837 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 05:00:20,837 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 05:00:20,840 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 05:00:21,382 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 05:00:22,384 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:00:22,882 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 05:00:22,883 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 05:00:22,884 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 05:00:22,886 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 05:00:23,387 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 05:00:24,388 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:00:24,923 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 05:00:24,924 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 05:00:24,924 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 05:00:24,926 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 05:00:25,392 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 05:00:25,492 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:00:26,394 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:00:26,983 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 05:00:26,984 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 05:00:26,984 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 05:00:26,988 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 05:00:27,397 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 05:00:28,400 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:00:28,574 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:00:29,063 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 05:00:29,064 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 05:00:29,064 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 05:00:29,070 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 05:00:29,403 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 05:00:30,134 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:00:30,135 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:00:30,405 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:00:30,638 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:00:31,093 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 05:00:31,094 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 05:00:31,094 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 05:00:31,097 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 05:00:31,408 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 05:00:32,409 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:00:33,137 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 05:00:33,138 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 05:00:33,138 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 05:00:33,142 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 05:00:33,414 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 05:00:34,416 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:00:35,186 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 05:00:35,187 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 05:00:35,187 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 05:00:35,189 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 05:00:35,419 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 05:00:35,770 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:00:36,421 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:00:37,249 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 05:00:37,250 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 05:00:37,250 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 05:00:37,253 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 05:00:37,424 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 05:00:38,316 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 05:00:38,317 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 05:00:38,317 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 05:00:38,320 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 05:00:38,427 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:00:38,428 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 05:00:40,430 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:00:41,331 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:00:45,209 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:00:45,209 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:00:46,381 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:00:51,382 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:00:56,382 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:00:58,575 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:01:00,151 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:01:00,152 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:01:02,321 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:01:06,466 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:01:07,206 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 05:01:07,207 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 05:01:07,207 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 05:01:07,211 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 05:01:07,372 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:01:07,469 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 05:01:08,471 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:01:09,259 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 05:01:09,260 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 05:01:09,260 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 05:01:09,263 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 05:01:09,474 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 05:01:10,475 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:01:11,312 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 05:01:11,313 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 05:01:11,313 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 05:01:11,315 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 05:01:11,478 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 05:01:12,480 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:01:12,489 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:01:13,349 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 05:01:13,350 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 05:01:13,351 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 05:01:13,353 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 05:01:13,483 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 05:01:14,485 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:01:15,151 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:01:15,152 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:01:15,374 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 05:01:15,375 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 05:01:15,375 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 05:01:15,378 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 05:01:15,487 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 05:01:16,489 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:01:17,415 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 05:01:17,416 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 05:01:17,417 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 05:01:17,420 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 05:01:17,493 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 05:01:17,583 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:01:18,495 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:01:19,471 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 05:01:19,472 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 05:01:19,473 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 05:01:19,476 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 05:01:19,497 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 05:01:20,499 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:01:21,515 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 05:01:21,516 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 05:01:21,517 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 05:01:21,519 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 05:01:22,503 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:01:22,504 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 05:01:22,694 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:01:23,597 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: partial_history 2023-09-08 05:01:23,598 DEBUG SenderThread:896010 [sender.py:send():369] send: history 2023-09-08 05:01:23,599 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: summary_record 2023-09-08 05:01:23,601 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 05:01:24,508 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:01:24,509 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 05:01:24,881 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:01:24,881 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:01:24,881 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:01:24,881 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:01:24,882 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:01:24,882 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:01:24,883 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:01:25,054 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:01:25,054 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:01:25,054 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:01:25,054 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:01:25,055 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:01:25,055 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:01:25,055 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:01:25,056 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:01:25,056 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:01:25,056 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:01:25,056 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:01:25,058 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:01:25,058 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:01:26,512 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:01:26,581 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:01:26,581 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:01:26,581 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:01:26,582 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:01:26,582 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:01:26,582 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:01:26,582 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:01:26,582 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:01:27,592 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:01:27,592 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:01:27,592 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:01:27,602 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:01:28,603 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:01:30,152 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:01:30,152 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:01:34,324 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:01:39,325 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:01:44,954 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:01:45,152 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:01:45,152 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:01:50,323 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:01:55,324 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:02:00,152 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:02:00,152 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:02:01,321 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:02:06,322 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:02:11,323 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:02:15,152 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:02:15,152 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:02:17,321 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:02:22,322 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:02:27,323 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:02:30,152 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:02:30,153 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:02:32,324 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:02:37,325 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:02:41,319 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:02:41,320 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:02:41,320 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:02:41,320 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:02:41,320 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:02:41,320 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:02:41,321 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:02:41,322 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:02:41,322 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:02:41,365 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:02:41,365 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:02:41,366 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:02:41,366 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:02:41,366 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:02:41,366 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:02:41,366 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:02:41,366 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:02:42,634 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:02:42,665 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:02:42,665 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:02:42,665 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:02:42,666 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:02:42,666 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:02:45,152 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:02:45,153 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:02:48,325 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:02:53,326 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:02:58,327 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:03:00,152 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:03:00,153 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:03:03,329 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:03:08,329 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:03:13,331 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:03:15,153 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:03:15,153 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:03:19,324 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:03:24,325 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:03:27,290 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:03:27,290 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:03:27,290 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:03:27,290 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:03:27,290 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:03:27,291 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:03:27,291 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:03:27,292 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:03:27,293 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:03:27,302 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:03:27,302 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:03:27,302 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:03:27,302 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:03:27,303 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:03:27,303 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:03:27,303 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:03:27,303 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:03:28,357 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:03:28,357 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:03:28,357 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:03:28,358 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:03:28,711 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:03:29,956 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:03:30,153 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:03:30,153 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:03:35,326 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:03:40,327 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:03:45,153 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:03:45,153 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:03:46,325 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:03:51,326 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:03:56,327 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:04:00,153 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:04:00,153 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:04:02,323 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:04:07,324 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:04:12,325 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:04:15,153 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:04:15,154 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:04:18,325 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:04:23,326 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:04:28,327 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:04:30,153 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:04:30,154 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:04:34,324 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:04:39,325 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:04:44,956 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:04:45,154 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:04:45,154 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:04:50,326 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:04:55,327 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:05:00,153 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:05:00,154 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:05:01,324 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:05:06,325 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:05:11,326 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:05:11,421 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:05:11,422 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:05:11,422 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:05:11,422 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:05:11,422 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:05:11,422 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:05:11,423 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:05:11,424 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:05:11,424 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:05:15,153 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:05:15,154 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:05:17,321 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:05:17,322 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:05:17,322 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:05:17,322 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:05:17,322 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:05:17,322 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:05:17,322 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:05:17,322 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:05:18,328 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:05:18,328 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:05:18,329 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:05:18,329 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:05:18,329 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:05:22,331 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:05:27,332 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:05:30,154 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:05:30,154 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:05:33,325 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:05:38,326 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:05:43,327 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:05:45,154 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:05:45,154 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:05:48,330 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:05:53,331 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:05:58,332 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:06:00,154 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:06:00,154 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:06:00,325 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:06:00,325 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:06:00,325 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:06:00,325 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:06:00,325 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:06:00,326 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:06:00,328 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:06:00,328 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:06:00,329 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:06:00,330 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:06:00,330 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:06:00,330 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:06:00,330 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:06:00,330 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:06:00,331 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:06:00,331 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:06:00,331 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:06:01,404 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:06:01,404 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:06:01,404 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:06:01,405 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:06:03,406 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:06:07,501 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:06:07,501 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:06:07,501 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:06:07,501 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:06:07,502 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:06:07,502 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:06:07,502 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:06:07,503 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:06:07,504 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:06:07,514 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:06:07,514 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:06:07,514 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:06:07,514 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:06:07,514 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:06:07,515 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:06:07,515 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:06:07,515 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:06:08,572 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:06:08,573 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:06:08,573 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:06:08,573 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:06:08,576 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:06:13,578 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:06:15,154 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:06:15,155 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:06:19,336 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:06:24,337 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:06:29,958 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:06:30,154 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:06:30,155 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:06:35,334 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:06:36,839 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:06:36,839 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:06:36,839 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:06:36,839 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:06:36,839 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:06:36,840 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:06:36,840 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:06:36,841 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:06:36,841 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:06:37,391 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:06:37,391 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:06:37,392 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:06:37,392 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:06:37,392 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:06:37,392 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:06:37,392 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:06:37,392 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:06:38,867 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:06:38,868 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:06:38,868 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:06:38,868 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:06:40,870 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:06:45,155 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:06:45,155 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:06:46,327 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:06:49,569 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:06:49,569 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:06:49,569 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:06:49,569 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:06:49,569 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:06:49,570 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:06:49,570 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:06:49,570 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:06:49,571 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:06:49,580 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:06:49,580 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:06:49,580 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:06:49,580 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:06:49,581 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:06:49,581 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:06:49,581 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:06:49,581 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:06:50,923 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:06:50,923 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:06:50,923 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:06:50,934 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:06:51,049 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:06:51,936 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:06:52,306 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:06:52,306 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:06:52,306 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:06:52,306 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:06:52,306 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:06:52,306 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:06:52,307 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:06:52,308 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:06:52,308 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:06:55,652 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:06:55,652 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:06:55,652 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:06:55,653 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:06:55,653 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:06:55,653 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:06:55,653 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:06:55,653 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:06:56,662 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:06:56,662 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:06:56,663 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:06:56,663 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:06:57,665 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:07:00,155 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:07:00,155 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:07:02,246 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:07:02,246 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:07:02,246 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:07:02,246 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:07:02,246 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:07:02,247 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:07:02,247 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:07:02,248 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:07:02,248 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:07:02,265 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:07:02,266 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:07:02,266 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:07:02,266 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:07:02,266 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:07:02,266 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:07:02,266 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:07:02,266 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:07:03,620 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:07:03,621 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:07:03,621 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:07:03,621 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:07:03,622 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:07:08,104 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:07:08,104 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:07:08,105 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:07:08,105 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:07:08,105 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:07:08,105 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:07:08,106 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:07:08,106 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:07:08,107 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:07:08,117 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:07:08,117 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:07:08,117 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:07:08,117 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:07:08,117 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:07:08,117 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:07:08,118 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:07:08,118 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:07:09,174 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:07:09,175 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:07:09,175 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:07:09,175 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:07:09,176 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:07:13,958 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:07:14,078 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:07:14,078 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:07:14,078 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:07:14,078 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:07:14,078 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:07:14,079 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:07:14,079 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:07:14,079 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:07:14,080 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:07:14,123 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:07:14,123 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:07:14,123 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:07:14,123 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:07:14,123 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:07:14,123 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:07:14,123 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:07:14,123 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:07:15,137 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:07:15,137 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:07:15,138 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:07:15,138 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:07:15,155 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:07:15,156 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:07:19,330 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:07:22,165 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:07:22,166 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:07:22,166 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:07:22,166 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:07:22,166 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:07:22,166 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:07:22,167 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:07:22,167 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:07:22,168 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:07:22,169 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:07:22,170 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:07:22,170 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:07:22,170 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:07:22,170 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:07:22,170 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:07:22,170 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:07:22,170 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:07:23,295 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:07:23,295 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:07:23,295 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:07:23,296 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:07:25,297 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:07:30,155 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:07:30,156 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:07:30,326 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:07:31,263 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:07:31,263 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:07:31,263 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:07:31,263 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:07:31,263 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:07:31,264 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:07:31,264 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:07:31,264 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:07:31,265 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:07:31,265 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:07:31,266 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:07:31,266 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:07:31,266 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:07:31,266 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:07:31,267 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:07:31,267 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:07:31,267 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:07:32,360 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:07:32,360 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:07:32,360 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:07:32,361 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:07:35,363 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:07:40,364 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:07:45,155 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:07:45,155 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:07:46,309 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:07:51,310 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:07:56,311 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:08:00,155 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:08:00,156 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:08:00,747 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:08:00,747 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:08:00,747 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:08:00,748 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:08:00,748 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:08:00,748 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:08:00,748 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:08:00,749 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:08:00,749 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:08:01,347 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:08:02,211 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:08:02,211 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:08:02,211 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:08:02,211 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:08:02,211 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:08:02,211 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:08:02,212 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:08:02,212 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:08:03,169 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:08:03,254 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:08:03,255 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:08:03,255 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:08:03,255 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:08:07,257 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:08:12,258 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:08:15,156 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:08:15,156 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:08:17,312 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:08:22,313 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:08:27,314 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:08:30,156 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:08:30,156 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:08:33,309 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:08:38,310 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:08:43,311 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:08:45,156 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:08:45,156 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:08:48,328 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:08:53,329 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:08:58,330 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:09:00,156 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:09:00,156 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:09:04,328 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:09:09,329 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:09:14,960 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:09:15,156 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:09:15,156 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:09:19,905 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:09:19,905 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:09:19,905 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:09:19,905 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:09:19,905 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:09:19,906 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:09:19,906 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:09:19,907 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:09:19,907 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:09:20,073 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:09:20,073 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:09:20,073 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:09:20,073 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:09:20,074 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:09:20,074 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:09:20,074 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:09:20,074 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:09:21,066 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:09:21,066 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:09:21,067 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:09:21,067 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:09:21,067 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:09:26,069 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:09:30,156 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:09:30,157 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:09:31,328 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:09:36,329 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:09:41,330 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:09:44,324 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:09:44,324 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:09:44,324 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:09:44,324 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:09:44,324 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:09:44,325 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:09:44,325 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:09:44,326 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:09:44,326 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:09:44,388 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:09:44,388 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:09:44,388 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:09:44,388 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:09:44,388 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:09:44,388 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:09:44,389 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:09:44,389 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:09:45,422 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:09:45,423 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:09:45,423 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:09:45,423 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:09:45,424 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:09:45,424 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:09:46,597 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:09:51,598 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:09:56,599 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:10:00,157 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:10:00,157 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:10:02,327 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:10:07,328 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:10:08,483 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:10:08,483 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:10:08,483 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:10:08,483 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:10:08,483 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:10:08,484 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:10:08,484 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:10:08,485 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:10:08,486 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:10:08,497 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:10:08,498 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:10:08,498 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:10:08,498 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:10:08,498 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:10:08,498 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:10:08,498 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:10:08,498 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:10:09,558 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:10:09,559 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:10:09,559 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:10:09,560 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:10:10,205 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:10:10,205 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:10:10,205 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:10:10,205 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:10:10,205 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:10:10,205 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:10:10,206 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:10:10,207 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:10:10,207 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:10:10,225 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:10:10,226 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:10:10,226 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:10:10,226 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:10:10,226 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:10:10,226 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:10:10,226 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:10:10,226 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:10:11,315 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:10:11,315 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:10:11,315 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:10:11,316 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:10:11,388 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:10:12,844 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:10:12,844 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:10:12,844 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:10:12,845 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:10:12,845 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:10:12,845 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:10:12,846 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:10:12,847 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:10:12,847 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:10:12,873 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:10:12,873 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:10:12,873 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:10:12,873 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:10:12,873 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:10:12,874 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:10:12,874 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:10:12,874 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:10:13,914 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:10:13,914 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:10:13,914 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:10:13,914 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:10:13,926 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:10:13,962 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:10:13,962 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:10:13,962 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:10:13,962 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:10:13,962 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:10:13,963 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:10:13,963 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:10:13,964 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:10:13,964 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:10:14,003 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:10:14,003 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:10:14,003 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:10:14,003 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:10:14,003 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:10:14,003 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:10:14,003 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:10:14,003 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:10:15,027 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:10:15,027 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:10:15,027 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:10:15,028 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:10:15,157 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:10:15,157 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:10:15,567 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:10:15,568 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:10:15,568 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:10:15,568 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:10:15,568 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:10:15,568 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:10:15,569 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:10:15,569 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:10:15,569 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:10:15,593 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:10:15,593 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:10:15,593 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:10:15,593 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:10:15,593 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:10:15,593 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:10:15,593 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:10:15,594 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:10:16,649 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:10:16,649 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:10:16,650 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:10:16,650 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:10:18,652 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:10:23,653 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:10:28,654 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:10:30,157 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:10:30,157 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:10:34,330 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:10:36,312 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:10:36,312 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:10:36,312 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:10:36,312 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:10:36,312 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:10:36,313 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:10:36,313 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:10:36,314 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:10:36,314 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:10:36,322 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:10:36,323 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:10:36,323 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:10:36,323 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:10:36,323 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:10:36,323 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:10:36,323 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:10:36,323 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:10:37,368 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:10:37,369 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:10:37,369 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:10:37,369 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:10:37,431 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:10:37,770 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:10:37,770 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:10:37,770 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:10:37,770 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:10:37,771 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:10:37,771 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:10:37,771 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:10:37,772 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:10:37,773 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:10:37,790 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:10:37,790 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:10:37,790 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:10:37,790 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:10:37,790 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:10:37,790 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:10:37,790 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:10:37,790 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:10:38,844 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:10:38,845 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:10:38,845 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:10:38,846 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:10:39,379 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:10:39,379 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:10:39,379 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:10:39,379 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:10:39,380 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:10:39,380 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:10:39,380 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:10:39,381 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:10:39,381 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:10:39,382 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:10:39,383 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:10:39,383 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:10:39,383 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:10:39,383 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:10:39,383 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:10:39,383 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:10:39,384 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:10:40,427 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:10:40,427 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:10:40,427 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:10:40,427 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:10:40,428 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:10:42,145 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:10:42,145 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:10:42,145 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:10:42,145 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:10:42,145 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:10:42,146 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:10:42,146 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:10:42,147 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:10:42,147 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:10:42,163 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:10:42,163 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:10:42,163 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:10:42,164 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:10:42,164 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:10:42,164 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:10:42,164 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:10:42,164 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:10:43,198 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:10:43,199 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:10:43,199 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:10:43,200 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:10:44,961 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:10:45,157 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:10:45,157 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:10:50,331 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:10:55,332 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:11:00,157 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:11:00,158 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:11:01,329 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:11:03,891 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:11:03,891 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:11:03,891 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:11:03,891 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:11:03,892 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:11:03,892 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:11:03,892 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:11:03,894 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:11:03,895 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:11:03,897 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:11:03,897 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:11:03,897 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:11:03,897 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:11:03,897 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:11:03,898 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:11:03,898 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:11:03,898 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:11:04,858 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:11:04,858 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:11:04,858 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:11:04,869 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:11:06,870 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:11:11,871 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:11:15,157 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:11:15,158 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:11:15,525 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:11:15,525 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:11:15,525 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:11:15,526 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:11:15,526 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:11:15,526 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:11:15,527 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:11:15,527 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:11:15,528 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:11:15,529 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:11:15,529 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:11:15,529 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:11:15,530 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:11:15,530 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:11:15,530 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:11:15,530 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:11:15,530 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:11:16,490 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:11:16,490 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:11:16,490 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:11:16,491 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:11:17,492 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:11:22,493 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:11:27,494 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:11:30,158 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:11:30,158 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:11:33,329 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:11:37,328 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:11:37,329 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:11:37,329 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:11:37,329 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:11:37,329 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:11:37,330 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:11:37,330 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:11:37,331 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:11:37,331 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:11:37,332 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:11:37,332 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:11:37,332 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:11:37,333 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:11:37,333 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:11:37,333 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:11:37,334 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:11:37,334 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:11:38,298 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:11:38,298 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:11:38,298 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:11:38,298 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:11:39,300 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:11:41,331 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:11:41,331 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:11:41,331 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:11:41,331 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:11:41,332 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:11:41,332 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:11:41,333 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:11:41,334 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:11:41,334 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:11:41,335 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:11:41,335 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:11:41,335 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:11:41,335 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:11:41,336 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:11:41,336 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:11:41,336 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:11:41,336 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:11:42,282 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:11:42,283 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:11:42,283 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:11:42,283 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:11:44,482 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:11:44,482 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:11:44,482 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:11:44,482 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:11:44,482 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:11:44,483 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:11:44,483 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:11:44,484 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:11:44,484 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:11:44,523 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:11:44,523 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:11:44,523 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:11:44,523 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:11:44,523 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:11:44,523 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:11:44,523 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:11:44,524 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:11:45,448 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:11:45,448 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:11:45,448 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:11:45,448 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:11:45,448 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:11:45,449 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:11:45,449 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:11:50,620 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:11:55,620 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:12:00,158 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:12:00,158 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:12:01,187 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:12:01,187 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:12:01,187 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:12:01,187 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:12:01,188 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:12:01,188 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:12:01,188 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:12:01,189 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:12:01,189 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:12:01,189 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:12:01,191 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:12:01,191 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:12:01,191 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:12:01,191 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:12:01,192 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:12:01,192 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:12:01,192 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:12:02,162 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:12:02,162 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:12:02,162 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:12:02,162 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:12:02,163 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:12:07,165 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:12:12,165 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:12:15,158 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:12:15,159 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:12:17,328 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:12:22,329 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:12:22,474 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:12:22,474 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:12:22,474 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:12:22,474 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:12:22,474 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:12:22,475 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:12:22,475 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:12:22,476 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:12:22,477 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:12:22,488 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:12:22,488 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:12:22,488 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:12:22,488 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:12:22,489 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:12:22,489 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:12:22,489 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:12:22,489 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:12:23,451 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:12:23,451 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:12:23,451 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:12:23,452 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:12:23,691 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:12:24,504 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:12:24,504 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:12:24,504 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:12:24,505 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:12:24,505 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:12:24,505 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:12:24,506 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:12:24,506 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:12:24,508 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:12:24,508 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:12:24,508 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:12:24,508 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:12:24,509 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:12:24,509 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:12:24,509 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:12:24,509 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:12:24,510 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:12:25,462 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:12:25,462 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:12:25,463 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:12:25,473 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:12:25,984 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:12:25,984 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:12:25,984 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:12:25,985 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:12:25,985 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:12:25,985 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:12:25,986 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:12:25,986 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:12:25,986 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:12:26,006 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:12:26,006 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:12:26,006 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:12:26,007 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:12:26,007 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:12:26,007 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:12:26,007 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:12:26,007 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:12:26,944 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:12:26,944 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:12:26,944 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:12:26,945 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:12:27,084 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:12:27,084 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:12:27,084 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:12:27,084 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:12:27,085 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:12:27,085 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:12:27,085 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:12:27,087 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:12:27,087 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:12:27,087 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:12:27,087 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:12:27,088 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:12:27,088 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:12:27,088 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:12:27,088 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:12:27,089 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:12:27,089 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:12:28,051 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:12:28,051 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:12:28,051 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:12:28,051 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:12:28,052 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:12:28,515 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:12:28,515 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:12:28,515 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:12:28,515 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:12:28,516 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:12:28,516 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:12:28,516 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:12:28,518 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:12:28,518 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:12:28,538 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:12:28,538 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:12:28,538 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:12:28,539 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:12:28,539 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:12:28,539 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:12:28,539 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:12:28,539 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:12:29,481 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:12:29,481 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:12:29,481 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:12:29,482 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:12:30,158 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:12:30,159 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:12:33,335 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:12:38,336 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:12:43,337 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:12:45,159 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:12:45,159 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:12:49,333 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:12:52,147 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:12:52,147 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:12:52,147 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:12:52,147 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:12:52,148 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:12:52,148 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:12:52,148 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:12:52,149 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:12:52,150 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:12:52,160 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:12:52,160 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:12:52,160 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:12:52,160 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:12:52,161 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:12:52,161 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:12:52,161 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:12:52,161 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:12:53,120 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:12:53,120 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:12:53,120 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:12:53,121 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:12:53,745 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:12:55,122 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:12:55,163 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:12:55,163 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:12:55,163 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:12:55,163 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:12:55,164 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:12:55,164 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:12:55,164 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:12:55,165 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:12:55,166 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:12:55,166 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:12:55,166 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:12:55,167 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:12:55,167 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:12:55,168 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:12:55,168 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:12:55,168 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:12:55,168 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:12:56,127 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:12:56,127 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:12:56,128 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:12:56,128 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:12:59,026 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:12:59,026 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:12:59,026 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:12:59,026 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:12:59,027 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:12:59,027 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:12:59,027 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:12:59,029 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:12:59,030 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:12:59,071 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:12:59,071 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:12:59,071 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:12:59,071 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:12:59,072 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:12:59,072 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:12:59,072 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:12:59,072 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:12:59,986 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:12:59,987 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:12:59,987 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:12:59,997 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:13:00,159 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:13:00,159 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:13:00,333 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:13:00,348 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:13:00,348 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:13:00,348 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:13:00,348 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:13:00,349 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:13:00,349 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:13:00,349 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:13:00,350 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:13:00,350 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:13:00,351 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:13:00,351 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:13:00,351 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:13:00,352 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:13:00,352 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:13:00,352 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:13:00,352 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:13:00,353 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:13:01,324 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:13:01,324 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:13:01,324 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:13:01,325 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:13:01,701 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:13:01,701 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:13:01,701 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:13:01,701 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:13:01,701 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:13:01,702 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:13:01,702 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:13:01,703 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:13:01,703 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:13:01,722 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:13:01,722 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:13:01,722 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:13:01,722 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:13:01,723 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:13:01,723 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:13:01,723 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:13:01,723 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:13:02,673 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:13:02,673 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:13:02,673 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:13:02,674 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:13:05,675 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:13:10,676 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:13:15,159 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:13:15,159 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:13:16,237 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:13:16,237 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:13:16,237 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:13:16,238 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:13:16,238 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:13:16,238 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:13:16,239 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:13:16,239 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:13:16,240 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:13:16,243 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:13:16,243 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:13:16,243 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:13:16,243 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:13:16,244 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:13:16,244 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:13:16,244 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:13:16,244 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:13:16,244 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:13:17,196 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:13:17,196 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:13:17,196 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:13:17,197 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:13:17,485 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:13:17,485 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:13:17,486 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:13:17,486 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:13:17,486 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:13:17,486 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:13:17,487 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:13:17,488 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:13:17,488 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:13:17,488 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:13:17,488 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:13:17,489 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:13:17,489 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:13:17,489 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:13:17,490 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:13:17,490 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:13:17,490 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:13:17,793 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:13:18,449 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:13:18,449 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:13:18,449 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:13:18,449 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:13:18,754 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:13:18,754 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:13:18,754 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:13:18,755 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:13:18,755 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:13:18,755 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:13:18,755 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:13:18,756 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:13:18,756 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:13:18,777 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:13:18,777 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:13:18,777 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:13:18,777 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:13:18,777 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:13:18,778 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:13:18,778 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:13:18,778 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:13:19,723 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:13:19,723 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:13:19,723 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:13:19,723 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:13:20,269 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:13:20,269 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:13:20,269 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:13:20,269 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:13:20,270 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:13:20,270 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:13:20,271 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:13:20,272 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:13:20,272 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:13:20,272 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:13:20,273 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:13:20,273 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:13:20,273 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:13:20,273 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:13:20,274 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:13:20,274 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:13:20,275 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:13:21,224 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:13:21,224 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:13:21,224 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:13:21,224 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:13:21,393 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:13:21,394 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:13:21,394 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:13:21,394 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:13:21,394 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:13:21,394 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:13:21,395 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:13:21,395 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:13:21,396 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:13:21,414 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:13:21,414 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:13:21,414 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:13:21,414 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:13:21,414 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:13:21,414 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:13:21,415 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:13:21,415 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:13:22,352 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:13:22,352 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:13:22,352 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:13:22,353 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:13:22,363 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:13:27,365 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:13:30,159 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:13:30,159 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:13:30,728 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:13:30,728 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:13:30,728 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:13:30,729 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:13:30,729 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:13:30,729 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:13:30,730 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:13:30,730 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:13:30,731 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:13:30,744 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:13:30,744 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:13:30,744 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:13:30,744 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:13:30,745 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:13:30,745 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:13:30,745 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:13:30,745 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:13:31,683 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:13:31,683 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:13:31,683 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:13:31,684 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:13:32,686 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:13:37,687 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:13:42,688 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:13:45,159 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:13:45,159 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:13:48,332 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:13:53,334 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:13:58,335 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:14:00,159 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:14:00,160 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:14:03,336 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:14:08,337 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:14:13,338 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:14:15,160 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:14:15,160 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:14:19,336 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:14:24,337 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:14:29,963 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:14:30,160 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:14:30,160 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:14:32,791 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:14:32,792 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:14:32,792 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:14:32,792 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:14:32,792 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:14:32,793 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:14:32,793 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:14:32,793 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:14:32,794 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:14:32,815 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:14:32,815 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:14:32,815 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:14:32,815 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:14:32,816 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:14:32,816 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:14:32,816 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:14:32,816 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:14:33,751 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:14:33,751 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:14:33,751 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:14:33,752 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:14:35,753 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:14:40,754 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:14:42,696 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:14:42,696 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:14:42,696 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:14:42,697 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:14:42,697 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:14:42,697 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:14:42,698 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:14:42,699 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:14:42,699 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:14:42,708 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:14:42,708 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:14:42,708 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:14:42,708 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:14:42,709 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:14:42,709 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:14:42,709 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:14:42,709 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:14:43,652 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:14:43,652 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:14:43,652 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:14:43,653 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:14:45,160 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:14:45,160 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:14:46,289 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:14:47,405 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:14:47,405 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:14:47,406 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:14:47,406 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:14:47,406 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:14:47,406 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:14:47,407 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:14:47,408 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:14:47,408 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:14:47,408 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:14:47,409 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:14:47,409 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:14:47,410 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:14:47,410 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:14:47,410 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:14:47,411 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:14:47,411 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:14:48,360 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:14:48,360 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:14:48,360 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:14:48,361 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:14:51,362 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:14:51,517 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:14:51,517 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:14:51,517 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:14:51,517 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:14:51,517 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:14:51,518 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:14:51,518 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:14:51,519 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:14:51,520 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:14:51,899 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:14:51,900 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:14:51,900 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:14:51,900 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:14:51,900 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:14:51,900 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:14:51,900 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:14:51,901 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:14:52,804 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:14:52,804 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:14:52,804 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:14:52,805 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:14:56,807 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:15:00,160 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:15:00,160 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:15:02,335 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:15:07,336 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:15:12,337 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:15:15,160 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:15:15,161 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:15:18,337 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:15:23,338 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:15:28,339 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:15:30,161 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:15:30,161 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:15:34,337 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:15:39,338 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:15:44,964 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:15:45,161 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:15:45,161 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:15:50,337 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:15:55,339 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:16:00,161 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:16:00,161 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:16:01,336 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:16:06,338 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:16:11,339 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:16:15,161 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:16:15,162 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:16:17,333 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:16:22,334 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:16:27,335 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:16:30,161 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:16:30,162 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:16:33,334 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:16:38,335 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:16:43,339 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:16:45,162 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:16:45,162 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:16:48,359 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:16:53,360 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:16:58,361 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:17:00,169 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:17:00,170 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:17:04,361 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:17:09,363 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:17:14,966 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:17:15,162 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:17:15,162 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:17:20,266 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:17:25,267 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:17:30,162 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:17:30,162 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:17:30,337 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:17:35,338 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:17:40,339 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:17:45,162 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:17:45,163 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:17:46,337 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:17:51,338 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:17:56,339 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:18:00,162 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:18:00,163 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:18:02,336 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:18:07,337 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:18:12,338 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:18:15,162 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:18:15,163 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:18:18,337 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:18:23,338 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:18:28,339 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:18:30,163 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:18:30,163 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:18:33,341 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:18:37,804 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:18:37,804 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:18:37,804 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:18:37,805 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:18:37,805 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:18:37,805 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:18:37,809 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:18:37,811 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:18:37,813 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:18:38,342 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:18:38,708 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:18:38,708 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:18:38,708 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:18:38,708 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:18:38,709 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:18:38,709 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:18:38,709 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:18:38,709 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:18:39,624 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:18:39,625 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:18:39,625 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:18:39,628 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:18:43,630 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:18:45,163 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:18:45,163 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:18:49,339 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:18:54,340 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:18:59,967 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:19:00,163 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:19:00,164 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:19:05,339 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:19:10,340 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:19:15,164 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:19:15,164 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:19:16,338 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:19:21,339 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:19:26,341 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:19:30,164 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:19:30,164 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:19:32,340 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:19:37,342 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:19:42,343 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:19:45,164 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:19:45,164 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:19:48,340 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:19:53,341 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:19:58,342 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:20:00,164 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:20:00,164 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:20:04,340 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:20:09,341 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:20:14,968 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:20:15,164 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:20:15,165 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:20:20,341 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:20:25,342 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:20:30,165 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:20:30,165 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:20:31,340 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:20:36,341 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:20:41,342 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:20:45,165 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:20:45,165 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:20:46,345 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:20:51,345 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:20:56,347 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:21:00,165 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:21:00,165 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:21:02,240 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:21:07,241 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:21:12,242 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:21:15,165 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:21:15,165 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:21:17,339 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:21:22,339 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:21:27,341 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:21:30,165 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:21:30,165 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:21:33,341 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:21:38,342 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:21:43,343 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:21:45,166 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:21:45,166 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:21:48,344 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:21:53,345 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:21:58,346 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:22:00,166 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:22:00,166 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:22:04,342 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:22:09,343 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:22:14,969 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:22:15,166 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:22:15,166 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:22:20,343 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:22:25,343 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:22:30,166 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:22:30,167 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:22:31,340 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:22:36,341 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:22:37,027 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:22:37,028 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:22:37,028 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:22:37,028 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:22:37,028 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:22:37,029 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:22:37,029 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:22:37,030 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:22:37,030 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:22:37,047 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:22:37,047 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:22:37,047 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:22:37,048 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:22:37,048 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:22:37,048 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:22:37,048 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:22:37,048 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:22:37,980 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:22:37,980 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:22:37,980 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:22:38,001 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:22:42,003 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:22:45,166 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:22:45,166 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:22:47,344 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:22:52,345 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:22:56,215 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:22:56,215 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:22:56,215 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:22:56,215 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:22:56,216 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:22:56,216 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:22:56,216 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:22:56,218 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:22:56,218 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:22:56,223 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:22:56,223 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:22:56,223 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:22:56,223 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:22:56,224 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:22:56,224 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:22:56,224 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:22:56,224 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:22:57,179 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:22:57,179 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:22:57,179 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:22:57,179 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:22:58,180 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:23:00,166 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:23:00,167 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:23:03,343 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:23:04,061 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:23:04,061 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:23:04,061 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:23:04,061 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:23:04,062 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:23:04,062 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:23:04,062 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:23:04,063 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:23:04,064 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:23:04,065 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:23:04,065 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:23:04,065 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:23:04,066 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:23:04,066 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:23:04,066 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:23:04,066 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:23:04,066 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:23:05,024 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:23:05,024 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:23:05,024 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:23:05,025 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:23:09,026 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:23:14,970 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:23:15,167 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:23:15,167 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:23:20,342 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:23:23,815 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:23:23,815 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:23:23,815 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:23:23,815 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:23:23,816 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:23:23,816 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:23:23,817 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:23:23,818 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:23:23,818 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:23:24,027 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:23:24,027 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:23:24,027 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:23:24,027 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:23:24,027 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:23:24,027 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:23:24,028 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:23:24,028 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:23:24,928 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:23:24,928 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:23:24,928 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:23:24,929 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:23:25,929 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:23:30,167 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:23:30,167 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:23:31,023 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:23:31,023 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:23:31,023 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:23:31,023 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:23:31,024 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:23:31,024 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:23:31,025 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:23:31,025 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:23:31,026 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:23:31,341 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:23:33,516 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:23:33,516 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:23:33,516 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:23:33,516 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:23:33,517 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:23:33,517 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:23:33,517 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:23:33,517 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:23:34,414 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:23:34,414 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:23:34,415 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:23:34,415 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:23:35,560 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:23:35,560 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:23:35,560 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:23:35,560 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:23:35,561 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:23:35,561 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:23:35,562 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:23:35,563 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:23:35,563 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:23:35,563 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:23:35,563 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:23:35,564 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:23:35,564 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:23:35,564 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:23:35,565 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:23:35,565 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:23:35,565 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:23:36,508 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:23:36,508 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:23:36,508 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:23:36,508 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:23:36,509 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:23:41,510 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:23:45,167 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:23:45,167 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:23:47,342 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:23:48,568 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:23:48,568 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:23:48,568 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:23:48,568 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:23:48,569 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:23:48,569 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:23:48,570 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:23:48,571 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:23:48,571 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:23:48,571 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:23:48,571 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:23:48,572 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:23:48,572 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:23:48,572 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:23:48,573 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:23:48,573 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:23:48,574 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:23:49,519 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:23:49,519 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:23:49,519 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:23:49,520 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:23:52,521 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:23:57,522 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:23:58,028 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:23:58,028 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:23:58,028 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:23:58,028 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:23:58,029 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:23:58,029 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:23:58,029 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:23:58,030 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:23:58,031 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:23:58,527 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:23:58,528 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:23:58,528 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:23:58,528 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:23:58,528 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:23:58,528 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:23:58,528 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:23:58,529 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:23:59,434 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:23:59,435 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:23:59,435 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:23:59,447 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:24:00,167 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:24:00,168 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:24:02,268 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:24:02,268 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:24:02,268 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:24:02,268 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:24:02,268 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:24:02,269 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:24:02,269 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:24:02,269 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:24:02,270 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:24:02,270 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:24:02,271 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:24:02,272 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:24:02,272 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:24:02,273 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:24:02,273 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:24:02,273 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:24:02,273 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:24:03,236 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:24:03,237 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:24:03,237 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:24:03,237 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:24:03,238 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:24:08,239 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:24:13,241 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:24:15,167 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:24:15,168 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:24:17,707 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:24:17,707 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:24:17,707 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:24:17,707 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:24:17,708 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:24:17,708 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:24:17,709 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:24:17,710 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:24:17,710 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:24:17,760 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:24:17,760 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:24:17,760 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:24:17,760 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:24:17,761 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:24:17,761 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:24:17,761 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:24:17,761 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:24:18,665 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:24:18,665 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:24:18,665 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:24:18,665 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:24:18,666 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:24:23,667 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:24:28,218 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:24:28,218 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:24:28,218 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:24:28,218 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:24:28,219 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:24:28,219 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:24:28,219 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:24:28,220 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:24:28,220 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:24:28,221 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:24:28,221 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:24:28,221 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:24:28,222 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:24:28,222 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:24:28,223 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:24:28,223 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:24:29,163 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:24:29,164 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:24:29,164 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:24:29,164 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:24:30,167 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:24:30,168 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:24:34,343 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:24:34,774 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:24:34,775 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:24:34,775 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:24:34,775 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:24:34,775 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:24:34,775 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:24:34,776 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:24:34,776 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:24:34,777 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:24:35,738 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:24:35,738 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:24:35,738 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:24:35,738 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:24:35,738 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:24:35,738 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:24:35,739 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:24:35,739 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:24:36,649 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:24:36,649 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:24:36,650 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:24:36,650 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:24:39,652 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:24:44,971 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:24:45,168 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:24:45,168 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:24:50,345 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:24:54,463 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:24:54,463 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:24:54,463 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:24:54,463 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:24:54,464 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:24:54,464 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:24:54,464 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:24:54,465 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:24:54,466 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:24:54,509 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:24:54,509 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:24:54,509 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:24:54,510 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:24:54,510 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:24:54,510 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:24:54,511 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:24:54,511 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:24:55,500 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:24:55,501 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:24:55,501 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:24:55,501 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:24:55,503 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:24:57,111 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:24:57,111 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:24:57,112 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:24:57,112 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:24:57,112 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:24:57,112 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:24:57,113 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:24:57,113 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:24:57,113 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:24:58,437 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:24:58,437 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:24:58,437 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:24:58,437 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:24:58,438 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:24:58,438 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:24:58,438 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:24:58,438 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:24:59,343 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:24:59,343 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:24:59,343 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:24:59,344 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:25:00,168 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:25:00,168 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:25:00,348 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:25:05,349 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:25:10,350 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:25:15,168 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:25:15,169 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:25:15,354 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:25:20,355 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:25:25,356 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:25:30,168 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:25:30,169 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:25:31,344 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:25:32,894 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:25:32,894 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:25:32,894 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:25:32,894 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:25:32,895 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:25:32,895 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:25:32,895 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:25:32,896 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:25:32,897 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:25:34,576 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:25:34,576 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:25:34,576 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:25:34,576 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:25:34,577 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:25:34,577 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:25:34,577 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:25:34,577 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:25:35,493 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:25:35,494 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:25:35,494 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:25:35,494 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:25:36,495 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:25:41,496 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:25:45,169 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:25:45,169 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:25:47,347 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:25:52,349 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:25:57,350 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:26:00,169 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:26:00,169 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:26:03,344 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:26:08,345 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:26:13,346 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:26:15,169 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:26:15,169 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:26:19,345 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:26:24,346 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:26:29,973 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:26:30,169 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:26:30,169 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:26:35,333 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:26:40,335 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:26:45,169 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:26:45,169 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:26:45,344 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:26:50,346 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:26:55,347 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:27:00,169 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:27:00,170 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:27:01,344 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:27:06,345 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:27:11,346 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:27:15,170 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:27:15,170 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:27:17,344 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:27:22,345 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:27:24,016 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:27:24,016 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:27:24,016 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:27:24,016 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:27:24,017 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:27:24,018 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:27:24,018 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:27:24,018 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:27:24,018 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:27:24,018 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:27:24,019 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:27:24,020 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:27:24,020 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:27:24,021 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:27:24,021 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:27:24,021 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:27:24,975 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:27:24,975 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:27:24,975 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:27:26,538 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:27:27,977 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:27:30,170 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:27:30,170 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:27:33,345 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:27:38,346 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:27:43,347 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:27:45,170 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:27:45,170 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:27:49,309 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:27:54,310 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:27:59,973 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:28:00,170 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:28:00,170 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:28:05,311 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:28:10,312 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:28:15,170 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:28:15,171 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:28:16,308 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:28:21,310 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:28:26,311 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:28:30,171 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:28:30,171 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:28:31,346 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:28:36,347 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:28:41,348 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:28:45,171 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:28:45,171 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:28:47,345 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:28:52,346 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:28:57,347 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:29:00,171 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:29:00,171 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:29:03,345 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:29:08,345 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:29:13,347 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:29:15,171 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:29:15,171 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:29:19,345 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:29:24,347 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:29:29,974 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:29:30,171 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:29:30,171 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:29:35,318 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:29:40,319 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:29:45,171 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:29:45,172 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:29:45,346 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:29:50,347 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:29:55,348 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:30:00,171 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:30:00,172 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:30:01,345 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:30:06,346 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:30:11,347 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:30:15,171 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:30:15,172 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:30:17,345 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:30:22,346 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:30:27,347 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:30:30,171 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:30:30,172 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:30:33,346 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:30:37,510 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:30:37,510 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:30:37,510 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:30:37,510 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:30:37,511 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:30:37,511 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:30:37,511 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:30:37,511 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:30:37,513 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:30:38,347 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:30:38,645 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:30:38,645 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:30:38,645 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:30:38,645 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:30:38,645 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:30:38,645 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:30:38,645 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:30:38,645 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:30:39,674 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:30:39,674 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:30:39,674 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:30:39,675 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:30:43,677 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:30:45,172 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:30:45,172 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:30:49,347 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:30:54,348 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:30:59,976 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:31:00,172 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:31:00,172 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:31:05,348 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:31:10,349 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:31:12,597 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:31:12,597 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:31:12,597 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:31:12,597 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:31:12,598 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:31:12,598 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:31:12,599 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:31:12,599 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:31:12,600 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:31:13,729 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:31:13,729 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:31:13,729 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:31:13,730 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:31:13,730 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:31:13,730 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:31:13,730 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:31:13,730 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:31:14,917 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:31:14,917 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:31:14,917 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:31:14,918 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:31:14,927 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:31:15,172 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:31:15,172 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:31:16,348 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:31:21,349 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:31:26,350 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:31:30,172 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:31:30,172 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:31:32,347 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:31:32,893 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:31:32,893 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:31:32,893 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:31:32,893 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:31:32,893 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:31:32,894 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:31:32,894 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:31:32,895 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:31:32,895 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:31:34,032 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:31:34,032 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:31:34,032 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:31:34,032 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:31:34,032 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:31:34,033 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:31:34,033 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:31:34,033 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:31:34,961 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:31:35,046 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:31:35,047 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:31:35,047 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:31:35,048 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:31:38,049 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:31:43,050 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:31:45,172 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:31:45,173 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:31:48,345 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:31:50,496 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:31:50,496 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:31:50,496 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:31:50,496 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:31:50,497 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:31:50,497 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:31:50,497 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:31:50,499 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:31:50,499 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:31:51,902 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:31:51,902 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:31:51,902 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:31:51,902 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:31:51,902 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:31:51,902 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:31:51,903 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:31:51,903 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:31:52,984 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:31:52,984 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:31:52,984 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:31:52,985 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:31:52,991 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:31:53,986 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:31:59,976 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:32:00,172 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:32:00,173 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:32:05,350 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:32:08,344 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:32:08,344 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:32:08,344 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:32:08,344 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:32:08,344 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:32:08,344 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:32:08,345 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:32:08,345 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:32:08,346 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:32:09,476 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:32:09,476 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:32:09,476 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:32:09,477 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:32:09,477 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:32:09,477 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:32:09,477 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:32:09,477 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:32:10,491 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:32:10,491 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:32:10,491 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:32:10,492 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:32:10,493 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:32:11,022 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:32:15,173 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:32:15,173 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:32:16,353 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:32:21,354 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:32:26,355 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:32:30,173 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:32:30,173 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:32:32,347 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:32:37,348 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:32:42,349 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:32:45,173 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:32:45,173 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:32:48,348 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:32:53,349 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:32:58,350 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:33:00,173 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:33:00,173 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:33:04,348 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:33:05,217 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:33:05,217 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:33:05,217 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:33:05,218 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:33:05,218 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:33:05,218 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:33:05,218 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:33:05,219 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:33:05,220 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:33:06,355 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:33:06,355 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:33:06,355 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:33:06,356 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:33:06,356 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:33:06,356 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:33:06,356 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:33:06,356 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:33:07,121 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:33:07,388 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:33:07,388 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:33:07,388 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:33:07,389 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:33:09,391 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:33:14,977 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:33:15,173 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:33:15,173 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:33:20,349 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:33:25,350 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:33:30,174 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:33:30,174 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:33:31,347 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:33:36,348 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:33:41,349 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:33:45,174 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:33:45,174 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:33:47,348 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:33:52,349 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:33:57,350 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:33:57,442 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:33:57,442 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:33:57,442 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:33:57,442 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:33:57,443 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:33:57,443 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:33:57,443 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:33:57,445 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:33:57,445 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:33:58,211 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:33:58,578 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:33:58,578 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:33:58,578 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:33:58,578 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:33:58,578 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:33:58,578 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:33:58,579 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:33:58,579 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:33:59,601 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:33:59,602 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:33:59,602 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:33:59,603 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:34:00,174 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:34:00,174 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:34:03,349 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:34:08,350 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:34:13,351 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:34:15,174 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:34:15,174 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:34:18,354 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:34:23,355 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:34:28,356 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:34:30,174 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:34:30,174 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:34:34,349 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:34:39,349 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:34:44,983 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:34:45,181 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:34:45,183 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:34:50,413 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:34:55,414 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:35:00,174 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:35:00,175 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:35:01,350 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:35:06,368 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:35:11,367 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:35:15,175 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:35:15,175 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:35:17,350 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:35:22,351 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:35:27,352 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:35:30,175 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:35:30,175 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:35:33,349 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:35:38,350 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:35:43,351 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:35:43,996 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:35:43,996 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:35:43,996 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:35:43,996 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:35:43,996 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:35:43,997 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:35:44,000 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:35:44,003 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:35:44,004 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:35:44,395 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:35:44,622 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:35:44,623 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:35:44,623 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:35:44,623 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:35:44,623 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:35:44,623 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:35:44,623 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:35:44,623 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:35:45,655 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:35:45,655 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:35:45,655 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:35:45,655 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:35:45,657 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:35:45,658 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:35:46,399 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:35:48,834 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:35:53,835 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:35:58,836 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:36:00,175 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:36:00,175 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:36:04,351 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:36:09,352 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:36:10,744 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:36:10,744 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:36:10,744 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:36:10,744 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:36:10,745 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:36:10,745 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:36:10,745 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:36:10,746 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:36:10,747 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:36:10,807 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:36:10,807 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:36:10,807 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:36:10,807 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:36:10,808 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:36:10,808 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:36:10,808 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:36:10,808 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:36:11,832 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:36:11,832 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:36:11,832 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:36:11,833 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:36:12,443 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:36:14,981 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:36:15,175 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:36:15,175 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:36:20,351 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:36:25,352 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:36:30,175 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:36:30,176 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:36:31,350 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:36:36,351 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:36:41,352 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:36:45,176 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:36:45,176 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:36:47,351 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:36:51,244 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:36:51,244 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:36:51,244 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:36:51,244 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:36:51,244 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:36:51,245 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:36:51,246 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:36:51,247 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:36:51,247 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:36:51,333 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:36:51,333 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:36:51,333 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:36:51,333 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:36:51,333 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:36:51,333 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:36:51,334 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:36:51,334 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:36:52,360 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:36:52,360 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:36:52,360 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:36:52,361 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:36:52,362 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:36:52,512 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:36:57,363 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:37:00,176 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:37:00,176 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:37:00,477 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:37:00,477 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:37:00,477 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:37:00,477 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:37:00,477 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:37:00,478 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:37:00,478 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:37:00,479 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:37:00,479 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:37:00,541 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:37:00,541 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:37:00,541 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:37:00,541 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:37:00,541 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:37:00,542 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:37:00,542 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:37:00,542 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:37:01,557 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:37:01,558 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:37:01,558 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:37:01,559 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:37:02,528 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:37:02,560 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:37:07,561 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:37:12,562 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:37:15,176 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:37:15,176 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:37:18,355 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:37:23,356 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:37:24,405 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:37:24,405 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:37:24,405 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:37:24,405 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:37:24,406 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:37:24,406 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:37:24,406 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:37:24,407 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:37:24,408 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:37:26,568 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:37:28,360 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:37:28,571 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:37:30,176 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:37:30,176 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:37:30,574 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:37:31,202 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:37:31,202 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:37:31,202 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:37:31,202 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:37:31,203 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:37:31,203 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:37:31,203 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:37:31,203 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:37:32,266 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:37:32,267 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:37:32,267 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:37:32,267 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:37:32,578 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:37:34,269 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:37:39,270 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:37:44,982 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:37:45,176 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:37:45,177 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:37:50,353 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:37:55,354 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:37:56,659 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:37:56,659 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:37:56,659 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:37:56,659 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:37:56,660 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:37:56,660 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:37:56,660 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:37:56,661 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:37:56,662 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:37:58,623 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:38:00,176 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:38:00,176 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:38:01,350 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:38:03,149 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:38:03,149 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:38:03,149 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:38:03,149 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:38:03,150 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:38:03,150 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:38:03,150 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:38:03,150 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:38:04,174 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:38:04,174 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:38:04,174 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:38:04,175 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:38:04,907 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:38:04,907 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:38:04,907 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:38:04,907 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:38:04,908 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:38:04,908 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:38:04,908 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:38:04,909 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:38:04,910 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:38:04,910 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:38:04,910 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:38:04,910 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:38:04,911 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:38:04,911 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:38:04,911 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:38:04,911 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:38:04,912 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:38:05,993 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:38:05,994 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:38:05,994 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:38:05,994 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:38:06,995 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:38:11,997 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:38:15,176 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:38:15,177 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:38:17,352 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:38:22,353 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:38:24,480 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:38:24,480 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:38:24,480 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:38:24,480 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:38:24,481 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:38:24,481 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:38:24,481 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:38:24,482 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:38:24,483 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:38:24,553 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:38:24,553 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:38:24,553 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:38:24,553 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:38:24,553 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:38:24,553 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:38:24,554 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:38:24,554 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:38:25,586 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:38:25,587 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:38:25,587 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:38:25,588 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:38:27,589 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:38:30,177 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:38:30,177 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:38:33,353 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:38:38,354 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:38:43,355 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:38:45,177 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:38:45,177 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:38:49,353 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:38:49,740 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:38:49,740 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:38:49,740 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:38:49,741 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:38:49,741 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:38:49,741 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:38:49,742 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:38:49,743 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:38:49,743 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:38:49,962 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:38:49,963 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:38:49,963 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:38:49,963 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:38:49,963 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:38:49,963 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:38:49,963 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:38:49,963 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:38:50,710 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:38:51,051 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:38:51,051 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:38:51,051 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:38:51,052 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:38:55,053 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:39:00,177 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:39:00,177 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:39:00,316 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:39:04,284 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:39:04,284 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:39:04,284 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:39:04,284 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:39:04,285 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:39:04,285 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:39:04,285 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:39:04,285 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:39:04,287 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:39:04,702 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:39:04,702 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:39:04,702 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:39:04,703 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:39:04,703 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:39:04,703 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:39:04,703 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:39:04,703 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:39:05,753 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:39:05,753 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:39:05,753 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:39:05,753 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:39:05,754 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:39:10,756 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:39:15,177 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:39:15,178 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:39:16,317 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:39:17,010 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:39:17,010 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:39:17,010 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:39:17,011 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:39:17,011 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:39:17,011 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:39:17,012 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:39:17,013 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:39:17,013 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:39:17,720 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:39:17,720 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:39:17,720 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:39:17,720 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:39:17,721 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:39:17,721 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:39:17,721 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:39:17,721 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:39:18,756 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:39:18,756 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:39:18,756 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:39:18,773 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:39:21,775 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:39:26,776 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:39:28,683 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:39:28,683 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:39:28,683 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:39:28,683 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:39:28,684 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:39:28,684 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:39:28,684 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:39:28,685 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:39:28,685 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:39:30,157 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:39:30,157 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:39:30,157 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:39:30,158 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:39:30,158 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:39:30,158 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:39:30,158 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:39:30,158 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:39:30,777 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:39:31,206 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:39:31,206 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:39:31,206 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:39:31,207 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:39:31,207 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:39:31,208 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:39:32,371 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:39:37,373 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:39:42,374 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:39:45,178 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:39:45,178 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:39:48,317 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:39:53,318 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:39:58,319 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:40:00,178 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:40:00,178 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:40:03,320 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:40:08,321 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:40:13,322 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:40:15,178 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:40:15,178 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:40:18,354 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:40:23,355 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:40:28,356 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:40:30,178 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:40:30,178 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:40:34,353 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:40:39,354 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:40:44,984 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:40:45,178 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:40:45,178 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:40:45,953 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:40:45,953 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:40:45,953 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:40:45,953 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:40:45,954 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:40:45,954 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:40:45,954 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:40:45,954 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:40:45,955 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:40:46,401 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:40:46,401 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:40:46,401 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:40:46,401 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:40:46,402 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:40:46,402 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:40:46,402 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:40:46,402 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:40:47,423 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:40:47,424 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:40:47,424 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:40:47,425 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:40:50,426 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:40:55,427 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:40:58,622 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:40:58,622 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:40:58,622 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:40:58,622 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:40:58,623 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:40:58,623 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:40:58,623 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:40:58,624 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:40:58,624 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:40:58,763 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:40:58,763 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:40:58,763 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:40:58,763 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:40:58,763 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:40:58,763 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:40:58,764 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:40:58,764 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:40:59,774 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:40:59,775 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:40:59,775 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:40:59,776 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:41:00,178 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:41:00,179 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:41:01,354 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:41:06,355 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:41:11,356 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:41:14,382 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: resume 2023-09-08 05:41:14,382 INFO HandlerThread:896010 [handler.py:handle_request_resume():719] starting system metrics thread or process 2023-09-08 05:41:14,382 INFO HandlerThread:896010 [system_monitor.py:start():181] Starting system monitor 2023-09-08 05:41:14,383 INFO SystemMonitor:896010 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-09-08 05:41:14,383 INFO SystemMonitor:896010 [interfaces.py:start():190] Started cpu monitoring 2023-09-08 05:41:14,383 INFO SystemMonitor:896010 [interfaces.py:start():190] Started disk monitoring 2023-09-08 05:41:14,383 INFO SystemMonitor:896010 [interfaces.py:start():190] Started gpu monitoring 2023-09-08 05:41:14,384 INFO SystemMonitor:896010 [interfaces.py:start():190] Started memory monitoring 2023-09-08 05:41:14,385 INFO SystemMonitor:896010 [interfaces.py:start():190] Started network monitoring 2023-09-08 05:41:14,812 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: pause 2023-09-08 05:41:14,812 INFO HandlerThread:896010 [handler.py:handle_request_pause():728] stopping system metrics thread or process 2023-09-08 05:41:14,812 INFO HandlerThread:896010 [system_monitor.py:finish():190] Stopping system monitor 2023-09-08 05:41:14,813 DEBUG SystemMonitor:896010 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-09-08 05:41:14,813 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined cpu monitor 2023-09-08 05:41:14,813 DEBUG SystemMonitor:896010 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-09-08 05:41:14,813 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined disk monitor 2023-09-08 05:41:14,813 DEBUG SystemMonitor:896010 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-09-08 05:41:15,831 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined gpu monitor 2023-09-08 05:41:15,831 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined memory monitor 2023-09-08 05:41:15,831 INFO HandlerThread:896010 [interfaces.py:finish():202] Joined network monitor 2023-09-08 05:41:15,832 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:41:15,832 DEBUG SenderThread:896010 [sender.py:send():369] send: stats 2023-09-08 05:41:15,833 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:41:17,009 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:41:22,010 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:41:27,011 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:41:30,179 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:41:30,179 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:41:32,353 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:41:37,354 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:41:42,355 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:41:45,179 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:41:45,179 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:41:48,354 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:41:53,355 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:41:58,356 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:42:00,179 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:42:00,180 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:42:04,355 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:42:09,356 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:42:14,984 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:42:15,179 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:42:15,179 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:42:20,356 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:42:25,357 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:42:30,179 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:42:30,179 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:42:31,354 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:42:36,355 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:42:41,356 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:42:45,179 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:42:45,179 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:42:47,355 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:42:52,356 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:42:57,357 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:43:00,179 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:43:00,180 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:43:03,353 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:43:08,354 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:43:13,355 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:43:15,179 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:43:15,180 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:43:19,352 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:43:24,353 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:43:29,985 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:43:30,179 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:43:30,180 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:43:35,354 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:43:40,356 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:43:45,180 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:43:45,180 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:43:46,353 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:43:51,354 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:43:56,355 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:44:00,180 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:44:00,180 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:44:02,354 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:44:07,355 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:44:12,356 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:44:15,180 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:44:15,180 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:44:18,354 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:44:23,355 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:44:28,356 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:44:30,180 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:44:30,181 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:44:33,360 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:44:38,361 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:44:43,362 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:44:45,180 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: stop_status 2023-09-08 05:44:45,181 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: stop_status 2023-09-08 05:44:49,356 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:44:52,141 DEBUG SenderThread:896010 [sender.py:send():369] send: exit 2023-09-08 05:44:52,142 INFO SenderThread:896010 [sender.py:send_exit():574] handling exit code: 0 2023-09-08 05:44:52,142 INFO SenderThread:896010 [sender.py:send_exit():576] handling runtime: 2297 2023-09-08 05:44:52,146 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 05:44:52,147 INFO SenderThread:896010 [sender.py:send_exit():582] send defer 2023-09-08 05:44:52,147 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: defer 2023-09-08 05:44:52,147 INFO HandlerThread:896010 [handler.py:handle_request_defer():170] handle defer: 0 2023-09-08 05:44:52,147 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: defer 2023-09-08 05:44:52,147 INFO SenderThread:896010 [sender.py:send_request_defer():598] handle sender defer: 0 2023-09-08 05:44:52,147 INFO SenderThread:896010 [sender.py:transition_state():602] send defer: 1 2023-09-08 05:44:52,148 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: defer 2023-09-08 05:44:52,148 INFO HandlerThread:896010 [handler.py:handle_request_defer():170] handle defer: 1 2023-09-08 05:44:52,148 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: defer 2023-09-08 05:44:52,148 INFO SenderThread:896010 [sender.py:send_request_defer():598] handle sender defer: 1 2023-09-08 05:44:52,148 INFO SenderThread:896010 [sender.py:transition_state():602] send defer: 2 2023-09-08 05:44:52,148 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: defer 2023-09-08 05:44:52,148 INFO HandlerThread:896010 [handler.py:handle_request_defer():170] handle defer: 2 2023-09-08 05:44:52,148 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: defer 2023-09-08 05:44:52,148 INFO SenderThread:896010 [sender.py:send_request_defer():598] handle sender defer: 2 2023-09-08 05:44:52,148 INFO SenderThread:896010 [sender.py:transition_state():602] send defer: 3 2023-09-08 05:44:52,148 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: defer 2023-09-08 05:44:52,148 INFO HandlerThread:896010 [handler.py:handle_request_defer():170] handle defer: 3 2023-09-08 05:44:52,148 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: defer 2023-09-08 05:44:52,148 INFO SenderThread:896010 [sender.py:send_request_defer():598] handle sender defer: 3 2023-09-08 05:44:52,148 INFO SenderThread:896010 [sender.py:transition_state():602] send defer: 4 2023-09-08 05:44:52,148 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: defer 2023-09-08 05:44:52,148 INFO HandlerThread:896010 [handler.py:handle_request_defer():170] handle defer: 4 2023-09-08 05:44:52,149 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: defer 2023-09-08 05:44:52,149 INFO SenderThread:896010 [sender.py:send_request_defer():598] handle sender defer: 4 2023-09-08 05:44:52,149 INFO SenderThread:896010 [sender.py:transition_state():602] send defer: 5 2023-09-08 05:44:52,149 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: defer 2023-09-08 05:44:52,149 INFO HandlerThread:896010 [handler.py:handle_request_defer():170] handle defer: 5 2023-09-08 05:44:52,149 DEBUG SenderThread:896010 [sender.py:send():369] send: summary 2023-09-08 05:44:52,151 INFO SenderThread:896010 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-09-08 05:44:52,152 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: defer 2023-09-08 05:44:52,152 INFO SenderThread:896010 [sender.py:send_request_defer():598] handle sender defer: 5 2023-09-08 05:44:52,152 INFO SenderThread:896010 [sender.py:transition_state():602] send defer: 6 2023-09-08 05:44:52,152 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: defer 2023-09-08 05:44:52,152 INFO HandlerThread:896010 [handler.py:handle_request_defer():170] handle defer: 6 2023-09-08 05:44:52,153 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: defer 2023-09-08 05:44:52,153 INFO SenderThread:896010 [sender.py:send_request_defer():598] handle sender defer: 6 2023-09-08 05:44:52,153 INFO SenderThread:896010 [sender.py:transition_state():602] send defer: 7 2023-09-08 05:44:52,153 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: status_report 2023-09-08 05:44:52,153 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: defer 2023-09-08 05:44:52,153 INFO HandlerThread:896010 [handler.py:handle_request_defer():170] handle defer: 7 2023-09-08 05:44:52,153 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: defer 2023-09-08 05:44:52,153 INFO SenderThread:896010 [sender.py:send_request_defer():598] handle sender defer: 7 2023-09-08 05:44:52,328 INFO Thread-12 :896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 05:44:53,079 INFO SenderThread:896010 [sender.py:transition_state():602] send defer: 8 2023-09-08 05:44:53,079 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: defer 2023-09-08 05:44:53,079 INFO HandlerThread:896010 [handler.py:handle_request_defer():170] handle defer: 8 2023-09-08 05:44:53,079 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: defer 2023-09-08 05:44:53,080 INFO SenderThread:896010 [sender.py:send_request_defer():598] handle sender defer: 8 2023-09-08 05:44:53,080 INFO SenderThread:896010 [job_builder.py:build():240] Attempting to build job artifact 2023-09-08 05:44:53,083 INFO SenderThread:896010 [job_builder.py:build():259] run is notebook based run 2023-09-08 05:44:53,083 INFO SenderThread:896010 [job_builder.py:build():274] no source found 2023-09-08 05:44:53,083 INFO SenderThread:896010 [sender.py:transition_state():602] send defer: 9 2023-09-08 05:44:53,083 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: defer 2023-09-08 05:44:53,083 INFO HandlerThread:896010 [handler.py:handle_request_defer():170] handle defer: 9 2023-09-08 05:44:53,083 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: defer 2023-09-08 05:44:53,083 INFO SenderThread:896010 [sender.py:send_request_defer():598] handle sender defer: 9 2023-09-08 05:44:53,083 INFO SenderThread:896010 [dir_watcher.py:finish():359] shutting down directory watcher 2023-09-08 05:44:53,142 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: poll_exit 2023-09-08 05:44:53,331 INFO SenderThread:896010 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:44:53,332 INFO SenderThread:896010 [dir_watcher.py:finish():389] scan: ./wandb/run-20230908_042527-jggbeix7/files 2023-09-08 05:44:53,333 INFO SenderThread:896010 [dir_watcher.py:finish():403] scan save: ./wandb/run-20230908_042527-jggbeix7/files/requirements.txt requirements.txt 2023-09-08 05:44:53,334 INFO SenderThread:896010 [dir_watcher.py:finish():403] scan save: ./wandb/run-20230908_042527-jggbeix7/files/config.yaml config.yaml 2023-09-08 05:44:53,334 INFO SenderThread:896010 [dir_watcher.py:finish():403] scan save: ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json wandb-summary.json 2023-09-08 05:44:53,358 INFO SenderThread:896010 [dir_watcher.py:finish():403] scan save: ./wandb/run-20230908_042527-jggbeix7/files/conda-environment.yaml conda-environment.yaml 2023-09-08 05:44:53,361 INFO SenderThread:896010 [dir_watcher.py:finish():403] scan save: ./wandb/run-20230908_042527-jggbeix7/files/wandb-metadata.json wandb-metadata.json 2023-09-08 05:44:53,361 INFO SenderThread:896010 [dir_watcher.py:finish():403] scan save: ./wandb/run-20230908_042527-jggbeix7/files/output.log output.log 2023-09-08 05:44:53,363 INFO SenderThread:896010 [sender.py:transition_state():602] send defer: 10 2023-09-08 05:44:53,363 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: poll_exit 2023-09-08 05:44:53,363 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: defer 2023-09-08 05:44:53,365 INFO HandlerThread:896010 [handler.py:handle_request_defer():170] handle defer: 10 2023-09-08 05:44:53,365 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: defer 2023-09-08 05:44:53,365 INFO SenderThread:896010 [sender.py:send_request_defer():598] handle sender defer: 10 2023-09-08 05:44:53,365 INFO SenderThread:896010 [file_pusher.py:finish():159] shutting down file pusher 2023-09-08 05:44:53,935 INFO wandb-upload_1:896010 [upload_job.py:push():133] Uploaded file ./wandb/run-20230908_042527-jggbeix7/files/requirements.txt 2023-09-08 05:44:54,124 INFO wandb-upload_2:896010 [upload_job.py:push():133] Uploaded file ./wandb/run-20230908_042527-jggbeix7/files/wandb-summary.json 2023-09-08 05:44:54,138 INFO wandb-upload_0:896010 [upload_job.py:push():133] Uploaded file ./wandb/run-20230908_042527-jggbeix7/files/config.yaml 2023-09-08 05:44:54,143 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: poll_exit 2023-09-08 05:44:54,144 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: poll_exit 2023-09-08 05:44:54,292 INFO wandb-upload_3:896010 [upload_job.py:push():133] Uploaded file ./wandb/run-20230908_042527-jggbeix7/files/conda-environment.yaml 2023-09-08 05:44:54,309 INFO wandb-upload_4:896010 [upload_job.py:push():133] Uploaded file ./wandb/run-20230908_042527-jggbeix7/files/output.log 2023-09-08 05:44:54,509 INFO Thread-11 (_thread_body):896010 [sender.py:transition_state():602] send defer: 11 2023-09-08 05:44:54,510 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: defer 2023-09-08 05:44:54,510 INFO HandlerThread:896010 [handler.py:handle_request_defer():170] handle defer: 11 2023-09-08 05:44:54,510 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: defer 2023-09-08 05:44:54,510 INFO SenderThread:896010 [sender.py:send_request_defer():598] handle sender defer: 11 2023-09-08 05:44:54,510 INFO SenderThread:896010 [file_pusher.py:join():164] waiting for file pusher 2023-09-08 05:44:54,510 INFO SenderThread:896010 [sender.py:transition_state():602] send defer: 12 2023-09-08 05:44:54,510 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: defer 2023-09-08 05:44:54,510 INFO HandlerThread:896010 [handler.py:handle_request_defer():170] handle defer: 12 2023-09-08 05:44:54,511 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: defer 2023-09-08 05:44:54,511 INFO SenderThread:896010 [sender.py:send_request_defer():598] handle sender defer: 12 2023-09-08 05:44:54,740 INFO SenderThread:896010 [sender.py:transition_state():602] send defer: 13 2023-09-08 05:44:54,740 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: defer 2023-09-08 05:44:54,740 INFO HandlerThread:896010 [handler.py:handle_request_defer():170] handle defer: 13 2023-09-08 05:44:54,740 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: defer 2023-09-08 05:44:54,740 INFO SenderThread:896010 [sender.py:send_request_defer():598] handle sender defer: 13 2023-09-08 05:44:54,741 INFO SenderThread:896010 [sender.py:transition_state():602] send defer: 14 2023-09-08 05:44:54,741 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: defer 2023-09-08 05:44:54,741 DEBUG SenderThread:896010 [sender.py:send():369] send: final 2023-09-08 05:44:54,742 INFO HandlerThread:896010 [handler.py:handle_request_defer():170] handle defer: 14 2023-09-08 05:44:54,742 DEBUG SenderThread:896010 [sender.py:send():369] send: footer 2023-09-08 05:44:54,742 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: defer 2023-09-08 05:44:54,742 INFO SenderThread:896010 [sender.py:send_request_defer():598] handle sender defer: 14 2023-09-08 05:44:54,742 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: poll_exit 2023-09-08 05:44:54,742 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: poll_exit 2023-09-08 05:44:54,743 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: poll_exit 2023-09-08 05:44:54,743 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: poll_exit 2023-09-08 05:44:54,743 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: server_info 2023-09-08 05:44:54,743 DEBUG SenderThread:896010 [sender.py:send_request():396] send_request: server_info 2023-09-08 05:44:54,744 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: get_summary 2023-09-08 05:44:54,745 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: sampled_history 2023-09-08 05:44:54,898 INFO MainThread:896010 [wandb_run.py:_footer_history_summary_info():3464] rendering history 2023-09-08 05:44:54,899 INFO MainThread:896010 [wandb_run.py:_footer_history_summary_info():3496] rendering summary 2023-09-08 05:44:54,899 INFO MainThread:896010 [wandb_run.py:_footer_sync_info():3423] logging synced files 2023-09-08 05:44:54,900 DEBUG HandlerThread:896010 [handler.py:handle_request():144] handle_request: shutdown 2023-09-08 05:44:54,901 INFO HandlerThread:896010 [handler.py:finish():854] shutting down handler 2023-09-08 05:44:55,743 INFO WriterThread:896010 [datastore.py:close():298] close: ./wandb/run-20230908_042527-jggbeix7/run-jggbeix7.wandb 2023-09-08 05:44:55,898 INFO SenderThread:896010 [sender.py:finish():1526] shutting down sender 2023-09-08 05:44:55,898 INFO SenderThread:896010 [file_pusher.py:finish():159] shutting down file pusher 2023-09-08 05:44:55,898 INFO SenderThread:896010 [file_pusher.py:join():164] waiting for file pusher