|
2023-08-31 16:21:12,589 INFO StreamThr :3427956 [internal.py:wandb_internal():86] W&B internal server running at pid: 3427956, started at: 2023-08-31 16:21:12.587723 |
|
2023-08-31 16:21:12,590 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: status |
|
2023-08-31 16:21:12,598 INFO WriterThread:3427956 [datastore.py:open_for_write():85] open: ./wandb/run-20230831_162112-pftwit53/run-pftwit53.wandb |
|
2023-08-31 16:21:12,600 DEBUG SenderThread:3427956 [sender.py:send():369] send: header |
|
2023-08-31 16:21:12,601 DEBUG SenderThread:3427956 [sender.py:send():369] send: run |
|
2023-08-31 16:21:12,606 INFO SenderThread:3427956 [sender.py:_maybe_setup_resume():737] checking resume status for ckadirt/brain2music/pftwit53 |
|
2023-08-31 16:21:13,166 INFO SenderThread:3427956 [dir_watcher.py:__init__():211] watching files in: ./wandb/run-20230831_162112-pftwit53/files |
|
2023-08-31 16:21:13,166 INFO SenderThread:3427956 [sender.py:_start_run_threads():1100] run started: pftwit53 with start time 1693498872.59191 |
|
2023-08-31 16:21:13,166 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: summary_record |
|
2023-08-31 16:21:13,168 INFO SenderThread:3427956 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end |
|
2023-08-31 16:21:13,171 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: check_version |
|
2023-08-31 16:21:13,172 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: check_version |
|
2023-08-31 16:21:13,241 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: run_start |
|
2023-08-31 16:21:13,251 DEBUG HandlerThread:3427956 [system_info.py:__init__():31] System info init |
|
2023-08-31 16:21:13,251 DEBUG HandlerThread:3427956 [system_info.py:__init__():46] System info init done |
|
2023-08-31 16:21:13,251 INFO HandlerThread:3427956 [system_monitor.py:start():181] Starting system monitor |
|
2023-08-31 16:21:13,251 INFO SystemMonitor:3427956 [system_monitor.py:_start():145] Starting system asset monitoring threads |
|
2023-08-31 16:21:13,252 INFO HandlerThread:3427956 [system_monitor.py:probe():201] Collecting system info |
|
2023-08-31 16:21:13,252 INFO SystemMonitor:3427956 [interfaces.py:start():190] Started cpu monitoring |
|
2023-08-31 16:21:13,252 INFO SystemMonitor:3427956 [interfaces.py:start():190] Started disk monitoring |
|
2023-08-31 16:21:13,253 INFO SystemMonitor:3427956 [interfaces.py:start():190] Started gpu monitoring |
|
2023-08-31 16:21:13,253 INFO SystemMonitor:3427956 [interfaces.py:start():190] Started memory monitoring |
|
2023-08-31 16:21:13,254 INFO SystemMonitor:3427956 [interfaces.py:start():190] Started network monitoring |
|
2023-08-31 16:21:13,306 DEBUG HandlerThread:3427956 [system_info.py:probe():195] Probing system |
|
2023-08-31 16:21:13,308 DEBUG HandlerThread:3427956 [system_info.py:_probe_git():180] Probing git |
|
2023-08-31 16:21:13,313 ERROR HandlerThread:3427956 [gitlib.py:root():89] git root error: Cmd('git') failed due to: exit code(128) |
|
cmdline: git rev-parse --show-toplevel |
|
stderr: 'fatal: detected dubious ownership in repository at '/fsx' |
|
To add an exception for this directory, call: |
|
|
|
git config --global --add safe.directory /fsx' |
|
2023-08-31 16:21:13,313 DEBUG HandlerThread:3427956 [system_info.py:_probe_git():188] Probing git done |
|
2023-08-31 16:21:13,313 DEBUG HandlerThread:3427956 [system_info.py:probe():240] Probing system done |
|
2023-08-31 16:21:13,313 DEBUG HandlerThread:3427956 [system_monitor.py:probe():210] {'os': 'Linux-5.15.0-1037-aws-x86_64-with-glibc2.31', 'python': '3.10.8', 'heartbeatAt': '2023-08-31T16:21:13.306106', 'startedAt': '2023-08-31T16:21:12.558810', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': '/fsx/proj-fmri/ckadirt/b2m/src/mlpdummy.py', 'codePath': 'src/mlpdummy.py', 'git': {'remote': None, 'commit': None}, 'email': None, 'root': '/fsx/proj-fmri/ckadirt/b2m', 'host': 'ip-26-0-144-149', 'username': 'ckadirt', 'executable': '/admin/home-ckadirt/miniconda3/envs/mindeye/bin/python', 'cpu_count': 48, 'cpu_count_logical': 96, 'cpu_freq': {'current': 3002.844541666667, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 2773.902, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 2591.475, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 2971.069, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 232.5829315185547, 'used': 89.25303268432617}, 'gpu': 'NVIDIA A100-SXM4-40GB', 'gpu_count': 8, 'gpu_devices': [{'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}], 'memory': {'total': 1121.815086364746}} |
|
2023-08-31 16:21:13,313 INFO HandlerThread:3427956 [system_monitor.py:probe():211] Finished collecting system info |
|
2023-08-31 16:21:13,313 INFO HandlerThread:3427956 [system_monitor.py:probe():214] Publishing system info |
|
2023-08-31 16:21:13,313 DEBUG HandlerThread:3427956 [system_info.py:_save_pip():51] Saving list of pip packages installed into the current environment |
|
2023-08-31 16:21:13,316 DEBUG HandlerThread:3427956 [system_info.py:_save_pip():67] Saving pip packages done |
|
2023-08-31 16:21:13,316 DEBUG HandlerThread:3427956 [system_info.py:_save_conda():74] Saving list of conda packages installed into the current environment |
|
2023-08-31 16:21:14,170 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_created():272] file/dir created: ./wandb/run-20230831_162112-pftwit53/files/conda-environment.yaml |
|
2023-08-31 16:21:14,170 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_created():272] file/dir created: ./wandb/run-20230831_162112-pftwit53/files/requirements.txt |
|
2023-08-31 16:21:14,170 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_created():272] file/dir created: ./wandb/run-20230831_162112-pftwit53/files/wandb-summary.json |
|
2023-08-31 16:21:19,578 DEBUG HandlerThread:3427956 [system_info.py:_save_conda():86] Saving conda packages done |
|
2023-08-31 16:21:19,578 DEBUG HandlerThread:3427956 [system_info.py:_save_code():89] Saving code |
|
2023-08-31 16:21:19,582 ERROR HandlerThread:3427956 [gitlib.py:root():89] git root error: Cmd('git') failed due to: exit code(128) |
|
cmdline: git rev-parse --show-toplevel |
|
stderr: 'fatal: detected dubious ownership in repository at '/fsx' |
|
To add an exception for this directory, call: |
|
|
|
git config --global --add safe.directory /fsx' |
|
2023-08-31 16:21:19,588 DEBUG HandlerThread:3427956 [system_info.py:_save_code():110] Saving code done |
|
2023-08-31 16:21:19,588 DEBUG HandlerThread:3427956 [system_info.py:_save_patches():127] Saving git patches |
|
2023-08-31 16:21:19,591 ERROR HandlerThread:3427956 [gitlib.py:root():89] git root error: Cmd('git') failed due to: exit code(128) |
|
cmdline: git rev-parse --show-toplevel |
|
stderr: 'fatal: detected dubious ownership in repository at '/fsx' |
|
To add an exception for this directory, call: |
|
|
|
git config --global --add safe.directory /fsx' |
|
2023-08-31 16:21:19,600 DEBUG HandlerThread:3427956 [system_info.py:_save_patches():169] Saving git patches done |
|
2023-08-31 16:21:19,604 INFO HandlerThread:3427956 [system_monitor.py:probe():216] Finished publishing system info |
|
2023-08-31 16:21:19,608 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: status_report |
|
2023-08-31 16:21:19,608 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: keepalive |
|
2023-08-31 16:21:19,608 DEBUG SenderThread:3427956 [sender.py:send():369] send: files |
|
2023-08-31 16:21:19,608 INFO SenderThread:3427956 [sender.py:_save_file():1354] saving file wandb-metadata.json with policy now |
|
2023-08-31 16:21:19,609 INFO SenderThread:3427956 [sender.py:_save_file():1354] saving file code/src/mlpdummy.py with policy now |
|
2023-08-31 16:21:19,616 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-08-31 16:21:19,616 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: stop_status |
|
2023-08-31 16:21:19,994 DEBUG SenderThread:3427956 [sender.py:send():369] send: telemetry |
|
2023-08-31 16:21:19,994 DEBUG SenderThread:3427956 [sender.py:send():369] send: metric |
|
2023-08-31 16:21:19,995 DEBUG SenderThread:3427956 [sender.py:send():369] send: telemetry |
|
2023-08-31 16:21:19,995 DEBUG SenderThread:3427956 [sender.py:send():369] send: metric |
|
2023-08-31 16:21:19,995 WARNING SenderThread:3427956 [sender.py:send_metric():1305] Seen metric with glob (shouldn't happen) |
|
2023-08-31 16:21:20,181 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230831_162112-pftwit53/files/conda-environment.yaml |
|
2023-08-31 16:21:20,182 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_created():272] file/dir created: ./wandb/run-20230831_162112-pftwit53/files/code/src/mlpdummy.py |
|
2023-08-31 16:21:20,182 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_created():272] file/dir created: ./wandb/run-20230831_162112-pftwit53/files/wandb-metadata.json |
|
2023-08-31 16:21:20,182 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_created():272] file/dir created: ./wandb/run-20230831_162112-pftwit53/files/output.log |
|
2023-08-31 16:21:20,182 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_created():272] file/dir created: ./wandb/run-20230831_162112-pftwit53/files/code/src |
|
2023-08-31 16:21:20,182 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_created():272] file/dir created: ./wandb/run-20230831_162112-pftwit53/files/code |
|
2023-08-31 16:21:20,188 INFO wandb-upload_0:3427956 [upload_job.py:push():133] Uploaded file /tmp/tmpcnz000j2wandb/zi0muto5-wandb-metadata.json |
|
2023-08-31 16:21:20,359 INFO wandb-upload_1:3427956 [upload_job.py:push():133] Uploaded file /tmp/tmpcnz000j2wandb/igjvcllv-code/src/mlpdummy.py |
|
2023-08-31 16:21:22,186 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230831_162112-pftwit53/files/output.log |
|
2023-08-31 16:21:23,961 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: status_report |
|
2023-08-31 16:21:26,201 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230831_162112-pftwit53/files/output.log |
|
2023-08-31 16:21:29,329 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: status_report |
|
2023-08-31 16:21:34,329 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: status_report |
|
2023-08-31 16:21:34,684 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: stop_status |
|
2023-08-31 16:21:34,684 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: stop_status |
|
2023-08-31 16:21:36,096 DEBUG SenderThread:3427956 [sender.py:send():369] send: exit |
|
2023-08-31 16:21:36,096 INFO SenderThread:3427956 [sender.py:send_exit():574] handling exit code: 1 |
|
2023-08-31 16:21:36,096 INFO SenderThread:3427956 [sender.py:send_exit():576] handling runtime: 22 |
|
2023-08-31 16:21:36,099 INFO SenderThread:3427956 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end |
|
2023-08-31 16:21:36,100 INFO SenderThread:3427956 [sender.py:send_exit():582] send defer |
|
2023-08-31 16:21:36,100 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer |
|
2023-08-31 16:21:36,100 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 0 |
|
2023-08-31 16:21:36,100 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer |
|
2023-08-31 16:21:36,100 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 0 |
|
2023-08-31 16:21:36,100 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 1 |
|
2023-08-31 16:21:36,101 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer |
|
2023-08-31 16:21:36,101 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 1 |
|
2023-08-31 16:21:36,101 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer |
|
2023-08-31 16:21:36,101 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 1 |
|
2023-08-31 16:21:36,101 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 2 |
|
2023-08-31 16:21:36,101 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer |
|
2023-08-31 16:21:36,101 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 2 |
|
2023-08-31 16:21:36,101 INFO HandlerThread:3427956 [system_monitor.py:finish():190] Stopping system monitor |
|
2023-08-31 16:21:36,101 INFO HandlerThread:3427956 [interfaces.py:finish():202] Joined cpu monitor |
|
2023-08-31 16:21:36,101 DEBUG SystemMonitor:3427956 [system_monitor.py:_start():159] Starting system metrics aggregation loop |
|
2023-08-31 16:21:36,102 INFO HandlerThread:3427956 [interfaces.py:finish():202] Joined disk monitor |
|
2023-08-31 16:21:36,102 DEBUG SystemMonitor:3427956 [system_monitor.py:_start():166] Finished system metrics aggregation loop |
|
2023-08-31 16:21:36,102 DEBUG SystemMonitor:3427956 [system_monitor.py:_start():170] Publishing last batch of metrics |
|
2023-08-31 16:21:36,277 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230831_162112-pftwit53/files/output.log |
|
2023-08-31 16:21:36,278 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230831_162112-pftwit53/files/wandb-summary.json |
|
2023-08-31 16:21:37,227 INFO HandlerThread:3427956 [interfaces.py:finish():202] Joined gpu monitor |
|
2023-08-31 16:21:37,228 INFO HandlerThread:3427956 [interfaces.py:finish():202] Joined memory monitor |
|
2023-08-31 16:21:37,228 INFO HandlerThread:3427956 [interfaces.py:finish():202] Joined network monitor |
|
2023-08-31 16:21:37,228 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: poll_exit |
|
2023-08-31 16:21:37,228 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer |
|
2023-08-31 16:21:37,229 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 2 |
|
2023-08-31 16:21:37,229 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 3 |
|
2023-08-31 16:21:37,229 DEBUG SenderThread:3427956 [sender.py:send():369] send: stats |
|
2023-08-31 16:21:37,229 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer |
|
2023-08-31 16:21:37,229 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: poll_exit |
|
2023-08-31 16:21:37,229 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 3 |
|
2023-08-31 16:21:37,230 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer |
|
2023-08-31 16:21:37,230 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 3 |
|
2023-08-31 16:21:37,230 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 4 |
|
2023-08-31 16:21:37,230 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer |
|
2023-08-31 16:21:37,230 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 4 |
|
2023-08-31 16:21:37,230 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer |
|
2023-08-31 16:21:37,230 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 4 |
|
2023-08-31 16:21:37,230 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 5 |
|
2023-08-31 16:21:37,230 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer |
|
2023-08-31 16:21:37,230 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 5 |
|
2023-08-31 16:21:37,231 DEBUG SenderThread:3427956 [sender.py:send():369] send: summary |
|
2023-08-31 16:21:37,233 INFO SenderThread:3427956 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end |
|
2023-08-31 16:21:37,233 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer |
|
2023-08-31 16:21:37,233 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 5 |
|
2023-08-31 16:21:37,233 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 6 |
|
2023-08-31 16:21:37,233 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer |
|
2023-08-31 16:21:37,233 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 6 |
|
2023-08-31 16:21:37,234 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer |
|
2023-08-31 16:21:37,234 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 6 |
|
2023-08-31 16:21:37,238 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: status_report |
|
2023-08-31 16:21:37,283 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230831_162112-pftwit53/files/wandb-summary.json |
|
2023-08-31 16:21:37,323 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 7 |
|
2023-08-31 16:21:37,323 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer |
|
2023-08-31 16:21:37,323 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 7 |
|
2023-08-31 16:21:37,324 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer |
|
2023-08-31 16:21:37,324 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 7 |
|
2023-08-31 16:21:38,097 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: poll_exit |
|
2023-08-31 16:21:38,288 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230831_162112-pftwit53/files/output.log |
|
2023-08-31 16:21:38,289 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230831_162112-pftwit53/files/config.yaml |
|
2023-08-31 16:21:38,923 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 8 |
|
2023-08-31 16:21:38,923 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: poll_exit |
|
2023-08-31 16:21:38,924 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer |
|
2023-08-31 16:21:38,924 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 8 |
|
2023-08-31 16:21:38,924 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer |
|
2023-08-31 16:21:38,924 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 8 |
|
2023-08-31 16:21:38,924 INFO SenderThread:3427956 [job_builder.py:build():240] Attempting to build job artifact |
|
2023-08-31 16:21:38,926 INFO SenderThread:3427956 [job_builder.py:build():274] no source found |
|
2023-08-31 16:21:38,926 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 9 |
|
2023-08-31 16:21:38,927 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer |
|
2023-08-31 16:21:38,927 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 9 |
|
2023-08-31 16:21:38,927 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer |
|
2023-08-31 16:21:38,927 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 9 |
|
2023-08-31 16:21:38,927 INFO SenderThread:3427956 [dir_watcher.py:finish():359] shutting down directory watcher |
|
2023-08-31 16:21:39,098 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: poll_exit |
|
2023-08-31 16:21:39,295 INFO SenderThread:3427956 [dir_watcher.py:finish():389] scan: ./wandb/run-20230831_162112-pftwit53/files |
|
2023-08-31 16:21:39,296 INFO SenderThread:3427956 [dir_watcher.py:finish():403] scan save: ./wandb/run-20230831_162112-pftwit53/files/wandb-summary.json wandb-summary.json |
|
2023-08-31 16:21:39,296 INFO SenderThread:3427956 [dir_watcher.py:finish():403] scan save: ./wandb/run-20230831_162112-pftwit53/files/requirements.txt requirements.txt |
|
2023-08-31 16:21:39,297 INFO SenderThread:3427956 [dir_watcher.py:finish():403] scan save: ./wandb/run-20230831_162112-pftwit53/files/wandb-metadata.json wandb-metadata.json |
|
2023-08-31 16:21:39,299 INFO SenderThread:3427956 [dir_watcher.py:finish():403] scan save: ./wandb/run-20230831_162112-pftwit53/files/conda-environment.yaml conda-environment.yaml |
|
2023-08-31 16:21:39,301 INFO SenderThread:3427956 [dir_watcher.py:finish():403] scan save: ./wandb/run-20230831_162112-pftwit53/files/config.yaml config.yaml |
|
2023-08-31 16:21:39,301 INFO SenderThread:3427956 [dir_watcher.py:finish():403] scan save: ./wandb/run-20230831_162112-pftwit53/files/output.log output.log |
|
2023-08-31 16:21:39,308 INFO SenderThread:3427956 [dir_watcher.py:finish():403] scan save: ./wandb/run-20230831_162112-pftwit53/files/code/src/mlpdummy.py code/src/mlpdummy.py |
|
2023-08-31 16:21:39,308 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 10 |
|
2023-08-31 16:21:39,308 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: poll_exit |
|
2023-08-31 16:21:39,309 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer |
|
2023-08-31 16:21:39,309 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 10 |
|
2023-08-31 16:21:39,309 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer |
|
2023-08-31 16:21:39,309 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 10 |
|
2023-08-31 16:21:39,309 INFO SenderThread:3427956 [file_pusher.py:finish():159] shutting down file pusher |
|
2023-08-31 16:21:39,746 INFO wandb-upload_0:3427956 [upload_job.py:push():133] Uploaded file ./wandb/run-20230831_162112-pftwit53/files/wandb-summary.json |
|
2023-08-31 16:21:39,814 INFO wandb-upload_1:3427956 [upload_job.py:push():133] Uploaded file ./wandb/run-20230831_162112-pftwit53/files/requirements.txt |
|
2023-08-31 16:21:39,848 INFO wandb-upload_2:3427956 [upload_job.py:push():133] Uploaded file ./wandb/run-20230831_162112-pftwit53/files/conda-environment.yaml |
|
2023-08-31 16:21:39,965 INFO wandb-upload_3:3427956 [upload_job.py:push():133] Uploaded file ./wandb/run-20230831_162112-pftwit53/files/config.yaml |
|
2023-08-31 16:21:40,100 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: poll_exit |
|
2023-08-31 16:21:40,100 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: poll_exit |
|
2023-08-31 16:21:40,182 INFO wandb-upload_4:3427956 [upload_job.py:push():133] Uploaded file ./wandb/run-20230831_162112-pftwit53/files/output.log |
|
2023-08-31 16:21:40,383 INFO Thread-11 (_thread_body):3427956 [sender.py:transition_state():602] send defer: 11 |
|
2023-08-31 16:21:40,384 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer |
|
2023-08-31 16:21:40,384 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 11 |
|
2023-08-31 16:21:40,384 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer |
|
2023-08-31 16:21:40,384 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 11 |
|
2023-08-31 16:21:40,384 INFO SenderThread:3427956 [file_pusher.py:join():164] waiting for file pusher |
|
2023-08-31 16:21:40,384 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 12 |
|
2023-08-31 16:21:40,384 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer |
|
2023-08-31 16:21:40,384 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 12 |
|
2023-08-31 16:21:40,385 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer |
|
2023-08-31 16:21:40,385 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 12 |
|
2023-08-31 16:21:40,463 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 13 |
|
2023-08-31 16:21:40,464 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer |
|
2023-08-31 16:21:40,464 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 13 |
|
2023-08-31 16:21:40,464 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer |
|
2023-08-31 16:21:40,464 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 13 |
|
2023-08-31 16:21:40,464 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 14 |
|
2023-08-31 16:21:40,464 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer |
|
2023-08-31 16:21:40,465 DEBUG SenderThread:3427956 [sender.py:send():369] send: final |
|
2023-08-31 16:21:40,465 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 14 |
|
2023-08-31 16:21:40,465 DEBUG SenderThread:3427956 [sender.py:send():369] send: footer |
|
2023-08-31 16:21:40,465 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer |
|
2023-08-31 16:21:40,465 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 14 |
|
2023-08-31 16:21:40,465 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: poll_exit |
|
2023-08-31 16:21:40,465 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: poll_exit |
|
2023-08-31 16:21:40,466 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: poll_exit |
|
2023-08-31 16:21:40,466 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: poll_exit |
|
2023-08-31 16:21:40,466 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: server_info |
|
2023-08-31 16:21:40,466 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: server_info |
|
2023-08-31 16:21:40,466 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: get_summary |
|
2023-08-31 16:21:40,468 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: sampled_history |
|
2023-08-31 16:21:40,613 INFO MainThread:3427956 [wandb_run.py:_footer_history_summary_info():3464] rendering history |
|
2023-08-31 16:21:40,613 INFO MainThread:3427956 [wandb_run.py:_footer_history_summary_info():3496] rendering summary |
|
2023-08-31 16:21:40,613 INFO MainThread:3427956 [wandb_run.py:_footer_sync_info():3423] logging synced files |
|
2023-08-31 16:21:40,614 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: shutdown |
|
2023-08-31 16:21:40,614 INFO HandlerThread:3427956 [handler.py:finish():854] shutting down handler |
|
2023-08-31 16:21:41,466 INFO WriterThread:3427956 [datastore.py:close():298] close: ./wandb/run-20230831_162112-pftwit53/run-pftwit53.wandb |
|
2023-08-31 16:21:41,613 INFO SenderThread:3427956 [sender.py:finish():1526] shutting down sender |
|
2023-08-31 16:21:41,613 INFO SenderThread:3427956 [file_pusher.py:finish():159] shutting down file pusher |
|
2023-08-31 16:21:41,614 INFO SenderThread:3427956 [file_pusher.py:join():164] waiting for file pusher |
|
|