| 2024-03-27 19:04:01,650 INFO StreamThr :1855917 [internal.py:wandb_internal():86] W&B internal server running at pid: 1855917, started at: 2024-03-27 19:04:01.649845 | |
| 2024-03-27 19:04:01,650 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status | |
| 2024-03-27 19:04:01,652 INFO WriterThread:1855917 [datastore.py:open_for_write():87] open: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/run-h708adzm.wandb | |
| 2024-03-27 19:04:01,652 DEBUG SenderThread:1855917 [sender.py:send():379] send: header | |
| 2024-03-27 19:04:01,652 DEBUG SenderThread:1855917 [sender.py:send():379] send: run | |
| 2024-03-27 19:04:02,097 INFO SenderThread:1855917 [dir_watcher.py:__init__():211] watching files in: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files | |
| 2024-03-27 19:04:02,097 INFO SenderThread:1855917 [sender.py:_start_run_threads():1124] run started: h708adzm with start time 1711566241.648942 | |
| 2024-03-27 19:04:02,100 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: check_version | |
| 2024-03-27 19:04:02,100 DEBUG SenderThread:1855917 [sender.py:send_request():406] send_request: check_version | |
| 2024-03-27 19:04:02,267 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: run_start | |
| 2024-03-27 19:04:02,297 DEBUG HandlerThread:1855917 [system_info.py:__init__():26] System info init | |
| 2024-03-27 19:04:02,297 DEBUG HandlerThread:1855917 [system_info.py:__init__():41] System info init done | |
| 2024-03-27 19:04:02,297 INFO HandlerThread:1855917 [system_monitor.py:start():194] Starting system monitor | |
| 2024-03-27 19:04:02,297 INFO SystemMonitor:1855917 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-03-27 19:04:02,297 INFO SystemMonitor:1855917 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-03-27 19:04:02,298 INFO HandlerThread:1855917 [system_monitor.py:probe():214] Collecting system info | |
| 2024-03-27 19:04:02,298 INFO SystemMonitor:1855917 [interfaces.py:start():190] Started disk monitoring | |
| 2024-03-27 19:04:02,298 INFO SystemMonitor:1855917 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-03-27 19:04:02,299 INFO SystemMonitor:1855917 [interfaces.py:start():190] Started memory monitoring | |
| 2024-03-27 19:04:02,299 INFO SystemMonitor:1855917 [interfaces.py:start():190] Started network monitoring | |
| 2024-03-27 19:04:02,321 DEBUG HandlerThread:1855917 [system_info.py:probe():150] Probing system | |
| 2024-03-27 19:04:02,322 DEBUG HandlerThread:1855917 [gitlib.py:_init_repo():56] git repository is invalid | |
| 2024-03-27 19:04:02,322 DEBUG HandlerThread:1855917 [system_info.py:probe():198] Probing system done | |
| 2024-03-27 19:04:02,322 DEBUG HandlerThread:1855917 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-89-generic-x86_64-with-glibc2.35', 'python': '3.10.14', 'heartbeatAt': '2024-03-27T19:04:02.321789', 'startedAt': '2024-03-27T19:04:01.647083', 'docker': None, 'cuda': None, 'args': ('--model_name_or_path', 'google-bert/bert-base-uncased', '--output_dir', './run', '--do_train', '--do_eval', '--do_predict'), 'state': 'running', 'program': '/home/gordon1109/qa_test/run_qa.py', 'codePathLocal': 'run_qa.py', 'codePath': 'run_qa.py', 'host': 'sally', 'username': 'gordon1109', 'executable': '/home/gordon1109/.miniconda3/envs/qa_tune/bin/python', 'cpu_count': 16, 'cpu_count_logical': 32, 'cpu_freq': {'current': 4424.5814375, 'min': 3000.0, 'max': 4500.0}, 'cpu_freq_per_core': [{'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3586.606, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}], 'disk': {'/': {'total': 788.7955780029297, 'used': 47.308345794677734}}, 'gpu': 'NVIDIA GeForce RTX 4090', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 4090', 'memory_total': 25757220864}], 'memory': {'total': 125.00867462158203}} | |
| 2024-03-27 19:04:02,322 INFO HandlerThread:1855917 [system_monitor.py:probe():224] Finished collecting system info | |
| 2024-03-27 19:04:02,322 INFO HandlerThread:1855917 [system_monitor.py:probe():227] Publishing system info | |
| 2024-03-27 19:04:02,322 DEBUG HandlerThread:1855917 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment | |
| 2024-03-27 19:04:03,098 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_created():271] file/dir created: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/conda-environment.yaml | |
| 2024-03-27 19:04:03,338 DEBUG HandlerThread:1855917 [system_info.py:_save_conda():222] Saving conda packages done | |
| 2024-03-27 19:04:03,339 INFO HandlerThread:1855917 [system_monitor.py:probe():229] Finished publishing system info | |
| 2024-03-27 19:04:03,340 DEBUG SenderThread:1855917 [sender.py:send():379] send: files | |
| 2024-03-27 19:04:03,340 INFO SenderThread:1855917 [sender.py:_save_file():1390] saving file wandb-metadata.json with policy now | |
| 2024-03-27 19:04:03,375 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: python_packages | |
| 2024-03-27 19:04:03,375 DEBUG SenderThread:1855917 [sender.py:send_request():406] send_request: python_packages | |
| 2024-03-27 19:04:03,375 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-27 19:04:03,376 DEBUG SenderThread:1855917 [sender.py:send_request():406] send_request: stop_status | |
| 2024-03-27 19:04:03,377 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-27 19:04:03,691 DEBUG SenderThread:1855917 [sender.py:send():379] send: telemetry | |
| 2024-03-27 19:04:04,071 INFO wandb-upload_0:1855917 [upload_job.py:push():131] Uploaded file /tmp/tmp1psp4pcbwandb/cmzt5n6v-wandb-metadata.json | |
| 2024-03-27 19:04:04,098 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/conda-environment.yaml | |
| 2024-03-27 19:04:04,098 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_created():271] file/dir created: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/requirements.txt | |
| 2024-03-27 19:04:04,098 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_created():271] file/dir created: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/wandb-metadata.json | |
| 2024-03-27 19:04:06,693 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:04:11,693 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:04:16,695 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:04:18,376 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-27 19:04:18,376 DEBUG SenderThread:1855917 [sender.py:send_request():406] send_request: stop_status | |
| 2024-03-27 19:04:18,418 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-27 19:04:22,570 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:04:26,105 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_created():271] file/dir created: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:04:27,583 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:04:28,105 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:04:30,106 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:04:32,106 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:04:32,801 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:04:33,106 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/config.yaml | |
| 2024-03-27 19:04:33,338 INFO StreamThr :1855917 [internal.py:wandb_internal():86] W&B internal server running at pid: 1855917, started at: 2024-03-27 19:04:33.338087 | |
| 2024-03-27 19:04:33,339 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status | |
| 2024-03-27 19:04:33,340 INFO WriterThread:1855917 [datastore.py:open_for_write():87] open: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/run-tszlk6iz.wandb | |
| 2024-03-27 19:04:33,340 DEBUG SenderThread:1855917 [sender.py:send():379] send: header | |
| 2024-03-27 19:04:33,340 DEBUG SenderThread:1855917 [sender.py:send():379] send: run | |
| 2024-03-27 19:04:33,375 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-27 19:04:33,376 DEBUG SenderThread:1855917 [sender.py:send_request():406] send_request: stop_status | |
| 2024-03-27 19:04:33,376 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-27 19:04:33,742 INFO SenderThread:1855917 [dir_watcher.py:__init__():211] watching files in: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files | |
| 2024-03-27 19:04:33,742 INFO SenderThread:1855917 [sender.py:_start_run_threads():1124] run started: tszlk6iz with start time 1711566273.337732 | |
| 2024-03-27 19:04:33,749 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: check_version | |
| 2024-03-27 19:04:33,749 DEBUG SenderThread:1855917 [sender.py:send_request():406] send_request: check_version | |
| 2024-03-27 19:04:33,921 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: run_start | |
| 2024-03-27 19:04:33,922 DEBUG HandlerThread:1855917 [system_info.py:__init__():26] System info init | |
| 2024-03-27 19:04:33,922 DEBUG HandlerThread:1855917 [system_info.py:__init__():41] System info init done | |
| 2024-03-27 19:04:33,922 INFO HandlerThread:1855917 [system_monitor.py:start():194] Starting system monitor | |
| 2024-03-27 19:04:33,922 INFO SystemMonitor:1855917 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-03-27 19:04:33,923 INFO HandlerThread:1855917 [system_monitor.py:probe():214] Collecting system info | |
| 2024-03-27 19:04:33,923 INFO SystemMonitor:1855917 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-03-27 19:04:33,923 INFO SystemMonitor:1855917 [interfaces.py:start():190] Started disk monitoring | |
| 2024-03-27 19:04:33,924 INFO SystemMonitor:1855917 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-03-27 19:04:33,924 INFO SystemMonitor:1855917 [interfaces.py:start():190] Started memory monitoring | |
| 2024-03-27 19:04:33,924 INFO SystemMonitor:1855917 [interfaces.py:start():190] Started network monitoring | |
| 2024-03-27 19:04:33,967 DEBUG HandlerThread:1855917 [system_info.py:probe():150] Probing system | |
| 2024-03-27 19:04:33,967 DEBUG HandlerThread:1855917 [gitlib.py:_init_repo():56] git repository is invalid | |
| 2024-03-27 19:04:33,967 DEBUG HandlerThread:1855917 [system_info.py:probe():198] Probing system done | |
| 2024-03-27 19:04:33,967 DEBUG HandlerThread:1855917 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-89-generic-x86_64-with-glibc2.35', 'python': '3.10.14', 'heartbeatAt': '2024-03-27T19:04:33.967587', 'startedAt': '2024-03-27T19:04:33.335983', 'docker': None, 'cuda': None, 'args': ('--node-ip-address=140.112.90.72', '--node-manager-port=33181', '--object-store-name=/tmp/ray/session_2024-03-27_19-04-27_986745_1855858/sockets/plasma_store', '--raylet-name=/tmp/ray/session_2024-03-27_19-04-27_986745_1855858/sockets/raylet', '--redis-address=None', '--metrics-agent-port=52536', '--runtime-env-agent-port=54815', '--logging-rotate-bytes=536870912', '--logging-rotate-backup-count=5', '--runtime-env-agent-port=54815', '--gcs-address=140.112.90.72:44817', '--session-name=session_2024-03-27_19-04-27_986745_1855858', '--temp-dir=/tmp/ray', '--webui=127.0.0.1:8265', '--cluster-id=8366c6b584aa9ce726c1927d2af9c014ef82eaaddd946cb356a7cba8', '--startup-token=32', '--worker-launch-time-ms=1711566270449', '--runtime-env-hash=-1413791333'), 'state': 'running', 'program': '/home/gordon1109/.miniconda3/envs/qa_tune/lib/python3.10/site-packages/ray/_private/workers/default_worker.py', 'codePathLocal': None, 'host': 'sally', 'username': 'gordon1109', 'executable': '/home/gordon1109/.miniconda3/envs/qa_tune/bin/python', 'cpu_count': 16, 'cpu_count_logical': 32, 'cpu_freq': {'current': 3542.56859375, 'min': 3000.0, 'max': 4500.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4964.736, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 3000.0, 'min': 3000.0, 'max': 4500.0}, {'current': 4500.0, 'min': 3000.0, 'max': 4500.0}], 'disk': {'/': {'total': 788.7955780029297, 'used': 47.309295654296875}}, 'gpu': 'NVIDIA GeForce RTX 4090', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 4090', 'memory_total': 25757220864}], 'memory': {'total': 125.00867462158203}} | |
| 2024-03-27 19:04:33,967 INFO HandlerThread:1855917 [system_monitor.py:probe():224] Finished collecting system info | |
| 2024-03-27 19:04:33,967 INFO HandlerThread:1855917 [system_monitor.py:probe():227] Publishing system info | |
| 2024-03-27 19:04:33,967 DEBUG HandlerThread:1855917 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment | |
| 2024-03-27 19:04:34,107 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:04:34,744 INFO Thread-25 :1855917 [dir_watcher.py:_on_file_created():271] file/dir created: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/conda-environment.yaml | |
| 2024-03-27 19:04:34,983 DEBUG HandlerThread:1855917 [system_info.py:_save_conda():222] Saving conda packages done | |
| 2024-03-27 19:04:34,983 INFO HandlerThread:1855917 [system_monitor.py:probe():229] Finished publishing system info | |
| 2024-03-27 19:04:34,983 DEBUG SenderThread:1855917 [sender.py:send():379] send: files | |
| 2024-03-27 19:04:34,984 INFO SenderThread:1855917 [sender.py:_save_file():1390] saving file wandb-metadata.json with policy now | |
| 2024-03-27 19:04:35,017 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: python_packages | |
| 2024-03-27 19:04:35,017 DEBUG SenderThread:1855917 [sender.py:send_request():406] send_request: python_packages | |
| 2024-03-27 19:04:35,018 DEBUG SenderThread:1855917 [sender.py:send():379] send: telemetry | |
| 2024-03-27 19:04:35,066 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-27 19:04:35,066 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-27 19:04:35,066 DEBUG SenderThread:1855917 [sender.py:send_request():406] send_request: stop_status | |
| 2024-03-27 19:04:35,351 DEBUG SenderThread:1855917 [sender.py:send():379] send: config | |
| 2024-03-27 19:04:35,352 DEBUG SenderThread:1855917 [sender.py:send():379] send: metric | |
| 2024-03-27 19:04:35,352 DEBUG SenderThread:1855917 [sender.py:send():379] send: telemetry | |
| 2024-03-27 19:04:35,352 DEBUG SenderThread:1855917 [sender.py:send():379] send: metric | |
| 2024-03-27 19:04:35,352 WARNING SenderThread:1855917 [sender.py:send_metric():1341] Seen metric with glob (shouldn't happen) | |
| 2024-03-27 19:04:35,352 DEBUG SenderThread:1855917 [sender.py:send():379] send: telemetry | |
| 2024-03-27 19:04:35,649 INFO wandb-upload_0:1855917 [upload_job.py:push():131] Uploaded file /tmp/tmpmtmt7bb2wandb/gxma4yqp-wandb-metadata.json | |
| 2024-03-27 19:04:35,742 INFO Thread-25 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/conda-environment.yaml | |
| 2024-03-27 19:04:35,742 INFO Thread-25 :1855917 [dir_watcher.py:_on_file_created():271] file/dir created: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/wandb-metadata.json | |
| 2024-03-27 19:04:35,743 INFO Thread-25 :1855917 [dir_watcher.py:_on_file_created():271] file/dir created: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/requirements.txt | |
| 2024-03-27 19:04:36,107 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:04:38,025 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:04:38,353 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:04:43,025 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:04:43,353 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:04:48,026 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:04:48,354 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:04:48,375 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-27 19:04:48,376 DEBUG SenderThread:1855917 [sender.py:send_request():406] send_request: stop_status | |
| 2024-03-27 19:04:48,377 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-27 19:04:50,017 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-27 19:04:50,018 DEBUG SenderThread:1855917 [sender.py:send_request():406] send_request: stop_status | |
| 2024-03-27 19:04:50,058 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-27 19:04:53,614 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:04:54,076 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:04:58,615 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:04:59,077 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:05:02,112 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:05:02,299 DEBUG SystemMonitor:1855917 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-03-27 19:05:02,300 DEBUG SenderThread:1855917 [sender.py:send():379] send: stats | |
| 2024-03-27 19:05:03,376 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-27 19:05:03,376 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-27 19:05:03,376 DEBUG SenderThread:1855917 [sender.py:send_request():406] send_request: stop_status | |
| 2024-03-27 19:05:03,644 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:05:04,080 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:05:04,748 INFO Thread-25 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/config.yaml | |
| 2024-03-27 19:05:05,019 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-27 19:05:05,019 DEBUG SenderThread:1855917 [sender.py:send_request():406] send_request: stop_status | |
| 2024-03-27 19:05:05,062 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-27 19:05:08,644 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:05:10,299 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:05:13,717 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:05:13,749 INFO Thread-25 :1855917 [dir_watcher.py:_on_file_created():271] file/dir created: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/output.log | |
| 2024-03-27 19:05:14,114 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:05:15,365 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:05:15,750 INFO Thread-25 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/output.log | |
| 2024-03-27 19:05:16,114 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:05:17,750 INFO Thread-25 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/output.log | |
| 2024-03-27 19:05:18,115 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:05:18,376 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-27 19:05:18,376 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-27 19:05:18,376 DEBUG SenderThread:1855917 [sender.py:send_request():406] send_request: stop_status | |
| 2024-03-27 19:05:18,722 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:05:19,751 INFO Thread-25 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/output.log | |
| 2024-03-27 19:05:20,021 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-27 19:05:20,021 DEBUG SenderThread:1855917 [sender.py:send_request():406] send_request: stop_status | |
| 2024-03-27 19:05:20,062 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-27 19:05:20,115 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:05:20,457 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:05:21,751 INFO Thread-25 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/output.log | |
| 2024-03-27 19:05:22,115 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:05:23,731 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:05:23,752 INFO Thread-25 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/output.log | |
| 2024-03-27 19:05:24,116 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:05:25,752 INFO Thread-25 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/output.log | |
| 2024-03-27 19:05:25,771 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:05:25,787 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-27 19:05:25,788 DEBUG SenderThread:1855917 [sender.py:send():379] send: metric | |
| 2024-03-27 19:05:25,788 DEBUG SenderThread:1855917 [sender.py:send():379] send: metric | |
| 2024-03-27 19:05:25,788 DEBUG SenderThread:1855917 [sender.py:send():379] send: metric | |
| 2024-03-27 19:05:25,788 DEBUG SenderThread:1855917 [sender.py:send():379] send: metric | |
| 2024-03-27 19:05:25,788 DEBUG SenderThread:1855917 [sender.py:send():379] send: metric | |
| 2024-03-27 19:05:25,788 DEBUG SenderThread:1855917 [sender.py:send():379] send: metric | |
| 2024-03-27 19:05:25,788 DEBUG SenderThread:1855917 [sender.py:send():379] send: history | |
| 2024-03-27 19:05:25,788 DEBUG SenderThread:1855917 [sender.py:send_request():406] send_request: summary_record | |
| 2024-03-27 19:05:25,788 INFO SenderThread:1855917 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-03-27 19:05:26,116 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:05:26,752 INFO Thread-25 :1855917 [dir_watcher.py:_on_file_created():271] file/dir created: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/wandb-summary.json | |
| 2024-03-27 19:05:28,116 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:05:28,263 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-27 19:05:28,263 DEBUG SenderThread:1855917 [sender.py:send():379] send: telemetry | |
| 2024-03-27 19:05:28,263 DEBUG SenderThread:1855917 [sender.py:send_request():406] send_request: summary_record | |
| 2024-03-27 19:05:28,264 INFO SenderThread:1855917 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-03-27 19:05:28,264 DEBUG SenderThread:1855917 [sender.py:send_request():406] send_request: summary_record | |
| 2024-03-27 19:05:28,265 INFO SenderThread:1855917 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-03-27 19:05:28,265 DEBUG SenderThread:1855917 [sender.py:send_request():406] send_request: summary_record | |
| 2024-03-27 19:05:28,265 INFO SenderThread:1855917 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-03-27 19:05:28,265 DEBUG SenderThread:1855917 [sender.py:send_request():406] send_request: summary_record | |
| 2024-03-27 19:05:28,265 INFO SenderThread:1855917 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-03-27 19:05:28,265 DEBUG SenderThread:1855917 [sender.py:send_request():406] send_request: summary_record | |
| 2024-03-27 19:05:28,265 INFO SenderThread:1855917 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-03-27 19:05:28,266 DEBUG SenderThread:1855917 [sender.py:send():379] send: history | |
| 2024-03-27 19:05:28,266 DEBUG SenderThread:1855917 [sender.py:send_request():406] send_request: summary_record | |
| 2024-03-27 19:05:28,266 INFO SenderThread:1855917 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-03-27 19:05:28,753 INFO Thread-25 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/wandb-summary.json | |
| 2024-03-27 19:05:28,753 INFO Thread-25 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/output.log | |
| 2024-03-27 19:05:29,378 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:05:29,753 INFO Thread-25 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/output.log | |
| 2024-03-27 19:05:29,935 INFO memory :1855917 [interfaces.py:monitor():140] Process proc.memory.rssMB has exited. | |
| 2024-03-27 19:05:29,936 DEBUG SystemMonitor:1855917 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-03-27 19:05:29,936 DEBUG SystemMonitor:1855917 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-03-27 19:05:29,936 DEBUG SystemMonitor:1855917 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-03-27 19:05:29,937 DEBUG SenderThread:1855917 [sender.py:send():379] send: stats | |
| 2024-03-27 19:05:30,117 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:05:30,753 INFO Thread-25 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/output.log | |
| 2024-03-27 19:05:30,938 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:05:32,301 DEBUG SenderThread:1855917 [sender.py:send():379] send: stats | |
| 2024-03-27 19:05:33,376 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-27 19:05:33,376 DEBUG SenderThread:1855917 [sender.py:send_request():406] send_request: stop_status | |
| 2024-03-27 19:05:33,376 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-27 19:05:34,562 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:05:35,942 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:05:36,755 INFO Thread-25 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /tmp/ray/session_2024-03-27_19-04-27_986745_1855858/artifacts/2024-03-27_19-04-30/_objective_2024-03-27_19-04-30/working_dirs/_objective_d70b7_00000_0_learning_rate=0.0000_2024-03-27_19-04-30/wandb/run-20240327_190433-tszlk6iz/files/config.yaml | |
| 2024-03-27 19:05:39,562 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:05:41,180 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:05:44,563 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:05:46,180 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:05:48,120 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:05:48,376 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-27 19:05:48,376 DEBUG SenderThread:1855917 [sender.py:send_request():406] send_request: stop_status | |
| 2024-03-27 19:05:48,378 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-27 19:05:49,571 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:05:50,121 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:05:51,181 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:05:52,121 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:05:54,121 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:05:54,837 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:05:56,122 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:05:56,181 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:05:58,122 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:05:59,876 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:06:01,123 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:06:01,182 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:06:02,301 DEBUG SenderThread:1855917 [sender.py:send():379] send: stats | |
| 2024-03-27 19:06:03,123 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:06:03,376 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-27 19:06:03,376 DEBUG SenderThread:1855917 [sender.py:send_request():406] send_request: stop_status | |
| 2024-03-27 19:06:03,377 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-27 19:06:04,906 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:06:05,124 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:06:06,182 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:06:07,124 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:06:09,124 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:06:09,922 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:06:11,125 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:06:11,183 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:06:13,125 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:06:14,950 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:06:15,126 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:06:16,183 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:06:17,126 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:06:18,376 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-27 19:06:18,376 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-27 19:06:18,376 DEBUG SenderThread:1855917 [sender.py:send_request():406] send_request: stop_status | |
| 2024-03-27 19:06:19,126 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:06:20,012 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:06:21,127 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:06:21,184 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:06:23,127 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:06:25,014 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:06:25,128 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:06:26,184 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:06:27,128 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:06:29,129 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:06:30,102 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:06:31,129 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:06:31,185 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:06:32,302 DEBUG SenderThread:1855917 [sender.py:send():379] send: stats | |
| 2024-03-27 19:06:33,129 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:06:33,376 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-27 19:06:33,376 DEBUG SenderThread:1855917 [sender.py:send_request():406] send_request: stop_status | |
| 2024-03-27 19:06:33,378 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-27 19:06:35,112 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:06:35,130 INFO Thread-12 :1855917 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/gordon1109/qa_test/wandb/run-20240327_190401-h708adzm/files/output.log | |
| 2024-03-27 19:06:36,185 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:06:40,150 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-27 19:06:41,186 DEBUG HandlerThread:1855917 [handler.py:handle_request():146] handle_request: status_report | |